|
Blender V5.3
|
#include <token_stream.hh>
Inherits lexit::TokenBuffer.
Inherited by blender::gpu::shader::parser::ParserBase.
Public Member Functions | |
| std::string_view | token_types_str () const |
| void | identify_keywords () |
| void | identify_template_tokens () |
| void | reset_template_tokens () |
| Public Member Functions inherited from lexit::TokenBuffer | |
| TokenBuffer ()=default | |
| TokenBuffer (const std::string_view str, const CharClass char_class_table[128]) | |
| void | process (const std::string_view str, const CharClass char_class_table[128]) |
| void | clear () |
| Discard all currently held data. Does not reallocate. | |
| void | reserve (const uint32_t count) |
| Allocate backing memory for the given number of tokens and move currently held data. | |
| void | tokenize (const CharClass char_class_table[128]) |
| Tokenizes the input string by grouping contiguous characters of the same class. | |
| void | merge_complex_literals (bool bsl_template_angle_bracket_lexing=false) |
| Merge complex literals such as floats, strings and comments. | |
| void | atomize_words (IdentifierMap &identifiers, const KeywordTable &keywords) |
| Assign keyword types and atoms for a small set of identifier. | |
| void | compute_lengths () |
| Compute small token length for speeding up certain tasks. | |
| uint32_t | size () const |
| Return the amount of token inside the buffer. | |
| std::string_view | substr (const Token &start, const Token &end, const bool with_trailing_whitespaces=false) const |
| Return the substring between the start and end tokens (included). | |
| void | append (TokenType type, TokenAtom atom, int32_t str_size, int32_t str_size_with_witespaces) |
| Append a Token at the end of the buffer. | |
| Token | operator[] (int index) const |
| TokenMut | operator[] (int index) |
| TokenIt | begin () |
| TokenIt | end () |
| Token | front () |
| Token | back () |
Static Public Attributes | |
| static const std::array< CharClass, 128 > | bsl_char_class_table |
| static const std::array< CharClass, 128 > | default_char_class_table |
Additional Inherited Members | |
| Public Attributes inherited from lexit::TokenBuffer | |
| std::string_view | str_ |
| AlignedArrayPtr< TokenType > | types_ |
| AlignedArrayPtr< uint32_t > | offsets_ |
| AlignedArrayPtr< uint32_t > | offsets_end_ |
| AlignedArrayPtr< uint8_t > | lengths_ |
| AlignedArrayPtr< TokenAtom > | atoms_ |
| uint32_t | size_ = 0 |
| uint32_t | allocated_size_ = 0 |
Turns string into token.
Definition at line 26 of file token_stream.hh.
| void blender::gpu::shader::parser::LexerBase::identify_keywords | ( | ) |
Definition at line 319 of file intermediate.cc.
References identify_keywords(), blender::gpu::shader::parser::multi_tok_lookup(), lexit::Number, blender::gpu::shader::parser::type_lookup(), and lexit::Word.
Referenced by identify_keywords(), blender::gpu::shader::parser::ExpressionLexer::lexical_analysis(), and blender::gpu::shader::parser::FullLexer::lexical_analysis().
| void blender::gpu::shader::parser::LexerBase::identify_template_tokens | ( | ) |
Definition at line 335 of file intermediate.cc.
References lexit::AngleClose, lexit::AngleOpen, identify_template_tokens(), lexit::Minus, lexit::TokenBuffer::size(), lexit::Template, lexit::TemplateClose, lexit::TemplateOpen, and lexit::TokenMut::type().
Referenced by blender::gpu::shader::parser::ParserBase::build_scope_tree(), identify_template_tokens(), and blender::gpu::shader::parser::ParserBase::parse_bsl().
| void blender::gpu::shader::parser::LexerBase::reset_template_tokens | ( | ) |
Definition at line 358 of file intermediate.cc.
References lexit::AngleClose, lexit::AngleOpen, reset_template_tokens(), lexit::TokenBuffer::size(), lexit::TemplateClose, lexit::TemplateOpen, and lexit::TokenMut::type().
Referenced by blender::gpu::shader::parser::ParserBase::build_scope_tree(), blender::gpu::shader::parser::ParserBase::parse_bsl(), and reset_template_tokens().
|
inline |
Compact visualization of token_types.
Definition at line 31 of file token_stream.hh.
References lexit::AlignedArrayPtr< T >::get(), lexit::TokenBuffer::size_, and lexit::TokenBuffer::types_.
Referenced by blender::gpu::shader::parser::IntermediateForm< parser::FullLexer, parser::FullParser >::debug_print().
|
static |
Definition at line 27 of file token_stream.hh.
Referenced by blender::gpu::shader::parser::FullLexer::lexical_analysis(), and blender::gpu::shader::parser::SimpleLexer::lexical_analysis().
|
static |
Definition at line 28 of file token_stream.hh.
Referenced by blender::gpu::shader::parser::ExpressionLexer::lexical_analysis().