|
Blender V5.3
|
Inherits lexit::TokenBuffer.
Public Member Functions | |
| AtomicLexer (StringRef str) | |
| BLI_INLINE_METHOD TokenAtom | hash (StringRef tok_str) |
| BLI_INLINE_METHOD Token | paste_token (const std::string &tok_str, const TokenType type, const bool followed_by_space) |
| TokenAtom | max_atom_value () const |
Type-safe identifier management. | |
| Line | line (int index) const |
| Directive | directive (int index) const |
| Public Member Functions inherited from lexit::TokenBuffer | |
| TokenBuffer ()=default | |
| TokenBuffer (const std::string_view str, const CharClass char_class_table[128]) | |
| void | process (const std::string_view str, const CharClass char_class_table[128]) |
| void | clear () |
| Discard all currently held data. Does not reallocate. | |
| void | reserve (const uint32_t count) |
| Allocate backing memory for the given number of tokens and move currently held data. | |
| void | tokenize (const CharClass char_class_table[128]) |
| Tokenizes the input string by grouping contiguous characters of the same class. | |
| void | merge_complex_literals (bool bsl_template_angle_bracket_lexing=false) |
| Merge complex literals such as floats, strings and comments. | |
| void | atomize_words (IdentifierMap &identifiers, const KeywordTable &keywords) |
| Assign keyword types and atoms for a small set of identifier. | |
| void | compute_lengths () |
| Compute small token length for speeding up certain tasks. | |
| uint32_t | size () const |
| Return the amount of token inside the buffer. | |
| std::string_view | substr (const Token &start, const Token &end, const bool with_trailing_whitespaces=false) const |
| Return the substring between the start and end tokens (included). | |
| void | append (TokenType type, TokenAtom atom, int32_t str_size, int32_t str_size_with_witespaces) |
| Append a Token at the end of the buffer. | |
| Token | operator[] (int index) const |
| TokenMut | operator[] (int index) |
| TokenIt | begin () |
| TokenIt | end () |
| Token | front () |
| Token | back () |
Public Attributes | |
| blender::OffsetIndices< int > | line_offsets |
| Vector< int > | directive_lines |
| TokenPastingBuffer | pasting_buf |
| Public Attributes inherited from lexit::TokenBuffer | |
| std::string_view | str_ |
| AlignedArrayPtr< TokenType > | types_ |
| AlignedArrayPtr< uint32_t > | offsets_ |
| AlignedArrayPtr< uint32_t > | offsets_end_ |
| AlignedArrayPtr< uint8_t > | lengths_ |
| AlignedArrayPtr< TokenAtom > | atoms_ |
| uint32_t | size_ = 0 |
| uint32_t | allocated_size_ = 0 |
Protected Member Functions | |
| void | lexical_analysis (std::string_view input) |
| BLI_NOINLINE void | lex_pass () |
Protected Attributes | |
| Vector< int > | line_offsets_buf_ |
| lexit::IdentifierMap | identifier_map |
Lexer variant for very fast tokenization for the preprocessor. Does not merge newlines and spaces together. Convert all identifier strings (words) into unique identifiers (TokenAtom) for fast comparison.
Definition at line 97 of file gpu_shader_preprocessor.cc.
|
inline |
Definition at line 105 of file gpu_shader_preprocessor.cc.
References lexical_analysis(), and str.
| Directive blender::gpu::AtomicLexer::directive | ( | int | index | ) | const |
Definition at line 490 of file gpu_shader_preprocessor.cc.
|
inline |
Definition at line 113 of file gpu_shader_preprocessor.cc.
References BLI_INLINE_METHOD, identifier_map, and lexit::IdentifierMap::lookup_or_add().
Referenced by lex_pass(), and paste_token().
|
inlineprotected |
All-in-one lexing pass.
Definition at line 154 of file gpu_shader_preprocessor.cc.
References lexit::TokenBuffer::atomize_words(), lexit::TokenBuffer::atoms_, BLI_NOINLINE, lexit::Define, directive_lines, lexit::Elif, lexit::Else, lexit::Endif, lexit::AlignedArrayPtr< T >::get(), hash(), identifier_map, lexit::If, lexit::Ifdef, lexit::Ifndef, lexit::Line, line_offsets, line_offsets_buf_, lexit::IdentifierMap::lookup_or_add(), lexit::IdentifierMap::make_keyword(), lexit::NewLine, lexit::Pragma, lexit::IdentifierMap::reserve(), lexit::Vector< T >::reserve(), lexit::Vector< T >::resize(), lexit::TokenBuffer::size(), lexit::Vector< T >::size(), lexit::TokenBuffer::size_, lexit::TokenBuffer::types_, and lexit::Undef.
Referenced by lexical_analysis().
|
inlineprotected |
Definition at line 137 of file gpu_shader_preprocessor.cc.
References lexit::char_class_table, lex_pass(), and lexit::TokenBuffer::process().
Referenced by AtomicLexer().
| Line blender::gpu::AtomicLexer::line | ( | int | index | ) | const |
Definition at line 485 of file gpu_shader_preprocessor.cc.
|
inline |
Definition at line 131 of file gpu_shader_preprocessor.cc.
References identifier_map, and lexit::IdentifierMap::max_atom_value().
|
inline |
Definition at line 120 of file gpu_shader_preprocessor.cc.
References lexit::TokenMut::atom(), BLI_INLINE_METHOD, hash(), blender::gpu::TokenPastingBuffer::paste_token(), pasting_buf, and lexit::Token::str().
| Vector<int> blender::gpu::AtomicLexer::directive_lines |
Definition at line 102 of file gpu_shader_preprocessor.cc.
Referenced by lex_pass().
|
protected |
Definition at line 146 of file gpu_shader_preprocessor.cc.
Referenced by hash(), lex_pass(), and max_atom_value().
| blender::OffsetIndices<int> blender::gpu::AtomicLexer::line_offsets |
Definition at line 99 of file gpu_shader_preprocessor.cc.
Referenced by lex_pass().
|
protected |
Definition at line 144 of file gpu_shader_preprocessor.cc.
Referenced by lex_pass().
| TokenPastingBuffer blender::gpu::AtomicLexer::pasting_buf |
Definition at line 118 of file gpu_shader_preprocessor.cc.
Referenced by paste_token().