Blender V5.3
blender::gpu::AtomicLexer Struct Reference

Inherits lexit::TokenBuffer.

Public Member Functions

 AtomicLexer (StringRef str)
BLI_INLINE_METHOD TokenAtom hash (StringRef tok_str)
BLI_INLINE_METHOD Token paste_token (const std::string &tok_str, const TokenType type, const bool followed_by_space)
TokenAtom max_atom_value () const
Type-safe identifier management.
Line line (int index) const
Directive directive (int index) const
Public Member Functions inherited from lexit::TokenBuffer
 TokenBuffer ()=default
 TokenBuffer (const std::string_view str, const CharClass char_class_table[128])
void process (const std::string_view str, const CharClass char_class_table[128])
void clear ()
 Discard all currently held data. Does not reallocate.
void reserve (const uint32_t count)
 Allocate backing memory for the given number of tokens and move currently held data.
void tokenize (const CharClass char_class_table[128])
 Tokenizes the input string by grouping contiguous characters of the same class.
void merge_complex_literals (bool bsl_template_angle_bracket_lexing=false)
 Merge complex literals such as floats, strings and comments.
void atomize_words (IdentifierMap &identifiers, const KeywordTable &keywords)
 Assign keyword types and atoms for a small set of identifier.
void compute_lengths ()
 Compute small token length for speeding up certain tasks.
uint32_t size () const
 Return the amount of token inside the buffer.
std::string_view substr (const Token &start, const Token &end, const bool with_trailing_whitespaces=false) const
 Return the substring between the start and end tokens (included).
void append (TokenType type, TokenAtom atom, int32_t str_size, int32_t str_size_with_witespaces)
 Append a Token at the end of the buffer.
Token operator[] (int index) const
TokenMut operator[] (int index)
TokenIt begin ()
TokenIt end ()
Token front ()
Token back ()

Public Attributes

blender::OffsetIndices< int > line_offsets
Vector< int > directive_lines
TokenPastingBuffer pasting_buf
Public Attributes inherited from lexit::TokenBuffer
std::string_view str_
AlignedArrayPtr< TokenType > types_
AlignedArrayPtr< uint32_t > offsets_
AlignedArrayPtr< uint32_t > offsets_end_
AlignedArrayPtr< uint8_t > lengths_
AlignedArrayPtr< TokenAtom > atoms_
uint32_t size_ = 0
uint32_t allocated_size_ = 0

Protected Member Functions

void lexical_analysis (std::string_view input)
BLI_NOINLINE void lex_pass ()

Protected Attributes

Vector< int > line_offsets_buf_
lexit::IdentifierMap identifier_map

Detailed Description

Lexer variant for very fast tokenization for the preprocessor. Does not merge newlines and spaces together. Convert all identifier strings (words) into unique identifiers (TokenAtom) for fast comparison.

Definition at line 97 of file gpu_shader_preprocessor.cc.

Constructor & Destructor Documentation

◆ AtomicLexer()

blender::gpu::AtomicLexer::AtomicLexer ( StringRef str)
inline

Definition at line 105 of file gpu_shader_preprocessor.cc.

References lexical_analysis(), and str.

Member Function Documentation

◆ directive()

Directive blender::gpu::AtomicLexer::directive ( int index) const

Definition at line 490 of file gpu_shader_preprocessor.cc.

◆ hash()

BLI_INLINE_METHOD TokenAtom blender::gpu::AtomicLexer::hash ( StringRef tok_str)
inline

◆ lex_pass()

◆ lexical_analysis()

void blender::gpu::AtomicLexer::lexical_analysis ( std::string_view input)
inlineprotected

◆ line()

Line blender::gpu::AtomicLexer::line ( int index) const

Definition at line 485 of file gpu_shader_preprocessor.cc.

◆ max_atom_value()

TokenAtom blender::gpu::AtomicLexer::max_atom_value ( ) const
inline

◆ paste_token()

BLI_INLINE_METHOD Token blender::gpu::AtomicLexer::paste_token ( const std::string & tok_str,
const TokenType type,
const bool followed_by_space )
inline

Member Data Documentation

◆ directive_lines

Vector<int> blender::gpu::AtomicLexer::directive_lines

Definition at line 102 of file gpu_shader_preprocessor.cc.

Referenced by lex_pass().

◆ identifier_map

lexit::IdentifierMap blender::gpu::AtomicLexer::identifier_map
protected

Definition at line 146 of file gpu_shader_preprocessor.cc.

Referenced by hash(), lex_pass(), and max_atom_value().

◆ line_offsets

blender::OffsetIndices<int> blender::gpu::AtomicLexer::line_offsets

Definition at line 99 of file gpu_shader_preprocessor.cc.

Referenced by lex_pass().

◆ line_offsets_buf_

Vector<int> blender::gpu::AtomicLexer::line_offsets_buf_
protected

Definition at line 144 of file gpu_shader_preprocessor.cc.

Referenced by lex_pass().

◆ pasting_buf

TokenPastingBuffer blender::gpu::AtomicLexer::pasting_buf

Definition at line 118 of file gpu_shader_preprocessor.cc.

Referenced by paste_token().


The documentation for this struct was generated from the following file: