Blender V5.3
blender::gpu::shader::parser::LexerBase Struct Reference

#include <token_stream.hh>

Inherits lexit::TokenBuffer.

Inherited by blender::gpu::shader::parser::ParserBase.

Public Member Functions

std::string_view token_types_str () const
void identify_keywords ()
void identify_template_tokens ()
void reset_template_tokens ()
Public Member Functions inherited from lexit::TokenBuffer
 TokenBuffer ()=default
 TokenBuffer (const std::string_view str, const CharClass char_class_table[128])
void process (const std::string_view str, const CharClass char_class_table[128])
void clear ()
 Discard all currently held data. Does not reallocate.
void reserve (const uint32_t count)
 Allocate backing memory for the given number of tokens and move currently held data.
void tokenize (const CharClass char_class_table[128])
 Tokenizes the input string by grouping contiguous characters of the same class.
void merge_complex_literals (bool bsl_template_angle_bracket_lexing=false)
 Merge complex literals such as floats, strings and comments.
void atomize_words (IdentifierMap &identifiers, const KeywordTable &keywords)
 Assign keyword types and atoms for a small set of identifier.
void compute_lengths ()
 Compute small token length for speeding up certain tasks.
uint32_t size () const
 Return the amount of token inside the buffer.
std::string_view substr (const Token &start, const Token &end, const bool with_trailing_whitespaces=false) const
 Return the substring between the start and end tokens (included).
void append (TokenType type, TokenAtom atom, int32_t str_size, int32_t str_size_with_witespaces)
 Append a Token at the end of the buffer.
Token operator[] (int index) const
TokenMut operator[] (int index)
TokenIt begin ()
TokenIt end ()
Token front ()
Token back ()

Static Public Attributes

static const std::array< CharClass, 128 > bsl_char_class_table
static const std::array< CharClass, 128 > default_char_class_table

Additional Inherited Members

Public Attributes inherited from lexit::TokenBuffer
std::string_view str_
AlignedArrayPtr< TokenType > types_
AlignedArrayPtr< uint32_t > offsets_
AlignedArrayPtr< uint32_t > offsets_end_
AlignedArrayPtr< uint8_t > lengths_
AlignedArrayPtr< TokenAtom > atoms_
uint32_t size_ = 0
uint32_t allocated_size_ = 0

Detailed Description

Turns string into token.

Definition at line 26 of file token_stream.hh.

Member Function Documentation

◆ identify_keywords()

◆ identify_template_tokens()

◆ reset_template_tokens()

◆ token_types_str()

std::string_view blender::gpu::shader::parser::LexerBase::token_types_str ( ) const
inline

Member Data Documentation

◆ bsl_char_class_table

const std::array< CharClass, 128 > blender::gpu::shader::parser::LexerBase::bsl_char_class_table
static
Initial value:
= [] {
std::array<CharClass, 128> table;
memcpy(table.data(), lexit::char_class_table, sizeof(lexit::char_class_table));
table['\n'] = CharClass::WhiteSpace;
table['<'] = CharClass::Separator;
table['>'] = CharClass::Separator;
return table;
}()
const CharClass char_class_table[128]
Definition tables.hh:12

Definition at line 27 of file token_stream.hh.

Referenced by blender::gpu::shader::parser::FullLexer::lexical_analysis(), and blender::gpu::shader::parser::SimpleLexer::lexical_analysis().

◆ default_char_class_table

const std::array< CharClass, 128 > blender::gpu::shader::parser::LexerBase::default_char_class_table
static
Initial value:
= [] {
std::array<CharClass, 128> table;
memcpy(table.data(), lexit::char_class_table, sizeof(lexit::char_class_table));
return table;
}()

Definition at line 28 of file token_stream.hh.

Referenced by blender::gpu::shader::parser::ExpressionLexer::lexical_analysis().


The documentation for this struct was generated from the following files: