Blender V5.3
lexit::TokenBuffer Struct Reference

#include <lexit.hh>

Inherited by blender::gpu::AtomicLexer, blender::gpu::TokenPastingBuffer, and blender::gpu::shader::parser::LexerBase.

Classes

struct  TokenIt
 Token iterator. More...

Public Member Functions

 TokenBuffer ()=default
 TokenBuffer (const std::string_view str, const CharClass char_class_table[128])
void process (const std::string_view str, const CharClass char_class_table[128])
void clear ()
 Discard all currently held data. Does not reallocate.
void reserve (const uint32_t count)
 Allocate backing memory for the given number of tokens and move currently held data.
void tokenize (const CharClass char_class_table[128])
 Tokenizes the input string by grouping contiguous characters of the same class.
void merge_complex_literals (bool bsl_template_angle_bracket_lexing=false)
 Merge complex literals such as floats, strings and comments.
void atomize_words (IdentifierMap &identifiers, const KeywordTable &keywords)
 Assign keyword types and atoms for a small set of identifier.
void compute_lengths ()
 Compute small token length for speeding up certain tasks.
uint32_t size () const
 Return the amount of token inside the buffer.
std::string_view substr (const Token &start, const Token &end, const bool with_trailing_whitespaces=false) const
 Return the substring between the start and end tokens (included).
void append (TokenType type, TokenAtom atom, int32_t str_size, int32_t str_size_with_witespaces)
 Append a Token at the end of the buffer.
Token operator[] (int index) const
TokenMut operator[] (int index)
TokenIt begin ()
TokenIt end ()
Token front ()
Token back ()

Public Attributes

std::string_view str_
AlignedArrayPtr< TokenType > types_
AlignedArrayPtr< uint32_t > offsets_
AlignedArrayPtr< uint32_t > offsets_end_
AlignedArrayPtr< uint8_t > lengths_
AlignedArrayPtr< TokenAtom > atoms_
uint32_t size_ = 0
uint32_t allocated_size_ = 0

Detailed Description

Definition at line 121 of file lexit.hh.

Constructor & Destructor Documentation

◆ TokenBuffer() [1/2]

lexit::TokenBuffer::TokenBuffer ( )
default

◆ TokenBuffer() [2/2]

lexit::TokenBuffer::TokenBuffer ( const std::string_view str,
const CharClass char_class_table[128] )
inline

Definition at line 146 of file lexit.hh.

References lexit::char_class_table, process(), and str.

Member Function Documentation

◆ append()

void lexit::TokenBuffer::append ( TokenType type,
TokenAtom atom,
int32_t str_size,
int32_t str_size_with_witespaces )
inline

Append a Token at the end of the buffer.

Definition at line 236 of file lexit.hh.

References atoms_, offsets_, offsets_end_, reserve(), size_, and types_.

Referenced by blender::gpu::TokenPastingBuffer::paste_token().

◆ atomize_words()

void lexit::TokenBuffer::atomize_words ( IdentifierMap & identifiers,
const KeywordTable & keywords )

Assign keyword types and atoms for a small set of identifier.

Definition at line 671 of file lexit.cc.

References lexit::divide_ceil(), end(), lengths_, lexit::Vector< T >::resize(), size(), size_, types_, and lexit::Word.

Referenced by blender::gpu::AtomicLexer::lex_pass().

◆ back()

Token lexit::TokenBuffer::back ( )
inline

Definition at line 311 of file lexit.hh.

References size_.

Referenced by blender::gpu::shader::parser::ScopeParser::translation_unit().

◆ begin()

TokenIt lexit::TokenBuffer::begin ( )
inline

Definition at line 298 of file lexit.hh.

References lexit::TokenBuffer::TokenIt::TokenIt().

◆ clear()

void lexit::TokenBuffer::clear ( )

Discard all currently held data. Does not reallocate.

Definition at line 55 of file lexit.cc.

References size_.

Referenced by process().

◆ compute_lengths()

void lexit::TokenBuffer::compute_lengths ( )

Compute small token length for speeding up certain tasks.

Definition at line 644 of file lexit.cc.

References lengths_, offsets_, offsets_end_, lexit::select(), and size_.

Referenced by process().

◆ end()

TokenIt lexit::TokenBuffer::end ( )
inline

Definition at line 302 of file lexit.hh.

References size_, and lexit::TokenBuffer::TokenIt::TokenIt().

Referenced by atomize_words(), substr(), and tokenize().

◆ front()

Token lexit::TokenBuffer::front ( )
inline

Definition at line 307 of file lexit.hh.

◆ merge_complex_literals()

void lexit::TokenBuffer::merge_complex_literals ( bool bsl_template_angle_bracket_lexing = false)

Merge complex literals such as floats, strings and comments.

Parameters
bsl_template_angle_bracket_lexingIf true, will parse angle bracket without a whitespace before or after as a template bracket.

Definition at line 1010 of file lexit.cc.

References assert, lexit::EndOfFile, lexit::AlignedArrayPtr< T >::get(), lexit::GThan, lexit::lex_angle(), lexit::lex_comment(), lexit::lex_float(), lexit::lex_string(), lexit::LThan, lexit::Number, offsets_, offsets_end_, size_, lexit::Slash, str_, lexit::String, and types_.

Referenced by blender::gpu::shader::parser::ExpressionLexer::lexical_analysis(), and blender::gpu::shader::parser::FullLexer::lexical_analysis().

◆ operator[]() [1/2]

TokenMut lexit::TokenBuffer::operator[] ( int index)
inline

Definition at line 250 of file lexit.hh.

References index, and lexit::TokenMut::TokenMut().

◆ operator[]() [2/2]

Token lexit::TokenBuffer::operator[] ( int index) const
inline

Definition at line 246 of file lexit.hh.

References index, and lexit::Token::Token().

◆ process()

◆ reserve()

void lexit::TokenBuffer::reserve ( const uint32_t count)

Allocate backing memory for the given number of tokens and move currently held data.

Does nothing if allocation is already large enough.

Definition at line 60 of file lexit.cc.

References allocated_size_, atoms_, lengths_, offsets_, offsets_end_, lexit::realloc_aligned_array(), size_, and types_.

Referenced by append(), tokenize(), and blender::gpu::TokenPastingBuffer::TokenPastingBuffer().

◆ size()

◆ substr()

std::string_view lexit::TokenBuffer::substr ( const Token & start,
const Token & end,
const bool with_trailing_whitespaces = false ) const
inline

Return the substring between the start and end tokens (included).

Parameters
with_trailing_whitespacesIf true, include the trailing whitespaces.

Definition at line 224 of file lexit.hh.

References end(), offsets_, offsets_end_, and str_.

Referenced by blender::gpu::shader::parser::ast::Node::str(), bsl::CodegenContext::StaticConditionStack::SubCondition::str(), and blender::gpu::shader::parser::ast::Node::str_full().

◆ tokenize()

void lexit::TokenBuffer::tokenize ( const CharClass char_class_table[128])

Tokenizes the input string by grouping contiguous characters of the same class.

This function iterates through the input string and identifies "runs" of characters that map to the same CharClass. For each new group, it records the type and the starting byte offset into the result arrays.

Only characters with the #CanMerge flag are merged together. Characters with a class greater than #ClassToTypeThreshold will just be assigned their class as TokenType. Otherwise, the first character of the token will be used as TokenType.

If the input string contains characters that are not inside the ASCII range, the result of the operation is undefined and might cause segmentation fault.

Parameters
char_class_tableA lookup table mapping ASCII values (0-127) to an 8-bit CharClass.

The goal of this function is to build the offset and type buffers describing the tokens. The type of the token is decided by the character class of its first character. This is a simple lookup from the char_class_table combined with a select. It is then merged with the preceding character if the classes are compatible (see CharClass::CanMerge).

The offsets define the boundaries of the tokens. If whitespaces are treated as tokens, one offset is emitted for the beginning of each token.

str: i n t a = 0 ; EndOfFile emit: 1 0 0 0 1 1 1 1 1 1 offsets: 0 4 6 7 8 9

If whitespaces are merged with preceding tokens, one offset is emitted for the beginning of each token and at either the start of the next token or the start of the next whitespace.

str: i n t a = 0 ; EndOfFile emit start: 1 0 0 0 1 1 1 1 0 1 emit end: 1 0 0 1 0 1 1 1 1 1 offsets start: 0 4 5 6 7 9 offsets end: 3 5 6 7 8 9

Definition at line 488 of file lexit.cc.

References assert, lexit::CanMerge, lexit::char_class_table, lexit::ClassToTypeThreshold, end(), lexit::EndOfFile, is_zero(), lexit::None, offsets_, offsets_end_, popcount(), reserve(), lexit::select(), size_, str, str_, types_, and lexit::WhiteSpace.

Referenced by process().

Member Data Documentation

◆ allocated_size_

uint32_t lexit::TokenBuffer::allocated_size_ = 0

Definition at line 137 of file lexit.hh.

Referenced by reserve().

◆ atoms_

AlignedArrayPtr<TokenAtom> lexit::TokenBuffer::atoms_

Definition at line 133 of file lexit.hh.

Referenced by append(), lexit::Token::atom(), blender::gpu::AtomicLexer::lex_pass(), and reserve().

◆ lengths_

AlignedArrayPtr<uint8_t> lexit::TokenBuffer::lengths_

Definition at line 131 of file lexit.hh.

Referenced by atomize_words(), compute_lengths(), and reserve().

◆ offsets_

◆ offsets_end_

◆ size_

◆ str_

◆ types_


The documentation for this struct was generated from the following files: