slang::Lexer class

The Lexer is responsible for taking source text and chopping it up into tokens. Tokens carry along leading "trivia", which is things like whitespace and comments, so that we can programmatically piece back together what the original file looked like.

There are also helper methods on this class that handle token manipulation on the character level.

Public static functions

static auto concatenateTokens(BumpAllocator& alloc, Token left, Token right) -> Token
Concatenates two tokens together; used for macro pasting.
static auto stringify(BumpAllocator& alloc, SourceLocation location, span<Trivia const> trivia, Token* begin, Token* end) -> Token
static void splitTokens(BumpAllocator& alloc, Diagnostics& diagnostics, const SourceManager& sourceManager, Token sourceToken, size_t offset, KeywordVersion keywordVersion, SmallVector<Token>& results)

Public functions

auto lex(KeywordVersion keywordVersion = LexerFacts::getDefaultKeywordVersion()) -> Token

Function documentation

static Token slang::Lexer::stringify(BumpAllocator& alloc, SourceLocation location, span<Trivia const> trivia, Token* begin, Token* end)

Converts a range of tokens into a string literal; used for macro stringification. The location and trivia parameters are used in the newly created token. The range of tokens to stringify is given by begin and end. If noWhitespace is set to true, all whitespace in between tokens will be stripped.

static void slang::Lexer::splitTokens(BumpAllocator& alloc, Diagnostics& diagnostics, const SourceManager& sourceManager, Token sourceToken, size_t offset, KeywordVersion keywordVersion, SmallVector<Token>& results)

Splits the given token at the specified offset into its raw source text. The trailing portion of the split is lexed into new tokens and appened to results

Token slang::Lexer::lex(KeywordVersion keywordVersion = LexerFacts::getDefaultKeywordVersion())

Lexes the next token from the source code. This will never return a null pointer; at the end of the buffer, an infinite stream of EndOfFile tokens will be generated