slang::parsing::Token class
#include <slang/parsing/Token.h>

Represents a single lexed token, including leading trivia, original location, token kind, and any related information derived from the token itself (such as the lexeme).

This class is a lightweight immutable structure designed to be copied around and stored wherever. The bulk of the token's data is stored in a heap allocated block. Most of the hot path only cares about the token's kind, so that's given priority.

Public static functions

static bool storageHasTokenTag(const void* storage): Returns true if the bytes at storage (which must be at least sizeof(Token) bytes) appear to contain a Token (rather than a pointer).

Public functions

bool isMissing() const: A missing token was expected and inserted by the parser at a given point.
std::string_view valueText() const: Value text is the "nice" lexed version of certain tokens; for example, in string literals, escape sequences are converted appropriately.
std::string_view rawText() const: Gets the original lexeme that led to the creation of this token.
std::string toString() const: Prints the token (including all of its trivia) to a string.
SVInt intValue() const: Data accessors for specific kinds of tokens.
bool isOnSameLine() const: Returns true if this token is on the same line as the token before it.
bool hasInlinedTrivia() const: Returns true if this token's trivia (if any) is encoded inline within the token's own bits, requiring no separate allocation in the info block.
size_t getTriviaSizeInBytes() const: Gets the number of bytes in the info block used to store this token's trivia, including any overflow count header.
size_t getSizeInBytes() const: Gets the allocated size of this token, in bytes, including its trivia.
Token withTrivia(BumpAllocator& alloc, const TriviaView& trivia) const: Modification methods to make it easier to deal with immutable tokens.

Public variables

TokenKind kind: The kind of the token; this is not in the info block because we almost always want to look at it (perf).

Function documentation

SVInt slang::parsing::Token::intValue() const

Data accessors for specific kinds of tokens.

These will generally assert if the kind is wrong.

bool slang::parsing::Token::isOnSameLine() const

Returns true if this token is on the same line as the token before it.

This is detected by examining the leading trivia of this token for newlines.

size_t slang::parsing::Token::getTriviaSizeInBytes() const

Gets the number of bytes in the info block used to store this token's trivia, including any overflow count header.

Returns 0 when the trivia is empty or encoded inline.

slang::parsing::Token class #include <slang/parsing/Token.h>