slang::Token class

Contents

Represents a single lexed token, including leading trivia, original location, token kind, and any related information derived from the token itself (such as the lexeme).

This class is a lightweight immutable structure designed to be copied around and stored wherever. The bulk of the token's data is stored in a heap allocated block. Most of the hot path only cares about the token's kind, so that's given priority.

Public functions

auto isMissing() const -> bool
A missing token was expected and inserted by the parser at a given point.
auto valueText() const -> string_view
auto rawText() const -> string_view
Gets the original lexeme that led to the creation of this token.
auto toString() const -> std::string
Prints the token (including all of its trivia) to a string.
auto intValue() const -> SVInt
auto withTrivia(BumpAllocator& alloc, span<Trivia const> trivia) const -> Token
Modification methods to make it easier to deal with immutable tokens.

Public variables

TokenKind kind

Function documentation

string_view slang::Token::valueText() const

Value text is the "nice" lexed version of certain tokens; for example, in string literals, escape sequences are converted appropriately.

SVInt slang::Token::intValue() const

Data accessors for specific kinds of tokens. These will generally assert if the kind is wrong.

Variable documentation

TokenKind slang::Token::kind

The kind of the token; this is not in the info block because we almost always want to look at it (perf).