slang::parsing::Token class
#include <slang/parsing/Token.h>

Represents a single lexed token, including leading trivia, original location, token kind, and any related information derived from the token itself (such as the lexeme).

This class is a lightweight immutable structure designed to be copied around and stored wherever. The bulk of the token's data is stored in a heap allocated block. Most of the hot path only cares about the token's kind, so that's given priority.

Public functions

auto isMissing() const -> bool
A missing token was expected and inserted by the parser at a given point.
auto valueText() const -> std::string_view
auto rawText() const -> std::string_view
Gets the original lexeme that led to the creation of this token.
auto toString() const -> std::string
Prints the token (including all of its trivia) to a string.
auto intValue() const -> SVInt
auto isOnSameLine() const -> bool
auto withTrivia(BumpAllocator& alloc, std::span<Trivia const> trivia) const -> Token
Modification methods to make it easier to deal with immutable tokens.

Public variables

TokenKind kind

Function documentation

std::string_view slang::parsing::Token::valueText() const

Value text is the "nice" lexed version of certain tokens; for example, in string literals, escape sequences are converted appropriately.

SVInt slang::parsing::Token::intValue() const

Data accessors for specific kinds of tokens. These will generally assert if the kind is wrong.

bool slang::parsing::Token::isOnSameLine() const

Returns true if this token is on the same line as the token before it. This is detected by examining the leading trivia of this token for newlines.

Variable documentation

TokenKind slang::parsing::Token::kind

The kind of the token; this is not in the info block because we almost always want to look at it (perf).