Token class
Represents a single lexed token, including leading trivia, original location, token kind, and any related information derived from the token itself (such as the lexeme).
This class is a lightweight immutable structure designed to be copied around and stored wherever. The bulk of the token's data is stored in a heap allocated block. Most of the hot path only cares about the token's kind, so that's given priority.
Public functions
- auto isMissing() const -> bool
- A missing token was expected and inserted by the parser at a given point.
-
auto valueText() const -> std::
string_view - Value text is the "nice" lexed version of certain tokens; for example, in string literals, escape sequences are converted appropriately.
-
auto rawText() const -> std::
string_view - Gets the original lexeme that led to the creation of this token.
-
auto toString() const -> std::
string - Prints the token (including all of its trivia) to a string.
- auto intValue() const -> SVInt
- Data accessors for specific kinds of tokens.
- auto isOnSameLine() const -> bool
- Returns true if this token is on the same line as the token before it.
-
auto withTrivia(BumpAllocator& alloc,
std::
span<Trivia const> trivia) const -> Token - Modification methods to make it easier to deal with immutable tokens.
Public variables
- TokenKind kind
- The kind of the token; this is not in the info block because we almost always want to look at it (perf).
Function documentation
bool slang::
Returns true if this token is on the same line as the token before it.
This is detected by examining the leading trivia of this token for newlines.