class
#include <slang/parsing/Token.h>
Token Represents a single lexed token, including leading trivia, original location, token kind, and any related information derived from the token itself (such as the lexeme).
This class is a lightweight immutable structure designed to be copied around and stored wherever. The bulk of the token's data is stored in a heap allocated block. Most of the hot path only cares about the token's kind, so that's given priority.
Public functions
- auto isMissing() const -> bool
- A missing token was expected and inserted by the parser at a given point.
- auto valueText() const -> std::string_view
- auto rawText() const -> std::string_view
- Gets the original lexeme that led to the creation of this token.
- auto toString() const -> std::string
- Prints the token (including all of its trivia) to a string.
- auto intValue() const -> SVInt
- auto isOnSameLine() const -> bool
- auto withTrivia(BumpAllocator& alloc, std::span<Trivia const> trivia) const -> Token
- Modification methods to make it easier to deal with immutable tokens.
Public variables
- TokenKind kind
Function documentation
std::string_view slang:: parsing:: Token:: valueText() const
Value text is the "nice" lexed version of certain tokens; for example, in string literals, escape sequences are converted appropriately.
bool slang:: parsing:: Token:: isOnSameLine() const
Returns true if this token is on the same line as the token before it. This is detected by examining the leading trivia of this token for newlines.
Variable documentation
TokenKind slang:: parsing:: Token:: kind
The kind of the token; this is not in the info block because we almost always want to look at it (perf).