Struct Token

pub struct Token {
    pub token_type: Tokentype,
    pub lexeme: String,
    pub pos: usize,
}

Represents a token in the source code

Fields

token_type: Tokentype

The type of the token

lexeme: String

The actual text of the token

pos: usize

Position index - used with LineInfo to determine line number

Implementations

impl Token

pub fn new(token_type: Tokentype, lexeme: String, pos: usize) -> Token

Creates a new token with the given type, lexeme, and position

Trait Implementations

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.