Struct LexerState

struct LexerState<'a> {
    input: &'a str,
    chars: Peekable<Chars<'a>>,
    current_pos: usize,
    current_line: usize,
    tokens_on_current_line: usize,
    tokens: Vec<Token>,
    line_tokens: Vec<(u16, u16)>,
    errors: Vec<CompilerError>,
}

Lexer state for tracking position during tokenization

Fields

input: &'a str

Source text being tokenized

chars: Peekable<Chars<'a>>

Iterator over source characters

current_pos: usize

Current position in source

current_line: usize

Current line number

tokens_on_current_line: usize

Number of tokens on current line

tokens: Vec<Token>

Tokens generated so far

line_tokens: Vec<(u16, u16)>

Line token counts for line info

errors: Vec<CompilerError>

Collected lexer errors

Implementations

impl<'a> LexerState<'a>

fn new(input: &'a str) -> Self

Creates a new lexer state for the given input

Arguments

• input - The source code to tokenize

Returns

A new LexerState object

fn advance(&mut self) -> Option<char>

Advances to the next character in the input

Arguments

• state - The current lexer state

fn peek(&mut self) -> Option<&char>

Peeks at the next character without consuming it

Arguments

• state - The current lexer state

fn add_token(&mut self, token_type: Tokentype, lexeme: String, start_pos: usize)

Adds a token to the token list

Arguments

• state - The current lexer state
• token_type - The type of token to add
• lexeme - The string representation of the token
• start_pos - The starting position of the token in the input

fn add_error( &mut self, error_code: ErrorCode, message: String, start_pos: usize, token_length: Option<usize>, )

Adds an error to the error list

Arguments

• state - The current lexer state
• error_code - The error code for this error
• message - The error message
• start_pos - The starting position of the error
• token_length - The length of the problematic token

fn record_line_break(&mut self)

Records a line break, updating line counts

Arguments

• state - The current lexer state

fn finish(self) -> CompileResult<LexerResult<'a>>

Finishes tokenization and returns the result

Arguments

• state - The current lexer state