Function tokenize
Source pub fn tokenize(input: &str) -> CompileResult<LexerResult<'_>>
Expand description
Converts source code text into a sequence of tokens with line information
§Arguments
input - The source code to tokenize
§Returns
A CompileResult containing LexerResult (tokens and line information) or lexer errors