Module lexer

Structs§

handle_ampersand 🔒: Handles ampersand character (logical AND)
handle_block_comment 🔒: Handles multi-line block comments
handle_dash 🔒: Handles dash character (minus or arrow)
handle_equals 🔒: Handles equals character (assignment or equality)
handle_exclamation 🔒: Handles exclamation mark (not or not equal)
handle_greater_than 🔒: Handles greater than character (greater than or greater than or equal)
handle_identifier 🔒: Handles alphabetic identifiers and keywords
handle_invalid_char 🔒: Handles invalid characters
handle_less_than 🔒: Handles less than character (less than or less than or equal)
handle_line_comment 🔒: Handles single-line comments
handle_number 🔒: Handles numeric literals (integers and floating point)
handle_pipe 🔒: Handles pipe character (logical OR)
handle_simple_token 🔒: Handles simple one-character tokens
handle_slash 🔒: Handles slash character (divide or comments)
handle_string 🔒: Handles string literals
handle_whitespace 🔒: Handles whitespace characters in the input
tokenize: Converts source code text into a sequence of tokens with line information