Remap Tokens During Parsing

The Rhai Engine first parses a script into a stream of tokens.

Tokens have the type Token which is only exported under internals.

The function Engine::on_parse_token, available only under internals, allows registration of a mapper function that converts (remaps) a Token into another.

Hot Tips: Use as safety checks

Since it is called for every token parsed from the script, this token mapper function can also be used to implement safety checks against, say, stack-overflow or out-of-memory situations during parsing.

See here for more details.

Function Signature

Tip: Raising errors

Raise a parse error by returning Token::LexError as the mapped token.

The function signature passed to Engine::on_parse_token takes the following form.

Fn(token: Token, pos: Position, state: &TokenizeState) -> Token

where:

ParameterTypeDescription
tokenTokenthe next symbol parsed
posPositionlocation of the token
state&TokenizeStatecurrent state of the tokenizer

Example

use rhai::{Engine, FLOAT, Token};

let mut engine = Engine::new();

// Register a token mapper function.
engine.on_parse_token(|token, pos, state| {
    match token {
        // Change 'begin' ... 'end' to '{' ... '}'
        Token::Identifier(s) if &s == "begin" => Token::LeftBrace,
        Token::Identifier(s) if &s == "end" => Token::RightBrace,

        // Change all integer literals to floating-point
        Token::IntegerConstant(n) => Token::FloatConstant((n as FLOAT).into()),
        
        // Disallow '()'
        Token::Unit => Token::LexError(
            LexError::ImproperSymbol("()".to_string(), "".to_string()).into()
        ),

        // Pass through all other tokens unchanged
        _ => token
    }
});