pub enum Token<'a> {
Show 39 variants
Identifier(&'a str),
Number(f64),
Text(&'a str),
LeftRoundBracket,
RightRoundBracket,
LeftSquareBracket,
RightSquareBracket,
LeftCurlyBracket,
RightCurlyBracket,
PairingOp,
AssignmentOp,
LabellingOp,
MappingOp,
ThenOp,
ReverseCompositionOp,
CompositionOp,
FunctionDefinitionOp,
AlternativeOp,
ConditionalOp,
OrOp,
AndOp,
EqualToOp,
NotEqualToOp,
LessThanOp,
LessThanOrEqualToOp,
GreaterThanOp,
GreaterThanOrEqualToOp,
PlusOp,
MinusOp,
ProductOp,
DivisionOp,
ModuloOp,
PowerOp,
SubcontextingOp,
NotOp,
LiftingOp,
Undefined(&'a str),
Comment(&'a str),
EndOfExpression,
}Expand description
All token kinds produced by the Lexer.
The lifetime 'a is tied to the source string: Identifier, Text,
Comment, and Undefined variants borrow slices directly from it, so
no heap allocation is needed for those payloads.
Variants§
Identifier(&'a str)
A name — starts with a letter or _, continues with letters, digits,
or _. Examples: foo, _bar, myVar1.
Number(f64)
A floating-point numeric literal. Integer, decimal, and scientific
notation are all parsed into an f64. Examples: 42, 3.14,
1.5e-3.
Text(&'a str)
A double-quoted string literal. The slice contains only the content
between the quotes — the delimiters are not included.
Example: "hello" → Text("hello").
There are no escape sequences: a backslash is an ordinary character,
and a text literal cannot contain ". A literal may span several lines.
Note: if the closing
"is absent the slice extends to the end of the source string (the lexer does not error on unterminated strings).
LeftRoundBracket
(
RightRoundBracket
)
LeftSquareBracket
[ — opens a list literal.
RightSquareBracket
] — closes a list literal.
LeftCurlyBracket
{ — opens a namespace literal.
RightCurlyBracket
} — closes a namespace literal.
PairingOp
, (precedence 10) — pairing / tuple construction.
AssignmentOp
= (precedence 12) — assignment.
LabellingOp
: (precedence 12) — labelling / named argument.
MappingOp
=> (precedence 13, right-associative) — mapping: applies the
right-hand function(s) to each item of the left-hand tuple.
ThenOp
~> (precedence 13, left-associative) — then: chains a function to
a promise.
ReverseCompositionOp
>> (precedence 14) — reverse composition / pipe (left-to-right).
CompositionOp
<< (precedence 14) — composition (right-to-left).
FunctionDefinitionOp
-> (precedence 15) — function definition (right-associative).
AlternativeOp
; (precedence 21) — alternative: evaluates the right-hand side only
when the left-hand side is Nothing.
ConditionalOp
? (precedence 22) — conditional: evaluates the right-hand side only
when the left-hand side is truthy.
OrOp
| (precedence 23) — logical OR.
AndOp
& (precedence 23) — logical AND.
EqualToOp
== (precedence 24) — equality comparison.
NotEqualToOp
!= (precedence 24) — inequality comparison.
LessThanOp
< (precedence 24) — less-than comparison.
LessThanOrEqualToOp
<= (precedence 24) — less-than-or-equal comparison.
GreaterThanOp
> (precedence 24) — greater-than comparison.
GreaterThanOrEqualToOp
>= (precedence 24) — greater-than-or-equal comparison.
PlusOp
+ (precedence 25) — addition.
MinusOp
- (precedence 25) — subtraction.
ProductOp
* (precedence 26) — multiplication.
DivisionOp
/ (precedence 26) — division.
ModuloOp
% (precedence 26) — modulo.
PowerOp
^ (precedence 27) — exponentiation.
SubcontextingOp
. (precedence 30) — sub-contexting: looks up the right-hand name
inside the namespace produced by the left-hand expression.
Example: math.sin retrieves sin from the math namespace.
NotOp
! (prefix) — logical negation.
LiftingOp
$ (prefix) — lifting: looks a name up starting from the outermost
scope.
Undefined(&'a str)
A character sequence the lexer could not classify as any valid token. The slice covers exactly the unrecognised source characters.
Examples: ', @, a malformed number like 1e+ or 12abc.
Comment(&'a str)
A #-prefixed line comment. The slice contains the comment text
after the # and before the terminating \n (or end of source).
The lexer yields comments as tokens so that a tool such as a formatter
can use them. The Parser skips them.
EndOfExpression
Signals that the source string has been fully consumed.
next_token returns this sentinel instead of Option::None so that
call-sites do not need to handle two distinct “no more input” states.
The Iterator impl converts it to None.