The previous algorithm would loop through all states but only process
those that didn't have their processed flag set. Much faster to only
iterate through those states that haven't been processed.
Allows most metacharacters, e.g. "*+()[", to appear unescaped within a
character class. This, along with hyphens at the end, allows "[+-]" to
match a plus or a minus.
Using set<>::operator<() doesn't generate a stable comparison wrt. the
nodes in the item. This is either set<>::operator<() not behaving as
a lexicographical compare like I expect or not taking into account the
comparison object the set is initialized with.
Fixes non-deterministic lexer generation.
Previously this cast a GrammarSymbol to a ParserSymbol and then wrote
out its index -- generating a wildly wrong value. This was okay at
runtime because the lexer only checks to see whether or not there is a
symbol and never uses its value.
However it looks like a pretty nasty error to anyone looking at the
generated source code. Fixed by converting the GrammarSymbol to the
correct ParserSymbol when the lexer is written out.
Uses the algorithms described in Compilers - Principles, Techniques,
and Tools, p.241 to generate spontaneous lookaheads and propagate them
through the state machine.
Adds GrammarSymbolSet to store symbol sets as a bitmask by symbol index
and a vector of GrammarSymbol*.
Multi-threads the calculation of goto items for lookahead propagation.
Sets the precedence and associativity of an implicit terminal from the
literal or identifier that sets it. It's an error for both the terminal
and the identifier to set it.
Implicit terminals can be identified by their literal value or by the
identifier that names them. But if both forms are used to specify an
associativity and precedence then it's not clear which one is correct.
Make it an error if associativity is specified for a terminal by its
literal and its identifier.
Scan for whitespace until no whitespace token matches and reset the
position to the start of scanning so that whitespace prefixes that are
also valid tokens aren't skipped.
Back-tracks to the most recently matched symbol and lexeme if a run
continues but doesn't match anoter symbol. Allows tokens that are
prefixes of other longer tokens to be matched after attempting to match
the longer token fails.
Previously action handlers for tokens and whitespace were stored in the
same vector. But indices for the tokens and whitespace were overlapped
because they are generated for two separate state machines. Separating
them into their own vectors allows the two to work together.
Improves error detection of undefined symbols to included symbols
referenced in precedence directives but not defined as terminals or
referenced from %precedence directives. Symbols referenced from
%precedence directives are assumed to be dummy terminals used to set
the precedence for a particular production.
