Quote Originally Posted by kaos View Post
Each token gets a unique id when the lexer file gets parsed. With this unique id several infos about the token are saved (the name, the language, etc)
But then how do you handle parsing numbers for instance? It must be done with a more complex DFA than a keyword and allow for many more than one single token... I guess you meant that the id is associated with the DFA matching state or am I missing something?