What input does a tool like Lex/Flex require to generate a tokenizer?
2025
What input does a tool like Lex/Flex require to generate a tokenizer?
- A.
Abstract syntax trees (ASTs)
- B.
Machine-specific optimization flags
- C.
Regular expressions defining tokens
- D.
Context-sensitive grammar rules
Attempted by 44 students.
Show answer & explanation
Correct answer: C
Computers cannot directly "read" a regular expression. Under the hood, Flex takes your list of regular expressions and mathematically converts them into a Deterministic Finite Automaton (DFA). This DFA is essentially a highly optimized state machine generated as raw C/C++ code.
When your compiled compiler runs, this state machine processes the input text character-by-character, matching the text patterns against your regular expressions, and outputs the corresponding tokens for the next phase of the compiler (the parser) to use.