Token questions punish two habits: counting the characters inside a string literal instead of counting the literal as one token, and blaming the lexer for an error that semantic analysis actually catches. Both habits are tested below, alongside finite automata, the input Lex and Flex expect, maximal munch and three token counts scanned position by position. Commit to an option before you read the answer, and if the distinction between tokens, lexemes and patterns is not firm yet, revise that first. For a broader sweep of the same phase, work through Lexical Analysis MCQs: 10 solved compiler design questions as well.
1. Lexical analysis and tokens: the six terms every question turns on
Term | Meaning | Concrete example |
|---|---|---|
Character set | Permitted input symbols | Letters, digits and punctuation |
Token | A category assigned by the scanner |
|
Lexeme | The actual matched text |
|
Pattern | The rule describing a token class |
|
Scanner output | The token stream sent onward |
|
Ignored input | Whitespace and comments, unless the language gives them significance | Spaces between C tokens |
For sum = value1 + 42;, scan sum, =, value1, +, 42, ;. These map to IDENTIFIER, ASSIGN, IDENTIFIER, PLUS, INTEGER_LITERAL, SEMICOLON. The count is 6: value1 and 42 are each one lexeme.
The lexer groups characters into token classes, the parser checks grammar, and semantic analysis checks declarations, types and meaning. Parsing in Compiler Design: Top-Down and Bottom-Up Explained develops the next phase.
2. Lexical-analyzer roles and phase boundaries: Q1-Q4
For every phase question, classify the issue as characters, grammar or meaning.
Q1. GATE 2021
Consider the following ANSI C program:
int main () {
Integer x;
return 0;
}Which one of the following phases in a seven-phase C compiler will throw an error?
A. Lexical analyzer
B. Syntax analyzer
C. Semantic analyzer
D. Machine dependent optimizer
Answer: C. Semantic analyzer. Integer is identifier-shaped, so the scanner emits a perfectly good identifier token, and the parser accepts an identifier sitting in the type position of a declaration. Nothing in the program declares Integer as a type, and an undeclared name is a meaning error, which is why the failure surfaces two phases later. Walk all seven phases on the GATE 2021 phase question, solved step by step.
Q2.
Which of the following is not the function of a lexical analyzer?
A. Storing token or identifier information in the symbol table.
B. Identification of lexemes in high-level code.
C. Tracking the line number and column number of each token.
D. Macro expansion in the code.
Answer: D. Macro expansion in the code. The preprocessor expands macros before the scanner ever sees the file. The other three are genuine scanner duties: it identifies lexemes, records the line and column of each token so later phases can report positions, and hands identifier entries to the symbol table. More questions of this shape sit in the Lexical Analysis and Tokens PYQ practice set.
Q3. GATE 2011 and TPSC 2025
In a compiler, keywords of a language are recognized during
A. parsing of the program
B. the code generation
C. the lexical analysis of the program
D. dataflow analysis
Answer: C. the lexical analysis of the program. The scanner matches an identifier-shaped lexeme, looks it up against the reserved-word list and emits WHILE or RETURN rather than IDENTIFIER. By the time the parser sees the stream, the keyword decision is already made. Compare with the GATE 2011 keyword-recognition solution.
Q4. Beltron Programmer 2025
What is the role of the character set in the formation of tokens during the lexical analysis phase of program compilation?
A. It governs how functions are linked during the compilation process.
B. It ensures that runtime memory is properly allocated for variables.
C. It specifies the logical grouping of code into syntactic units.
D. It defines the range of permissible symbols used to construct valid tokens.
Answer: D. It defines the range of permissible symbols used to construct valid tokens. The character set decides that value1 is even a legal sequence of input symbols; the pattern [A-Za-z_][A-Za-z0-9_]* then classifies it as IDENTIFIER. Linking, memory allocation and syntactic grouping belong to three different phases. The reasoning is laid out in the solved character-set question.
3. Regular expressions, finite automata and Lex/Flex: Q5-Q8
Remember regular expression -> regular language -> NFA/DFA -> scanner. Arbitrary nested syntax needs more than a finite automaton.
Q5. GATE 2011
The lexical analysis for a modern computer language such as Java needs the power of which one of the following machine models in a necessary and sufficient sense?
A. Finite state automata
B. Deterministic pushdown automata
C. Non-deterministic pushdown automata
D. Turing machine
Answer: A. Finite state automata. Every Java token class (identifiers, literals, operators) is a regular language, so a finite automaton is sufficient. It is also necessary, since even matching [A-Za-z_$][A-Za-z0-9_$]* needs state. Pushdown power is only required once nesting has to be counted, which is the parser's job. See the GATE 2011 automata-power solution.
Q6. Beltron Programmer 2025
What type of grammar is accepted by LEX for token recognition?
A. Unrestricted grammar
B. Regular grammar
C. Context-free grammar
D. Context-sensitive grammar
Answer: B. Regular grammar. LEX takes regular patterns and compiles them into a finite-state recognizer, which is exactly the machine a regular grammar describes. Context-free rules go to YACC or Bison, one layer up. The distinction is worked out in the solved LEX grammar question.
Q7. Beltron Programmer 2025
Which of the following statements about token recognition using finite automata is true?
A. Finite Automata are not suitable for recognizing operators in source code.
B. Lexical analyzers require context-sensitive grammars for accurate token recognition.
C. Every token defined by a regular expression can be recognized by a Deterministic Finite Automaton (DFA).
D. Only Nondeterministic Finite Automata (NFA) can be used to recognize tokens in a compiler.
Answer: C. Every token defined by a regular expression can be recognized by a Deterministic Finite Automaton (DFA). A regular expression, an NFA and a DFA all describe the same class of languages, so subset construction turns any token pattern into a DFA. That also kills A and D: operators such as ++ and <<= are finite patterns, and nothing forces the scanner to stay non-deterministic. Check your reasoning against the solved DFA-recognition question.
Q8. Beltron Programmer 2025
What input does a tool like Lex/Flex require to generate a tokenizer?
A. Abstract syntax trees (ASTs)
B. Machine-specific optimization flags
C. Regular expressions defining tokens
D. Context-sensitive grammar rules
Answer: C. Regular expressions defining tokens. You write pattern and action pairs, for example [0-9]+ returning an integer-literal token and [A-Za-z_][A-Za-z0-9_]* returning an identifier token, and Lex or Flex generates the scanner from them. Abstract syntax trees are produced later by the parser, and optimization flags never reach the tokenizer. See the solved Lex/Flex input question.
4. Longest-prefix matching in lexical analysis: Q9
Maximal munch selects the token matching the longest available input prefix, then restarts at the first unconsumed character.
Q9. GATE 2018
A lexical analyzer uses the following patterns to recognize three tokens T1, T2, and T3 over the alphabet {a,b,c}.
T1: a? (b|c)*a
T2: b? (a|c)*b
T3: c? (b|a)*cNote that
x?means 0 or 1 occurrence of the symbol x. Note also that the analyzer outputs the token that matches the longest possible prefix.If the string
bbaacabcis processed by the analyzer, which one of the following is the sequence of tokens it outputs?
A. T1T2T3
B. T1T1T3
C. T2T1T3
D. T3T3
Answer: D. T3T3. Longest prefix wins, so measure all three from the first character. T1 cannot pass the first a: its (b|c)* consumes bb and the required final a closes it at bba, length 3. T2 manages only bb, length 2. T3 skips its optional c, lets (b|a)* consume bbaa and closes on the c: bbaac, length 5, so T3 is emitted. Restart at abc, where T3 again takes everything with (b|a)* = ab, length 3. 5 + 3 = 8 characters, the whole string, so the output is T3 then T3. Compare your trace with the GATE 2018 maximal-munch solution.
5. Token-counting MCQs with exact scans: Q10-Q12
Apply four rules: a complete string is one token; ++ is one token; each delimiter counts separately; whitespace does not count.
Q10. UGC NET 2014
How many tokens will be generated by the scanner for the following statement?
x = x * (a + b) - 5;
A. 12
B. 11
C. 10
D. 07
Answer: A. 12. Scan left to right: x(1), =(2), x(3), *(4), ((5), a(6), +(7), b(8), )(9), -(10), 5(11), ;(12). Every parenthesis and the semicolon are separate tokens, and the spaces contribute nothing. The same scan appears in the UGC NET 2014 token-count solution.
Q11. GATE 2000
The number of tokens in the following C statement is
printf("i = %d, &i = %x", i, &i);
A. 3
B. 26
C. 10
D. 21
Answer: C. 10. Scan printf(1), ((2), the whole string literal (3), ,(4), i(5), ,(6), &(7), i(8), )(9), ;(10). The trap is the literal: everything between the quotes, including the %d, the comma and the &i inside it, is one token, while the & and i outside the quotes are two. Full working is on the GATE 2000 printf token-count solution.
Q12. ISRO 2020
The number of tokens in the following C code segment is
switch(inputvalue)
{
case 1 : b = c * d; break;
default : b = b++; break;
}A. 27
B. 29
C. 26
D. 24
Answer: C. 26. Scan the header first: switch(1), ((2), inputvalue(3), )(4), {(5). Then the case arm: case(6), 1(7), :(8), b(9), =(10), c(11), *(12), d(13), ;(14), break(15), ;(16). Then the default arm: default(17), :(18), b(19), =(20), b(21), ++(22), ;(23), break(24), ;(25), and the closing }(26). Maximal munch keeps ++ as one token; splitting it into two plus signs is what produces the 27 in option A. See the ISRO 2020 token-count solution.
6. How exams test lexical analysis and where token questions go wrong
Question cue | First action | Typical trap |
|---|---|---|
Compiler phase | Classify character, grammar or meaning | Calling every bad identifier a lexical error |
Regex or automaton | Ask whether the token language is regular | Choosing a PDA because compilers use grammars |
Longest prefix | Write each consumed prefix and its length | Stopping at the first valid match |
Token count | Mark literals and multi-character operators first | Counting characters inside strings or splitting |
Examiners reuse four shapes, and the table above is the order to apply them in: name the phase, name the machine model, trace the longest prefix, count the tokens. The first two are recall and take seconds. The last two are scans, and they are only reliable when you write the prefix or the numbered token list out rather than eyeballing it. More compiler-design practice sits under GATE CS Exam Preparation.
In 60 seconds, box literals, circle names, underline longest-match operators, count delimiters, and ignore spacing. Redo Q1, Q9, Q11 and Q12 blind.
7. Token questions: the short version and your next step
The lexer consumes characters and emits tokens.
A lexeme is the actual matched text.
Token patterns are regular.
Finite automata implement those patterns.
Maximal munch takes the longest valid prefix.
A complete string and a multi-character operator each count as one token.
Try again after one day. Hide the options and write the Q9 to Q12 token streams.
For structured topic tests and mock practice, continue with the GATE Test Series after that attempt.
