Programming Language Syntax Explained: Grammar, Tokens and Semantics
Follow one assignment from its source characters to tokens, an AST, type checks and runtime evaluation. Learn exactly where syntax ends and semantics begins.
KnowledgeGate Team
Exam prep & CS education

Valid tokens do not guarantee valid syntax, and valid syntax does not guarantee meaningful or successful execution. The expression total = price + 3 * qty; contains 24 source characters, including six spaces, and scans into eight tokens. With the precedence grammar below, it builds one intended parse tree and evaluates to total = 19 when price = 4 and qty = 5. For broader programming practice, use Coding & DSA.
Put characters, tokens, grammar and semantics into one pipeline
Characters are raw source symbols. A lexer groups them into lexemes and assigns token types. A grammar defines legal token sequences, while a parser uses it to build a parse tree or a simpler abstract syntax tree (AST). Static semantics checks declarations, types and other context-sensitive rules. Dynamic semantics describes evaluation or execution.
Layer | Input | Output | Question answered |
|---|---|---|---|
characters |
| raw stream | what was written? |
lexer | raw stream | 8 tokens | what are the units? |
parser plus grammar | 8 tokens |
| is the structure legal? |
static semantics | AST plus symbol table | well-typed AST | is the program meaningful before execution? |
dynamic semantics | AST plus environment |
| what does it do? |
Grammar is the rule set; a parser is the program using it. Syntax covers legal form, while semantics covers meaning. A parse tree proves structural legality, not valid names or types.

Tokens and lexemes: scan without losing the categories
In total, the character sequence is the lexeme and ID its token type. In 3, the lexeme is 3, the type is INT, and the associated value is 3. Punctuation still has categories: + is PLUS and ; is SEMI.
Scanning left to right gives eight tokens: ID("total"), ASSIGN, ID("price"), PLUS, INT(3), STAR, ID("qty"), SEMI. This lexer discards the six separating spaces, so they add no tokens.
For longest match, assume identifiers consume the longest letter sequence and only complete if is a keyword. Then ifx = 10; becomes ID("ifx"), ASSIGN, INT(10), SEMI, not IF, ID("x"), ASSIGN, INT(10), SEMI. Continue with Lexical Analysis in Compiler Design: Tokens and Lexemes.
Grammar and parsing: force multiplication below addition
Use this precedence grammar for the miniature language:
Stmt -> id '=' Expr ';'
Expr -> Expr '+' Term | Term
Term -> Term '*' Factor | Factor
Factor -> id | num | '(' Expr ')'It parses as Stmt(id(total), '=', Expr(Expr(Term(Factor(id(price)))), '+', Term(Term(Factor(num(3))), '*', Factor(id(qty)))), ';'). Removing punctuation and grammar-only nodes gives Assign(total, Add(Var(price), Mul(Int(3), Var(qty)))).
Separate Expr and Term levels make the structure price + (3 * qty). The original parses. total price + 3 * qty; fails because = is missing. total = price + * qty; fails because the required Term cannot begin with *. These are syntax judgments, not value judgments.
Static and dynamic semantics: turn the tree into 19
Take the symbol table total:int, price:int, qty:int. Check upward:
INT(3)has typeint, andqtyhas typeint.Therefore,
3 * qtyhas typeint.priceisint, soprice + (3 * qty)has typeint.Assigning that result to
total:intis permitted.
That is static semantics. Under {price: 4, qty: 5, total: 0}, substitute qty = 5 and calculate 3 * 5 = 15. Substitute price = 4 and calculate 4 + 15 = 19. Updating only total produces {price: 4, qty: 5, total: 19}, the chosen dynamic meaning of assignment.
Syntax alone is insufficient. In an extension that adds string literals, subtraction and division, total = price + "3"; tokenizes and parses, but static semantics rejects int + string. The well-typed total = price / (qty - 5); reaches 4 / (5 - 5) = 4 / 0 and fails dynamically. Neither is a syntax error.
Ambiguity, precedence and associativity change the answer
The ambiguous grammar E -> E '+' E | E '*' E | id | num permits both Add(Var(price), Mul(Int(3), Var(qty))) and Mul(Add(Var(price), Int(3)), Var(qty)) for our expression.
The first tree gives 4 + (3 * 5) = 4 + 15 = 19. The second gives (4 + 3) * 5 = 7 * 5 = 35. The same tokens have different structures. Our precedence grammar forms multiplication at Term, below addition at Expr, so only 19 survives.
Associativity resolves another choice. For 20 - 5 - 3, left association gives (20 - 5) - 3 = 15 - 3 = 12; right association gives 20 - (5 - 3) = 20 - 2 = 18. Continue with Ambiguous Grammars, Precedence and Associativity for GATE.

Common traps and the repair for each one
Counting characters or spaces as tokens: list token categories after lexing.
Treating
totalas a token type: writeID("total").Calling grammar the parser: separate the rule set from its implementation.
Assuming parse success proves type correctness: run static checks next.
Reading production order as precedence: inspect the non-terminal hierarchy.
Calling every failure a syntax error: name the earliest layer that cannot continue.
Apply the routine. total = price + 3 * qty; reaches evaluation and produces 19. total = price + * qty; has lexing as its last successful layer and stops at parsing. Under the string-enabled extension, total = price + "3"; reaches parsing but stops at static checking on int + string.
These results belong to this miniature language. Real languages differ in lexical rules, grammar notation, conversions, evaluation order and runtime behaviour.
How exams can test syntax, grammar, tokens and semantics
Practice questions may ask you to count tokens, distinguish tokens from lexemes, choose a derivation, find ambiguity, apply precedence and associativity, read an AST, classify errors or evaluate a tree. These are durable problem formats rather than a forecast of any paper's exact weightage or question count.
Use this 30-second sequence: copy the source, write its tokens, test the grammar, sketch the AST's operator skeleton, apply declarations and types, then substitute values. Report the earliest failing layer or final result. Here the checkpoints are 8 tokens -> valid Stmt -> Assign(total, Add(Var(price), Mul(Int(3), Var(qty)))) -> all int -> total = 19. Once the source-to-result trace is clear, practise the same distinctions across 12 varied items in Programming Language Syntax MCQs: 12 Solved Questions on Grammar, Tokens and Semantics, which focuses on question solving rather than reteaching this single pipeline.
Short version and the next study step
Lexemes are matched source fragments, tokens are their labelled categories, grammar defines legal structure, and semantics supplies context-sensitive validity and meaning. Here, eight tokens form Assign(total, Add(Var(price), Mul(Int(3), Var(qty)))); the symbol table makes every operation int; and {price: 4, qty: 5} produces total = 19. Use C Language for language-level concepts, MCQs and coding questions. If you can classify the malformed snippets and explain both 19 and 35, you understand the layers.
Keep learning
Data Abstraction Explained: Hide Implementation Details with a Worked Java Example
See how a stable Account contract protects its invariant while the Java implementation changes from a running balance to a transaction ledger.
Programming Paradigms Explained: Concepts, Classifications and Worked Examples
Learn how imperative, procedural, object-oriented, functional and declarative styles organise the same computation, and how to classify code from visible evidence.
Introduction to Visual Basic (VB): Concepts, Worked Code and Exam Patterns
Learn how a VB.NET form turns text-box input and a button click into a traced result. Then practise operators, loops, procedures, scope and debugging.
Types of Programming Languages: Levels, Paradigms and Execution Models
Learn why low-level, object-oriented, compiled and dynamically typed describe different axes. Build a clear classification map through one result expressed three ways.





