Programming Language Syntax Explained: Grammar, Tokens and Semantics

Follow one assignment from its source characters to tokens, an AST, type checks and runtime evaluation. Learn exactly where syntax ends and semantics begins.

KnowledgeGate Team

Exam prep & CS education

Updated 12 Nov 20265 min read
Programming Language Syntax Explained: Grammar, Tokens and Semantics

Valid tokens do not guarantee valid syntax, and valid syntax does not guarantee meaningful or successful execution. The expression total = price + 3 * qty; contains 24 source characters, including six spaces, and scans into eight tokens. With the precedence grammar below, it builds one intended parse tree and evaluates to total = 19 when price = 4 and qty = 5. For broader programming practice, use Coding & DSA.

Put characters, tokens, grammar and semantics into one pipeline

Characters are raw source symbols. A lexer groups them into lexemes and assigns token types. A grammar defines legal token sequences, while a parser uses it to build a parse tree or a simpler abstract syntax tree (AST). Static semantics checks declarations, types and other context-sensitive rules. Dynamic semantics describes evaluation or execution.

Layer

Input

Output

Question answered

characters

total = price + 3 * qty;

raw stream

what was written?

lexer

raw stream

8 tokens

what are the units?

parser plus grammar

8 tokens

Assign AST

is the structure legal?

static semantics

AST plus symbol table

well-typed AST

is the program meaningful before execution?

dynamic semantics

AST plus environment

total becomes 19

what does it do?

Grammar is the rule set; a parser is the program using it. Syntax covers legal form, while semantics covers meaning. A parse tree proves structural legality, not valid names or types.

A six-stage pipeline traces source text through lexing, parsing, checking, and evaluation, ending with total = 19 from price = 4 and qty = 5.

Tokens and lexemes: scan without losing the categories

In total, the character sequence is the lexeme and ID its token type. In 3, the lexeme is 3, the type is INT, and the associated value is 3. Punctuation still has categories: + is PLUS and ; is SEMI.

Scanning left to right gives eight tokens: ID("total"), ASSIGN, ID("price"), PLUS, INT(3), STAR, ID("qty"), SEMI. This lexer discards the six separating spaces, so they add no tokens.

For longest match, assume identifiers consume the longest letter sequence and only complete if is a keyword. Then ifx = 10; becomes ID("ifx"), ASSIGN, INT(10), SEMI, not IF, ID("x"), ASSIGN, INT(10), SEMI. Continue with Lexical Analysis in Compiler Design: Tokens and Lexemes.

Grammar and parsing: force multiplication below addition

Use this precedence grammar for the miniature language:

Code
Stmt   -> id '=' Expr ';'
Expr   -> Expr '+' Term | Term
Term   -> Term '*' Factor | Factor
Factor -> id | num | '(' Expr ')'

It parses as Stmt(id(total), '=', Expr(Expr(Term(Factor(id(price)))), '+', Term(Term(Factor(num(3))), '*', Factor(id(qty)))), ';'). Removing punctuation and grammar-only nodes gives Assign(total, Add(Var(price), Mul(Int(3), Var(qty)))).

Separate Expr and Term levels make the structure price + (3 * qty). The original parses. total price + 3 * qty; fails because = is missing. total = price + * qty; fails because the required Term cannot begin with *. These are syntax judgments, not value judgments.

Static and dynamic semantics: turn the tree into 19

Take the symbol table total:int, price:int, qty:int. Check upward:

  1. INT(3) has type int, and qty has type int.

  2. Therefore, 3 * qty has type int.

  3. price is int, so price + (3 * qty) has type int.

  4. Assigning that result to total:int is permitted.

That is static semantics. Under {price: 4, qty: 5, total: 0}, substitute qty = 5 and calculate 3 * 5 = 15. Substitute price = 4 and calculate 4 + 15 = 19. Updating only total produces {price: 4, qty: 5, total: 19}, the chosen dynamic meaning of assignment.

Syntax alone is insufficient. In an extension that adds string literals, subtraction and division, total = price + "3"; tokenizes and parses, but static semantics rejects int + string. The well-typed total = price / (qty - 5); reaches 4 / (5 - 5) = 4 / 0 and fails dynamically. Neither is a syntax error.

Ambiguity, precedence and associativity change the answer

The ambiguous grammar E -> E '+' E | E '*' E | id | num permits both Add(Var(price), Mul(Int(3), Var(qty))) and Mul(Add(Var(price), Int(3)), Var(qty)) for our expression.

The first tree gives 4 + (3 * 5) = 4 + 15 = 19. The second gives (4 + 3) * 5 = 7 * 5 = 35. The same tokens have different structures. Our precedence grammar forms multiplication at Term, below addition at Expr, so only 19 survives.

Associativity resolves another choice. For 20 - 5 - 3, left association gives (20 - 5) - 3 = 15 - 3 = 12; right association gives 20 - (5 - 3) = 20 - 2 = 18. Continue with Ambiguous Grammars, Precedence and Associativity for GATE.

A side-by-side ambiguity figure compares two trees for price + 3 * qty, producing 19 with precedence and 35 with the alternative grouping.

Common traps and the repair for each one

  • Counting characters or spaces as tokens: list token categories after lexing.

  • Treating total as a token type: write ID("total").

  • Calling grammar the parser: separate the rule set from its implementation.

  • Assuming parse success proves type correctness: run static checks next.

  • Reading production order as precedence: inspect the non-terminal hierarchy.

  • Calling every failure a syntax error: name the earliest layer that cannot continue.

Apply the routine. total = price + 3 * qty; reaches evaluation and produces 19. total = price + * qty; has lexing as its last successful layer and stops at parsing. Under the string-enabled extension, total = price + "3"; reaches parsing but stops at static checking on int + string.

These results belong to this miniature language. Real languages differ in lexical rules, grammar notation, conversions, evaluation order and runtime behaviour.

How exams can test syntax, grammar, tokens and semantics

Practice questions may ask you to count tokens, distinguish tokens from lexemes, choose a derivation, find ambiguity, apply precedence and associativity, read an AST, classify errors or evaluate a tree. These are durable problem formats rather than a forecast of any paper's exact weightage or question count.

Use this 30-second sequence: copy the source, write its tokens, test the grammar, sketch the AST's operator skeleton, apply declarations and types, then substitute values. Report the earliest failing layer or final result. Here the checkpoints are 8 tokens -> valid Stmt -> Assign(total, Add(Var(price), Mul(Int(3), Var(qty)))) -> all int -> total = 19. Once the source-to-result trace is clear, practise the same distinctions across 12 varied items in Programming Language Syntax MCQs: 12 Solved Questions on Grammar, Tokens and Semantics, which focuses on question solving rather than reteaching this single pipeline.

Short version and the next study step

Lexemes are matched source fragments, tokens are their labelled categories, grammar defines legal structure, and semantics supplies context-sensitive validity and meaning. Here, eight tokens form Assign(total, Add(Var(price), Mul(Int(3), Var(qty)))); the symbol table makes every operation int; and {price: 4, qty: 5} produces total = 19. Use C Language for language-level concepts, MCQs and coding questions. If you can classify the malformed snippets and explain both 19 and 35, you understand the layers.