C++ Tokens Explained: Keywords, Identifiers, Literals and Worked Examples
Learn how C++ source code becomes tokens, then practise exact token counts, identifier rules, operator boundaries, precedence and output prediction.
KnowledgeGate Team
Exam prep & CS education

A C++ line looks like one instruction, but the compiler first encounters smaller units such as int, total_marks, =, 12, + and ;. Learners often mix up characters, words, tokens and syntax, then miscount a string, a negative value or a multi-character operator. Tokenisation separates source text into classified units before parsing, and those boundaries determine how C++ expressions are interpreted. The Coding & DSA courses connect this foundation to program tracing and interview practice.
From source characters to C++ tokens
A token is the smallest meaningful lexical unit passed onward for syntactic analysis. Consider this source line:
int total_marks = 12 + 3 * 4;Characters are the individual letters, digits, spaces and symbols typed. Lexemes are the actual spellings found in the source, such as int and total_marks. Token classes describe what those lexemes are, such as keyword and identifier.
The line has exactly nine tokens, in order: int, total_marks, =, 12, +, 3, *, 4, ;. That is one keyword, one identifier, three integer literals, three operators and one punctuator. Spaces make the boundaries readable, but they are not among the nine tokens.
C++ first recognises preprocessing tokens, handles directives and macros, then parses the resulting tokens. Lexical Analysis in Compiler Design: Tokens and Lexemes explains how regex patterns become DFAs and how scanner buffering works. C++ token questions instead ask which spellings form tokens, how many appear in a statement, and what the resulting expression outputs.

The five token classes most questions expect
Use this working classification:
Keywords: reserved spellings such as
int,if,returnandwhile.Identifiers: names such as
total_marks,stdandcout.Literals: written values such as
12,2.5,'K'and"KG".Operators: spellings such as
=,+,*,+=and<<.Punctuators: structural spellings such as
;,,,(,),{,}and::.
C++ lexical grammar groups operator and punctuation spellings closely, while classroom questions often separate them by role. Follow the classification stated in a question when it is explicit.
std::cout is three tokens, std, ::, cout, not one identifier. long long count = 7; has six tokens: long, long, count, =, 7, ;. The two long spellings are separate keyword tokens.
Tokenisation identifies a spelling, not its full meaning. << is one operator token in both std::cout << 24 and 6 << 2. The first use performs stream insertion. The second is a bitwise left shift and produces 24.
Keywords, identifiers and literals without memorisation traps
In ASCII-focused questions, an identifier uses letters, digits and underscores, but cannot start with a digit or be a keyword. Thus _score2 and marks_2026 are valid spellings. 2score cannot begin as one identifier, total-marks contains the - operator, and class is a keyword. Modern C++ identifiers can also use permitted non-ASCII characters.
Production code has another trap. Names such as __cache and _Node are reserved to the implementation, while _value is reserved at global namespace scope. Prefer ordinary descriptive names such as cache_size and node_count.
42 is an integer literal, while 42u is an unsigned integer literal. 3.5 is a floating-point literal, and 3.5f is one with an f suffix. 'K' is a character literal, "KG" is a string literal, and '\n' is one character literal containing an escape sequence.
The minus sign is separate. int loss = -42; has six tokens: int, loss, =, -, 42, ;. The - is an operator token and 42 is the literal token.
Worked program: count the tokens, evaluate the expression, predict the output
Trace a complete program:
#include <iostream>
int main() {
int total_marks = 12 + 3 * 4;
std::cout << "Total: " << total_marks << '\n';
}The declaration contributes the same nine tokens. Parsing and evaluation then apply precedence: 3 * 4 = 12, followed by 12 + 12 = 24. Assignment stores 24 in total_marks.
The output statement has exactly ten tokens: std, ::, cout, <<, "Total: ", <<, total_marks, <<, '\n', ;. The space and colon inside "Total: " remain inside one string-literal token. Each << is one token, and '\n' is one character-literal token.
The visible output is:
Total: 24It is followed by a newline. Treat #include <iostream> separately as a preprocessing directive. In this include context, <iostream> is recognised as a header-name preprocessing token, so it does not belong in either statement count.

Tokenisation traps that change the answer
C++ generally forms the longest valid token it can from the next source characters. Given int a = 5, b = 2; int result = a+++b;, the compact ending becomes a, ++, +, b. It behaves as a++ + b: the value used from a is 5, result becomes 5 + 2 = 7, and then a becomes 6. Write a++ + b to make the intended boundary clear.
In int value = 3; value <<= 1;, <<= is one compound-assignment token. Shifting 3 left by one bit gives 6, which is assigned back to value. If the next expression is value << 1, << is one token and produces 12 from the current value 6, but it does not assign 12 back by itself.
Comments and quotes change what can be counted. In sum/* subtotal */+5, the block comment separates sum from +, leaving the meaningful sequence sum, +, 5. In std::string label = "C++ tokens";, the space and two plus characters inside the quotes belong to the single string-literal token "C++ tokens".
Finally, recognisable tokens do not guarantee a valid program. int = value 5; contains recognisable spellings, but their order does not form a valid declaration. Token recognition is only the first structural step.
How exams and interviews test C++ tokens
Typical questions ask you to count tokens in a line, classify a highlighted spelling, choose a valid identifier, or predict output where precedence or a token boundary matters.
Try these four checks, then compare immediately:
float rate = 7.5f;has five tokens:float,rate,=,7.5f,;.In
std::string name = "KG";,stdandstringare separate identifiers with::between them, while"KG"is one string literal._score2is a valid ordinary spelling,2scoreis not an identifier spelling,classis a keyword, and__scoreis reserved to the implementation.int result = 5 + 2 * 3;has nine tokens and stores11, because2 * 3 = 6is evaluated first, then5 + 6 = 11.
An oral interview may ask, “Is -18 one token?” In the expression -18, unary - and integer literal 18 are separate tokens. Together, the expression evaluates to the negative value -18.
Short version and the next practice step
Tokens are meaningful lexical units.
Token spellings and token classes are different ideas.
Keywords are reserved.
Identifiers name program entities.
Literals carry written values.
Operators and punctuators connect and structure a program.
Whitespace and comments usually separate tokens, while quoted spaces remain inside a string literal.
For a 15-minute exercise, tokenise int points = 8 + 2 * 5; and std::cout << "P=" << points << '\n'; before compiling. The declaration has nine tokens, the output statement has ten, 2 * 5 = 10, and 8 + 10 = 18. Therefore points stores 18, and the output is P=18 followed by a newline.
Now replace the expression with (8 + 2) * 5. The declaration has eleven tokens because ( and ) are tokens. Evaluate 8 + 2 = 10, then 10 * 5 = 50.
Use the C++ Tutorial: Complete Learning Path as the broader next read. For a structured route, continue with the C++ Programming Course: Concepts, MCQs and Coding. For mixed-language placement practice, choose Coding for Placements: C, C++, Java and Python. Before you run the next example, tokenise one line by hand.
Keep learning
Tokens in C++ MCQs: 11 Solved Questions on Identifiers, Operators and Whitespace
Solve 11 real exam and placement-test C++ token questions, then learn the small rules that settle identifier, whitespace, operator, scope and output traps.
Compiler Design Semester Guide: Trace One Program from Tokens to Target Code
Follow one assignment through lexical analysis, parsing, semantic checks, TAC, optimization and abstract target instructions, with every hand-off made visible.
Java Strings and String Pool: Literals, intern(), == and equals() with Worked Examples
Trace pooled and non-pooled Java String references, compare identity with content, and learn to count objects without hiding the assumptions.
JavaScript Strings Tutorial: Methods, Template Literals, and Runnable Examples
Learn strings from their delimiters and zero-based indexes through searching, cleaning, and coercion. Each example is runnable, including a complete input-to-key transformation.





