C++ Tokens Explained: Keywords, Identifiers, Literals and Worked Examples

Learn how C++ source code becomes tokens, then practise exact token counts, identifier rules, operator boundaries, precedence and output prediction.

KnowledgeGate Team

Exam prep & CS education

Updated 14 Nov 20266 min read
C++ Tokens Explained: Keywords, Identifiers, Literals and Worked Examples

A C++ line looks like one instruction, but the compiler first encounters smaller units such as int, total_marks, =, 12, + and ;. Learners often mix up characters, words, tokens and syntax, then miscount a string, a negative value or a multi-character operator. Tokenisation separates source text into classified units before parsing, and those boundaries determine how C++ expressions are interpreted. The Coding & DSA courses connect this foundation to program tracing and interview practice.

From source characters to C++ tokens

A token is the smallest meaningful lexical unit passed onward for syntactic analysis. Consider this source line:

cpp
int total_marks = 12 + 3 * 4;

Characters are the individual letters, digits, spaces and symbols typed. Lexemes are the actual spellings found in the source, such as int and total_marks. Token classes describe what those lexemes are, such as keyword and identifier.

The line has exactly nine tokens, in order: int, total_marks, =, 12, +, 3, *, 4, ;. That is one keyword, one identifier, three integer literals, three operators and one punctuator. Spaces make the boundaries readable, but they are not among the nine tokens.

C++ first recognises preprocessing tokens, handles directives and macros, then parses the resulting tokens. Lexical Analysis in Compiler Design: Tokens and Lexemes explains how regex patterns become DFAs and how scanner buffering works. C++ token questions instead ask which spellings form tokens, how many appear in a statement, and what the resulting expression outputs.

Three-level diagram splitting the line int total_marks = 12 + 3 * 4; into nine token boxes and their classes.

The five token classes most questions expect

Use this working classification:

  • Keywords: reserved spellings such as int, if, return and while.

  • Identifiers: names such as total_marks, std and cout.

  • Literals: written values such as 12, 2.5, 'K' and "KG".

  • Operators: spellings such as =, +, *, += and <<.

  • Punctuators: structural spellings such as ;, ,, (, ), {, } and ::.

C++ lexical grammar groups operator and punctuation spellings closely, while classroom questions often separate them by role. Follow the classification stated in a question when it is explicit.

std::cout is three tokens, std, ::, cout, not one identifier. long long count = 7; has six tokens: long, long, count, =, 7, ;. The two long spellings are separate keyword tokens.

Tokenisation identifies a spelling, not its full meaning. << is one operator token in both std::cout << 24 and 6 << 2. The first use performs stream insertion. The second is a bitwise left shift and produces 24.

Keywords, identifiers and literals without memorisation traps

In ASCII-focused questions, an identifier uses letters, digits and underscores, but cannot start with a digit or be a keyword. Thus _score2 and marks_2026 are valid spellings. 2score cannot begin as one identifier, total-marks contains the - operator, and class is a keyword. Modern C++ identifiers can also use permitted non-ASCII characters.

Production code has another trap. Names such as __cache and _Node are reserved to the implementation, while _value is reserved at global namespace scope. Prefer ordinary descriptive names such as cache_size and node_count.

42 is an integer literal, while 42u is an unsigned integer literal. 3.5 is a floating-point literal, and 3.5f is one with an f suffix. 'K' is a character literal, "KG" is a string literal, and '\n' is one character literal containing an escape sequence.

The minus sign is separate. int loss = -42; has six tokens: int, loss, =, -, 42, ;. The - is an operator token and 42 is the literal token.

Worked program: count the tokens, evaluate the expression, predict the output

Trace a complete program:

cpp
#include <iostream>

int main() {
    int total_marks = 12 + 3 * 4;
    std::cout << "Total: " << total_marks << '\n';
}

The declaration contributes the same nine tokens. Parsing and evaluation then apply precedence: 3 * 4 = 12, followed by 12 + 12 = 24. Assignment stores 24 in total_marks.

The output statement has exactly ten tokens: std, ::, cout, <<, "Total: ", <<, total_marks, <<, '\n', ;. The space and colon inside "Total: " remain inside one string-literal token. Each << is one token, and '\n' is one character-literal token.

The visible output is:

Code
Total: 24

It is followed by a newline. Treat #include <iostream> separately as a preprocessing directive. In this include context, <iostream> is recognised as a header-name preprocessing token, so it does not belong in either statement count.

Four-panel trace: the nine declaration tokens, an expression tree giving 24, total_marks in memory, and the output Total: 24.

Tokenisation traps that change the answer

C++ generally forms the longest valid token it can from the next source characters. Given int a = 5, b = 2; int result = a+++b;, the compact ending becomes a, ++, +, b. It behaves as a++ + b: the value used from a is 5, result becomes 5 + 2 = 7, and then a becomes 6. Write a++ + b to make the intended boundary clear.

In int value = 3; value <<= 1;, <<= is one compound-assignment token. Shifting 3 left by one bit gives 6, which is assigned back to value. If the next expression is value << 1, << is one token and produces 12 from the current value 6, but it does not assign 12 back by itself.

Comments and quotes change what can be counted. In sum/* subtotal */+5, the block comment separates sum from +, leaving the meaningful sequence sum, +, 5. In std::string label = "C++ tokens";, the space and two plus characters inside the quotes belong to the single string-literal token "C++ tokens".

Finally, recognisable tokens do not guarantee a valid program. int = value 5; contains recognisable spellings, but their order does not form a valid declaration. Token recognition is only the first structural step.

How exams and interviews test C++ tokens

Typical questions ask you to count tokens in a line, classify a highlighted spelling, choose a valid identifier, or predict output where precedence or a token boundary matters.

Try these four checks, then compare immediately:

  1. float rate = 7.5f; has five tokens: float, rate, =, 7.5f, ;.

  2. In std::string name = "KG";, std and string are separate identifiers with :: between them, while "KG" is one string literal.

  3. _score2 is a valid ordinary spelling, 2score is not an identifier spelling, class is a keyword, and __score is reserved to the implementation.

  4. int result = 5 + 2 * 3; has nine tokens and stores 11, because 2 * 3 = 6 is evaluated first, then 5 + 6 = 11.

An oral interview may ask, “Is -18 one token?” In the expression -18, unary - and integer literal 18 are separate tokens. Together, the expression evaluates to the negative value -18.

Short version and the next practice step

  • Tokens are meaningful lexical units.

  • Token spellings and token classes are different ideas.

  • Keywords are reserved.

  • Identifiers name program entities.

  • Literals carry written values.

  • Operators and punctuators connect and structure a program.

Whitespace and comments usually separate tokens, while quoted spaces remain inside a string literal.

For a 15-minute exercise, tokenise int points = 8 + 2 * 5; and std::cout << "P=" << points << '\n'; before compiling. The declaration has nine tokens, the output statement has ten, 2 * 5 = 10, and 8 + 10 = 18. Therefore points stores 18, and the output is P=18 followed by a newline.

Now replace the expression with (8 + 2) * 5. The declaration has eleven tokens because ( and ) are tokens. Evaluate 8 + 2 = 10, then 10 * 5 = 50.

Use the C++ Tutorial: Complete Learning Path as the broader next read. For a structured route, continue with the C++ Programming Course: Concepts, MCQs and Coding. For mixed-language placement practice, choose Coding for Placements: C, C++, Java and Python. Before you run the next example, tokenise one line by hand.