NLP Processing Stages Explained: Trace One Sentence from Tokens to Pragmatic Meaning

Tokenisation, morphology, syntax, semantics, discourse and pragmatics are easy to memorise as a list. The trouble begins when a question gives an example and asks which stage produced a token, tree, meaning representation or contextual inference. A semicolon-linked sentence keeps the words constant while every stage produces a different output. A two-sentence variant then exposes the extra work required for cross-sentence reference.

1. NLP processing stages: the map before the details

In the classical teaching sequence, written input moves through normalisation and tokenisation, lexical and morphological analysis, syntactic analysis, semantic analysis, discourse integration and pragmatic analysis. Another syllabus may call tokenisation preprocessing or separate morphology from lexical analysis. The labels can move, but the operation performed at each stage remains the deciding evidence.

Each stage asks a different question:

• Tokens, lexicon and morphology: What are the units and word forms?

• Syntax: How are those units grammatically connected?

• Semantics: What literal proposition is expressed?

• Discourse: How do entities and events connect across sentences?

• Pragmatics: What does the speaker probably intend in context?

Written-text analysis starts at tokenisation. Speech NLP first needs an acoustic or phonological front end before the language stages can operate.

2. Lexical and morphological analysis: identify forms before relationships

Tokenisation splits the input into units. Lexicon lookup supplies candidate entries, part-of-speech tagging assigns grammatical categories, lemmatisation finds a dictionary form, and morpheme analysis identifies meaningful parts. Thus deposited = deposit + PAST, rupees = rupee + PLURAL, and credited = credit + PAST. A lemma is a valid dictionary form; a stem may only be a mechanically shortened form.

Lexical analysis also exposes ambiguity. bank may mean a financial institution or a river edge. Credit may mean adding funds or acknowledging merit. Do not resolve these senses yet. This stage supplies candidates and local features; later context does the heavier disambiguation.

For preprocessing, feature extraction and a bigram calculation, use Natural Language Processing in AI: Core Concepts, a Worked Bigram Example, and How Exams Test It. Stage-boundary questions need a different lens: word form is morphology, grammatical relation is syntax, literal proposition is semantics, cross-clause reference is discourse, and contextual inference is pragmatics.

3. Syntactic and semantic analysis: structure first, literal meaning next

Syntax describes grammatical structure through constituency or dependency relations. In Riya deposited 5000 rupees in the bank; it credited her account immediately., the semicolon joins two clauses inside one sentence. The first clause contains subject Riya, verb deposited, object 5000 rupees and prepositional phrase in the bank. The second contains subject it, verb credited, object her account and adverb immediately.

Semantics represents literal predicate-argument meaning after grammatical roles and word senses are selected. The deposit event has amount=5000; the credit event must retain amount=UNKNOWN because the sentence does not state that amount. For a formal parsing bridge, top-down and bottom-up parsing explains compiler parsing strategies, although current NLP parsers are not restricted to those methods.

4. Discourse and pragmatics: meaning beyond one sentence

Discourse integration links entities and events across clauses and sentences. Replace the semicolon with a full stop: Riya deposited 5000 rupees in the bank. It credited her account immediately. The second sentence requires It -> bank_1 and her -> Riya, while the two events retain their order in the discourse. Sentence-by-sentence parsing alone cannot complete those links.

Pragmatics uses world knowledge and communicative context to choose an intended reading. Money, deposit and account cues favour the financial-institution sense of bank. They also support the inference that the deposit led to the credit.

Inference must remain separate from assertion. e1 likely causes e2 and credited amount = 5000 are plausible, but the wording explicitly supplies only the deposited amount and the sequence of clauses. Likewise, the literal semantics of Can you pass the salt? asks about ability, while its usual pragmatic force is a request.

5. Fully worked example: trace the same input through every NLP stage

Use this one-sentence input:

Riya deposited 5000 rupees in the bank; it credited her account immediately.

Under a whitespace-plus-punctuation convention:

• Tokens: [Riya, deposited, 5000, rupees, in, the, bank, ;, it, credited, her, account, immediately, .] = 14 tokens.

• Punctuation contributes ; and ., so 14 - 2 = 12 lexical tokens after punctuation is excluded.

• The semicolon joins two independent clauses without creating a second sentence.

The selected morphology is Riya/PROPN, deposited/deposit+V+PAST, 5000/NUM, rupees/rupee+N+PLURAL, bank/N, it/PRON, credited/credit+V+PAST, her/PRON-POSS, account/N and immediately/ADV.

The constituency shorthand is:

[S [S [NP Riya] [VP deposited [NP 5000 rupees] [PP in [NP the bank]]]]
   ; [S [NP it] [VP credited [NP her account] [ADVP immediately]]]]

The literal representations are:

deposit(e1, agent=Riya, amount=5000, unit=INR, location=bank_1, tense=past)
credit(e2, agent=bank_1, patient=account_of(Riya), amount=UNKNOWN, time=immediate, tense=past)

Discourse links it -> bank_1, her -> Riya and e1 before e2. Context supports bank = financial institution and e1 likely causes e2. It still does not state that the credited amount is 5000, so copying that value into e2 would turn an inference into a false assertion.

6. Common traps: where stage labels get mixed up

Five repairs handle the most common mistakes:

1. If an operation splits units, it is tokenisation. If it analyses internal word structure, it is morphology.

2. A tree or dependency relation is syntax. A predicate or role representation is semantics.

3. Literal meaning and likely contextual meaning are not interchangeable. Compare amount=UNKNOWN with the tempting but unstated amount=5000.

4. In the two-sentence variant, It -> bank_1 needs cross-sentence context, so it belongs to discourse or coreference, not only sentence semantics.

5. There is no universal labelling scheme. Read the operation definitions in the question, especially when morphology is separate or tokenisation is called preprocessing.

A modern transformer need not execute six visible boxes in sequence. The stages are an explanatory decomposition of tasks, not a claim about the only possible NLP architecture.

7. How exams test NLP processing stages

Four common practice formats test whether you can recognise the output of a stage:

1. Order: Rearrange pragmatic, lexical, discourse, syntactic, semantic. The classical answer is lexical, syntactic, semantic, discourse, pragmatic.

2. Match: Map deposited -> deposit + PAST to morphology, a bracketed tree to syntax, and It -> bank_1 to discourse or coreference.

3. Disambiguate: Identify bank = financial institution as word-sense resolution informed by context.

4. Reject an unsupported value: Spot a representation that writes credit amount=5000 instead of UNKNOWN.

8. The short version and the next step

Use this five-line decision rule:

• Word form means lexical or morphological analysis.

• Grammatical relation means syntax.

• Literal proposition means semantics.

• Cross-sentence link means discourse.

• Intended contextual reading means pragmatics.

Two safeguards complete the method: 14 versus 12 depends on whether punctuation is counted, and the credited amount remains UNKNOWN.

Continue into models and applications with the Artificial Intelligence course. To compare adjacent learning routes first, browse the Skills category.