Character Encoding Schemes: ASCII, ISCII, Unicode and UTF-8 Worked Examples

Memorising that ASCII is 7-bit and UTF-8 is variable-length is not enough when a question asks for actual bytes, the number of possible codes or why one scheme stores an emoji while another cannot. The string Aह😀 becomes 41 E0 A4 B9 F0 9F 98 80 in UTF-8, and its UTF-16BE and UTF-32BE forms expose the storage trade-off. A byte order mark is excluded consistently, so the totals compare only the encoded text. The GATE CS Exam Preparation route places this encoding trace alongside number systems and computer organisation.

Character encoding schemes begin with four different ideas

A character repertoire is a set of supported abstract characters. A code point is a character's assigned number, such as A -> U+0041 and ह -> U+0939. A code unit is an encoding's fixed-size building block; an encoded byte sequence is the stored or transmitted bytes.

Unicode assigns code points across writing systems; UTF-8, UTF-16 and UTF-32 encode them as code units and bytes. Unicode is not a 16-bit encoding, and one grapheme may contain several code points.

Under a compatible encoding, 01000001 means A; elsewhere it may be an integer, instruction fragment or part of another value.

ASCII and ISCII solve different limited problems

ASCII is a 7-bit code with 2^7 = 128 values. For A, decimal 65 = 64 + 1, so its 7-bit form is 1000001. In an 8-bit byte, it is 01000001 = 0x41.

An 8-bit byte has 2^8 = 256 patterns. ASCII uses 128; the other 128 are outside standard 7-bit ASCII, not one universal "extended ASCII".

ISCII, the Indian Script Code for Information Interchange, is an older 8-bit Indian-script standard. It reuses a code region across related scripts with script-selection context. In the ISCII-91 Devanagari table, ह maps to byte 0xD8. The byte is not self-describing: with another Indic script active, that code position represents the analogous character in that script.

Unicode, UTF-8, UTF-16 and UTF-32 compared

UTF-8 uses 0xxxxxxx, 110xxxxx 10xxxxxx, 1110xxxx 10xxxxxx 10xxxxxx and 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx. Continuation bytes start with 10; the leading prefix gives the length.

Across UTFs, A = U+0041, ह = U+0939 and 😀 = U+1F600; only their code units and bytes change.

UTF-8 worked example: encode Aह😀 byte by byte

A fits 0xxxxxxx: 01000001 = 0x41.

Split U+0939 into 0000 | 100100 | 111001, then add three-byte prefixes:

11100000 10100100 10111001 = E0 A4 B9

Split U+1F600 into 000 | 011111 | 011000 | 000000, then add four-byte prefixes:

11110000 10011111 10011000 10000000 = F0 9F 98 80

Concatenation gives 41 E0 A4 B9 F0 9F 98 80, or 1 + 3 + 4 = 8 bytes without a byte order mark. If regrouping binary as hexadecimal is unfamiliar, revise Number Systems and Base Conversions Explained.

UTF storage comparison for the same text

In UTF-16BE without a byte order mark, A is 00 41 and ह is 09 39. The emoji needs a surrogate pair. Start with U+1F600 - 0x10000 = 0xF600:

• High surrogate: 0xD800 + (0xF600 >> 10) = 0xD83D

• Low surrogate: 0xDC00 + (0xF600 & 0x3FF) = 0xDE00

UTF-16BE is 00 41 | 09 39 | D8 3D DE 00, totalling 2 + 2 + 4 = 8 bytes.

UTF-32BE is 00 00 00 41 | 00 00 09 39 | 00 01 F6 00, exactly 3 x 4 = 12 bytes without a byte order mark. ASCII cannot encode the full string because it lacks ह and 😀.

For an English-only check, 100 ASCII code points occupy 100 UTF-8 bytes and 400 UTF-32BE bytes, without a byte order mark. UTF-8 is smaller for this input, not every possible string.

UTF-8 decoding checks catch invalid sequences

Reverse E0 A4 B9. Its 1110 prefix announces three bytes. Remove prefixes 1110, 10 and 10 to recover 0000 | 100100 | 111001. Concatenation gives 0000100100111001 = 0x0939, or U+0939, ह.

Check mechanically: read the length from the leader, confirm continuations start with 10, recover the payload, reject illegal or incomplete sequences, then map the point. E0 41 B9 is invalid because 0x41 starts with 01, not 10.

Compare Floating Point Representation: IEEE 754 Format. Its field rules cannot decode text bytes, and UTF rules cannot decode a floating-point word.

Character encoding exam patterns and common traps

Exam questions usually combine abbreviation expansion, capacity, byte conversion, storage comparison and scheme selection. Use the declared encoding as the first constraint, then show the bit grouping and state whether a byte order mark is included.

Correct these traps:

• "ASCII is an 8-bit standard." It is 7-bit, though commonly stored in a byte.

• "Unicode and UTF-8 are synonyms." Unicode assigns code points; UTF-8 encodes them.

• "UTF-32 uses 32 bytes." It uses 32 bits, or 4 bytes, per code unit.

• "Fixed-width means one visible character." A grapheme can span code points.

• "Any byte above 127 is extended ASCII." The particular encoding must be named.

The GATE Test Series is an optional route for mixed and topic-wise practice.

Character encoding schemes: the short version and next step

• ASCII assigns 128 seven-bit codes.

• ISCII is an 8-bit Indian-script standard with contextual script selection.

• Unicode assigns code points, while UTFs encode those points.

• Aह😀 takes 8 bytes in UTF-8, 8 in UTF-16BE and 12 in UTF-32BE under the no-BOM convention.

• The declared format tells you what a bit pattern means.

Closed notes, reproduce A -> 41, ह -> E0 A4 B9 and 😀 -> F0 9F 98 80, then decode E0 A4 B9 to U+0939. If stuck, redraw the four UTF-8 prefixes. For structured study, continue with GATE Guidance by Sanchit Sir.