How Text Encoding Works

Send the word café through a system with the wrong settings and it can come back as café. Nothing was deleted — every byte made the trip intact. The garbling happens because computers never store text: they store numbers, and somewhere along the way the two sides disagreed about which number means which character. Understanding that mapping — encoding — explains mojibake, surprising file sizes, and why one letter can cost one byte or three.

Every character is secretly a number

Computers can only store numbers, so writing text requires an agreed dictionary between characters and numbers. That dictionary is called an encoding. Press A and what travels is the number 65; the screen that later displays "A" is doing the reverse lookup. As long as every device in the chain uses the same dictionary, the letter survives. When any link uses a different one, you get a wrong character, a replacement box, or soup like é. Everything about text encoding is built on this one idea: the character is not stored — its number is.

ASCII: the original 128

In 1963, American teleprinters standardized ASCII: 7 bits, giving codes 0–127. Ninety-five codes were printable characters — A as 65, a as 97, 0 as 48 — and 33 were control codes like newline that told a printer what to do rather than what to type. You can still see the entire map today in any ASCII Table, because ASCII is the foundation everything later grew from.

One hundred and twenty-eight codes cover English and nothing else. When bytes settled at 8 bits, codes 128–255 were free space, and every region filled it differently: Latin-1 served Western Europe, KOI8 served Russian, Shift-JIS served Japanese. The same byte value could mean é in Paris and a totally different character in Moscow. That ambiguity is the seed of every mojibake you have ever seen.

Unicode: one number for every character

Unicode, assembled from 1991 onward, assigns every character in every script a unique identifier called a code point, conventionally written U+xxxx: A is U+0041, é is U+00E9, € is U+20AC, 日 is U+65E5. Over 150,000 code points now cover living scripts, mathematics, and emoji.

The crucial subtlety: Unicode is a catalog of numbers, not bytes. It says "é is number 233" but says nothing about how to write 233 into a file. Filling that gap is the job of an encoding such as UTF-8 — the bridge between code points and actual bytes on disk or on the wire.

UTF-8: the compromise that won

UTF-8 represents a code point in one to four bytes: short codes get short byte runs, rare codes get longer ones. Two rules make it work. First, code points U+0000–U+007F — exactly ASCII — are one byte, unchanged, so every existing ASCII file is automatically valid UTF-8 and adoption cost almost nothing. Second, longer code points use a leading byte that announces how many bytes follow (110… for two, 1110… for three, 11110… for four), and every continuation byte starts with 10, so a decoder can always tell mid-sequence bytes apart and resync after damage. These properties carried UTF-8 to more than 98% of the web.

Why é becomes two bytes

é is code point 233 (U+00E9), binary 11101001 — too big for a single UTF-8 byte, so it takes the two-byte template, which offers 11 bits of space:

code point é  =  U+00E9 =            1110 1001   (233)
2-byte template  =  110xxxxx 10xxxxxx   (11 bits of space)
11-bit value     =  000 1110 1001
result           =  11000011 10101001  =  C3 A9

One character, two bytes: C3 A9. Meanwhile A (U+0041) stays a single byte (41), € (U+20AC, 16 bits) needs three bytes (E2 82 AC), and 日 (U+65E5) also takes three (E6 97 A5). This is why "café" is 4 characters but 5 bytes in UTF-8 — and why a counter that reports characters and bytes separately is more informative than one that reports a single number.

Mojibake: when bytes are read with the wrong map

Mojibake is bytes encoded with one dictionary decoded with another. The classic: é stored in UTF-8 as C3 A9, then read as Latin-1 — which maps every single byte directly to a character — turns C3 into à and A9 into ©, displaying é as é. Let the mistake compound (misread as Latin-1, then re-saved as UTF-8) and you get é, the tell-tale fingerprint of double-encoded text. The fix is almost never re-typing; it is making both sides agree on UTF-8 — the file's declared charset, the HTTP header, the database's encoding.

Domain names solve the same problem with a trick of their own. The DNS predates Unicode, so internationalized domains are converted to pure-ASCII Punycode: café.example becomes xn--caf-dma.example. The Punycode Converter translates in both directions if you want to see your own name compressed into xn-- form.

Try it yourself

Browse the ASCII Table to see all 128 codes with their decimal, hex, and binary values side by side. The ASCII Text Converter converts text to code values and back — paste café and watch the multi-byte characters appear. The Character Counter reports characters and bytes for any text, so you can watch café turn from 4 characters into 5 bytes. For how the binary and hex notation in this article works, see Number Base Systems Explained; for the encoding that packs arbitrary bytes into text for transport, see How Base64 Encoding Works.