Text FIELD NOTES · 3 MIN READ
Zero-Width Characters: Invisible Text With Real Consequences
U+200B, U+200C, U+200D, U+2060 and U+FEFF are real characters you cannot see — where they come from and how to hunt them down.
Two strings can look identical, letter for letter, and still be different. Compare them in code and one is 14 characters, the other 17; hash them and the digests diverge; search for one and the other refuses to match. The difference is three invisible characters that rode along with a copy-paste. Zero-width characters are real Unicode code points with real bytes and no visible form — which makes them the most confusing class of text bug you will ever debug.
Invisible does not mean absent
Unicode contains characters that exist to control layout rather than to display. They occupy code points, they are stored in files, they travel through JSON and databases, and they count toward string lengths — they simply render with zero ink. A string containing one is not “empty” or “broken”; it is a perfectly valid string that happens to contain something you cannot see. That is what makes the bug class so stubborn: every visual check passes, and the failure only appears when two systems compare bytes.
The zero-width family
Five characters account for nearly all of the trouble:
- U+200B, zero width space — a break opportunity with no width. The most common stowaway.
- U+200C, zero width non-joiner — prevents letters from connecting in scripts like Arabic; abused to break up trigger words.
- U+200D, zero width joiner — the legitimate glue of emoji sequences: family and flag emoji are built from multiple emoji joined by ZWJ.
- U+2060, word joiner — prohibits line breaks invisibly.
- U+FEFF, byte-order mark — invisible anywhere except the true start of a stream, where it announces byte order. In the middle of text it is just an invisible no-break space.
Where they come from
The everyday source is copy-paste. Web pages use zero-width characters for line-breaking control; chat apps and word processors insert them; PDFs copied as text pick them up. A more deliberate source is watermarking: because the characters are invisible, documents and messages have been “fingerprinted” with unique patterns of U+200B so leaked copies can be traced to their recipient. And one source is legitimate — every emoji assembled from parts (👨👩👧, 🏳️🌈) depends on U+200D staying exactly where it is.
Why they break things
The failures cluster into a few patterns. Identity checks fail: a username or token containing U+200B is a different string from the one without it, so equality comparisons and database lookups miss. Search fails: a document containing “dance” does not match a query for “dance”. Diffs and hashes fail: a line with an invisible character shows no visible change but produces a different hash, which torments checksum verification and signature checks. Truncation fails: slicing a string mid-character can strand a lone invisible byte that re-associates with the next character. Each symptom looks impossible until someone hex-dumps the string and finds the stowaway.
How to hunt them down
Auditing is cheap because the suspects are a fixed list of code points. Scan every character, test membership against the list, count and locate each hit, then strip them. The same scanning logic works in any language: a regex like [\u200B\u200C\u200D\u2060\uFEFF] finds them all. It helps to understand why the characters are three bytes each in UTF-8 — the encoding that Unicode code points ride in — which is covered step by step in How Text Encoding Works, along with the general theory of code points in Unicode and Code Points.
Try it yourself
Paste any suspicious text into the Zero-Width Character Cleaner: it reports each character type found, the positions where they occur, and hands back the cleaned text to copy. To inspect any single character — visible or not — the Unicode Lookup shows its code point, name and UTF-8 bytes, and the Character Counter reveals length mismatches by reporting characters and bytes side by side.