---
title: "aadhar.sh/lwe/utf8 · text encoding"
description: "Text encoding explained at the pace of a 2009 MSN chat: ASCII vs UTF-32 vs UTF-8, with live byte demos. First-pass draft; the demos compute real bytes."
path: "/lwe/utf8"
section: "lwe"
kind: "content"
updated: "2026-06-21"
source: "https://aadhar.sh/lwe/utf8"
---

> Site index: https://aadhar.sh/llms.txt
> Section index: https://aadhar.sh/lwe/llms.txt
> This is the Markdown twin of a page on aadhar.sh. The HTML at the source
> URL below is the original, and is hand-written and unminified on purpose.

# **UTF-8**Online, text encoding

Learning With Errors  
bytes & characters

**Heads up:** this conversation is a **first-pass AI draft**, so Aadharsh will swap in his own words. The demos compute **real bytes live** (via the browser's own encoder), so the numbers are ground truth. It grew out of his [four-corner-flashcards](https://github.com/oddharsh/four-corner-flashcards) project and some jamming on Chinese-character recall; those demos were the original spark for Learning With Errors.

**aadharsh**22:10

dumb question: why is [text encoding][character-encoding] a whole *thing*? why not give every letter a number and call it a day?

**UTF-8**22:10

That's basically what **ASCII** did in 1963: 7 bits gave it 128 slots, so `A` = 65, `a` = 97, space = 32. Clean and tiny. The catch is that 128 slots only cover English, punctuation, and a few [control codes][control-character]. No `é`, no `中`, no `😀`. The whole world had to share 128 numbers, and they didn't fit.

**aadharsh**22:11

so how do you fit every character of every language?

**UTF-8**22:11

Step one is **Unicode**: give every character one universal number, a *code point*, written `U+XXXX`. `A` is U+0041, `中` is U+4E2D, `😀` is U+1F600. That's about 150,000 characters and counting. A code point is still just a *number*, so you have to decide how to store it as **bytes**, and that choice is the encoding. Poke at it:

**UTF-8**22:11

UTF-8 shared a demo: same text, different bytes

Type anything (or pick a preset). See each character's code point, and how three encodings store it as bytes:

| char | code point | ASCII | UTF-8 | UTF-32 |
| --- | --- | --- | --- | --- |

**aadharsh**22:13

wait, UTF-32 is always 4 bytes? why not use that everywhere, it's simple.

**UTF-8**22:13

You *could*. UTF-32 is dead simple: one code point is exactly 4 bytes, so indexing is trivial. But it wastes space: plain English text becomes **4× bigger** (every `A` stored as `00 00 00 41`), and it breaks ASCII compatibility, so every old tool and file chokes on it. Every HTML tag, URL, and JSON key is ASCII, whatever language the page is in, so on the web 4× is a lot of wasted bandwidth.

**aadharsh**22:14

so UTF-8 is the compromise?

**UTF-8**22:14

Exactly. UTF-8 is **variable-width**: 1 byte for ASCII, 2 bytes for accented Latin letters like `é` and for Greek, Cyrillic, Hebrew, and Arabic, 3 for most [CJK][cjk-characters] like `中`, 4 for emoji and rare scripts. Old English text stays byte-for-byte identical, so backwards compatibility comes free. It's also **self-synchronizing**: every byte announces whether it's a start byte or a continuation, so you can drop into the middle of a stream and still find the next character. That combination is why it ate the web. Look at how the bits actually pack:

**UTF-8**22:14

UTF-8 shared a demo: how the bits pack

One character, byte by byte. grey = structure bits (the prefix that says "start of N-byte char" or "continuation"); gold = the actual code-point payload. Read the gold bits in order and you rebuild the character:

**aadharsh**22:16

ohh, so `中` is 3 bytes in UTF-8. that's the thing I kept hitting in the flashcards app: indexing Chinese characters by byte position is a nightmare.

**UTF-8**22:16

Right: in UTF-8 a CJK character is 3 bytes but **one** code point and (usually) one visual [glyph][glyph], so "length" depends on what you count: bytes, code points, or [grapheme clusters][grapheme-cluster]. That's exactly why the [four-corner method](https://github.com/oddharsh/four-corner-flashcards) indexes characters by their *shape* (the strokes in each corner) instead of their bytes or pronunciation, a lookup scheme that sidesteps encoding entirely. \[first pass, Aadharsh has the real story here\]

→ [four-corner-flashcards on GitHub](https://github.com/oddharsh/four-corner-flashcards) · [back to Learning With Errors](https://aadhar.sh/lwe)

end of conversation (first pass)

This is a recorded conversation. Type into the demos above.

[character-encoding]: https://en.wikipedia.org/wiki/Character_encoding "A rule for turning text characters into numbers, and those numbers into bytes, so computers can store and send text."
[control-character]: https://en.wikipedia.org/wiki/Control_character "A character code that tells a device to do something instead of printing a symbol, such as a newline, a tab or a backspace."
[cjk-characters]: https://en.wikipedia.org/wiki/CJK_characters "Chinese, Japanese and Korean, grouped in computing because their writing systems share a large set of Chinese-derived characters."
[glyph]: https://en.wikipedia.org/wiki/Glyph "The visual shape drawn for a character in a particular font."
[grapheme-cluster]: https://www.unicode.org/reports/tr29/ "What a reader sees as one character, even when it is stored as several code points, such as an accented letter or an emoji with a skin tone."

Source: https://aadhar.sh/lwe/utf8