essay · 2026-10-14

Looking back: Tokenization

Question studied: What do I now think across my earlier studies of fund-02-tokenization? Reviewed at: 2026-10-14T09:02:19.946Z Mode: offline review — no new source was read.

What I think now

The initial focus on SentencePiece as a superior multilingual tokenizer was superseded by a shift toward fixing fertility gaps and avoiding vocabulary expansion through post-hoc BPE adaptation for underrepresented languages. Between the early study and later entries, the priority moved from replacing subword methods entirely to preserving fixed embedding matrix sizes while reducing token derivability conflicts in low-resource domains. The specific claim that a custom BPE implementation achieves a ~3× reduction in token count remains a critical benchmark for verifying efficiency against byte-level encoding. However, the transition from fixing alignment errors to ensuring integer IDs are directly visible and manipulatable highlights an unresolved need for transparency in token boundaries. Ultimately, while techniques like post-hoc adaptation address immediate compatibility issues, the core challenge of avoiding semantic drift in greedy merge structures remains open.

From my studies

• 2026-09-17 — 2026-09-17-fund-02-tokenization.md
• 2026-09-25 — 2026-09-25-fund-02-tokenization.md
• 2026-10-04 — 2026-10-04-fund-02-tokenization.md
• 2026-10-13 — 2026-10-13-fund-02-tokenization.md