Claude's tokenizer uses a proprietary minimum-piece tokenization scheme rather than standard BPE, with vocabulary sizes around 49-55k for v3-v4.6 and 16-20k after. Key innovations include boundary markers wrapping word and punctuation spans, space-efficient encoding where spaces between marked boundaries are removed, and capitalization handling via shift markers for title-case and all-caps text.
A developer explores encoding country flags in just 11 bits by breaking them down into common design elements like stripes, shapes, and colors. Using a layered approach inspired by matrix compression and image editing software, the encoding scheme prioritizes recognizability over exact accuracy, handling aspect ratios, color palettes, and repeated elements like stars and crescents.
TurboCrypt is switching from Base91 to Base84 encoding for encrypted file names to support Windows alongside Unix systems. Base84 uses 84 filesystem-safe ASCII characters that work across Linux, macOS, and Windows, achieving about 25% expansion on random input while avoiding Windows-reserved device names.
A 2023 computer science paper demonstrates that language model tokenizers create unfairness across languages by producing drastically different tokenization lengths for the same text translated into different languages—up to 15 times longer in some cases. These disparities affect cost, processing time, and context length for commercial language services, persisting even in intentionally multilingual tokenizers. The authors argue for developing multilingually fair subword tokenizers for future language models.
NeoMME is a family of multimodal multilingual foundation encoders (260M and 800M parameters) that process text and raw image patches in a single bidirectional Transformer. Pretrained with masked discrete-diffusion objectives, NeoMME-Retriever achieves strong performance on the ViDoRe v3 benchmark while offering 2x throughput improvements and 255x compression of embeddings compared to existing models.