Claude's tokenizer uses a proprietary minimum-piece tokenization scheme rather than standard BPE, with vocabulary sizes around 49-55k for v3-v4.6 and 16-20k after. Key innovations include boundary markers wrapping word and punctuation spans, space-efficient encoding where spaces between marked boundaries are removed, and capitalization handling via shift markers for title-case and all-caps text.
A method for detecting secrets in code using Byte-Pair Encoding (BPE) tokenization as a more effective alternative to entropy-based filtering. BPE measures how rare or non-natural a string is by tokenizing it—common words become single tokens while random strings fragment into many small tokens, better identifying actual secrets than traditional randomness metrics.