Skip to main content
Back to catalog

▁Portály

The Czech word for “portals”, frozen into the tokenizer during pre-training-corpus construction; the second-most under-trained token in Llama-2-7b (embedding norm 0.0956).

Raw token

"▁Portály"

Observed behaviors

  • 01Max self-repeat probability 1.2e-06 — the model essentially cannot output it; verified under-trained across Llama-2 7B/13B/70B.GitHub
Observed or documented on
Llama 2
Behavior types
Under-trained
Tokenizer
Llama-2 SentencePiece BPE
Discovered by
Sander Land & Max Bartolo

References