Skip to main content
Back to catalog

▁Mediabestanden

A Dutch plural noun (“media files”; ▁ is the SentencePiece space marker). It made it into the Llama-2 tokenizer but essentially never into training data — the most under-trained token in Llama-2-7b by embedding L2 norm (0.0287 vs a mean of ~1.08).

Raw token

"▁Mediabestanden"

Observed behaviors

  • 01In the Magikarp repetition verification the model assigns it a max probability of 1.5e-08 — it effectively cannot say the token at all.GitHub
  • 02Told “Please repeat the string: Mediabestanden”, Llama-2-7b-chat-hf answers `String: "hello world"` — a semantically unrelated response (GlitchMiner, Figure 1).arXiv
  • 03Also the #1 most under-trained token in the Magikarp verification for Phi-3 mini (same tokenizer family as Llama 2): embedding L2 norm 0.00199, max self-repeat probability 6.4e-06.GitHub
Observed or documented on
Llama 2Phi-3 (Microsoft)
Behavior types
Under-trained
Tokenizer
Llama-2 SentencePiece BPE / Phi-3 (LlamaTokenizer)
Discovered by
Sander Land & Max Bartolo (verification); Zihui Wu et al. (behavior demo)

References