Llama 2

6 glitch tokens

These tokens have been publicly documented on this model or model family. Inclusion does not mean that every version still shows the same behavior.

▁Mediabestanden

A Dutch plural noun (“media files”; ▁ is the SentencePiece space marker). It made it into the Llama-2 tokenizer but essentially never into training data — the most under-trained token in Llama-2-7b by embedding L2 norm (0.0287 vs a mean of ~1.08).

Llama 2Phi-3 (Microsoft)Under-trained

oreferrer

Fragment of the HTML attribute `rel="noreferrer"`: present in the Llama-2 vocabulary but nearly absent from training, with an embedding norm far below the threshold (0.113).

Llama 2Under-trained

▁Portály

The Czech word for “portals”, frozen into the tokenizer during pre-training-corpus construction; the second-most under-trained token in Llama-2-7b (embedding norm 0.0956).

Llama 2Under-trained

IABot

Fragment of the name of the Internet Archive Bot, Wikipedia's dead-link repair bot.

Llama 2Unspeakable

abestanden

A German word-stem fragment (“to submit/file”).

Llama 2Unspeakable

ederbörd

A German-looking fragment of unknown origin.

Llama 2Unspeakable