Skip to main content
Back to catalog

oreferrer

Fragment of the HTML attribute `rel="noreferrer"`: present in the Llama-2 vocabulary but nearly absent from training, with an embedding norm far below the threshold (0.113).

Raw token

"oreferrer"

Observed behaviors

  • 01Llama-2-7b-chat-hf fails to recognize/repeat it, treating it as an uncorrelated term (GlitchMiner Figure 1).arXiv
  • 02Verified under-trained: max self-repeat probability of 2e-05 in the Magikarp verification.GitHub
Observed or documented on
Llama 2
Behavior types
Under-trained
Tokenizer
Llama-2 SentencePiece BPE
Discovered by
Sander Land & Max Bartolo; Zihui Wu et al.

References