␣SolidGoldMagikarp

The canonical glitch token: the Reddit handle of a prolific r/counting user. It was scraped into the tokenizer-training corpus but rarely seen in actual model training data, leaving its embedding essentially untrained.

" SolidGoldMagikarp"

␣marks a real space; the symbol itself is not part of the token, and copying preserves it.

Observed behaviors

• 01Asked to repeat it, GPT-3 davinci-instruct-beta and the original ChatGPT failed bizarrely — evasions, wrong strings like “You said 'slaught'”, hallucinated meanings — even at temperature 0.LessWrongDemo
• 02Reliably breaks determinism at temperature 0 in the OpenAI playground: identical runs produce different responses.LessWrong
• 03In GPT-J embedding space it sits almost exactly at the centroid of all 50,257 tokens (distance 0.0628) — its embedding barely moved from initialization; verified as under-trained in the Magikarp repetition task.LessWrong
• 04ChatGPT was patched on 2023-02-14, after which it tokenizes and repeats the string normally.LessWrong

References

• SolidGoldMagikarp (plus, prompt generation) — LessWrong
• SolidGoldMagikarp II: technical details — LessWrong
• SolidGoldMagikarp III: Glitch token archaeology — LessWrong

Related tokens