AI Benchmarking Controversy Hits Pokémon World
Not even Pokémon is safe from AI benchmarking controversy.
A recent viral post on X claimed that Google’s Gemini model had outperformed Anthropic’s Claude model in the original Pokémon video game trilogy. The Gemini model was reported to have reached Lavendar Town in a developer’s Twitch stream, while Claude was stuck at Mount Moon as of late February.
Gemini is literally ahead of Claude atm in Pokémon after reaching Lavender Town
119 live views only btw, incredibly underrated stream pic.twitter.com/8AvSovAI4x
— Jush (@Jush21e8) April 10, 2025
However, it was later revealed that Gemini had an advantage. The developer maintaining the Gemini stream had created a custom minimap to assist the model in identifying game elements like cuttable trees, giving it an edge in decision-making compared to Claude.
While Pokémon may not be a rigorous AI benchmark, it serves as an example of how different implementations can skew results. For instance, Anthropic’s Claude model achieved varying scores on the SWE-bench Verified benchmark based on the use of a custom scaffold. Similarly, Meta’s Llama 4 Maverick model showed improved performance on the LM Arena benchmark after fine-tuning.
The use of custom implementations and non-standard approaches in AI benchmarks like Pokémon raises concerns about the comparability of models. With the evolving landscape of AI technology, it may become increasingly challenging to make fair and accurate assessments of AI capabilities.
