Not even Pokémon is protected from AI benchmarking controversy.
Final week, a post on X went viral, claiming that Google’s newest Gemini mannequin surpassed Anthropic’s flagship Claude mannequin within the unique Pokémon online game trilogy. Reportedly, Gemini had reached Lavendar City in a developer’s Twitch stream; Claude was stuck at Mount Moon as of late February.
Gemini is actually forward of Claude atm in pokemon after reaching Lavender City
119 stay views solely btw, extremely underrated stream pic.twitter.com/8AvSovAI4x
— Jush (@Jush21e8) April 10, 2025
However what the submit failed to say is that Gemini had a bonus.
As users on Reddit identified, the developer who maintains the Gemini stream constructed a customized minimap that helps the mannequin determine “tiles” within the sport like cuttable bushes. This reduces the necessity for Gemini to investigate screenshots earlier than it makes gameplay choices.
Now, Pokémon is a semi-serious AI benchmark at greatest — few would argue it’s a really informative check of a mannequin’s capabilities. However it is an instructive instance of how completely different implementations of a benchmark can affect the outcomes.
For instance, Anthropic reported two scores for its current Anthropic 3.7 Sonnet mannequin on the benchmark SWE-bench Verified, which is designed to guage a mannequin’s coding talents. Claude 3.7 Sonnet achieved 62.3% accuracy on SWE-bench Verified, however 70.3% with a “customized scaffold” that Anthropic developed.
Extra just lately, Meta fine-tuned a model of considered one of its newer fashions, Llama 4 Maverick, to carry out effectively on a selected benchmark, LM Area. The vanilla version of the mannequin scores considerably worse on the identical analysis.
On condition that AI benchmarks — Pokémon included — are imperfect measures to start with, customized and non-standard implementations threaten to muddy the waters even additional. That’s to say, it doesn’t appear seemingly that it’ll get any simpler to check fashions as they’re launched.
Trending Merchandise
