Pitting AI Against Super Mario Bros: A Tougher Benchmark?
While Pokémon has long been considered a tough benchmark for AI, a recent study suggests that Super Mario Bros. may be an even tougher challenge. Researchers at the Hao AI Lab, based at the University of California San Diego, recently conducted experiments involving AI and live Super Mario Bros. games. The results were surprising, with Anthropic’s Claude 3.7 emerging as the top performer, followed closely by Claude 3.5. However, Google’s Gemini 1.5 Pro and OpenAI’s GPT-4o faced significant challenges in the gaming environment.
It’s important to note that the version of Super Mario Bros. used in the study was not the original 1985 release. Instead, the game ran in an emulator and was integrated with a framework called GamingAgent, developed by the Hao Lab. This framework gave the AIs control over Mario, allowing them to navigate the game world and interact with obstacles and enemies.
GamingAgent provided the AI with basic instructions and in-game screenshots, allowing them to generate Python code to control Mario. Despite this assistance, the game still required the models to learn complex maneuvers and develop gameplay strategies. Interestingly, the researchers found that reasoning models, which typically excel in problem-solving tasks, performed worse than non-reasoning models in this real-time gaming environment.
According to Hao, reasoning models struggle in games like Super Mario Bros. because they take longer to make decisions, which can be detrimental in fast-paced situations where split-second timing is crucial. This highlights the unique challenges that real-time games pose for AI models.
While games have been used to benchmark AI for decades, some experts have raised concerns about drawing direct parallels between gaming skills and overall technological advancement. Games, being abstract and relatively simple, may not fully capture the complexities of real-world scenarios that AI systems are designed to tackle.
These recent gaming benchmarks underscore what Andrej Karpathy, a research scientist at OpenAI, has described as an “evaluation crisis” in the field of AI. With rapidly advancing technologies and evolving benchmarks, it can be challenging to assess the true capabilities of AI models.
Despite these challenges, watching AI navigate the world of Super Mario Bros. offers a fascinating glimpse into the capabilities and limitations of artificial intelligence in real-time gaming scenarios.
