The promises of artificial intelligence (AI) have long been tantalizing, with visions of a future where machines are smarter than humans and can solve all of our most pressing problems. However, recent research is beginning to paint a different picture, one in which even the most powerful AI systems struggle with basic puzzles that most humans find trivial.
Leaders in the AI industry, such as Sam Altman of OpenAI and Demis Hassabis of Google DeepMind, have been vocal about the potential for AI to revolutionize society in the coming decades. Altman predicts a future of “radical abundance” and groundbreaking scientific discoveries, while Hassabis envisions AI solving global challenges like curing diseases and colonizing other planets.
These visions hinge on the idea that large language models (LLMs) like ChatGPT will continue to improve with more training data and computational power. However, recent developments have raised doubts about this scaling law. For example, OpenAI’s GPT-4.5 model showed only modest improvements over its predecessor despite significant investment, and Meta is reportedly planning a $15 billion investment to achieve “superintelligence.”
In an effort to overcome these challenges, AI companies have turned to reasoning models like OpenAI’s o1. These models use iterative processes to mimic human problem-solving, but recent research has shown that they can struggle with even simple logic puzzles. For example, models tested on tasks like transporting items across a river or solving the Tower of Hanoi game failed to consistently use explicit algorithms and struggled with increasing complexity.
Experts like Artur Garcez from City, University of London, believe that these issues highlight the limitations of current AI systems. While it’s possible that these models can be improved to tackle complex problems, simply increasing their size or computational resources may not be enough. Nikos Aletras from the University of Sheffield notes that AI models excel at tasks they’ve been trained on but can struggle with new scenarios, as highlighted by recent research findings.
Furthermore, studies have shown that longer “chain of thought” processes can actually decrease an AI model’s performance. Research conducted at the University of Maryland found that increasing the number of tokens used by a model led to a decrease in accuracy on tests of mathematical reasoning, suggesting that more thinking time does not always equate to better performance.
In conclusion, while the future of AI holds great promise, it also poses significant challenges that must be addressed. As researchers continue to uncover the limitations of current AI systems, it becomes clear that achieving truly intelligent machines will require more than just increased computational power. Only by understanding and addressing these limitations can we hope to realize the full potential of artificial intelligence in the years to come. A recent study conducted on the impact of increasing the number of tokens on AI benchmark scores revealed a significant decrease of approximately 17 per cent. This finding indicates that extending the token count may not always lead to improved performance in AI models.
Furthermore, researchers testing DeepSeek’s AI models on simple maze navigation tasks discovered a discrepancy between the AI’s “chain of thought” output and its final answer. Subbarao Kambhampati and his team at Arizona State University observed that the AI’s internal reasoning, as reflected in its intermediate tokens, often contained errors that did not affect the eventual solution. Surprisingly, providing the AI with nonsensical “chains of thought” sometimes resulted in better outcomes.
Kambhampati emphasizes the need to refrain from interpreting these intermediate tokens as indicative of the AI’s thought process. He warns against anthropomorphizing AI models and suggests that the notion of AI “thinking” or “reasoning” may be inaccurate.
Similarly, Anna Rogers from the IT University of Copenhagen highlights the historical trend of associating AI techniques with cognitive analogies that are later disproven. She cautions against overestimating AI’s ability to reason or think based on current models’ performance.
Andreas Vlachos from the University of Cambridge acknowledges the practical applications of Large Language Models (LLMs) in text generation but notes the challenges in making them proficient in solving complex problems. Vlachos underscores the disparity between the AI’s training in next-word prediction and the expectations placed on it to engage in reasoning tasks.
While some experts express skepticism about AI’s ability to reason effectively, OpenAI remains optimistic. A spokesperson for OpenAI asserts that methods like the chain of thought can enhance AI performance on intricate problems. They are actively working on enhancing these capabilities through improved training, evaluation, and model design. On the other hand, DeepSeek did not provide a response to inquiries on the matter.
In conclusion, while advancements in AI technology have shown promise in various applications, there are still limitations to be addressed in terms of reasoning and problem-solving capabilities. The ongoing research and development efforts aim to bridge the gap between AI models’ training objectives and the complex challenges they are expected to tackle in the future.
