Great news for xAI: Grok is now pretty good at answering questions about Baldur’s Gate

Different Priorities in AI Labs

Various AI labs have different focuses and priorities. OpenAI typically caters to consumer users, while Anthropic, its rival, tends to target enterprises. Recently, it came to light that Elon Musk’s xAI has been placing a significant emphasis on video game walkthroughs.

Business Insider’s Grace Kay recently published a comprehensive report on xAI, the AI startup acquired by SpaceX, shedding light on how Musk’s leadership style can create challenges for employees. One striking anecdote from the report highlighted Musk’s dissatisfaction with a chatbot’s responses related to the video game “Baldur’s Gate,” leading high-level engineers to pivot their focus to improve these responses before launch.

According to sources familiar with the matter, a model release was delayed as Musk was unhappy with the chatbot’s answers about “Baldur’s Gate.” This resulted in top engineers being reassigned to enhance the responses before the launch.

While it may seem frustrating for engineers to shift their attention from complex AI problems to assisting Musk in a video game, it raises the question of whether Musk achieved the gaming skills he desired.

Testing xAI’s Gaming Knowledge

To determine Musk’s gaming proficiency, our RPG enthusiast, Ram Iyer, devised five general questions about Baldur’s Gate. These questions were posed to xAI and three major models in a benchmarking exercise dubbed BaldurBench.

For transparency, all chat transcripts are available for public viewing: Grok, ChatGPT, Claude, and Gemini.

Upon analysis, Grok provided detailed and informative responses, albeit heavy on gamer jargon. ChatGPT and Gemini, while drawing from similar sources, differed in their stylistic presentation. Notably, Claude was cautious about potentially spoiling the gaming experience, advising players to prioritize fun over stress.

Given xAI’s focus on achieving parity, it’s interesting to note that Grok’s responses aligned closely with other models post the reported sprint. While this may not be conclusive, it demonstrates xAI’s capability to adapt and excel in varied domains.