Chatbots work best when you speak to them with formal language

Using a more casual tone when conversing with AI chatbots can hinder the accuracy of the information they provide. This observation implies that either users should adopt stricter linguistic standards or that the AI systems need better training to comprehend informal language.

Researchers Fulei Zhang and Zhou Yu from Amazon explored the differences in communication styles when interacting with human agents versus chatbot assistants driven by large language models (LLMs). They utilized the Claude 3.5 Sonnet model to assess conversations across various dimensions, finding that human communication is generally more accurate grammatically and more polite compared to interactions with chatbots, alongside a narrower vocabulary range.

For instance, their analysis revealed that conversations between humans were 14.5 percent more polite and formal compared to chatbot discussions, 5.3 percent more fluent, and 1.4 percent more lexically diverse based on the scores derived from Claude.

“Users adapt their linguistic style in human-LLM conversations, producing messages that are shorter, more direct, less formal, and grammatically simpler,” the researchers stated in their paper, which they did not respond to for an interview. “This behavior is likely shaped by users’ perceptions of LLM chatbot[s] as less socially attuned or less capable of nuanced interpretation.”

However, this informality has its drawbacks. In another assessment, the researchers trained an AI model named Mistral 7B on 13,000 authentic human-to-human conversations and applied it to interpret 1,357 actual messages directed to AI chatbots. They categorized each instance within both datasets with an “intent” from a restricted list, summarizing the user’s objective in each case. Nevertheless, because Mistral was trained on human interaction, it struggled to appropriately label intents in chatbot conversations.

Zhang and Yu experimented with different methods to enhance Mistral’s comprehension. As a first step, they used Claude AI to rewrite users’ more brusque messages into a more human-like format, fine-tuning Mistral with these adaptations. This approach resulted in a 1.9 percent decrease in the accuracy of intent labeling from its default responses.

Subsequently, they instructed Claude to produce a “minimal” revision—more concise and straightforward (e.g., “paris next month. flights hotels?” for inquiries regarding travel options)—yet encountered a 2.6 percent accuracy decline for Mistral. On the other hand, a more “enriched” version with formal language also caused a 1.8 percent drop in accuracy. Only by training Mistral on both minimal and enriched rewrites did they achieve a 2.9 percent enhancement in performance.

Noah Giansiracusa from Bentley University expresses that he is not surprised by the differing communication styles between people and bots, but he suggests that these differences should not be avoided.

“The observation that individuals communicate distinctly with chatbots compared to other humans can be enticingly misinterpreted as a flaw of the chatbot—but I would argue that it’s advantageous when people recognize they are interacting with bots and adjust their demeanor accordingly,” Giansiracusa commented. “I believe this awareness is healthier than obsessively trying to bridge the gap between humanity and AI.”