OpenAI Realizes It Made a Terrible Mistake

OpenAI claims to have figured out what’s driving “hallucinations,” or AI models’ strong tendency to make up answers that are factually incorrect.

It’s a major problem plaguing the entire industry, greatly undercutting the usefulness of the tech. Worse yet, experts have found that the problem is getting worse as AI models get more capable.

As a result, despite incurring astronomical expenses in their deployment, frontier AI models are still prone to making inaccurate claims when faced with a prompt they don’t know the answer to.

Whether there’s a solution to the problem remains a hotly debated subject, with some experts arguing that hallucinations are intrinsic to the tech itself. In other words, large language models may be a dead end in our quest to develop AIs with a reliable grasp on factual claims.

In a paper published last week, a team of OpenAI researchers attempted to come up with an explanation. They suggest that large language models hallucinate because when they’re being created, they’re incentivized to guess rather than admit they simply don’t know the answer.

Hallucinations “persist due to the way most evaluations are graded — language models are optimized to be good test-takers, and guessing when uncertain improves test performance,” the paper reads.

Conventionally, the output of an AI is graded in a binary way, rewarding it when it gives a correct response and penalizing it when it gives an incorrect one.

In simple terms, in other words, guessing is rewarded — because it might be right — over an AI admitting it doesn’t know the answer, which will be graded as incorrect no matter what.

As a result, through “natural statistical pressures,” LLMs are far more prone to hallucinate an answer instead of “acknowledging uncertainty.”

“Most scoreboards prioritize and rank models based on accuracy, but errors are worse than abstentions,” OpenAI wrote in an accompanying blog post.

In other words, OpenAI says that it — and all its imitators across the industry — have made a grave structural error in how they’ve been training AI.

There’ll be a lot riding on whether the issue is correctable going forward. OpenAI claims that “there is a straightforward fix” to the problem: “Penalize confident errors more than you penalize uncertainty, and give partial credit for appropriate expressions of uncertainty.”

Going forward, evaluations need to ensure that “their scoring discourages guessing,” the blog post reads. “If the main scoreboards keep rewarding lucky guesses, models will keep learning to guess.”

“Simple modifications of mainstream evaluations can realign incentives, rewarding appropriate expressions of uncertainty rather than penalizing them,” the company’s researchers concluded in the paper. “This can remove barriers to the suppression of hallucinations, and open the door to future work on nuanced language models, e.g., with richer pragmatic competence.”

How these adjustments to evaluations will play out in the real world remains to be seen….