Can You Trust AI Chatbots for Current News?

The paper Evaluating Commercial AI Chatbots as News Intermediaries was published in May 2026 by researchers including Mirac Suzgun, Dan Jurafsky and James Zou.

Across 14 days from 9 to 22 February 2026, the team created factual questions from same-day BBC News reporting. The dataset covered US and Canada, Arabic, Afrique, Hindi, Russian and Turkish regional services.

Six systems were evaluated: Gemini 3 Flash and Pro, Grok 4, Claude 4.5 Sonnet, GPT-5 and GPT-4o mini. Testing same-day stories matters because memorised training knowledge cannot answer events that happened hours earlier.

The best systems achieved more than 90% accuracy in multiple-choice evaluation. That is encouraging evidence that retrieval-enabled chatbots can answer many emerging-news questions correctly.

Accuracy dropped when the same systems had to produce free responses. The paper reports an 11 to 13 percentage-point decline for leading systems and a 16 to 17-point decline across the cohort.

Free response is harder because the model must retrieve, select, phrase and scope the answer without options providing structure. A response can contain the correct core fact whilst adding unsupported detail.

This also explains why a chatbot can perform well in a benchmark yet still create risky copy. Accuracy scoring may focus on one requested fact, whilst a published paragraph contains several additional claims.

The researchers found retrieval failures caused more than 70% of errors. A chatbot may search the wrong terms, fail to surface the relevant regional report, or retrieve an article that is related but does not establish the requested fact.

This shifts attention from model intelligence to the full product pipeline: query generation, search coverage, ranking, page access, evidence extraction and citation placement.

A model can reason well over the wrong evidence and still produce a wrong answer. News evaluation must inspect sources, not just prose quality.

Paywalls, live blogs, corrections, local-language pages and rapidly updated headlines make the retrieval problem harder than a static web question.

Questions containing a subtle false premise are particularly risky. A cooperative assistant may accept the user's framing and construct an answer around an event or detail that did not occur.

A safer system should challenge the premise, state what could and could not be verified, and cite the closest authoritative reporting. Failure to push back is a warning sign.

Ask the chatbot to identify the claim that must be true before answering, then require a source that establishes that claim directly. This turns hidden assumptions into checkable statements.

A citation can be real and still fail to support the sentence attached to it. It may point to an article about the same subject, a later rewrite or a page that contradicts the answer.

Open every important citation. Check publication time, event date, named region and exact support. Prefer the original statement, court filing, paper, dataset or first-hand report where available.

Watch for citation laundering, where several outlets repeat one unsupported claim. Multiple links do not equal independent confirmation if they all trace back to the same weak origin.

For direct quotations, search the source page for the words and preserve enough context to avoid changing the speaker's meaning.

The study's six regional services are an important design choice. News retrieval is not evenly distributed across languages, locations and publishers.

A system may perform strongly on widely syndicated English-language reporting but struggle with a local source, transliterated name or region-specific vocabulary. Translation can also blur legal and political terms.

When the story originates outside the user's language, inspect the original regional report as well as a translation. Note where a claim comes from and avoid converting an uncertain translation into a categorical statement.

1. Separate the date the event happened from the publication date.
2. Open the cited page and confirm it directly supports the claim.
3. Prefer a primary source or first-hand report for the central fact.
4. Compare a second independent newsroom for developing stories.
5. Check regional, language and naming details.
6. Ask whether the question contains a false premise.
7. State uncertainty and corrections clearly.

Use the chatbot to discover leads and summarise verified material, not as the final authority for a consequential claim.

Commercial chatbots can answer many same-day factual questions accurately, particularly when the task is constrained. The strongest systems in this study performed well on multiple choice.

Reliability falls when retrieval fails, free responses invite unsupported detail or the question contains a false premise. The fluent answer is the end of a pipeline, not evidence that every stage worked.

Use AI for orientation and synthesis, then verify at source. For model comparisons, see the AI Tools Review benchmarks hub and our GPT-5.6 analysis.