Can You Trust AI Chatbots for Current News?

The paper Evaluating Commercial AI Chatbots as News Intermediaries was published in May 2026 by researchers including Mirac Suzgun, Dan Jurafsky and James Zou.

Across 14 days from 9 to 22 February 2026, the team created factual questions from same-day BBC News reporting. The dataset covered US and Canada, Arabic, Afrique, Hindi, Russian and Turkish regional services.

Six systems were evaluated: Gemini 3 Flash and Pro, Grok 4, Claude 4.5 Sonnet, GPT-5 and GPT-4o mini. Testing same-day stories matters because memorised training knowledge cannot answer events that happened hours earlier.

The best systems achieved more than 90% accuracy in multiple-choice evaluation. That is encouraging evidence that retrieval-enabled chatbots can answer many emerging-news questions correctly.

Accuracy dropped when the same systems had to produce free responses. The paper reports an 11 to 13 percentage-point decline for leading systems and a 16 to 17-point decline across the cohort.

Free response is harder because the model must retrieve, select, phrase and scope the answer without options providing structure. A response can contain the correct core fact whilst adding unsupported detail.

This also explains why a chatbot can perform well in a benchmark yet still create risky copy. Accuracy scoring may focus on one requested fact, whilst a published paragraph contains several additional claims.

The researchers found retrieval failures caused more than 70% of errors. A chatbot may search the wrong terms, fail to surface the relevant regional report, or retrieve an article that is related but does not establish the requested fact.

This shifts attention from model intelligence to the full product pipeline: query generation, search coverage, ranking, page access, evidence extraction and citation placement.

A model can reason well over the wrong evidence and still produce a wrong answer. News evaluation must inspect sources, not just prose quality.

Paywalls, live blogs, corrections, local-language pages and rapidly updated headlines make the retrieval problem harder than a static web question.

Questions containing a subtle false premise are particularly risky. A cooperative assistant may accept the user's framing and construct an answer around an event or detail that did not occur.

A safer system should challenge the premise, state what could and could not be verified, and cite the closest authoritative reporting. Failure to push back is a warning sign.

Ask the chatbot to identify the claim that must be true before answering, then require a source that establishes that claim directly. This turns hidden assumptions into checkable statements.

A citation can be real and still fail to support the sentence attached to it. It may point to an article about the same subject, a later rewrite or a page that contradicts the answer.

Open every important citation. Check publication time, event date, named region and exact support. Prefer the original statement, court filing, paper, dataset or first-hand report where available.

Watch for citation laundering, where several outlets repeat one unsupported claim. Multiple links do not equal independent confirmation if they all trace back to the same weak origin.

For direct quotations, search the source page for the words and preserve enough context to avoid changing the speaker's meaning.

The study's six regional services are an important design choice. News retrieval is not evenly distributed across languages, locations and publishers.

A system may perform strongly on widely syndicated English-language reporting but struggle with a local source, transliterated name or region-specific vocabulary. Translation can also blur legal and political terms.

When the story originates outside the user's language, inspect the original regional report as well as a translation. Note where a claim comes from and avoid converting an uncertain translation into a categorical statement.

The 2,100-question paper is not an outlier. In October 2025 the European Broadcasting Union published News Integrity in AI Assistants, the largest study of its kind, coordinated with the BBC. Journalists from 22 public service media organisations across 18 countries, working in 14 languages, evaluated more than 3,000 responses from ChatGPT, Copilot, Gemini and Perplexity.

The headline finding: 45% of all answers contained at least one significant issue. Around 31% of responses showed serious sourcing problems, such as missing, misleading or incorrect attributions, and roughly one in five contained major accuracy issues, including hallucinated detail and outdated information.

Performance also varied sharply by product. Gemini fared worst in that evaluation, with significant issues in 76% of its responses, largely driven by sourcing: 72% of its answers had a significant sourcing issue, against below 25% for the other assistants tested. The EBU's media director described the failings as "systemic, cross-border, and multilingual".

Read together, the two studies point in the same direction from different angles. The EBU and BBC work scored full journalistic quality, including attribution and context, whilst the 2026 paper isolated where in the pipeline the failures occur. Both conclude that the weak link is evidence handling rather than fluency.

The stakes are rising because usage is rising. The Reuters Institute's Digital News Report 2025 found that 7% of online news consumers use AI assistants to get news, climbing to 15% of under-25s. The audience least likely to cross-check against a bulletin or front page is adopting the tools fastest.

That shift changes the failure mode. A wrong paragraph on a low-traffic website misleads few people; a wrong synthesis delivered confidently inside a chat interface, with no editor and no correction mechanism, can quietly become someone's entire understanding of a story.

It also removes the visual cues readers normally rely on. A newspaper correction, a "developing story" banner or a clearly dated archive page all signal how much confidence to place in a claim. A chatbot answer presents everything in the same fluent register, whether the underlying evidence is a court judgment or a half-retrieved live blog.

You cannot fix retrieval from the prompt box, but you can make failures easier to spot. A few habits consistently help:

• Ask for sources inline, per claim. Requesting a citation attached to each factual sentence, rather than a list at the end, makes it obvious which claims arrived unsupported.
• Ask what the model could not verify. An explicit instruction to separate confirmed facts from inference invites the system to expose its own uncertainty instead of smoothing it over.
• Include dates in the question. "What happened at the hearing on 14 July" retrieves better than "what happened at the hearing", and reduces the risk of an answer built from last month's coverage.
• Ask for the disagreement. On developing stories, asking "what do different outlets report differently?" surfaces the contested details that a single synthesised answer would flatten.

None of this makes a chatbot a primary source. It converts an opaque answer into an inspectable one, which is the most a prompt can achieve.

For publishers, the studies carry an uncomfortable double message. Chatbots increasingly stand between newsrooms and audiences, yet they misattribute, blur opinion into fact and sometimes cite a rewrite rather than the original reporting. The EBU study's sourcing figures show attribution, the currency of journalism, is precisely where assistants are weakest.

For readers, the practical takeaway is to treat a chatbot the way a good editor treats a stringer's first call from the scene: plausibly right, genuinely useful, and unpublishable until checked. The convenience is real. So is the failure rate, and it is highest exactly where checking feels least convenient, in fast-moving, regionally specific or emotionally charged stories.

It is also worth remembering that these systems improve between studies. Retrieval pipelines, source ranking and citation grounding are active engineering fronts, and results measured in early 2026 will not describe products in 2027. The verification habits below, however, do not expire.

1. Separate the date the event happened from the publication date.
2. Open the cited page and confirm it directly supports the claim.
3. Prefer a primary source or first-hand report for the central fact.
4. Compare a second independent newsroom for developing stories.
5. Check regional, language and naming details.
6. Ask whether the question contains a false premise.
7. State uncertainty and corrections clearly.

Use the chatbot to discover leads and summarise verified material, not as the final authority for a consequential claim.

Commercial chatbots can answer many same-day factual questions accurately, particularly when the task is constrained. The strongest systems in this study performed well on multiple choice.

Reliability falls when retrieval fails, free responses invite unsupported detail or the question contains a false premise. The fluent answer is the end of a pipeline, not evidence that every stage worked.

Use AI for orientation and synthesis, then verify at source. For model comparisons, see the AI Tools Review benchmarks hub and our GPT-5.6 analysis.