ProviderBench v0.1

Summary

ProviderBench tested five frontier AI systems against 102 provider phone numbers, each flagged as likely inaccurate by Sunny Network Integrity and confirmed wrong by live verification calls.

The systems could not reliably correct those errors using web search. Across five leading products, 22–35% of answered queries repeated the exact phone number already proven wrong. When unanswered queries were also counted, the systems failed 28–58% of lookups. Even the best-performing product failed more than one in four.

The conclusion is straightforward: retrieval is not verification. AI systems searching the public web often reproduce the same stale, syndicated information found in conventional provider directories. Reliable provider data requires independent detection and real-world verification, not simply another search layer over existing sources.

Introduction

People ask AI assistants how to reach doctors, and the directories those answers come from are full of stale phone numbers. ProviderBench measures one thing: when an AI product gives you a phone number for a specific provider, does it actually reach them?

Ground truth comes from live phone calls placed the same day the models were evaluated. There is one question type and one deterministic error verdict. The providers come from listings Sunny had identified as likely incorrect, where stale data concentrates.

The result: with web search enabled, 22–35% of answers repeated a phone number a call had already proven wrong.

Dataset

The dataset is 102 provider listings from a major carrier's directory, each identified by Sunny Network Integrity as likely inaccurate and confirmed incorrect by a live phone call on July 13, 2026. Listings where the call confirmed the provider was reachable were excluded: with one verified number per provider, a different answer can't be graded.

How models are graded

Five products were tested: Claude Fable 5, Claude Opus 4.8, GPT-5.5, GPT-5.6, and Perplexity Sonar Pro, each in two arms: no tools (no retrieval) and web search (the product's native search). Sonar Pro is search-native and only ran with search.

Grading is conservative. The only confirmed error is giving the exact number a call showed to be wrong. Any other number is recorded as a different number. It may be right or not, but it is never counted as correct.

Headline results are the web-search arm, with abstention penalized: the model had the tools to find the number, so returning nothing is a failed lookup. The failure rate counts wrong numbers plus abstentions, out of all 102 records. The no-tools arm isn't scored; it's discussed at the end.

An example

Below is a real record: real provider, real listed number, real responses. The provider's name and location are withheld, marked [the provider] and […]; the responses are otherwise verbatim.

Results

With web search, answer rates ranged from 65% to 96%, and 22–35% of those answers repeated the exact number verified wrong. Counting abstentions as failures, GPT-5.5 failed 29 of 102 lookups (28%) and Claude Opus 4.8 failed 59 (58%). The best product still failed more than a quarter of the time.

Full results

The same wrong numbers

Wrong answers clustered. On records both models answered, Claude Fable 5 and Claude Opus 4.8 gave the same verified-incorrect numbers with 91% overlap (Jaccard); GPT-5.5 and GPT-5.6, 89%. Cross-family pairs ranged from 50% to 72%. 31 of the 102 records had at least two products give the same wrong number, and 11 had all five. The likely cause is shared sources: the same stale listings, syndicated across the sites web search reads.

The 11 records where all five products agreed

Every product returned the same wrong number for these providers:

A split in risk tolerance

Without tools, the products split by lab. Both Claude models declined to answer all 102 questions; GPT-5.5 answered 54% and GPT-5.6 27%. (Sonar Pro has no comparable mode.)

None of these answers repeated the verified-wrong number, and none could be confirmed right either. The split reflects design choices, not accuracy. Given a question it cannot verify without tools, a Claude model is tuned to return no answer, while a GPT model is tuned to produce one from training data. That looks like a difference in risk tolerance: Anthropic's products avoid asserting phone numbers that cannot be checked, while OpenAI's accept some rate of error in exchange for being useful when the output is right. Withholding an answer costs the user one that may have been correct; producing one risks presenting stale data as fact.

The conservative behavior carried over: with web search on, the Claude models still abstained far more than the others. The protection did not. When they did answer, their error rates were the highest, which suggests the conservatism applies to output generated from training data, not to content returned by search.

Limitations

• One carrier's directory, one metropolitan area.
• Phone numbers only, not addresses, hours, or network status.
• Ground truth from a single day of calls; numbers change.
• 102 records, so per-model rates carry real uncertainty.
• An adversarial sample: every record was identified by Sunny as likely incorrect. These rates describe stale listings, not directories in general.
• One verified number per provider, so answers naming a different number can't be graded; listings confirmed reachable were excluded for the same reason.
• Results describe the July 2026 versions of these products.

Discussion

Given search tools, every product answered more often, and a fifth to a third of those answers repeated a number already proven wrong, sometimes with citations attached. Retrieval isn't verification: the pages these models read are the same stale listings that created the problem.

A fuller evaluation would cover more markets, more fields (addresses, hours, network participation), and repeat verification over time. We'll report results for future models.

Built and run by Sunny Health AI, July 2026. Ground truth: live verification calls placed July 13, 2026. Judge model: Claude Opus 4.8; verdicts assigned by deterministic digit-matching.

© 2026 Sunny Health AI, Inc. All rights reserved.