Why we reward our agents for refusing.

Picture a compensation committee on a Thursday afternoon. An HR business partner asks an AI assistant what share of the engineering team is paid below the midpoint of their band. It answers 39%, in a clean sentence, and the meeting moves on. Two weeks later someone notices that the 28 contractors, who have no pay band, were counted as below midpoint because their missing salary was read as zero. By then the budget has been allocated.

Nothing in that answer was malicious, and the model was not broken. It did what most AI systems are trained and tested to do: it answered.

Why models learn to guess

A paper published in September 2025 by Kalai, Nachum, Vempala and Zhang argues that hallucination is partly an incentive problem. Most benchmarks grade answers as right or wrong. Under that grading, in the authors' words, “abstaining is strictly sub-optimal”: a guess might earn a point, while “I don't know” never does.

A companion note from the same team put figures on it, from the developer's own evaluation of two of its models. One declined to answer 52% of questions and was wrong on 26%. An older one declined 1% of the time and was wrong on 75%. Their accuracy was almost identical, 22% and 24%. On a leaderboard that only counts correct answers, the two look the same. In an HR team, one of them is wrong three times as often and gives no signal about which answers to doubt.

It gets worse with the models people now prefer. A June 2025 benchmark of twenty frontier models across twenty datasets found that fine-tuning a model to reason degrades its ability to abstain by 24% on average, and concluded that abstention is “an unsolved problem, and one where scaling models is of little use”. The better the model argues, the less often it stops.

What a wrong answer costs in HR

A wrong figure in HR rarely ends with a corrected slide. It can reopen a pay review, trigger a letter to an employee, or leave an HR director unable to answer a works council. Under the EU pay-transparency directive, employees can ask in writing for their own pay and the average pay of people doing the same work or work of equal value, and the employer has two months to answer. That answer has to be right the first time.

How our agents decide to stop

We build around a rule our team repeats often: grounded or silent. Before an answer reaches the person who asked, it has to clear four checks.

1. Is there a source the person may read? If a figure cannot be traced to a record, a calculation or a policy paragraph, it is left out, and the agent says what is missing.
2. Was the data actually read? If the read stopped early or hit a permission boundary, an empty result proves nothing. The agent says “not found in what I could see”, never “does not exist”.
3. Was the arithmetic done by code? Counts, averages and comparisons across many records are computed and checked, never estimated by the model.
4. Who decides? Anything that touches one person's pay, promotion or conduct ends with a named owner. The agent prepares the brief and the decision stays with them.

A refusal has to be useful

“I can't answer that” helps nobody. A good refusal declines the part it cannot support, answers the rest, and says exactly where the line fell. Here is the compensation question again, answered the way we design for.

The HR business partner can take that into the committee and defend every part of it: what was counted, what was left out, and why.

What to ask your vendor

1. What are your error rate and your abstention rate, measured separately?
2. When the agent declines, what exactly does the person see? Ask for a real example.
3. How does the system tell “no data” apart from “no access” or “not read”?
4. Which calculations are done by code, and which by the model?

Before any assistant goes in front of a compensation committee, ask it a question it cannot fully answer and read what it says.