Clarity for AI buildersSponsored by AI Routing
adoption

Artificial Analysis Decision Index v2 adds confidence calibration to its headline score

Artificial Analysis Decision Index v2 adds confidence calibration to its headline score

Artificial Analysis has released version 2 of its Decision Index, replacing its headline metric with a Decision Score that rewards both correct answers and well-calibrated confidence. The share of the index based on private datasets rose from 25% to 45%, a change Artificial Analysis says is intended to make the evaluation more useful in practice and harder to game.

Nace AI’s Drex Max v1.1 and Fastino Labs’ GLiDE v1.2 both scored 73 at the reported precision, with Mammoth 27B at 72. Artificial Analysis says Drex narrowly leads on the new index even though GLiDE leads on its accuracy metrics, 60 to 58. That difference illustrates what the new score changes: a model can give more correct answers yet rank lower if its confidence is less reliable.

The Decision Score puts correct answers above 50 and incorrect answers below 50, with confidence determining how far the score moves within either half. A confident wrong answer scores worse than a less confident one. Artificial Analysis describes the metric as averaging the payoff across confidence thresholds rather than choosing a single cutoff for when to accept an answer or escalate it to a person or a more capable model.

The new private AA-Paperclip benchmark accounts for 30% of the index. It tests five decisions and ten fact extractions on each of 1,500 documents, including resumes, receipts, invoices and research papers. A document passes only if all 15 answers are correct. Drex Max v1.1 answered 97% of individual questions correctly but passed 63% of documents; Claude Haiku 5.5 (Max) answered 99% correctly and passed 92%.

The all-or-nothing measure shows how a small gap in answer-level accuracy can translate into a much larger gap in completed documents.

AA-Symmetry accounts for another 15% and tests whether models give consistent answers and probabilities when equivalent questions are reworded. Artificial Analysis reports that Mammoth 27B chose the same answer across variants on 99% of questions, while noting that frequent, consistent abstention helped drive its high score. The update also adds EvalSafe-O*NET-AA, revises the options for declining to answer in AA-Omniscience-Decision and HLE-MultiChoice, reduces the weight of NVIDIA’s When2Call, and removes WildGuardTest because its results were highly saturated.

The published results also show trade-offs in cost and latency. Artificial Analysis estimates that GLiDE v1.2 costs $780 per million tasks versus $102 for Drex Max v1.1, in part because GLiDE bills the document input for each question rather than once per document. GPT-6 Luna (Max) scored 76 but cost $1,530 per million tasks and took 11.34 seconds per task, compared with 275 milliseconds for Drex.

These are results for the index’s workload, not general estimates for every deployment.