Clarity for AI buildersSponsored by AI Routing
availability

Google DeepMind and Artificial Analysis launch benchmark for AI agents working with complex visual files

Google DeepMind and Artificial Analysis launch benchmark for AI agents working with complex visual files
Google DeepMind and Artificial Analysis launched a benchmark that tests whether AI agents can turn complex visual files into usable work products. Artificial Analysis says the Multimodal Knowledge Agent (MKA) benchmark covers 200 private tasks across 10 industries. Agents must use inputs such as architectural drawings, engineering schematics and clinical scans to produce files such as slide decks, spreadsheets, reports and marked-up images. The visual material is necessary to complete the tasks, not just background context. That makes the benchmark a test of whether agents can carry information from a visual source into a finished deliverable, rather than simply answer questions about an image. Expert rubrics average about 70 weighted criteria per task. Artificial Analysis separates baseline checks, called Regression criteria, from harder Climb criteria intended to distinguish stronger models. The headline metric, Climb Weighted Satisfaction Rate (WSR), measures the weighted share of those harder criteria that an agent satisfies. It is not the percentage of tasks completed or a measure of how often agents pass the baseline checks. Artificial Analysis runs the tasks in Stirrup, its open-source agent harness, with code execution, web search, web fetching and image-viewing tools, allowing up to 250 turns. GPT-6 Luna (medium) grades the submitted files. The leaderboard therefore reflects performance with that tool access and grading setup, rather than unaided model performance. Anthropic’s Claude Fable 5.2 (max) led the reported results with a Climb WSR of 57%, followed by Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%. Artificial Analysis reported average costs of $31.97 per task for Fable and $6.29 for Carbon. At the published scores, Fable cost about five times as much per task as Carbon, making the score leader and the lower-cost option distinct choices within this evaluation. The overall ranking also obscures differences by workload. Artificial Analysis reports that Fable led or tied for the lead in five industries, while Carbon led Healthcare and Opus led Education and Construction. The report identifies Word documents and text files as the hardest output categories, with no tested model exceeding 52% Climb WSR in either. For teams evaluating agents for a specific workflow, the relevant industry and output format may be more informative than the overall ranking alone.