Clarity for AI buildersSponsored by AI Routing
adoption

DocSpring’s Ponytail test found leaner code but less complete changes

DocSpring’s Ponytail test found leaner code but less complete changes

DocSpring says Ponytail cut coding-agent costs and output in a small internal test, but missed changes its own setup covered.

In a published evaluation on its Rails and React monorepo, DocSpring compared its existing AGENTS.md-based workflow with the same setup plus Ponytail, a plugin that pushes coding agents to question whether code is needed and write smaller solutions. The company decided not to install the plugin, instead adapting three of its ideas into its own instructions.

The results illustrate a distinction between reducing implementation size and reducing task scope. Smaller diffs can contain less unnecessary code while still missing required behavior. DocSpring’s evaluation found evidence of both outcomes, making completeness—not just code volume—the central trade-off in its decision.

What the comparison measured

According to DocSpring’s account, the evaluation used Ponytail 5.3.3, Claude Code 2.1 and Opus 5.5. It covered four historical tasks: a dashboard rollup, a sign-up change, an avatar fix and a welcome modal. Fresh sessions started from the commit before each task, with DocSpring’s AGENTS.md instructions loaded. Each task ran twice with Ponytail and twice without it, producing 16 sessions.

Ponytail sessions produced 56% fewer source lines and 59% fewer test lines. Reported cost fell 34%, time fell 38%, and output tokens fell 41%.

A separate agent session judged the diffs without knowing which setup produced them. It preferred DocSpring’s existing setup, scoring it 6.8 out of 10 overall against Ponytail’s 5.9. Completeness scores were 2.4 out of 3 without Ponytail and 2.1 with it. The judge also identified more removable code in the baseline diffs: 253 lines, compared with 94 in Ponytail’s.

The misses were concrete. Both Ponytail runs on the sign-up task changed the Google and GitHub paths but missed the path where users entered their names. Both avatar runs left a test that would fail and made an inaccurate claim about the code in their final responses. On the rollup task, every session—including those without Ponytail—added an unrequested data-repair task.

Those examples limit what the headline savings establish. The lower line counts were not simply evidence of more efficient implementations: some accompanied omitted work. Conversely, the unwanted repair tasks show that DocSpring’s existing instructions did not eliminate overbuilding either.

Why project context mattered

DocSpring already maintained 42 AGENTS.md files totaling about 65,000 words, alongside workflow skills for investigation, reviews, CI fixes and browser QA. Its document-generation and e-signature work makes incorrect output particularly consequential, the company explained.

The post contrasts this environment with Ponytail’s own benchmark, which disabled CLAUDE.md files and automatic memory. As reported by DocSpring, that benchmark covered 39 tasks, each run five times per setup, and found 53% fewer lines, 26% lower cost and 41% faster completion with Ponytail. Hidden-check pass rates were 96.7% with the plugin and 95.6% without it.

These comparisons answer different questions. Ponytail’s reported benchmark isolates the plugin from existing project instructions; DocSpring’s test asks what it adds to an established workflow. Neither result, on the supplied evidence, establishes how it will perform across other repositories. DocSpring explicitly cautioned that its four-task sample provides only a rough direction, and its quality scores came from an agent judge rather than a reported human assessment.

What DocSpring changed instead

DocSpring adopted a “smallest complete change” rule: cover all required entry points, callers, tests and documentation in the same change, while avoiding unrequested options, compatibility paths and repair tasks. Agents must ask before repairing stored data.

It also required review findings to identify a plausible real-world trigger, while retaining thorough checks for new code, incorrect document values and data loss. Fixes larger than the code under review, or findings concerning code introduced by an earlier review round, go to the author for a decision. A separate lean-code pass now checks for unnecessary abstractions, unused options, duplication and work outside the ticket.

Together, these changes separate two decisions that a general instruction to “write less” can blur: which work is necessary, and how much implementation that work needs. DocSpring retained full task coverage as a requirement while adding explicit checks against unnecessary code.

The company also reported publishing selected principles, templates and workflow skills as the MIT-licensed DocSpring/docspring-agent-playbook. That package is an outcome of its internal workflow changes, not evidence that the same rules have been validated elsewhere.