In the controlled small-project benchmark, Product Traceability 2.0 made Claude Code 38.5% cheaper, reduced recorded end-to-end build time by 38.3%, and required 54.6% fewer agent round trips than plain Claude Code Opus 5 (1M context). The saving appeared in every application tested.
Each application was tested through complete ten-session builds, with three runs per configuration. The resulting products passed 98% to 100% of their external checks, and no previously working feature regressed.
That result is striking because Version 1.0 failed the controlled test.
In May, I published an article about a Claude Code skill I had built for myself: Product Traceability. The idea was simple: keep the product requirements, changes, important decisions, rejected alternatives, and links to the code inside the repository, so a fresh AI coding session does not have to reconstruct the product from scratch.
At the end of that article, I promised a proper A/B test because its original productivity and caching figures were retrospective estimates, not controlled results. I later withdrew the cache explanation. The new study would hold the work, model, runtime, and grader constant.
I finally ran it. Version 1.0 lost spectacularly.
Both Control and Version 1.0 recorded zero regressions, but Version 1.0 cost 2x to 3x times as much as plain Claude Code. The result showed that Version 1.0 was expensive. I had to take the run apart to learn why.
The failure did not end the idea. It exposed what needed to change. Here is how the failure was diagnosed, what changed, and how the redesigned system was tested.
Why Version 1.0 failed
Version 1.0 of Product Traceability asked the main coding agent to keep four Markdown files synchronized after every product change:
- requirements.md— the living product requirements;
- change.log— the sequence of changes;
- decisions.md— choices, rationale, and rejected alternatives; and
- traceability-matrix.md— the link between requirements and implementation.
The information was useful. The knowledge and memory maintenance mechanisms were not.
The teardown that led to Version 2.0
The first result told me that Version 1.0 was expensive, not why. The same direction appeared across all three diagnostic applications, although each had only one Version 1.0 trajectory at that stage, making those ratios diagnostic rather than population estimates. I then parsed all 20 sessions from the smallest application: ten from Control and ten from Version 1.0. I deduplicated the recorded tool calls, classified them by target and purpose, and reconciled those categories with the provider-reported model usage and cost for each assistant turn.
That ruled out a simple explanation: Version 1.0 was not doing more direct application work. Both arms made exactly the same LLM calls against the application file—the same number of edits, writes, and reads. Control made zero calls against the four Product Record files, while skill spent more compute on work with Product Records than the application itself. A post-hoc turn-level allocation estimated that record-only work accounted for about $7.83 of the $10.48 cost premium. That allocation is diagnostic, not a randomized causal estimate—the method had to apportion output tokens by character share—but both decompositions pointed to the same operational bottleneck.
The Product Record itself was not the expensive part. Maintaining it with a frontier coding model inside the foreground loop was. The teardown therefore imposed a clear design constraint on Version 2.0: preserve product history while moving its upkeep out of the coding agent’s loop.
The coding agent should build the product. Product Traceability should keep the record.
What Product Traceability 2.0 actually does
That test killed Version 1.0, but it did not kill the underlying problem. Code and Git are good at preserving what exists. They are much worse at preserving why it exists, which alternative was rejected, or which requirement replaced an older one.
The main coding agent always receives the complete user request. Product Traceability does not shorten or rewrite the task prompt.
Instead, lifecycle hooks observe the request, changed files, and final answer. After a turn that changes the product, a small auxiliary model reads bounded excerpts and proposes requirement updates and, at most, two reusable decisions. The request remains the source for requirement changes; an evidence guard accepts a decision only when its quoted support appears in the diff or final explanation. Deterministic code then rebuilds the human-readable Product Record and prepares a compact cross-session context for the next fresh session.
The Product Record is organized around stable semantic objects rather than transcript fragments. For example, a request such as “show active tasks before completed tasks, and sort each group by priority” can become two separately addressable requirements:
- REQ-007— active tasks precede completed tasks;
- REQ-008— tasks within each status group are ordered by priority.
A later request can refine or cancel one rule without silently changing the other. Product decisions can store not only what was selected, but what was explicitly rejected and why—the kind of information that may never appear in the final code.
For human developers, the system compiles readable requirements, decisions, changes, and traceability documents. For later AI sessions, it supplies prioritized cross-session product context and keeps detailed decision topics available on demand. The full archive does not have to be poured into every prompt.
Most importantly, the main coding agent no longer opens or edits the record. Across all nine final Product Traceability trajectories, it made zero main-agent record-file edits.
The next question was whether that architecture could reduce work without compromising the resulting product.
Testing the redesign
The controlled retest used three small, single-file web applications built with HTML and JavaScript:
- Todo Manager App
- Calculator App
- Tetris Game
Each product evolved through ten requirement batches, and every batch started in a fresh Claude Code session. The agent never saw the hidden full PRD. It saw the product change the way software changes in real life: features accumulated, requirements conflicted, an earlier decision was partially reversed, defects appeared later, and some requests relied on earlier agreements without restating them.
That fresh-session design matters. In one long conversation, the transcript itself can serve as memory. The experiment was designed to test what happens when the next agent has the code and repository history, but not the previous conversation.
The final primary comparison contained two configurations:
- Control: standard Claude Code with source access, Git history, and Claude Code’s built-in memory, but no Product Traceability.
- Product Traceability 2.0: the same Claude Code runtime plus the complete Product Traceability system.
Both used Claude Code 2.1.269, Claude Opus 5, its 1-million-token context window, the same starting source, the same prompts, the same effort level, and the same external browser grader. Each application/configuration pair was repeated three times and run one at a time in a seeded randomized schedule.
That produced 18 primary trajectories and 180 fresh coding sessions. Cost is the sum of provider-reported costs across a complete ten-session build, including Product Traceability’s auxiliary extraction calls. Time is the recorded end-to-end wall time for that complete build. The study also counted every model’s output and new tokens, not only the main coding agent.
With the applications, configurations, and measurements fixed, the opening headline can now be read for what it is: a comparison of complete ten-session builds, not isolated edits. In this study, “38% faster” means Product Traceability completed those builds in 38.3% less recorded time than plain Claude Code. Results on larger projects may differ.
The gain appeared in every application
The opening figure shows that the result was not driven by one lucky application. The direction was the same in all three, although the size of the improvement varied.
Todo was the app used while tuning the system, but Calculator and Tetris—frozen checks on apps outside that optimization loop—showed similar cost reductions.
Here are the actual final Product Traceability applications from the first replication run:
The quality result needs careful wording. Product Traceability passed every final Todo and Tetris check. One Calculator trajectory missed two of 99 final checks. Across all configurations, the grader was already near its ceiling, and the cells contained only three runs each.
Where the saving appears
The strongest operational change was not a magical compression ratio. It was fewer interactions.
The largest movement was in agent round trips, which fell 54.6%. All-model output tokens fell 33.7%, while all-model new tokens fell by a smaller 8.5%.
Plain Claude Code averaged roughly 80 main-agent round trips per ten-session build. Product Traceability needed roughly 40. Fewer turns mean fewer generated explanations, fewer repeated reads, fewer verification loops, and less accumulated context being processed again.
Post-hoc diagnostics from the Todo optimization sequence show the likely shape of the saving. Repeated source orientation fell from about 10.7 calls to 0.2, and repeated verification fell from about 25.3 calls to 2.8 in the final configuration. Earlier costly builds also created disposable test harnesses and paged through the same source repeatedly. The final system instead supplied a current source map, encouraged one source read, kept checks small, batched edits, and verified once at the end.
These diagnostics explain a plausible mechanism; they do not allocate a causal percentage to each tactic. The controlled experiment tested the complete Product Traceability configuration as a package: policy, hooks, extraction, compilation, source mapping, and cross-session context changed together.
That distinction matters. The 38% result belongs to the complete Product Traceability 2.0 configuration, not to “memory alone” or any one tactic.
The unexpected Product Traceability Light result
The teardown raised a second question: if compact projects benefited so much from better workflow discipline, did they need a persistent Product Record at all?
While fixing Version 1.0, I distilled six working rules that appeared to remove waste from the coding loop: read the source once, keep checks small, batch edits, verify once, respect explicit product rules, and end concisely.
I froze those rules in a small CLAUDE.md with no hooks, no extractor, and no persistent Product Record. That configuration became Product Traceability Light.
Product Traceability Light was then evaluated under the same runtime with three trajectories per application, adding 90 fresh coding sessions. It averaged 40.7% lower cost and 49.6% less recorded build time than Control and used 50.5% fewer round trips. That is an excellent result—and an important warning against telling the wrong causal story. A persistent record was not necessary to produce the operational gain on these compact projects.
Light is therefore the practical recommendation for prototypes, utilities, experiments, and other short-lived work where the main goal is moving quickly. It has almost no runtime machinery.
The complete Product Traceability system is intended for products that evolve over weeks or months, change hands, or need requirements, rationale, rejected options, and an audit trail to survive across sessions. That durable history is the capability Light does not provide.
There was already a hint of the tradeoff in the study: two of three Light Todo builds moved logic into a second local file even though an earlier requirement called for a single file. The automated grader missed that structural breach. Product Traceability preserved the rule in every run.
Light sharpens the recommendation rather than weakening it: workflow discipline is often enough for compact, short-lived work; durable product history becomes the differentiator as software lives longer and changes hands.
What this study does—and does not—establish
This is the first controlled A/B evaluation of Product Traceability. The earlier article used observational, pre/post evidence; this one holds the runtime, model, prompts, starting source, grading, and repository shape constant.
It is also a deliberately small first study:
- three applications;
- three runs per configuration and application;
- compact HTML-and-JavaScript codebases;
- one model and one Claude Code version;
- a near-ceiling external grader; and
- a complete-configuration comparison rather than a factorial test of every component.
The author designed the system, benchmark, and grader. The controlled experiment did not test large repositories, additional languages, multi-contributor handoffs, long-lived stale decisions, or automatic requirement-to-test linking.
Performance on Medium-Size Projects
A separate preliminary pilot drew three consecutive tasks from an anonymized real-world JavaScript product whose repository, when the pilot was prepared, contained approximately 2,800 tracked files, 113,000 lines of JavaScript, 300 test files, a PostgreSQL database, and nearly five months of history. One isolated trajectory was run for Control and one for Product Traceability. Product Traceability cost 66% less, took 52% less recorded time, and used 63% fewer agent round trips; both configurations passed the same 3 of 7 hidden checks. A post-run audit found no genuine project-suite regression from Product Traceability, while Control introduced one during the first task and recovered it during the second. With only one trajectory per configuration and three tasks, these results are directional evidence—not a stable effect estimate or validation at scale.
The larger-repository pilot is therefore directionally consistent with the small-project result, but it did not reproduce the full study. Completing the planned medium-sized-repository design—three configurations, three trajectories per configuration, and all nine tasks—would require 81 fresh Claude Code sessions plus approximately 27 Product Traceability extraction calls. A straight-line planning estimate based on the observed Control and Product Traceability pilot, with the still-unrun Light configuration estimated from the small-application benchmark, is about $430 in model charges, 18 hours of recorded agent runtime, 4,000 agent round trips, 14 million new model tokens, and roughly 470 million cached-context token reads. The cache figure represents repeated reads, not 470 million unique tokens of text; sequential grading would add approximately 1.6 hours. Because the six later tasks and the Light configuration remain unmeasured at this scale, a responsible operating budget would be approximately $500–$600 and 24–30 hours for a serial run, including setup, grading, and review. These are planning estimates, not observed results.
The conclusion I did not expect
Version 1.0 and Version 2.0 tested the same underlying idea with opposite runtime architectures. Version 1.0 made the most expensive model in the system maintain product memory and paid for it in turns. Version 2.0 treats product history as infrastructure: capture evidence off the main loop, compile what can be deterministic, retrieve only what is useful, and let the coding agent code.
Version 1.0 cost 2–2.9 times as much as plain Claude Code. After the redesign, Version 2.0 cost 38.5% less than plain Claude Code in the controlled small-project benchmark and demonstrated similar directional results on a medium-size project.
The larger lesson is that LLMs became significantly better at coding and existing coding harness is not necessarily leveraging new capabilities to full extent. Additional memory and cross-session knowledge management helps AI agents to develop software faster, but batching operations and timing test execution also creates a big difference. I am now more worried how this gained efficiency would affect the quality of the software in long-run, test coverage and cause unintended technical debt. Some of the tactics sound risky, careless and spread YOLO vibes. In my future work I will closely monitor quality and plan for a larger controlled study against several medium-sized projects.
Read the paper or try it
The complete paper, task specifications, graders, frozen configurations, analysis code, and raw sessions from all 27 study runs are available in the Product Traceability repository. The project is open source and MIT licensed.
git clone https://github.com/vmysla/agent-skill-product-traceability.git
cd agent-skill-product-traceability
./install.sh # Product Traceability 2.0
./install.sh --light # Product Traceability LightThe scientific paper is available here: Product Traceability for Coding Agents: A Controlled Evaluation Across Evolving Web Applications.
Product Traceability 2.0 was developed by Vlad Mysla. The study used Claude Code 2.1.269 with Claude Opus 5 and a 1-million-token context window. Claude Code and Claude are products of Anthropic; this project is independent and unaffiliated.