Why I paused an AI project: the cache bill missing from the architecture

2026-09-19 · agents, jev, prompt-caching, engineering, product

Jevis had a detailed design for frontier-model planning and cheaper execution. Prompt-cache economics exposed the hypothesis we still had to test.

Today I paused an AI project I had been preparing to build seriously.

It had a name, Jevis, a registered domain, and a second design specification running to more than forty pages. Work on the framework and website had started. Our discussion had even reached the point of deciding to build the whole design rather than keep debating feature priorities.

Then a question interrupted the momentum: what happens to the models' caches when we orchestrate them this way?

That question did not establish that multi-model collaboration is unworkable. It exposed something narrower: we had not demonstrated our central promise of delivering more reliable work within a limited budget.

This is a design-stage retrospective, not a benchmark report. We did not complete an end-to-end controlled experiment that could support a commercial conclusion. The numerical example below is hypothetical, not a measured Jevis saving or loss.

Why the idea was compelling

Frontier models are good at difficult judgments, but their cost and usage allowances are scarce. Many implementation tasks appear not to need the strongest model. Could we assign architecture to a frontier model and bounded execution to cheaper models, combining capability with efficiency?

Jev made that idea more concrete. TypeSafe describes it as a System One model that returns typed decisions: choices, scores and probabilities rather than free-form text. It can support routing and checks; it is not a generative model that writes code or summaries itself. TypeSafe's introduction

Our proposed division was straightforward: the frontier model defines correctness, ordinary models perform the repetitive implementation, and Jev helps transfer and evaluate the specification.

For an interface migration, a strong model might define the migration rules and invariants. Multiple workers would then edit separate files. Jev would supply signals on bounded questions: does this implementation satisfy a requirement, does it need escalation, is this material relevant? Code and the primary agent would remain responsible for actions.

I still think that division can be useful.

A serious design is still a collection of hypotheses

Version two was more careful than “hard tasks to expensive models, easy tasks to cheap ones.”

Its central questions were whether an output would be reused across subsequent work and whether its errors were detectable. A seemingly simple architectural choice could affect dozens of tasks. A difficult log-parsing job might have an unambiguous, checkable answer.

English translation and redraw of the original specification, page 6, figure 1. Horizontal axis: error detectability. Vertical axis: whether an output is reused. These are proposed decision boundaries, not experimentally validated ones.

The architecture had ten modules covering profiling, ingestion, specification, allocation, execution, state carrying, evaluation, a deterministic safety gate, quota governance and feedback. At its center was a “carrier”: goals, specifications, invariants, progress and workspace facts, projected into the context each worker needed.

English translation and redraw of the original specification, page 8, figure 2. Specifications enter the central carrier; workers read relevant projections; evaluation writes progress back. The three loops carry state, evaluation and learning. Latency estimates from the original have been omitted; this is a design proposal, not a measured result.

We had considered retries, escalation, permissions, task boundaries and rework. The specification explicitly distinguished the carrier from a cache: it represented the state of the workflow.

That distinction was conceptually correct. The missing step was economic: portable state does not imply inexpensive handoff.

Our carrier answered “what should the next worker know?” A provider's prompt cache answered “which input computation can this request reuse?” Sending the JSON to another worker does not transfer the provider's inference cache with it.

The baseline we overlooked: a frontier model already in context

It is easy to compare two price sheets and conclude that implementation belongs on the cheaper model.

Halfway through a long coding task, however, the alternatives may be different. One is a frontier model with an established, reusable prompt prefix. The other has a lower unit price but needs to receive the task material afresh.

Prompt caching matches input prefixes, not merely semantic similarity. Changes early in a prompt can reduce reuse later in it; an unchanged earlier portion may still hit. A new session is not necessarily entirely cold, and an existing session is not guaranteed to remain warm. Scope, lifetime, model and request layout matter. OpenAI documentation, Claude documentation

The relevant comparison becomes: let the already-contextualized frontier model continue, or pay for another worker's context, execution, checking and possible return to the frontier model?

Model selection is consequently history-dependent. Task difficulty and list price do not capture who has already paid to process the scene.

We cannot assume that different providers or models share caches. Equally, we cannot infer a cache miss solely from the creation of a subagent. Both shortcuts produce unreliable estimates.

A deliberately small example

Assume a frontier model's ordinary input costs 10 arbitrary units per million tokens and its cache reads cost 1. An ordinary model's uncached input costs 2. Both need the same 100,000 tokens of history.

History component of the next request	Hypothetical cost
Frontier model fully reuses its existing cache	0.10
Ordinary model reads the same material uncached at its ordinary input rate	0.20
Ordinary model receives only a 10,000-token task slice at its ordinary input rate	0.02

These invented rates are neither vendor quotes nor total task costs. They exclude new input, output, cache writes, Jev, validation and retries. Previous cache-building expenditure is treated as sunk for this next-action decision; a whole-task comparison must count both alternatives from the beginning.

The example establishes only that a cheaper model can have a more expensive next input, and that sufficiently small, adequate task slices can still make delegation attractive.

“Adequate” carries much of the difficulty. Omit an important constraint and the worker may reread files, replan, or produce something the frontier model has to repair. We cannot assume lossless slicing and then use that assumption as evidence of savings.

Compaction needs a payback calculation too

Another attractive idea was to let Jev identify obsolete history and remove it.

Fewer tokens need not mean a smaller bill. Removing material in the middle of cached history can reduce reuse of subsequent content. The decision process, rebuilding the affected cache and retrieving mistakenly omitted evidence must be weighed against future savings.

OpenAI's current documentation notes that compaction can reduce cache reuse on the next request while still reducing total input cost. It also distinguishes model generations: for GPT-5.6 and later, writes cost 1.25 times ordinary input and reads cost 0.1 times ordinary input. The old blanket assumption of no additional OpenAI cache-write charge is unsafe. OpenAI documentation

Claude also distinguishes writes and reads, with rates depending on retention and model. API prices should not be substituted for an exact formula governing Claude or Codex subscription allowances. Claude documentation

A useful decision test is whether expected savings over the remaining requests exceed the extra judgment, transformation, cache-rebuilding and expected rework costs. Cache rebuilding is already part of input billing; do not count it twice.

One remaining turn leaves little opportunity to recover an up-front cost. Many subsequent turns with a newly stable prefix can change the outcome. Near a context limit, the counterfactual must also account for whether continuing without compaction is possible at all.

Filtering new tool output before it enters the conversation differs from repeatedly rewriting old history. It generally offers a better opportunity to preserve the existing prefix, but still needs evidence that selection costs and missing information do not erase the benefit.

We had turned assumptions into modules

Looking back, several claims were doing more work than their evidence justified.

“Generate the specification once, reuse it many times” requires sufficiently similar tasks and a complete specification.

“Cheap execution has closed acceptance criteria” requires checks that catch consequential errors, rather than merely produce reassuring scores.

“The carrier prevents workers from rereading the scene” requires projections that are both small and complete.

“Jev is cheap, so additional judgments are negligible” needs a workflow-level measurement of call count, repeated state, waiting and mistakes.

Version two aimed to maximize delivery within a budget, not simply minimize cash expenditure. That makes subscription capacity important too. A prepaid allowance still has opportunity cost: capacity consumed by routine execution may be unavailable for a difficult diagnosis later.

We were becoming increasingly precise about implementation boundaries without being equally precise about why the assembled system would outperform the native workflow.

Module boundaries demonstrate implementability. The product hypothesis needs different evidence.

What I would make the MVP test first

I would start with an existing agent and representative real tasks, comparing three approaches: the native workflow; a retained primary session delegating only bounded subtasks; and full dynamic orchestration.

They would share task starting points and acceptance criteria. Short tasks, long sessions and repeated batch work would be evaluated separately. Cold and warm conditions would be recorded explicitly, with test order controlled so one alternative did not accidentally inherit another's warm-up advantage. A favorable demonstration would not substitute for repeated trials.

The unit of observation would be an accepted task: total cost, cache reads and writes, completion time, failures, retries, human correction and final quality. Observable subscription consumption would be recorded separately, without inventing a precise conversion to dollars.

Tests and deterministic rules would verify what they can. Semantic checks would have independent or human review. A high Jev score cannot be the sole evidence that Jev-based acceptance works.

The result might show that orchestration helps only a particular class of repetitive work. That would be a useful product boundary, not a disappointing footnote.

Why pause now

Caching did not defeat Jevis in a completed experiment. We had not run that experiment.

It challenged the certainty behind further investment. I had been acting as though value was established and engineering was the remaining task. The comparison baseline was still unresolved.

We subsequently explored Jev skills and plugins. Official guidance and community review and filtering tools already exist. A plugin could reduce the scope we take over, but would not automatically remove cache costs, incorrect judgments or maintenance. I decided not to relaunch the idea immediately under a smaller package. Official skills, Jev Review, jev-pruner

Today's decision is to pause this project. It is not a rejection of Jev or a verdict on the future of multi-agent systems.

The easy-to-miss distance between an MVP and reality lies between “this workflow runs” and “this workflow is worth what it costs the user.” AI makes the first milestone faster. It can also help us turn untested premises into an elaborate system faster.

Next time, I want the first deliverable to be the experiment that might convince me not to build.

Based on the September 19, 2026 project discussion, the author's v2 design specification and primary documentation checked that day. Prepared with AI assistance; the cover and two explanatory illustrations are generated conceptual visuals and the English diagrams are translated redraws of the original design; the Chinese version retains the source figures. No end-to-end performance result is claimed.

We also operate TopxAI, a multi-model API service supporting TypeSafe's latest public release, Jev 1.13.0, as of September 19, 2026. Check the live catalogue as versions change. To evaluate it yourself, start with our Jev integration guide. Explore the model; measure the workflow.

Other languages: 简体中文