Most teams trying to reduce AI costs begin in the same place: they shorten prompts. Remove a paragraph here, compress a system instruction there, celebrate shaving a few hundred tokens.
But the largest savings rarely come from sending shorter prompts. They come from asking a more important question: which model should see those tokens in the first place?
Many agent systems send every part of a workflow through the most capable, and most expensive, model available. The model interprets the request, creates the plan, reads every document, calls every endpoint, checks every field, retries every failure, and writes the final answer.
That is like hiring a strategy consultant to design your operating plan, and then paying the same consultant to copy rows between spreadsheets.
At Joxy, this is the difference between an agent workflow that is affordable to run every day and one that only makes sense as a demo. The token-heavy middle of our workflows (reading, calling, checking) dwarfs the planning at the start, and pricing all of it at the frontier rate is what breaks the economics.
A better architecture separates the work: big models plan. Small models execute. Big models return when execution gets stuck.
Two kinds of work
A complex agent workload contains two very different kinds of work.
The first is judgment-heavy: understanding an ambiguous request, decomposing it into steps, identifying dependencies, evaluating risk, and handling exceptions. The second is token-heavy: reading documents, fetching records, calling APIs, updating fields, and comparing expected results with actual ones.
These tasks do not require the same model capability.
Anthropic’s coordinator-pattern cookbook demonstrates the separation with a frontier coordinator and smaller research workers. The coordinator plans and synthesizes; the workers do the token-heavy reading in isolated contexts. In the authors’ example runs, the split architecture was roughly 2.5 times cheaper and three times faster, with 84 to 98 percent of input tokens billed at the worker-model rate. Anthropic cautions that these ratios vary by run: the architecture is the repeatable insight, not any single benchmark number.
In concrete terms, this is how we run Joxy: a Claude model handles the planning, and Gemini or an open-source model handles the execution. The same split works across today’s lineups, whether the frontier seat is Claude Fable or GPT-5.5, because the per-token price gap between frontier and small tiers is typically 5x to 15x. Every token you move down a tier is a direct discount on the same work.
The lesson: do not assign models by workflow. Assign them by cognitive difficulty.
Planning is where frontier reasoning earns its price
The most expensive model should make the most expensive decisions: what the user is actually trying to accomplish, what information is missing, which actions depend on earlier ones, what creates financial or reputational risk, where approval is required, and what successful completion looks like.
Once those decisions are made, much of the remaining work is procedural. Consider a request like:
Create a new advertising campaign, add three ad groups, attach the
approved creative, and schedule it for Monday.To the user, that sounds like one action. Operationally it involves identifying the correct account, checking for an existing campaign, confirming objective and budget, requesting approval, creating the campaign, saving the returned ID, creating the dependent ad groups and ads, then reading everything back to verify relationships, budgets, statuses, and schedules.
The difficult part is not calling create_campaign. The difficult part is deciding whether create_campaign is the right action, with the right arguments, at the right moment, against the right account. That is planning work.
The Planner
The Planner converts a high-level outcome into a chronological, dependency-aware execution plan: missing inputs, existing-object checks, read and write operations, approval groups, parent-child dependencies, idempotency rules, validation requirements, retry conditions, and escalation paths.
It does not execute tools. It creates an execution contract another model can follow without guessing.
In our architecture, an integration guide acts as the source of truth for available tools, argument names, IDs, approval rules, permission limits, and API quirks. The Planner is explicitly prevented from inventing unsupported endpoints.
That restriction is essential. A capable model is very good at producing an endpoint that looks plausible. Unfortunately, APIs do not accept “plausible” as an argument type. The Planner should be creative about solving the user’s problem, not creative about which tools supposedly exist.
The Worker
The Worker should not receive the entire business problem and be asked to figure things out again. It receives a narrow execution envelope: the action, the approved tool, the exact arguments, the dependencies that must exist, the result to save, the expected state afterward, the verification method, the retry policy, and the stopping condition.
For example:
Create the ad group using the saved campaign ID. Save the returned
ad-group ID. Read the object back and verify its name, campaign
relationship, targeting, budget, and status. Stop if any expected
field does not match.That is a much smaller reasoning problem than “build our advertising campaign.” The small model is not replacing the big model’s intelligence. It is operating inside a structure created by that intelligence.
Verify everything
A tool can return success while producing the wrong outcome: the wrong parent, the wrong budget, a draft state instead of a published one, a silently normalized field.
Every important action should follow a verification loop:
Execute → Read back → Compare → ContinueThe Worker does not move to a dependent step until the current output has been verified or explicitly escalated. A 200 OK means the server accepted the request. It does not mean the business objective was completed correctly.
Route by capability, not price
Sending every task to the cheapest model creates a different kind of expense: incorrect tool calls, duplicate objects, failed retries, human cleanup, and lost trust. The target is not the cheapest model. It is the cheapest capable model.
This is also OpenAI’s recommended process: run the workflow with the most capable model first and establish an eval baseline (tool selection, argument accuracy, completion rate, latency, cost), then substitute smaller models step by step, keeping each substitution only when it still meets the standard.
This creates a capability ladder:
A workflow should move down that ladder whenever reliability allows.
The economics
Total cost = planning cost + execution cost + exception cost.
In a poorly routed system, everything is billed at the frontier rate: planning, execution, retries, verification, and raw document reading. In a routed system, only planning and genuine exceptions are. The ceiling is high: when execution dominates the token volume and runs on a tier 10x to 15x cheaper, with cache reads at a tenth of input price on the stable prefix, input-token spend can fall by up to 90% against an all-frontier baseline. Most workflows land lower, and Anthropic's own coordinator runs came in around 2.5x cheaper, but the ceiling is what the architecture makes possible. The savings become meaningful when execution represents the majority of token consumption, which is why the pattern works especially well for document review, log analysis, codebase sweeps, CRM operations, campaign construction, and data migration.
Anthropic’s coordinator example gives each worker its own context. Workers read the raw material and return distilled findings (the relevant conclusion, the supporting evidence, the uncertainty, the recommended next action) rather than copying source content into the coordinator’s context. Independent work runs in parallel, and the expensive coordinator never processes the raw material at all.
Do not send the expensive model the whole warehouse when it only needs the inventory report.
Cache the tokens you cannot route away
Even a well-routed workflow re-sends the same tokens constantly: the system prompt, the tool schemas, the integration guide, the document under review. Prompt caching stops you from paying full price for that stable prefix on every call.
The economics are strongest on Claude. A cache write costs 1.25x the normal input price (5-minute lifetime) and every read within the window costs 0.1x. That means the cache pays for itself on the very first reuse: two questions over the same context cost 1.35x one pass instead of 2x, a saving of roughly a third, and every question after that costs a tenth of what it would have. The 1-hour cache writes at 2x, so it breaks even around the second reuse and profits from the third.
Gemini splits caching in two. Implicit caching is automatic: when a request happens to match a recent prefix, the discount is applied with no code changes and no storage fee, so any hit is pure savings. Explicit caching guarantees the discount (75 to 90 percent on Pro models, depending on the generation; check current rates) but charges rent: a storage fee per token per hour that the cache exists ($4.50 per million tokens per hour on Pro, $1.00 on Flash at the time of writing). That changes the break-even from a question count to a rate. A cache read several times within a short window wins decisively; a cache that sits idle for an hour on Pro can cost more than simply re-sending the context. As a rule of thumb, explicit caching on Gemini pays when the same large context is queried more than three or four times within its lifetime, and caches should be deleted the moment a job finishes.
Two practical rules make caching work in this architecture. First, structure prompts so the stable content comes first and the volatile content comes last, because caches match on an identical prefix: system prompt, then integration guide, then tool schemas, then the task. A change invalidates the cache from that point onward, so the earlier the edit, the more you re-pay. Second, cache at the tier where the tokens actually live. Caching applies at every tier, and the saving is rate times volume: the frontier model saves the most per token, while the Worker, which carries most of the traffic in a routed system, saves the most in aggregate. Either way the discount stacks with routing rather than replacing it.
Caching and routing are independent levers. Routing moves tokens to a cheaper meter. Caching stops the meter from running twice on the same tokens. Use both.
The Unblocker
Routing and caching keep healthy execution cheap. But cheap execution only works until reality stops matching the plan: a rejected field, a missing permission, an expired ID, a deprecated endpoint.
The Worker first applies a bounded retry policy for clearly transient failures (timeouts, rate limits, service unavailability). Before retrying a write, it must verify the previous request did not actually succeed. Otherwise a timeout becomes a duplicate campaign or a duplicate invoice.
If the error survives the retry limit, the Worker stops the affected path and sends the Unblocker a structured failure packet: the failed step, the attempted function and arguments, the error, the raw response, the retry count, partial outputs, and the last verified step.
The Unblocker receives only the failed turn, not the entire workflow. Its job is surgical: diagnose the failure, inspect the integration guide, consult current official documentation when needed, correct the smallest possible part of the failed action, verify, and return control to the Worker. Our Unblocker prompt formalizes this limited scope: it may not continue unrelated future work or bypass approvals.
Routine work stays cheap. Difficult exceptions earn expensive reasoning.
Operational memory
Every success and every discovered failure should improve the system. In this architecture, operational memory lives in a connection-specific integration guide: confirmed endpoints, working arguments, account IDs, permission scopes, rate-limit behavior, known quirks, deprecated fields, and successful unblock patterns.
When the Unblocker discovers that an argument format changed, that knowledge is written back to the guide. The next Planner avoids generating the broken step. The next Worker avoids executing it. The next Unblocker starts with the solution already documented.
Plan → Execute → Fail → Diagnose → Learn → Plan betterThe models do not need retraining for the system to improve. The workflow improves because experience becomes documentation.
Delegation is not abdication
OpenAI distinguishes between handoffs, where control moves to a specialist, and agents as tools, where a manager remains responsible and invokes specialists as bounded capabilities. For cost-controlled execution, the manager pattern is the better fit.
The Planner retains ownership of the intended outcome, the approved scope, the execution state, the dependency graph, and the final validation. Workers complete bounded assignments and report back. They do not independently redefine the mission.
Without central control, every worker may reinterpret the request, duplicate context, expand scope, and burn tokens re-deciding questions the Planner already answered. Good delegation reduces reasoning duplication. Bad delegation multiplies it.
Do not split work until the contract changes
More agents do not automatically make a better system. Every delegation adds a prompt, a context, a trace, a result to validate, and a failure point.
OpenAI’s guidance is to start with one agent and add specialists only when capability isolation, policy isolation, or trace legibility materially improves. Anthropic’s cookbook reports the same caveat: each worker has a setup cost, and excessively narrow briefs can increase the total bill.
The right unit of delegation is not the smallest possible task. It is the smallest task that remains independently understandable, reliably executable, easily verifiable, safely retryable, and economically worthwhile.
Atomic does not mean microscopic.
Verify the premise, not just the steps
An agent can execute every planned step correctly while the plan itself starts from a wrong assumption.
Anthropic’s cookbook provides an unusually honest example: both the coordinator team and the solo control rigorously verified every requested fact, but the initial list used to decompose the research question contained an incorrect park. The facts were verified; the premise was not.
Before executing a large workflow, verify critical premises: the correct account, the correct target object, the requested date, the active API version, the assumed existing state, and the user’s authority to make the change.
A perfectly executed plan based on a false premise is still a failure.
The complete architecture
The flow:
Plan big → Preview → Approve → Execute small → Verify
→ Escalate exceptions → Learn → ResumeThe real optimization
The objective is not to eliminate frontier models. It is to stop using frontier intelligence for work that no longer requires frontier judgment.
Use big models where mistakes are expensive: planning, ambiguity, risk, exceptions, final synthesis. Use small models where repetition dominates: reading, calling, updating, checking, reporting. Use deterministic software wherever no model is necessary. And cache the stable context so that no model, big or small, pays for the same tokens twice.
The most efficient agent system is not the one with the shortest prompt. It is the one that routes every decision, action, and token to the least expensive component capable of handling it reliably.
Big models should design the journey. Small models should drive the miles. And big models should return only when the road disappears.