Long-running AI tasks have a failure mode that is both common and difficult to notice: the agent simply stops.
It may time out. Its process may crash. A tool call may never return. The execution environment may terminate before the agent writes its latest status.
From the user’s perspective, none of those distinctions matter. They asked the system to complete a task. Then the system went quiet.
This is the problem Joxy exists to solve: complex, repetitive work on real tools, the kind most agents cannot survive. Our running example is Kat, our Meta Ads agent, executing a full campaign build: campaigns, a dozen ad sets, dozens of ads, creatives uploaded and attached, and every object read back and verified. That is 80-plus steps against an API that rate-limits, times out, and rejects, over a run that takes minutes to hours. Most agents attempt this as one long chain in one context: a single dropped call, a single crash, or a single context overflow anywhere in the sequence, and the whole build dies with nothing to show for it. This failure mode bit us repeatedly until we built supervision into the architecture itself.
A reliable agent platform needs more than an agent capable of doing the work. It needs another component whose job is to make sure the work does not quietly die, and does not quietly claim victory either. That component is the Watcher.
Long tasks need supervision
Many agent workflows cannot be completed in a single turn. A campaign build means creating dozens of dependent objects in strict order, uploading assets, waiting on external APIs, reading every object back to verify it, and retrying temporary failures along the way.
During that time, the user cannot tell the difference between an agent that is working and an agent that has died. Both appear silent.
This creates two separate reliability problems. The first is an execution problem: is the agent still alive? The second is a communication problem: does the user know the agent is still working?
The Watcher handles both. It monitors task state, detects abandoned work, verifies completion, restarts failed turns when appropriate, and keeps the user informed while execution continues.
Status must live outside the agent
A Worker should never be the only source of truth about whether it is alive. If the Worker crashes, its internal state disappears with it.
Instead, every long-running task should have an external task record: task ID, status, current turn, last heartbeat time, last completed step, retry count, start time, latest user-update time, exit criteria, and final result or failure information.
The Worker updates this record as it progresses:
status: in_progress
last_heartbeat: 10:42:00
current_step: creating ad set 9 of 12
retry_count: 0
The Watcher checks this record on a regular schedule.
A missing response does not always mean failure
Silence alone is not enough to declare an agent dead. A long tool call may still be running. A model may still be generating.
The Watcher therefore evaluates the task state, not just the absence of a response:
Has the task completed?
Yes → Verify the result, then stop watching.
No final response exists.
Is the task still marked in progress?
Yes → Check the latest heartbeat.
No → Treat the turn as abandoned.
Is the heartbeat recent?
Yes → The agent is working.
No → Treat the agent as potentially dead.
Both signals matter. A task marked in_progress forever is not proof that anything is happening. Without a heartbeat, “in progress” can become a polite way of saying “lost.”
The heartbeat proves the Worker is alive
While performing a long task, the Worker periodically writes a heartbeat. It does not need a full explanation, only proof that execution is still active:
{
"task_id": "campaign_build_1842",
"status": "in_progress",
"last_heartbeat": "2026-07-14T10:42:00Z",
"current_step": "Attaching creatives to ads",
"completed_units": 47,
"total_units": 82
}
The heartbeat interval should be shorter than the Watcher’s failure threshold. For example: heartbeat every 30 seconds, Watcher check every 60 seconds, agent considered stale after 120 seconds without a heartbeat. The exact timings can vary by workload; the architecture matters more than the numbers.
The Watcher should never restart a turn merely because the user has not yet received a response. It restarts only when the execution state shows the task is no longer progressing.
A final response does not always mean success
Dying silently is only half the problem. The other half is worse: the agent that confidently declares the task complete when it isn’t.
Agents do this in predictable ways. They hit a context limit and wrap up early. They summarize a plan as if it were executed. They create 9 of 12 ad sets, lose count, and report the structure as complete. A heartbeat cannot catch any of this. The agent was alive the whole time. It just wasn’t right.
The defense is the same principle as the heartbeat, applied at the finish line: never let the Worker be the only source of truth about its own success. Every task should define verifiable exit criteria at creation time, stored in the task record: files that must exist, records that must be written, tests that must pass, counts that must match.
"exit_criteria": [
{ "check": "ad_sets_exist", "expected": 12 },
{ "check": "ads_attached", "expected": 36 },
{ "check": "all_objects_verified", "expected": true }
]
When the Worker reports completion, the task does not move to completed. It moves to verifying. The Watcher (or a dedicated verifier) checks the completion claim against the exit criteria. Only evidence promotes the task to completed. A failed verification is treated exactly like a dead Worker: the turn is replayed from the last verified step, with the verification failure included in the retry context so the replacement Worker knows what was actually missing.
Like the timing thresholds, the criteria should be deterministic checks in software, not a model’s opinion about its own output. An agent grading its own homework is how false success happens in the first place.
When the agent is working, do nothing
If the task is in_progress and the heartbeat is recent, the Watcher does not interfere. No restart, no second Worker, no duplicate task.
This matters because an overactive Watcher creates more problems than it solves: duplicate tool calls, conflicting writes, multiple user responses, corrupted execution state, and wasted model costs.
When the agent dies, replay the turn
When the Watcher determines a Worker has died, it restarts the failed turn, not the entire user request.
The retry receives what it needs to resume safely: the original request, the execution plan, the last successful step, saved tool outputs, the retry number, idempotency information, and the exit criteria to verify against.
Instead of “start the entire task again,” the Watcher says:
Resume from the last verified step. Do not repeat completed writes.
This distinction is essential for workflows with external actions. If the previous agent created an ad set but died before recording the returned ID, blindly replaying the write creates a duplicate ad set in the live account. The replacement Worker should verify whether the previous action completed before retrying.
Inspect state → Verify previous action → Resume or retry → Save result
Assume failure → Repeat everything
Use a bounded model-retry policy
A Watcher should not restart failed work forever. Infinite retries create infinite cost and hide genuine system failures.
For workloads using models such as Claude Opus and Gemini, the Watcher can apply a bounded policy. For example, alternating up to two attempts per model:
Initial Worker fails
→ Opus retry 1
→ Gemini retry 1
→ Opus retry 2
→ Gemini retry 2
→ User-facing failure response
The exact order matters less than making the policy explicit and bounded. Each attempt is recorded in the task state:
{
"retry_count": 3,
"retry_history": [
{ "model": "opus", "result": "timeout" },
{ "model": "gemini", "result": "tool_error" },
{ "model": "opus", "result": "failed_verification" }
]
}
A retry is a controlled recovery operation, not a fresh conversation.
Keep the user informed every two minutes
Execution reliability alone is not enough. Even when the Worker is healthy, extended silence makes the system appear broken.
While a long task remains active, the Watcher sends the user a brief progress update approximately every two minutes:
Still on it. 47 of 82 steps are complete: the campaigns and ad sets
exist, and creatives are being attached now.
Or, when exact progress is unavailable:
The task is still running normally. I'll share the result when it's complete.
These updates come from the external task state, not from guesses. The Watcher reports what it knows: current step, units completed, whether a retry occurred, whether anything needs user action. It should not invent percentages or claim the task is “almost done” without evidence.
Separate execution updates from recovery updates
Users should not receive a technical incident report every time an internal retry occurs, but meaningful delays should be communicated honestly.
Normal updateThe task is still running, and the agent is continuing through
the remaining steps.
Recovery update
The original execution stopped unexpectedly, so the system resumed
from the last completed step. Your task is still in progress.
Reassure the user without hiding that recovery occurred. They don’t need stack traces or model-routing details. They do need to know the system has not forgotten their request.
The Watcher is a state machine, not a chatbot
The Watcher should be mostly deterministic. Its responsibilities require clear state transitions, not open-ended reasoning:
queued · running · waiting_on_tool · verifying · retrying · completed · failed · cancelled
Its logic reduces to:
IF status = completed or cancelled
stop watching
IF status = running AND heartbeat is recent
send progress update if two minutes have passed
continue watching
IF status = verifying
run exit-criteria checks
pass → mark completed
fail → treat as a failed turn
IF heartbeat is stale
mark current attempt as failed
verify previous writes
trigger the next retry
IF retry limit is exceeded
mark task as failed
send a user-facing response
The model may perform the recovered turn. But the decision to retry, verify, wait, notify, or stop should be governed by explicit software rules. Deterministic supervision is cheaper and more predictable than asking another language model whether the first language model looks dead, or whether it is telling the truth about being done.
Prevent multiple Watchers from restarting the same task
In a distributed system, two Watcher processes may inspect the same stale task at the same time. Without protection, both restart it.
Before recovering a task, a Watcher should atomically claim it:
recovery_owner: watcher_12
recovery_lease_expires_at: 10:47:00
Only the lease holder may trigger a retry. The same applies to user updates: the task record should store the timestamp of the last update sent, so two Watchers don’t both message the user at once.
Nothing says “our system is under control” like four identical reassurance messages.
Know when to stop
After the retry policy is exhausted, the Watcher stops restarting and produces a clear response: what could not be completed, what was completed and verified, whether any external changes were made, and what the user can do next.
I wasn't able to complete the task after several recovery attempts.
The first 46 steps completed and verified: the campaigns and all 12
ad sets exist. Execution repeatedly stopped while uploading the third
creative. No duplicate
writes were created, and the completed results have been preserved.
The worst outcome is not failure. The worst outcome is silent, ambiguous failure, or worse, a confident “done” that isn’t. A controlled failure response lets the user, or another system, decide what happens next.
The complete architecture
A dependable long-task system needs five components:
The operating flow:
Create task → Define exit criteria → Start Worker → Write heartbeats
→ Watch state → Send two-minute updates → Detect stale execution
→ Verify previous work → Retry the failed turn
→ Verify completion → Complete or exhaust retries → Respond to the user
The real purpose of the Watcher
The Watcher is not there to make the Worker smarter. It is there to make the overall system dependable.
A capable model can still crash. A correct tool call can still time out before its result is recorded. A confident agent can still be wrong about being finished. Reliability cannot depend on the intelligence of the active agent. It must come from the architecture around it.
The Worker does the job. The heartbeat proves it is alive. The Watcher notices when it is not. The exit criteria prove the work is real. The retry policy brings the task back. And the progress updates make sure the user is never left wondering whether anyone is still on the job.