/design/a2a/09-durable-a2a.md

9. Durable A2A — What the Claim Means, and How We Make It Hold

Status: implemented (Direction B — the A2A client). This doc defines what "durable A2A" must mean for the claim to be defensible, and the durability mechanisms it describes are now in the code and validated by ai/src/test/.../a2a/A2ADurabilityTest.java. Property status tags: HOLDS = satisfied & tested; PARTIAL = satisfied with a documented caveat; GAP = not yet addressed. (Earlier revisions of this doc used these tags to flag the work; they now reflect shipped state.)

9.1 The claim, stated precisely

Durable A2A: once a Conductor workflow initiates an A2A interaction with a remote agent, that interaction is guaranteed to run to a terminal outcome (completed, failed, canceled, or a clean input/auth hand-off) despite crashes or restarts of the Conductor server, loss of the worker, network partitions, and transient failure or slowness of the remote agent — without losing the conversation, without hanging forever, and without duplicating irreversible agent-side actions beyond what at-least-once delivery plus an idempotency key can prevent.

Three load-bearing words: guaranteed, terminal, without losing/hanging/duplicating. Every one of them is a testable obligation, enumerated in §9.7. If any fails, we don't get to say "durable."

A claim "holds" when it is (1) precisely scoped, (2) backed by a mechanism, and (3) proven by a test that injects the failure. §9.8 is explicit about the boundary of the promise — the honest line past which no client can go — because a claim that overreaches doesn't hold, it just hasn't been caught yet.

9.2 Why this is a real differentiator, not marketing

A2A itself is a thin, mostly stateless request/response protocol over HTTP. Durability is not a protocol feature — it is a property of the implementation that orchestrates the interaction lifecycle across failures. The reference A2A hosts (the ADK/CrewAI/LangGraph samples, the purchasing-concierge demo) orchestrate from an in-memory host process: if that process crashes mid-order, the order — and the agent conversation behind it — is gone. There is no resume.

Conductor is a durable execution engine. By implementing A2A as native system tasks driven by the durable task queue and persisted execution store, the orchestration of every A2A interaction inherits crash-safety, automatic resumption, at-least-once execution, bounded retries, and full execution visibility. That is the entire story in one line:

The same purchasing-concierge demo, run on Conductor, survives a server restart mid-order. The in-memory host does not. That difference is "durable A2A."

This positioning holds in both directions of 08-conductor-implications.md:

• Direction B (client — what we built): an AGENT step is a durable unit of work that drives a remote agent to completion across failures. This doc is about Direction B.
• Direction A (server — future): when a Conductor workflow is exposed as an A2A agent, the agent's task is a durable workflow execution. Durability is native, not bolted on. Noted here only to show the positioning is coherent end-to-end.

9.3 What "durable" decomposes into

Eight properties. Each guards a specific failure mode. For each: what Conductor gives us for free, and the current status of the A2A code.

#	Property	Guards against	Inherited from Conductor	A2A status
P1	Crash-safe persistence & resumption	server restart / redeploy mid-interaction	Persistent execution store; durable decider queue; IN_PROGRESS system tasks re-evaluated on a new instance	HOLDS — resume state in task output; proven by T1
P2	Guaranteed progress to terminal (liveness)	agent down/silent forever; lost webhook	timeoutSeconds / responseTimeoutSeconds when set	HOLDS — absolute deadline + consecutive-failure cap in execute(); push backstop poll; proven by T4a/T4b/T5
P3	Effectively-once side effects	double-send after crash between send and persist	at-least-once + retry (the hazard, not the cure)	HOLDS (for cooperating agents) — deterministic, restart-stable messageId idempotency key; proven by T2/T3; boundary in §9.8
P4	At-least-once execution + bounded retry w/ backoff	transient network/5xx/429	task retry (3×, linear backoff); HTTP RetryInterceptor; 429 Retry-After	HOLDS
P5	Idempotent, replay-safe callbacks	duplicate / replayed push webhooks	— (our code)	HOLDS — IN_PROGRESS status-guard + constant-time token compare + token expiry; concurrent-callback race settled by the engine rejecting updateTask on a terminal task
P6	Durable multi-turn continuity	losing the conversation across turns	persistent workflow variables	HOLDS (contextId/taskId surfaced in output, threaded by the workflow)
P7	Observability of in-flight state	"is it stuck or working?"	task status, execution history, UI, metrics	HOLDS — state/taskId/contextId + a2aStartedAt/a2aPollFailures in output; Micrometer counters via the shared Monitors registry (a2a_client_calls, a2a_client_poll_failures, a2a_rpc_errors, a2a_ssrf_blocked, a2a_server_requests, a2a_server_resumes); MDC correlation keys (a2aWorkflowId/a2aTaskId/a2aRemoteTaskId/a2aContextId/…); structured warn logs
P8	Durable secrets at rest	auth headers persisted in plaintext	Conductor secret references / external payload storage	PARTIAL — no header logging; use ${workflow.secrets...} / Conductor secrets for auth headers rather than inline cleartext (usage guidance, not enforced)

The honest headline: every property now HOLDS except P8, which is a usage-guidance item (don't put raw credentials in task input — reference Conductor secrets). The two that were the real work — P2 (liveness) and P3 (idempotency) — are closed and tested.

9.4 The hard one: exactly-once and the dual-write problem (P3)

This is where most "durable" claims quietly fail, so it gets its own section.

message/send is not idempotent in general — it can make the agent take an irreversible action (charge a card, send an email, book a flight). The durable-execution hazard is the gap between performing the side effect and durably recording that we performed it:

AgentTask.start():
  1. build message (messageId)
  2. a2aService.sendMessage(...)        ← agent may now START IRREVERSIBLE WORK
  3. task.addOutput(taskId); IN_PROGRESS
  ── return to engine ──
  4. executionDAOFacade.updateTask()    ← FIRST durable record of step 2

If the server crashes between 2 and 4, the agent has acted but Conductor has no record. The task is still SCHEDULED in the store; the durable queue redelivers it (unack timeout); a worker runs start() again. Today step 1 generates a fresh random messageId (UUID.randomUUID() in AgentTask.buildMessage), so the re-send looks like a brand-new message → the agent does the work twice.

You cannot eliminate this window from the client side alone — it is the same impossibility as exactly-once delivery. What you can do is the industry-standard pattern (Stripe idempotency keys, Temporal deterministic ids): make the request carry a stable idempotency key so the receiver can dedupe, and make at-least-once + dedupe = effectively-once.

The key insight: a deterministic, restart-stable messageId

The messageId must be:

• identical across retries and restarts of the same logical call (so a re-send is recognized as the same message), and
• distinct per logical invocation (so a different loop iteration is a genuinely new message).

Conductor's TaskModel gives us exactly the right stable identity (verified — all fields exist):

messageId = "a2a-" + sha256(workflowInstanceId + ":" + referenceTaskName + ":" + iteration)

• Stable across retries: Conductor task retry creates a new taskId but reuses the same referenceTaskName and iteration → same key. (Basing the key on taskId would be wrong — it changes per retry.)
• Stable across restarts: all three inputs are persisted before start() runs.
• Unique per DO_WHILE iteration: iteration differs → new key, as it should.
• User-overridable: if the caller sets message.messageId, honor it.

This turns the deterministic id into a true idempotency key. Combined with at-least-once delivery, an A2A agent that dedupes on messageId gets effectively-once.

The honest contract (this is what makes the claim hold)

A2A does not mandate that agents dedupe on messageId. So the precise, defensible promise is:

Conductor guarantees a stable idempotency key and at-least-once delivery with a small, bounded duplication window. For agents that honor the key, the effect is exactly-once. For agents that do not, duplicates are minimized but not eliminated — because the side effect lives at the agent, true exactly-once is the agent's responsibility, as it must be in any distributed system.

Overstating this (claiming unconditional exactly-once) is exactly how the claim fails to hold. Stating the boundary is how it holds.

Optional hardening: recovery-by-query (capability-gated)

For agents on A2A v1.0 that support tasks/list with a contextId filter, we can close the window further: default contextId = workflowInstanceId-derived, and on a re-run of start(), query for an existing task in this context before re-sending; if found, resume it instead of re-sending. This is an enhancement (v1.0 + agent support required), not the baseline. The baseline is the deterministic messageId.

9.5 Failure-mode catalog

The matrix the claim must survive. "Today" = current code; "Target" = with §9.6 changes.

Failure	Today	Target
Crash before send	task still SCHEDULED → re-run start() → sends once. ✅	unchanged ✅
Crash after send, before persist	re-run start() → re-sends with new random id → possible double-action ❌	re-send with same deterministic messageId → agent dedupes (effectively-once) ✅
Crash during poll (IN_PROGRESS)	engine re-evaluates; execute() re-reads taskId from output, resumes polling ✅	unchanged ✅ + persisted attempt counter survives
Crash during streaming	in-memory aggregation lost; re-run re-streams from scratch ❌ (best-effort)	document as best-effort; degrade to poll on disconnect; deterministic id limits duplication ✅
Agent down while polling	execute() swallows error, returns false, polls forever (no effective timeout) ❌	bounded: max consecutive transient failures → terminal FAILED; total deadline ✅
Agent slow (valid, long)	keeps polling — correct ✅	unchanged; covered by configurable deadline, not response-timeout ✅
Push webhook never arrives (agent died / URL unreachable)	isAsyncComplete task waits forever (no poll, no default timeout) ❌	backstop poll at a slow interval and/or mandatory deadline ✅
Duplicate push callback	status-guard makes 2nd a no-op ✅; concurrent pair races on updateTask ⚠️	guard + rely on engine's terminal-state rejection; document ✅
Replayed push token	constant-time compare + 24h expiry ✅	unchanged ✅
Network partition mid-call	A2AException (retryable) → task retried; re-send hazard as above ❌→	deterministic id makes retry safe ✅
Poison agent (always 4xx)	NonRetryableException → FAILED_WITH_TERMINAL_ERROR, no retry ✅	unchanged ✅

9.6 Proposed changes (concrete)

Ordered by importance to the claim.

C1 — Deterministic messageId (closes P3). ✅ SHIPPED. AgentTask.buildMessage, when the caller hasn't supplied one, derives messageId = "a2a-" + workflowInstanceId + ":" + referenceTaskName + ":" + iteration instead of UUID.randomUUID(). (Readable concatenation rather than a hash — the value is an opaque string; debuggability wins and uniqueness/stability are what matter.) Stable across retries/restarts, unique per iteration. The single highest-value change.

C2 — Liveness guards so nothing hangs (closes P2).

• Track a consecutive-transient-failure counter in task output (e.g. a2aPollFailures). In execute(), increment on a transient poll error, reset on success; after maxPollFailures (default e.g. 10) → terminal FAILED with a clear reason instead of polling forever.
• Enforce a deadline: record the start time in output; if now - start > maxDurationSeconds (configurable; sensible default tied to the push-token TTL, e.g. 24h) → terminal FAILED ("A2A agent did not reach a terminal state within the deadline"). Do not rely on responseTimeoutSeconds — each poll's updateTask resets it, so it never fires for a polling task (verified).
• Have AgentTaskMapper default timeoutSeconds/timeoutPolicy to a finite, overridable value rather than 0 (unbounded), as a backstop independent of our own deadline logic.

C3 — Durable push: backstop poll (closes the push hole in P2). Pure push (isAsyncComplete=true, no polling) hangs forever if the webhook is lost. For the durable posture, push mode should also poll at a slow backstop interval (e.g. every few minutes) so the task still completes if the callback never arrives — the webhook just makes it faster. This means not setting isAsyncComplete, and instead returning a large getEvaluationOffset while the push config is registered. Net: ~one backstop poll per N minutes vs ~one per few seconds for pure polling — the efficiency win of push, without the liveness risk. (Keep pure-push available as an explicit opt-in for users who accept the deadline as the only backstop.)

C4 — Default contextId = workflowInstanceId. ❌ DROPPED (spec-correctness). On reflection this is spec-questionable: A2A contextId is server-generated — the agent assigns it on the first response. A client pre-assigning a contextId on a new conversation can confuse strict agents. The durability mechanism (P3) is the deterministic messageId, which needs no client-chosen contextId. We keep the spec-correct flow: the agent generates contextId, we capture it in output, the workflow threads it into the next turn. (The recovery-by-query enhancement in §9.4 remains a v1.0-only optional follow-up.)

C5 — Streaming honesty + degrade-to-poll. ✅ SHIPPED. Documented as best-effort, not durable (in-memory aggregation is lost on crash). A stream that drops after yielding a taskId degrades to tasks/get polling automatically (the aggregated non-terminal task → IN_PROGRESS → execute() polls). A stream that yields nothing is treated as transient and retried (no false COMPLETE). Durability-sensitive users should prefer poll/push.

C6 — Secrets at rest (P8). ◐ PARTIAL (guidance). No header values are logged. The remaining item is usage guidance — auth headers in task input are persisted; reference Conductor secrets / ${workflow.secrets...} rather than inline cleartext. Not mechanically enforced.

C7 — Callback idempotency + observability (P5/P7). ✅ SHIPPED. IN_PROGRESS status-guard kept; the engine rejecting updateTask on an already-terminal task settles the concurrent-callback race. Counters (a2aStartedAt, a2aPollFailures) surfaced in output; structured warn logs on transient poll failures with the failure count and bound.

9.7 Proof obligations — the claim holds only if these pass

Each maps to a property and must be an automated test that injects the failure.

Each maps to a property and an automated test that injects the failure. All green.

Test	Proves	Where	Status
T1 crash-recovery	P1	A2ADurabilityTest.t1_crashRecovery_resumesOnAFreshInstance (fresh A2AService+AgentTask resume the persisted TaskModel) and t1b_crashRecovery_survivesPersistenceRoundTrip (the durable task state is serialized to JSON — as the execution DAO stores it — and a cold TaskModel reconstructed from that JSON alone resumes to completion)	✅
T2 idempotency key	P3	t2_messageId_isStableAcrossRetries — two attempts with the same (workflowId, ref, iteration) but different taskId send an identical messageId (+ t2_callerCanOverrideMessageId)	✅
T3 distinct per iteration	P3	t3_messageId_distinctPerIteration — different iteration → different messageId	✅
T4 liveness / dead agent	P2	t4_deadAgent_failsWithinFailureCap (failure cap) + t4_deadline_failsTerminally (absolute deadline) → terminal FAILED, not infinite polling	✅
T5 push backstop	P2	t5_pushBackstop_completesWithoutWebhook — push mode, no webhook ever fires; backstop poll completes it; offset confirmed slow	✅
T6 duplicate / expired callback	P5	A2ACallbackResourceTest — 2nd push is a no-op; expired & mismatched tokens rejected	✅
T7 retry safety	P3/P4	folded into T2 (a Conductor retry is a new taskId, same identity → same messageId)	✅

Why these prove crash-recovery without an OS-level kill: the engine's AsyncSystemTaskExecutor reloads the TaskModel from the persistence store on every execution cycle (loadTaskQuietly → getTaskModel(taskId)) — it holds no in-memory state between cycles. So "a restarted worker re-drives the task" is operationally identical to "T1b reconstructs a cold TaskModel from the persisted JSON and a fresh AgentTask resumes it." T1b exercises exactly that data boundary in CI.

Full-process proof (demonstrated). The OS-level version now exists as a runnable demo — ai/src/test/resources/a2a/durable-demo/run-durable-demo.sh: it starts a real persistent (SQLite) Conductor + a remote A2A agent, places an order via a AGENT workflow, kill -9s the server mid-order, restarts it on the same store, and the order resumes and completes. Verified output: workflow status: COMPLETED — receipt: Order ORD-… confirmed. This is the genuine crash-survival proof (real process kill, real persistence, real resume). It uses the conductor.a2a.client.allow-private-network opt-in (the SSRF guard blocks loopback/private agent URLs by default; the flag is also a legitimate feature for agents on a trusted private network). A CI-automated test-harness variant (cf. AIReasoningEndToEndTest) remains a nice-to-have.

9.8 The boundary of the promise (so the claim doesn't overreach)

What durable A2A on Conductor does guarantee:

• The interaction survives Conductor crashes/restarts and resumes automatically (P1).
• It always reaches a terminal state within a bounded time — it never hangs forever (P2).
• It is retried safely with a stable idempotency key; agents that dedupe get exactly-once (P3/P4).
• The conversation and its context persist across turns and failures (P6).
• Operators can see and reason about in-flight state (P7).

What it cannot guarantee, and why that's fine:

• Unconditional exactly-once side effects at the agent. Impossible for any client when the side effect is remote and the agent doesn't dedupe — this is the at-least-once-vs-exactly-once theorem, not a Conductor limitation. We provide the idempotency key; the agent must honor it.
• Durability of an in-flight SSE stream's partial output. Streaming trades durability for latency by design; poll/push are the durable paths.
• Recovery of work an agent did but never reported and cannot be re-queried. Mitigated by the deterministic key and (on v1.0) recovery-by-query, but bounded by what the agent exposes.

Stating these is not weakness — it is precisely what lets "durable A2A" be a claim that holds under scrutiny rather than a slogan that fails on the first incident review.

9.9 One-line positioning

Durable A2A: every agent interaction is a crash-safe, automatically-resumed, idempotently-keyed unit of durable work that is guaranteed to reach a terminal outcome — because Conductor runs it, not an in-memory host loop.