/docs/design/v1/mp_observability/blend_v3_observability.md

CacheBlend V3 Observability — Design

Status: Proposal · Scope: unify CB V3 tracing across the vLLM plugin (scheduler + worker) and the LMCache blend server into one distributed trace, plus the metrics each side exposes.

1. Goal

A single CB request touches three processes:

vLLM scheduler ──CB_UNIFIED_LOOKUP──▶ LMCache blend server
vLLM worker    ──CB_RETRIEVE_V3─────▶ LMCache blend server
vLLM worker    ── model forward (FULL_RECOMP → CHECK → PARTIAL)

Today these are observed by two disjoint systems:

side	mechanism	output
LMCache blend server	EventBus → BlendTracingSubscriber (subscribers/tracing/cb_server.py)	OTel spans (cb.request + children), OTLP export
vLLM plugin (connector/shim/attn)	_cb_span / _cb_stats_emit (lmcache_cacheblend/connector.py)	ad-hoc JSONL (CB_PROFILE=1)

They never share a trace: you cannot see, in one view, that a slow request's time went into the server-side L2 load vs the worker-side scatter vs the PARTIAL forward. Goal: one cb.request trace spanning all three processes, with sub-spans owned by whichever process did the work, plus aligned metrics.

2. The unified trace model

One trace per request_id. Process owner in brackets; cross-process children linked by trace-context propagation (§5).

cb.request                         [scheduler — root]  request_id, model, world_size, n_prompt_tokens
│
├─ cb.schedule                     [scheduler]  the get_num_new_matched_tokens defer loop
│  ├─ cb.lookup.rpc                [scheduler]  CB_UNIFIED_LOOKUP incl. N poll re-issues; attr: n_polls
│  │  └─ cb.lookup                 [SERVER]  ← cross-process child; attr prefix_chunks
│  │     ├─ cb.fingerprint_match   [server]  n_probes, table_hits, matches (token-stride=1, any offset)
│  │     │  (cb.coordinator_match instead, when a coordinator is configured)  matches, timed_out
│  │     │  (no cb.prefix_lookup span — prefix is traced by mp.lookup_prefetch)
│  │     ├─ cb.sparse_prefetch     [server]  n_keys, l1_hits, l2_misses
│  │     │  └─ cb.l2_load          [server·IO]  chunks, bytes, ms        (coalesced L2→L1)
│  │     └─ cb.classify            [server]  found, stale, per_rank_ok
│  │        ↳ on end: stamp hit_rate / prefix_coverage_tokens / n_non_prefix_tokens on cb.request
│  └─ cb.build_meta                [scheduler]  broadcast metadata to workers
│
└─ cb.execute                      [worker]
   ├─ cb.start_load_kv             [worker]  submit + stream-wait (may fire 2×: partial→full block alloc)
   │  └─ cb.retrieve               [SERVER]  ← cross-process child
   │     └─ cb.scatter             [server·GPU]  scattered_tokens, n_prefix, n_shifted (re-RoPE'd), dropped
   │        (re-RoPE folded in — interleaved per-batch, not a separate span)
   └─ cb.model_forward             [worker]  the sliced forward
      ├─ cb.full_recomp            [worker·GPU]  layers 0..cl-1
      ├─ cb.check                  [worker·GPU]  layer cl; imp_count, recomp_ratio
      └─ cb.partial                [worker·GPU]  layers cl+1..L; dispatch=flex|unified, imp_empty

This is the contract both sides implement against. The server owns cb.lookup and cb.retrieve subtrees; the plugin owns everything else.

2.1 V3 reuse is token-granular (#3582)

As of #3582, CB matches and scatters at token granularity, not vLLM-block / chunk granularity — which is what the spans/attrs below must reflect:

• Matching runs at probe_stride=1, so the shared body is found at any token offset (cb_unified_lookup no longer filters non-prefix matches to a chunk-aligned cur_st). cb.fingerprint_match reports token-offset matches, not aligned chunks.
• Scatter writes per-token via multi_layer_kv_transfer with slot_mapping = block_id[pos // bs] * bs + pos % bs. The reused token range is written slot-by-slot, so a partial vLLM block holding both matched and recomputed tokens is written correctly with no block-alignment trim on the write — the old whole-block scatter path and block-aligned drop checks are gone. cb.scatter's unit is scattered_tokens / slot_writes.
• L2 storage stays chunk-granular (256-token chunks): a non-block-aligned match still fetches whole chunks (cb.l2_load = chunks/bytes), then cb.scatter writes only the matched token sub-range. So cb.l2_load is measured in chunks while cb.scatter is measured in tokens/slots.
• cb.start_load_kv may fire twice (vLLM allocates the request's blocks partial-then-full). The first pass writes only slots inside the already-allocated block table — a slot-bound guard (cur_ed > num_slots), not a block-alignment trim; the second writes the rest. Expect two cb.retrieve children, the first with scattered_tokens < total.

3. LMCache-server side — what to expose

The server already emits cb.request / cb.lookup / cb.retrieve via the EventBus. Two changes:

(a) Finer V3 events for the lookup/retrieve subtrees. V3 currently emits only CB_LOOKUP_START/END and CB_RETRIEVE_START/END — too coarse for the subtree in §2. Add paired events (CPU-sync for compute, publish_on_stream for GPU ops so timing is GPU-accurate):

new event pair	span	timing source
CB_FINGERPRINT_MATCH_*	cb.fingerprint_match	CPU
CB_COORDINATOR_MATCH_*	cb.coordinator_match (fleet directory leg; mutually exclusive with the local matcher)	CPU + IO
(prefix lookup)	mp.lookup_prefetch (reused; prefix_chunks attr on cb.lookup)	CPU
CB_SPARSE_PREFETCH_*	cb.sparse_prefetch (+ existing L2 prefetch span as cb.l2_load)	CPU + IO
CB_SCATTER_*	cb.scatter (re-RoPE folded in via n_shifted)	publish_on_stream (GPU)

BlendTracingSubscriber.SPAN_DEFS gains the matching entries; all nest under cb.lookup / cb.retrieve via the existing SpanRegistry.

(b) Simplify the deferral logic for the V3 model. The current root-close deferral (_waiting_for_store_final, the STORE_FINAL_SUBMITTED bridge) is V2-only and inert under V3 (V3 never emits those). Under V3 the request ends at CB_RETRIEVE_END (no async store-final after inference). Gate the cb.request close on _pending_gpu_ops[sid] == 0 only, and drop the V2 store-final bridge from the V3 path. (The V2-only event handlers stay for blend_legacy.)

Span attributes (server): request_id, prefix_coverage_tokens, fingerprint_hits, storage_hits, stale_chunks, hit_tokens, requested_tokens, hit_rate, prefix_hit_tokens, non_prefix_hit_tokens, scatter_ms, scattered_tokens, slot_writes, partial_blocks, n_shifted_tokens, n_prefix_tokens (token-granular per §2.1 — not chunks).

V3 hit rate. hit_rate = hit_tokens / requested_tokens, where the numerator counts both reuse paths: hit_tokens = prefix_hit_tokens + non_prefix_hit_tokens. The two ranges are disjoint (the non-prefix complement is cur_st >= prefix coverage), so they sum without double-counting. Both components are also recorded individually on cb.request so a dashboard can split prefix-reuse vs re-RoPE'd non-prefix reuse.

Metrics (already present, keep): the lmcache_blend.* counters (lookup_requests, lookup_hit_tokens, lookup_storage_hits, lookup_stale_chunks, retrieve_requests, retrieve_failures, chunks_evicted, …). Note the V2-only store counters won't populate under V3 — document, or recompute the "stored" notion from the unified path.

4. vLLM-plugin side — what to expose

The plugin's _cb_span spans (sched.gnnmt, gnnmt.cb_unified_submit/poll, sched.build_meta, slk.*, shim.wrapper.{prepare,fwd}, flex.*, cb_admission_check) already cover the §2 plugin subtree — but as JSONL, not OTel. Make _cb_span dual-mode:

• when an OTel tracer is available → emit an OTel span (start/end, attributes);
• always (under CB_PROFILE) → keep the JSONL line (cheap local profiling).

Mapping plugin span → unified name: sched.gnnmt→cb.schedule, gnnmt.cb_unified_*→cb.lookup.rpc, sched.build_meta→cb.build_meta, slk.*→cb.start_load_kv, shim.wrapper.fwd→cb.model_forward, flex.* + the layer hooks → cb.full_recomp/cb.check/cb.partial.

Tracer source. vLLM has its own OTel (--otlp-traces-endpoint) and creates a per-request span. Prefer to reuse vLLM's tracer so CB spans nest under vLLM's request span intra-process; if vLLM tracing is off, the plugin owns a tracer pointed at the same OTLP endpoint as the LMCache server. Gate on a single CB_TRACING=1 (or reuse --enable-tracing semantics) so it's off by default.

5. Unification — linking the three processes

The blocker (from the surface map): the RPC envelope carries no trace-context — IPCCacheServerKey and CBUnifiedLookupResult have no traceparent field. Two ways to bridge:

Option A — propagate W3C trace-context through the RPC (recommended). Add an optional trace_context: str | None (W3C traceparent) to the CB RPC payloads (the lookup key + the retrieve args). The scheduler/worker inject the current span's context; the server extracts it and starts cb.lookup / cb.retrieve as remote children of it. Result: a true parent→child distributed trace across processes. Cost: one optional protocol field (backward-compatible — None when tracing off), an inject/extract at the two RPC boundaries.

Option B — deterministic trace-id from request_id (zero protocol change). Both sides derive a 128-bit trace-id = hash(request_id) and tag every span with it (+ request_id attribute). Backends group by trace-id, so the spans land in one trace — but there are no cross-process parent links (sibling spans, not nested). Use if the protocol field is undesirable short-term.

Recommendation: Option A. The field is tiny, optional, and gives real parent/child causality (e.g. "the 90 ms gnnmt was 50 ms server L2-load + 40 ms poll-wait"). Keep request_id as a span attribute regardless, so Option B is a trivial fallback. The SpanRegistry already handles intra-process nesting on each side; Option A only adds the cross-process edge.

6. Phasing

1. Plugin OTel — make _cb_span dual-mode (OTel + JSONL); reuse vLLM's tracer; gate with CB_TRACING. (plugin repo)
2. V3 server sub-spans — add the §3(a) events + SPAN_DEFS; simplify the §3(b) V3 deferral. (LMCache) — DONE: cb.fingerprint_match / cb.sparse_prefetch nest under cb.lookup (prefix lookup reuses mp.lookup_prefetch; prefix_chunks is a cb.lookup attr); cb.scatter (re-RoPE folded) nests under cb.retrieve; hit_rate = prefix + non-prefix. cb.l2_load GB/s is already covered by the existing L2ThroughputSubscriber (L2_LOAD_TASK_*), correlated by request; nesting that span under cb.sparse_prefetch is a cross-subsystem follow-up.
3. Cross-process link — add the optional trace_context RPC field; inject on the plugin side, extract on the server side. (both repos, in lockstep)
4. Dashboards — one trace view + the lmcache_blend.* / plugin latency metrics aligned on request_id.

Each phase is independently useful (1 and 2 give per-process traces; 3 unifies).

7. Open questions

• Reuse vLLM's tracer/provider, or a CB-owned one? (affects nesting under vLLM's request span vs a standalone cb.request root)
• Is the protocol field (Option A) acceptable for upstream, or start with the deterministic-trace-id fallback (Option B)?
• Sampling: per-request tracing is expensive at scale — head sampling at the scheduler (propagated via the same trace_context) so a sampled-out request is cheap on all three processes.