/docs/design/cli/commands/describe.md

# `lmcache describe` — Design & Implementation Plan

**Status:** Proposal  |  **Date:** 2026-03-19

## Context

The CLI framework (Phase 0) is complete — `BaseCommand`, `Metrics`, `MockCommand`,
and entry point are all working. The next step (Phase 1 per
[commands.md](../commands.md)) is to implement `lmcache describe kvcache`, which
provides a rich status dashboard of a running LMCache KV cache service.

`describe engine` (Phase 2) is now implemented — see the
[`describe engine`](#lmcache-describe-engine) section below.

---

## Command UX

```bash
$ lmcache describe kvcache --url http://localhost:8000

============ LMCache KV Cache Service ============
Health:                                  OK
ZMQ endpoint:                            tcp://localhost:5555
HTTP endpoint:                           http://localhost:8000
Engine type:                             blend
Chunk size:                              256
L1 capacity (GB):                        60.00
L1 used (GB):                            42.30 (70.5%)
Eviction policy:                         LRU
Cached objects:                          1024
Active sessions:                         3
Uptime:                                  2h 14m 32s
------ Model: meta-llama/Llama-3.1-70B-Instruct ---
World size:                              4
GPU IDs:                                 0, 1, 2, 3
Num layers:                              80
Num blocks:                              2048
Cache size per token (bytes):            327680
--- Kernel group 0 (meta-llama/Llama-3.1-70B-Instruct) ---
Kernel group index:                      0
Engine group index:                      0
Object group index:                      0
Num layers:                              80
Slots per block:                         128
Dtype:                                   torch.float16
MLA:                                     False
Attention backend:         vLLM non-MLA flash attention
Engine KV shape:           NL x [2, NB, BS, NH, HS]
Engine KV tensor shape:    80 x [2, 2048, 128, 8, 128]
----------- L2: NixlStoreL2Adapter ------------
Type:                          NixlStoreL2Adapter
Health:                                  OK
Backend:                                 nixl_rdma
Stored objects:                          512
Pool used:                     480 / 512 (93.8%)
==================================================
```

JSON output collects model and L2 adapter sections into lists for easier
programmatic access:

```json
{
  "title": "LMCache KV Cache Service",
  "metrics": {
    "health": "OK",
    "url": "http://localhost:8000",
    "engine_type": "blend",
    ...
    "models": [
      {
        "model": "meta-llama/Llama-3.1-70B-Instruct",
        "world_size": 4,
        "gpu_ids": "0, 1, 2, 3",
        "num_layers": 80,
        "num_blocks": 2048,
        "cache_size_per_token": 327680
      }
    ],
    "kernel_groups": [
      {
        "model": "meta-llama/Llama-3.1-70B-Instruct",
        "kernel_group_idx": 0,
        "engine_group_idx": 0,
        "object_group_idx": 0,
        "num_layers": 80,
        "slots_per_block": 128,
        "dtype": "torch.float16",
        "is_mla": false,
        "attention_backend": "vLLM non-MLA flash attention",
        "engine_kv_shape": "NL x [2, NB, BS, NH, HS]",
        "engine_kv_concrete_shape": "80 x [2, 2048, 128, 8, 128]"
      }
    ],
    "l2_adapters": [
      {
        "type": "NixlStoreL2Adapter",
        "health": "OK",
        "backend": "nixl_rdma",
        "stored_object_count": 512,
        "pool_used": "480 / 512 (93.8%)"
      }
    ]
  }
}
```

Per-model sections are generated for each unique `(model_name, world_size)` pair
registered with the engine. The model section carries the context-wide fields —
`num_layers`, `num_blocks`, and `cache_size_per_token` — and is followed by one
**kernel group** section per kernel group, since a hybrid model's groups can
differ in geometry.

Each kernel group section includes:

- **Kernel / engine / object group index** — the group's identity:
  `kernel_group_idx` enumerates the manager's kernel groups, `engine_group_idx`
  is the paged-block address space (0 for non-hybrid), and `object_group_idx` is
  the owning object group.
- **Num layers** and **Slots per block** — the group's layer count and
  `shape_desc.bs`.
- **Dtype** and **MLA** — the group's torch dtype and MLA flag.
- **Attention backend** — which attention implementation is active (e.g.,
  `vLLM non-MLA flash attention`, `vLLM MLA`, `SGLang MHA`), derived from the
  `EngineKVFormat` enum.
- **Engine KV shape** — the symbolic tensor layout using short names matching the
  `EngineKVFormat` enum (NB=num_blocks, NL=num_layers, BS=block_size, NH=num_heads,
  HS=head_size, PBS=page_buffer_size). E.g., `NL x [2, NB, BS, NH, HS]`.
- **Engine KV tensor shape** — the same layout with actual numeric values substituted
  from the group's `shape_desc` (e.g., `80 x [2, 2048, 128, 8, 128]`), so it is
  group-accurate.

L2 adapter sections are generated for each adapter in
`storage_manager.l2_adapters`. Fields shown depend on the adapter type:

- **Type** and **Health** — always shown.
- **Backend** — Nixl adapter backend (e.g., `nixl_rdma`).
- **Base path** — filesystem adapter storage directory.
- **Stored objects** — number of cached objects (Mock, Nixl).
- **Used** — size used vs capacity with percentage (Mock).
- **Pool used** — pool slots used vs total with percentage (Nixl).

---

## `lmcache describe engine`

`describe engine` is the engine-side counterpart to `describe kvcache`. Where
`kvcache` inspects the LMCache service, `engine` inspects the **inference
engine** (vLLM) that LMCache is paired with, reading only the engine's own HTTP
surface.

```bash
$ lmcache describe engine --url http://localhost:8000

================ Inference Engine ================
Model:                  meta-llama/Llama-3.1-8B-Instruct
Max context (tokens):   131072
Status:                 OK
Running requests:       3
==================================================
```

```json
{
  "title": "Inference Engine",
  "metrics": {
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "max_context": 131072,
    "status": "OK",
    "running_requests": 3
  }
}
```

### Data sources

Unlike `kvcache` (a single `/status` call), `engine` composes three vLLM
endpoints, so no LMCache-server cooperation is required:

| Display label | Machine key | Source |
|---|---|---|
| Model | `model` | `/v1/models` → `data[0].id` |
| Max context (tokens) | `max_context` | `/v1/models` → `data[0].max_model_len` |
| Status | `status` | `/health` HTTP 200 → `"OK"` / `"UNHEALTHY"` |
| Running requests | `running_requests` | `/metrics` → sum of `vllm:num_requests_running` series |

`--url` defaults to `http://localhost:8000` (the engine default; `kvcache`
defaults to `http://localhost:8080`).

### Error handling

| Condition | Behavior |
|---|---|
| `/v1/models` unreachable / errors | Print error to stderr, exit 1 (same as `kvcache`) |
| `/health` not reachable or non-200 | `Status: UNHEALTHY` (does not fail the command) |
| `/metrics` unreachable or metric absent | `Running requests: N/A` (best-effort; metric is informational) |
| Empty model list | `Model` / `Max context` render as `N/A` |

The `/health` and `/metrics` lookups are intentionally non-fatal: an engine that
is up but has metrics disabled, or is momentarily unhealthy, still yields a
useful report rather than a hard failure. Only the primary `/v1/models` fetch
exits non-zero on failure.

---

## Design Decisions

### 1. Sub-target as positional argument

```
lmcache describe kvcache --url http://localhost:8000
lmcache describe engine  --url http://localhost:8000
```

Uses a positional `target` argument with `choices=["kvcache", "engine"]`,
matching the `describe {kvcache,engine}` pattern in
[commands.md](../commands.md). Each target resolves its own default `--url`
(`8080` for `kvcache`, `8000` for `engine`) via the `DEFAULT_URLS` map.

### 2. `--url` points to the HTTP endpoint

The original design doc example shows `--url localhost:5555` (ZMQ port), but also
states that `describe kvcache` "gathers data from ... `/status` (HTTP)". The
HTTP `/status` endpoint already exposes **all** data needed (engine type, chunk
size, L1 memory, eviction policy, cached objects, health, sessions, etc.). Using
HTTP as the sole data source keeps the CLI simple — no ZMQ client needed.

`--url` accepts the HTTP base URL (e.g., `http://localhost:8000`). The command
normalizes it (adds `http://` if missing) and appends `/status`.

### 3. Output fields mapped from `/status`

| Display label | Machine key | Source in `/status` response |
|---|---|---|
| Health | `health` | `is_healthy` → `"OK"` / `"UNHEALTHY"` |
| ZMQ endpoint | `zmq_endpoint` | `zmq_endpoint` **(new — see Server-Side Changes)** |
| HTTP endpoint | `http_endpoint` | `http_endpoint` **(new — see Server-Side Changes)** |
| Engine type | `engine_type` | `engine_type` |
| Chunk size | `chunk_size` | `chunk_size` |
| L1 capacity (GB) | `l1_capacity_gb` | `storage_manager.l1_manager.memory_total_bytes` / 1024^3 |
| L1 used (GB) | `l1_used_gb` | `storage_manager.l1_manager.memory_used_bytes` / 1024^3, with `memory_usage_ratio` × 100 for % |
| Eviction policy | `eviction_policy` | `storage_manager.eviction_controller.eviction_policy` |
| Cached objects | `cached_objects` | `storage_manager.l1_manager.total_object_count` |
| Active sessions | `active_sessions` | `active_sessions` |
| Uptime | `uptime` | `uptime_seconds` **(new — see Server-Side Changes)**, formatted as `Xh Ym Zs` |

### 4. HTTP client: stdlib `urllib`

No new dependencies. Uses `urllib.request` following the same pattern as the
existing `lmcache/tools/mp_status_viewer/__main__.py`.

### 5. Error handling

| Condition | Behavior |
|---|---|
| Connection refused / timeout | Print error to stderr, exit 1 |
| HTTP 503 (engine not initialized) | Print "Server unhealthy: engine not initialized", exit 1 |
| Missing fields in response | Display as `N/A` (Metrics default for `None` values) |

---

## Server-Side Changes

Three fields in the design doc's `describe kvcache` output are **not currently
available** from `/status`. The following changes surface them.

### 1. Add `start_time` to `MPCacheServer` → expose `uptime_seconds`

**File:** `lmcache/v1/multiprocess/server.py`

`MPCacheServer.__init__()` (line 147) records `self._start_time = time.monotonic()`
at construction. `report_status()` (line 696) includes a new field:

```python
"uptime_seconds": time.monotonic() - self._start_time,
```

The CLI formats this as a human-readable string (e.g., `2h 14m 32s`).

### 2. Pass endpoint addresses into `MPCacheServer` → expose in status

**File:** `lmcache/v1/multiprocess/server.py`

Currently `MPCacheServer` does not know the ZMQ or HTTP addresses — those live in
`MPServerConfig` and `HTTPFrontendConfig`, which are only available in
`run_cache_server()` / `run_http_server()`.

**Option A — engine constructor params:** Add optional `zmq_endpoint: str | None`
and `http_endpoint: str | None` kwargs to `MPCacheServer.__init__()`. Callers
(`run_cache_server` at line 787, and the blend variant) pass these when available.
`report_status()` includes them.

**Option B — set after construction:** Add setter methods or attrs that
`run_cache_server()` / `run_http_server()` set after creating the engine, before
returning it. This avoids changing the constructor signature.

**Recommendation:** Option A is simpler and more explicit.

```python
# In run_cache_server() (line 787):
engine = MPCacheServer(
    storage_manager_config=storage_manager_config,
    chunk_size=mp_config.chunk_size,
    hash_algorithm=mp_config.hash_algorithm,
    zmq_endpoint=f"tcp://{mp_config.host}:{mp_config.port}",
)

# In run_http_server() lifespan (line 77):
# After engine is created, set http_endpoint:
engine.http_endpoint = f"http://{http_config.http_host}:{http_config.http_port}"
```

Note: The ZMQ endpoint is known at `run_cache_server()` time, but the HTTP
endpoint is only known in `run_http_server()`. Since `run_http_server()` calls
`run_cache_server(return_engine=True)` and gets back the engine, it can set
`http_endpoint` after construction. So a hybrid approach works:
- `zmq_endpoint` passed via constructor (always available)
- `http_endpoint` set as an attribute after construction (only when HTTP frontend
  is enabled)

`report_status()` returns both:

```python
"zmq_endpoint": self.zmq_endpoint,
"http_endpoint": getattr(self, "http_endpoint", None),
```

### 3. Same changes for `BlendCacheEngine`

**File:** `lmcache/v1/multiprocess/blend_server.py` (and `blend_server_v2.py`)

Mirror the same `start_time`, `zmq_endpoint`, and `http_endpoint` additions if
`BlendCacheEngine` has its own `report_status()`. If it delegates to
`MPCacheServer`, no separate change is needed.

### Summary of server-side changes

| Field | Where | Change |
|---|---|---|
| `uptime_seconds` | `MPCacheServer.__init__` + `report_status()` | Record `time.monotonic()` at init, compute delta in status |
| `zmq_endpoint` | `MPCacheServer.__init__` + `run_cache_server()` | New constructor kwarg, passed from `MPServerConfig` |
| `http_endpoint` | `run_http_server()` lifespan + `report_status()` | Set on engine after construction when HTTP is enabled |

### 4. Expose engine KV format, shape, and attention backend in `kv_cache_layout`

**Files:** `lmcache/v1/gpu_connector/utils.py`, `lmcache/v1/platform/cuda/cache_context.py`, `lmcache/v1/multiprocess/server.py`

Helper functions in `utils.py` (derived from `legible_print_engine_kv_format()`):
- `get_engine_kv_shape_description(engine_kv_format)` — symbolic shape (e.g., `NL x [2, NB, BS, NH, HS]`)
- `get_attention_backend(engine_kv_format)` — backend name (e.g., `vLLM non-MLA flash attention`)
- `get_concrete_engine_kv_shape(kv_caches, engine_kv_format)` — whole-context shape with actual values
- `get_concrete_engine_kv_shape_from_shape_desc(shape_desc, engine_kv_format)` — **group-accurate** shape with actual values, read from a single kernel group's `PageBufferShapeDesc` (used by `report_status`)

`report_status()` is organised **per kernel group**: a small set of context-wide
fields at the top level, plus a `kernel_groups` list where each entry is
self-describing. The format-derived fields (`engine_kv_format`, `engine_kv_shape`,
`attention_backend`, `is_mla`) and the group-accurate `engine_kv_concrete_shape`
live inside each group:

```python
"kv_cache_layout": {
    "num_layers": 80,
    "num_blocks": 2048,
    "cache_size_per_token": 327680,
    "kernel_groups": [
        {
            "kernel_group_idx": 0,
            "engine_group_idx": 0,
            "object_group_idx": 0,
            "num_layers": 80,
            "layer_indices": [0, 1, ...],
            "tokens_per_block": 128,
            "slots_per_block": 128,
            "dtype": "torch.float16",
            "engine_kv_concrete_shape": "80 x [2, 2048, 128, 8, 128]",
            "is_mla": false,
            "engine_kv_format": "NL_X_TWO_NB_BS_NH_HS",
            "engine_kv_shape": "NL x [2, NB, BS, NH, HS]",
            "attention_backend": "vLLM non-MLA flash attention",
        },
    ],
}
```

---

## CLI Implementation

### New file: `lmcache/cli/commands/describe.py`

```python
class DescribeCommand(BaseCommand):
    name() → "describe"
    help() → "Show detailed status of a running LMCache service."

    add_arguments(parser):
        parser.add_argument("target", choices=["kvcache", "engine"],
                            help="What to describe.")
        parser.add_argument("--url", default=None,
                            help="Server URL (per-target default applied)")

    execute(args):
        if args.url is None:
            args.url = DEFAULT_URLS[args.target]
        if args.target == "kvcache":
            self._describe_kvcache(args)
        elif args.target == "engine":
            self._describe_engine(args)

    _describe_kvcache(args):
        1. Normalize URL (ensure http:// prefix)
        2. Fetch JSON from {url}/status (timeout=10s)
        3. On error: print to stderr, sys.exit(1)
        4. Extract fields from nested response dict
        5. Format uptime_seconds → "Xh Ym Zs"
        6. Format L1 used bytes → "XX.XX (YY.Y%)"
        7. Build flat Metrics via self.create_metrics() (width=48)
        8. metrics.emit()
```

Module-level helpers:

```python
def _fetch_json(url: str, timeout: int = 10) -> dict:
    """GET *url*, return parsed JSON. Raises on HTTP/network errors."""

def _normalize_url(url: str) -> str:
    """Ensure URL has http:// scheme, strip trailing slash."""

def _fmt_uptime(seconds: float) -> str:
    """Format seconds as 'Xh Ym Zs'."""

def _fmt_used_gb(used_bytes: int, ratio: float) -> str:
    """Format as 'XX.XX (YY.Y%)'."""
```

### Modify: `lmcache/cli/commands/__init__.py`

Add import and registry entry:

```python
from lmcache.cli.commands.describe import DescribeCommand

ALL_COMMANDS: list[BaseCommand] = [
    MockCommand(),
    DescribeCommand(),
]
```

### Patterns to follow

- **Reuse `BaseCommand.create_metrics()`** — auto-handles `--format` and `--output`
  flags (see `base.py`).
- **Flat metrics, no sections** — all fields at top level, matching the design doc
  output style. Use `metrics.add(key, label, value)` directly.
- **Width = 48** — matches the divider width in `commands.md` examples.

---

## Verification

1. **Unit test:** Test `_normalize_url()`, `_fmt_uptime()`, `_fmt_used_gb()`, and
   field extraction logic with a synthetic `/status` response dict (no live
   server needed).
2. **Manual test against running server:**
   ```bash
   lmcache describe kvcache --url http://localhost:8000
   lmcache describe kvcache --url http://localhost:8000 --format json
   lmcache describe kvcache --url localhost:8000          # auto-prefix http://
   lmcache describe kvcache --url http://localhost:8000 --output status.json
   lmcache describe kvcache --url http://localhost:9999   # connection refused → exit 1
   ```
3. **JSON output:** Verify machine keys are snake_case and values are raw types
   (not display-formatted strings), except `l1_used_gb` and `uptime` which include
   human-readable formatting.