/docs/design/v1/distributed/l2_adapters/dax.md

DAX L2 Adapter Design

This document describes the built-in dax L2 adapter for LMCache multiprocess mode and how it shares implementation with the non-MP DAX storage backend.

Goals

• Reuse one synchronous DAX core for MP and non-MP DAX storage.
• Keep the MP controller flow unchanged: adapters still use the normal submit, event-fd, and query-result contract.
• Keep the adapter facade stable while the runtime DAX device pool changes.
• Keep DAX volatile-only. Keys are indexed in process memory and are not recovered from device bytes after restart.

Components

lmcache/v1/storage_backend/dax/core.py defines DaxCore[KeyT]. The core owns the mapped DAX arena, fixed-size slot allocation, in-memory index, LRU order, in-flight writes, external lock refcounts, active read borrow counts, close coordination, and direct ctypes.memmove copies.

lmcache/v1/storage_backend/plugins/dax_backend.py is the non-MP wrapper. It keeps existing non-MP behavior such as the local CPU backend requirement, TP=1 validation, optional async put, and the staging-slab batched restore path.

lmcache/v1/distributed/l2_adapters/dax_l2_adapter.py is the MP adapter. It self-registers adapter type dax, owns separate event notifiers and worker pools for store, lookup, and load operations, and uses one or more DaxCore[ObjectKey] instances behind a stable facade.

lmcache/v1/multiprocess/http_apis/reconfigure_api.py exposes runtime reconfiguration endpoints:

• GET /reconfigure/dax/status
• POST /reconfigure/dax/add
• POST /reconfigure/dax/remove
• POST /reconfigure/dax/resize

The HTTP layer routes backend, operation, and adapter-specific JSON payloads into the generic L2 adapter reconfiguration API on StorageManager. StorageManager only routes operation plus payload to a reconfigurable adapter; DAX path, mode, and migration semantics stay inside DaxL2Adapter. The same interface is intended for future adapters such as P2P, so the HTTP layer does not inspect private adapter lists or DAX core state directly.

Slot State

Each committed key points to one fixed-size slot. A slot is reusable only when:

• The key has been removed from the index.
• No external lock is held for the key.
• No store is in flight for the key.
• No active read has borrowed the slot.

Delete operations remove unlocked keys from the index immediately. If a read has borrowed the slot, the slot is marked pending-free and recycled when the borrow count reaches zero.

MP Flow

Store:

1. StoreController calls submit_store_task(keys, objects).
2. The adapter chooses an active DAX device. Existing keys prefer their current mapped device; new keys use the active device with the lowest slot usage.
3. A store worker copies each object into a DAX slot through DaxCore.put_many.
4. The adapter records task-level success as all(per_key_results).
5. The store event fd is signaled and store listeners are notified for the keys that were actually accepted by the core.

Lookup and load:

1. PrefetchController calls submit_lookup_and_lock_task(keys).
2. The adapter checks key -> device mappings first, then scans readable devices if needed.
3. The adapter calls DaxCore.exists_many(keys, lock=True) and returns a full bitmap, including holes.
4. Load workers call DaxCore.load_many_into(keys, objects) on the device that currently owns each key.
5. submit_unlock(keys) releases the external lock refcounts on every DAX core. This is deliberate because migration can update key -> device mappings between lookup and unlock.

Runtime Hotplug

The DAX facade keeps the event fds and worker pools stable. Runtime hotplug only mutates the device pool behind the facade, so StoreController, PrefetchController, and the vLLM MP connector do not need ZMQ protocol changes or poll-set re-registration.

Add:

1. Validate hotplug_enabled, path, and size.
2. Map a new DaxCore[ObjectKey].
3. Append a DaxDeviceEntry(state="active").
4. Return per-device status. Existing KV entries stay on their current devices.

Remove with migration:

1. Mark the source device draining so new stores do not choose it.
2. Reject the operation if externally locked or borrowed slots would be deleted.
3. Snapshot source keys and reserve source reads.
4. Copy each reserved payload from the source DAX pointer into another active DAX core with put_reserved_from_ptr.
5. Update key -> device mappings, delete the source entries, then close the source core.

Resize:

• Grow remaps the same core to a larger size after active reads and writes drain. No KV payload movement is needed.
• Shrink first proves that every live slot fits below the new slot count. If not, the out-of-range keys must migrate to another active device or the request fails. Shrink never silently evicts data.

Restart Behavior

The adapter stores keys and metadata only in memory. Closing the adapter and opening a new adapter against the same DAX device starts with an empty index. Old bytes may remain on the device, but they are unreachable because PR1 does not define any on-device metadata, scan, checkpoint, or recovery format.

Capacity And Eviction

Usage is slot-based, not payload-byte-based. get_usage() reports occupied slot capacity because the DAX arena is exhausted by slot count. The eviction controller calls delete(keys), which skips externally locked keys and reclaims slots after active read borrows drain.

Runtime capacity is the sum of active, draining, migrating, resizing, and removing device capacities. Closed, removed, and failed devices are excluded.

Current Limits

• Runtime hotplug does not perform kernel-level CXL or DAX reconfiguration.
• No per-TP partitioning.
• No restart recovery.
• Only single-buffer objects are supported.