/ods/extensions/services/llama-server

llama-server

Core LLM inference engine for ODS

Overview

llama-server is the local LLM inference backend, powered by llama.cpp. It loads GGUF-format models and exposes an OpenAI-compatible HTTP API on port 8080. GPU acceleration is provided via CUDA (NVIDIA) or ROCm (AMD); CPU fallback is available for systems without a supported GPU.

All other services that perform AI inference — Open WebUI, LiteLLM, Privacy Shield, and the dashboard chat endpoint — connect to llama-server internally.

Features

• OpenAI-compatible API: Drop-in replacement for the OpenAI Chat Completions and Completions endpoints
• GGUF model support: Load any GGUF-quantized model from data/models/
• GPU acceleration: CUDA (NVIDIA) and ROCm/HIP (AMD) backends
• Configurable context window: Token limit tunable via CTX_SIZE
• Prometheus metrics: /metrics endpoint for throughput and token stats
• Multi-GPU offload: All GPU layers offloaded with --n-gpu-layers 999
• Hardware-tier model selection: Installer auto-selects model size based on detected VRAM

Configuration

Environment variables (set in .env):

Variable	Default	Description
GGUF_FILE	Qwen3.5-9B-Q4_K_M.gguf	Model filename inside data/models/
CTX_SIZE	16384	Context window size in tokens
OLLAMA_PORT	8080	External port (maps to internal 8080)
GPU_BACKEND	nvidia	GPU backend: nvidia or amd
LLAMA_ARG_FLASH_ATTN	auto	llama.cpp Flash Attention mode: auto, on, or off
LLAMA_ARG_CACHE_TYPE_K	f16	KV cache key precision. Use q8_0 to reduce long-context memory pressure
LLAMA_ARG_CACHE_TYPE_V	f16	KV cache value precision. Use q8_0 to reduce long-context memory pressure
LLAMA_ARG_N_CPU_MOE	unset	Optional MoE-only CPU expert offload (--n-cpu-moe). Leave unset for dense models
LLAMA_ARG_SPEC_TYPE	unset	Optional speculative decoding mode (--spec-type). Use only with supported GGUF/runtime combinations
LLAMA_ARG_SPEC_DRAFT_N_MAX	unset	Optional speculative draft token cap (--spec-draft-n-max)
LLAMA_SERVER_MEMORY_LIMIT	64G	Docker memory limit for the container

Long-context profile

For larger context windows on memory-constrained GPUs, keep the model unchanged and tune the attention/KV cache first:

CTX_SIZE=32768
LLAMA_ARG_FLASH_ATTN=on
LLAMA_ARG_CACHE_TYPE_K=q8_0
LLAMA_ARG_CACHE_TYPE_V=q8_0

This is opt-in. The defaults remain auto Flash Attention and f16 KV cache to preserve existing behavior.

MoE expert offload

For Mixture-of-Experts GGUF models, llama.cpp can keep the first N MoE expert layers on CPU/RAM:

LLAMA_ARG_N_CPU_MOE=25

Tune this value per machine. Lower values keep more work on GPU and can be faster if enough VRAM is available; higher values reduce VRAM pressure. Leave this unset for dense models.

MTP speculative decoding

Newer llama.cpp builds support MTP speculative decoding for GGUFs that include compatible MTP data. ODS exposes the flags but does not enable them automatically, because normal GGUFs and older llama.cpp builds will reject or ignore these settings.

LLAMA_ARG_SPEC_TYPE=draft-mtp
LLAMA_ARG_SPEC_DRAFT_N_MAX=3

Use this only with a llama.cpp image or native binary built after MTP support landed, and with a model family that explicitly publishes MTP-capable GGUFs. Normal GGUFs without MTP layers should leave these variables unset.

In router or multi-model setups, do not put MTP settings in a shared default section when any routed model lacks MTP layers. Apply spec-type = draft-mtp and spec-draft-n-max = 3 only to the MTP-capable model section so non-MTP models keep loading normally.

AMD-specific variables

Variable	Default	Description
VIDEO_GID	44	GID of the video group (getent group video | cut -d: -f3)
RENDER_GID	992	GID of the render group (getent group render | cut -d: -f3)

API Endpoints

llama-server exposes an OpenAI-compatible REST API:

Method	Path	Description
GET	/health	Health check
GET	/metrics	Prometheus inference metrics
POST	/v1/chat/completions	Chat completions (OpenAI format)
POST	/v1/completions	Text completions
GET	/v1/models	List loaded models

Example

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Architecture

┌─────────────────────────────────────────────────────────┐
│  docker-compose.base.yml  (GPU-agnostic command + ports) │
│        +                                                  │
│  docker-compose.nvidia.yml  OR  docker-compose.amd.yml   │
│        (image + GPU device passthrough)                   │
└──────────────────────────┬──────────────────────────────┘
                           │
                    ┌──────▼──────────┐
                    │  llama-server   │
                    │  (llama.cpp)    │
                    │  :8080 (int)    │
                    │  :8080 (ext)    │
                    └──────┬──────────┘
                           │  OpenAI-compatible API
          ┌────────────────┼──────────────────┐
          │                │                  │
    ┌─────▼─────┐   ┌──────▼───────┐  ┌──────▼──────┐
    │ Open WebUI│   │   LiteLLM    │  │Privacy Shield│
    └───────────┘   └──────────────┘  └─────────────┘

Files

• manifest.yaml — Service metadata and feature definitions

Troubleshooting

Container not starting:

docker compose ps llama-server
docker compose logs llama-server

Model not found:

• Confirm the GGUF file exists: ls ods/data/models/
• Check GGUF_FILE in .env matches the filename exactly

Out of VRAM:

• Reduce CTX_SIZE in .env (try 8192 or 4096)
• Use a smaller quantized model (Q4 instead of Q8)

AMD GPU not detected:

• Verify group IDs: getent group video | cut -d: -f3 and getent group render | cut -d: -f3
• Update VIDEO_GID and RENDER_GID in .env
• Confirm /dev/kfd and /dev/dri exist on the host

Check inference metrics:

curl http://localhost:8080/metrics

License

Part of ODS — Local AI Infrastructure