/adr/0013-reuse-shared-in-memory-test-databases.md

ADR-0013: Reuse one in-memory test database per version instead of one container per test

• Status: Accepted
• Date: 2026-06-08
• Deciders: Backend team
• Tags: backend, testing, ci

Context

Our database tests run the same checks against many versions of each engine (PostgreSQL, MySQL, MariaDB, MongoDB). We tried two setups before this one and both broke.

Before — one big docker-compose. We started every version of every engine up front and kept them all running. That is ~50 containers alive at the same time and a 16 GB CI machine ran out of memory:

docker-compose up   (everything, all the time)
  postgres:12 .. postgres:18
  mysql:5.7 .. mysql:8.4
  mariadb:10.6 .. mariadb:12.0
  mongo:4.0 .. mongo:8.0
  ──────────────────────────────
  ~50 containers alive at once  →  out of RAM

Naive fix — a fresh container per test. Every single test booted its own database and killed it afterwards. A database cold-starts in 20–60s and one package has 40+ tests, so the suite hit the 15-minute timeout:

test 1 → boot DB → run → kill DB
test 2 → boot DB → run → kill DB
...        (40+ times per package)        →  too slow

We want both: fast tests and bounded memory, without tests stepping on each other.

Decision

For each database version we boot one container, run all the test functions against it, then shut it down before the next version:

boot postgres:16   (once)
  ├─ run test 1
  ├─ run test 2
  └─ run test 3
shut down postgres:16
→ boot postgres:17 ...

Two more things make this fast and safe: the database files live in RAM (so each boot and restore is quick) and the test packages run in parallel (so we get the speed back) while each one stays isolated (so peak memory stays bounded).

• One container per version, reused across the version's tests. Each matrix package declares its version list once and loops it with an outer t.Run. Inside that subtest it boots the server with StartPostgres / StartMysql / … (containers/{postgres,mysql,mariadb,mongodb}.go), then runs the former test functions as inner subtests. StartXxx registers t.Cleanup, which Go runs when the version subtest returns — so the container is torn down before the next version boots and only one matrix container is alive per package at a time. Each go test package is its own process, so its containers belong only to it. The one rule for test authors: when you create a fixed-name object (a table or user without a random suffix), start with DROP ... IF EXISTS — within a version the server is reused and the subtests run one after another.

• Database files in RAM. Each engine's data dir is mounted on tmpfs rw,size=512m (size is pinned — otherwise Docker reserves half the host RAM per container). Crash-safety is off because we throw these servers away: Postgres fsync=off / full_page_writes=off / synchronous_commit=off; MySQL & MariaDB innodb-flush-log-at-trx-commit=0 / innodb-doublewrite=0 / sync-binlog=0 / skip-log-bin. Files: containers/{postgres,mysql,mariadb,mongodb}.go.

• Run packages in parallel. go test -p=8 (TEST_PARALLEL_WORKERS in backend/Makefile). One matrix container is alive per package, so peak memory stays around 10–11 GB instead of the old all-at-once that ran out of RAM.

How parallel workers stay isolated

Eight packages run at the same time and share the same Postgres, Valkey and backup infrastructure, so each one grabs its own worker slot using a Postgres advisory lock (config.go → applyTestWorkerSlot / claimTestWorkerSlot). The slot gives it a private copy of everything shared:

• its own metadata database, named …_w{slot};
• its own Valkey/Redis database (the slot number);
• its own cache prefix "w{slot}:", which also tags the backup-node registry (backups/backups/backuping/nodes/registry.go) — every Redis key and pub/sub channel.

So two packages running side by side never touch each other's databases, cache entries, backup nodes or pub/sub channels.

Alternatives considered

• docker-compose, all versions up at once. Rejected: ~50 containers exhaust RAM on 16 GB CI and the packages share one set of databases with no isolation.
• A fresh container per test. Rejected: 40+ boots per package × 20–60s cold start → 15-minute timeouts.
• Database files on disk instead of RAM. Rejected: the slow part is fsync on cold start and on the write-heavy restore; RAM removes it for servers we throw away anyway.

Consequences

Positive

• The two packages that used to time out at 900s now finish in ~30s; the whole suite went from timing out to ~1.5–4 minutes.
• Peak memory is bounded and predictable — one matrix container per package — and parallel packages are fully isolated.

Negative

• Because a server is reused across a version's subtests, authors must DROP ... IF EXISTS for fixed-name objects and must not run those subtests with t.Parallel.
• Data in RAM is volatile — fine for throwaway test servers, but there is no crash safety.
• If a CI runner is short on memory, lower TEST_PARALLEL_WORKERS (e.g. to 6).

Neutral / follow-ups

• On a hard -timeout or os.Exit, t.Cleanup is skipped; the Makefile/CI label-sweep cleans up any leftover containers.

References

• backend/internal/util/testing/containers/{postgres,mysql,mariadb,mongodb}.go (the StartXxx helpers)
• backend/internal/features/tests/logical/postgresql/backup_restore_test.go — the per-version orchestrator pattern
• backend/internal/features/tests/physical/postgresql/{pg17,pg18} — the physical backup→restore E2E split one package per PostgreSQL major, so the two versions run as isolated, parallel test binaries (each with its own control plane and throwaway source + restore-target containers)
• backend/internal/config/config.go (applyTestWorkerSlot / claimTestWorkerSlot)
• backend/internal/features/backups/backups/backuping/nodes/registry.go
• backend/Makefile