Create instance workloads as single-replica Deployments instead of bare Pods so Kubernetes can recreate Pods after deletion, eviction, or node/runtime failures.
Update stop/delete and orphan cleanup to remove Deployments before residual Pods, and improve status sync to keep instances creating while a Deployment exists without a Pod and avoid stale Failed/Evicted Pods affecting instance state.
- Disable Webtop/KasmVNC bidirectional clipboard sync by default to reduce accidental paste during remote desktop input
- Derive desktop runtime /dev/shm defaults from instance memory: 1G for 4G, 2G for 8G, and 4G for 12G+
- Preserve explicit SHM_SIZE_GB overrides and keep shell runtime behavior unchanged
- Add tests for openclaw/hermes clipboard policy and /dev/shm default selection
- Support multiple OpenClaw channel resources and fix alias injection
Add cursor-based GET /teams/:id/tasks and GET /teams/:id/events APIs.
Team detail page loads older messages on scroll/click and deduplicates
collaboration chat messages with improved thread ordering.
- Add migrations plus Team/Member/Task/Event APIs; Redis Streams consumer projects inbox/events into DB as source of truth
- Inject Team Secret (Redis URL, team token) via envFrom; shared RWX PVC at /team; sync ConfigMap roster to /team/team.json
- Create member Pods through InstanceService; extend K8s (PVC/Secret/Pod/ConfigMap); stale-task sweep and Team/member deletion with cleanup
- Add /teams, /teams/new, /teams/🆔 creation wizard (roster, presets, shared env/OpenClaw plan), per-member desktops, collaboration timeline, debug dispatch (defaults to Leader when target omitted)
- add configurable /dev/shm mounts for instance pods with a bounded SHM_SIZE_GB override
- introduce pod security modes and use chromium-compat for OpenClaw instead of privileged
- keep privileged mode only as an explicit admin fallback
- remove the node-level clawmanager-node-tuner manifest to avoid changing host security defaults
- fix k3s HTTPS and API/proxy service port mappings
- add tests for SHM parsing and pod security mode behavior
BREAKING: PV hostPath prefix changed from /tmp/clawreef/ to /data/clawreef/
Problem:
- PV hostPath was hardcoded to /tmp/clawreef/user-{id}/instance-{id}
- /tmp/ is a volatile directory that may be cleaned by systemd-tmpfiles
or cleared on node reboot (depending on OS configuration)
- This caused data loss when worker-02 was rebooted by the hypervisor
Fix:
- Add configurable hostPathPrefix to RuntimePVCConfig (default: /data/clawreef)
- Support K8S_PV_HOST_PATH_PREFIX environment variable override
- Update deployment manifests to use /data/clawmanager/ for MySQL and MinIO
Migration:
- Existing deployments should move data from /tmp/clawreef/ to /data/clawreef/
and create a symlink for backward compatibility:
mkdir -p /data/clawreef
mv /tmp/clawreef/* /data/clawreef/
rm -rf /tmp/clawreef && ln -s /data/clawreef /tmp/clawreef
TestNormalizeOpenClawResourceContentPreservesUnknownChannelFields and
TestNormalizeOpenClawResourceContentPreservesFeishuSiblingAccounts were
declared twice causing build failure.
Every channel form editor (telegram, dingtalk-connector, slack, feishu)
silently drops fields on Save. The user flow that loses data:
1. Open a channel resource in the JSON tab, add any field not owned
by the form (e.g. webhook, custom capabilities, additional feishu
accounts).
2. Switch to the Form tab and change anything (e.g. rotate a secret).
3. Click Save.
4. The field from step 1 is gone from the saved config — and stays
gone after a reload.
The data loss occurs in two places; both must be fixed:
Frontend (OpenClawConfigCenterPage.tsx):
Each update*ChannelContentText rebuilt the output from a hard-coded
allowlist instead of merging form fields into the parsed config. Any
field not in the allowlist was dropped on Save. For feishu this
collapsed a multi-account config to a single `main` account.
Backend (openclaw_config_service.go):
normalizeOpenClawResourceContent calls normalize*ChannelConfigForEnv
on every create/update to validate and rebuild the stored config.
Those helpers use the same allowlist pattern, so even if the
frontend sends a complete config, the backend strips tenant-authored
keys before persisting.
Fix:
- Frontend: introduce a single mergeChannelConfig(existing, allowlisted)
helper and route every editor's output through it. Each handler
builds an allowlisted object containing only its known keys; the
helper overlays it onto the parsed existing config so unknown keys
survive. The merge point is now structural rather than a per-handler
convention, so a new editor (e.g. an MS Teams handler added in a
downstream patch) can't regress by forgetting the spread. Feishu's
accounts deep-merge is preserved by building the merged accounts
inside the allowlisted object before handing off to the helper.
- Backend: introduce mergeOpenClawChannelConfigForStorage, which
overlays the allowlist-normalized output onto the original parsed
payload at storage time. Normalized keys still win; unknown keys
pass through. For feishu the accounts map is merged member-wise.
Env-rendering semantics are unchanged. appendCompiledOpenClawEnvPayload
continues to call the *ForEnv helpers at render time against the
stored content, so the runtime env payload sees the same allowlist as
before.
Tests:
- Two existing ResourcePayloadFromModelNormalizesStored*JSON tests
asserted the pre-fix behavior (dropping `legacyField`) and are
updated to assert it survives.
- New TestNormalizeOpenClawResourceContentPreservesUnknownChannelFields
(table-driven, three editors) guards the preservation invariant.
- New TestNormalizeOpenClawResourceContentPreservesFeishuSiblingAccounts
guards the feishu deep-merge.
Manual verification against a live instance: add `webhook` via the
JSON tab, switch to Form tab, rotate a secret, Save, reload — the
webhook field is retained.
Kubernetes defaults imagePullPolicy to Always when the image tag is
:latest. This causes pod creation failures in air-gapped and enterprise
environments where nodes cannot reach external registries.
Changes:
- Add ImagePullPolicy field to PodConfig struct
- Default to IfNotPresent in CreatePod when not explicitly set
- Support IMAGE_PULL_POLICY env var for operator override
- Apply policy to both CreateInstance and StartInstance paths
Closes#94
Every channel form editor (telegram, dingtalk-connector, slack, feishu)
silently drops fields on Save. The user flow that loses data:
1. Open a channel resource in the JSON tab, add any field not owned
by the form (e.g. webhook, custom capabilities, additional feishu
accounts).
2. Switch to the Form tab and change anything (e.g. rotate a secret).
3. Click Save.
4. The field from step 1 is gone from the saved config — and stays
gone after a reload.
The data loss occurs in two places; both must be fixed:
Frontend (OpenClawConfigCenterPage.tsx):
Each update*ChannelContentText rebuilt the output from a hard-coded
allowlist instead of merging form fields into the parsed config. Any
field not in the allowlist was dropped on Save. For feishu this
collapsed a multi-account config to a single `main` account.
Backend (openclaw_config_service.go):
normalizeOpenClawResourceContent calls normalize*ChannelConfigForEnv
on every create/update to validate and rebuild the stored config.
Those helpers use the same allowlist pattern, so even if the
frontend sends a complete config, the backend strips tenant-authored
keys before persisting.
Fix:
- Frontend: introduce a single mergeChannelConfig(existing, allowlisted)
helper and route every editor's output through it. Each handler
builds an allowlisted object containing only its known keys; the
helper overlays it onto the parsed existing config so unknown keys
survive. The merge point is now structural rather than a per-handler
convention, so a new editor (e.g. an MS Teams handler added in a
downstream patch) can't regress by forgetting the spread. Feishu's
accounts deep-merge is preserved by building the merged accounts
inside the allowlisted object before handing off to the helper.
- Backend: introduce mergeOpenClawChannelConfigForStorage, which
overlays the allowlist-normalized output onto the original parsed
payload at storage time. Normalized keys still win; unknown keys
pass through. For feishu the accounts map is merged member-wise.
Env-rendering semantics are unchanged. appendCompiledOpenClawEnvPayload
continues to call the *ForEnv helpers at render time against the
stored content, so the runtime env payload sees the same allowlist as
before.
Tests:
- Two existing ResourcePayloadFromModelNormalizesStored*JSON tests
asserted the pre-fix behavior (dropping `legacyField`) and are
updated to assert it survives.
- New TestNormalizeOpenClawResourceContentPreservesUnknownChannelFields
(table-driven, three editors) guards the preservation invariant.
- New TestNormalizeOpenClawResourceContentPreservesFeishuSiblingAccounts
guards the feishu deep-merge.
Manual verification against a live instance: add `webhook` via the
JSON tab, switch to Form tab, rotate a secret, Save, reload — the
webhook field is retained.
Address three HIGH-severity issues from the memory-leak and bloat
analysis (issue #56).
1. Graceful shutdown (main.go)
- Replace gin's r.Run() with an explicit http.Server so the process
can intercept SIGINT / SIGTERM.
- On signal: drain active HTTP requests (10 s timeout), then stop
SyncService, WebSocket Hub, and InstanceAccessService cleanup
goroutine in order.
- Ensures database connections, K8s watchers, and background loops
are released cleanly on deploy or restart.
2. WebSocket Hub init race (websocket_service.go)
- GetHub() used a bare nil-check with no synchronisation; two
goroutines could each create a Hub and start a Run() loop.
- Replaced with sync.Once to guarantee exactly one Hub instance.
- Added a stop channel to Hub.Run() so the hub can be shut down
gracefully, closing all connected clients.
3. InstanceAccessService goroutine leak (instance_access_service.go)
- cleanupExpiredTokens() looped on ticker.C with no exit path,
leaking the goroutine for the lifetime of the process.
- Added a stopChan; cleanupExpiredTokens now selects on both the
ticker and the stop signal.
- Exposed Stop() on the service; InstanceHandler.Shutdown() calls
it during graceful shutdown.
Tests:
- TestGetHubReturnsSameInstance: singleton guarantee
- TestGetHubConcurrentAccess: 50-goroutine race test
- TestHubStopClosesClients: verifies client cleanup on Stop()
- TestInstanceAccessServiceStopTerminatesCleanup: Stop() is safe and
the service remains functional for token ops afterward
All existing tests continue to pass; full project build and regression
verified.
Ref: #56
When a resource is updated, all compiled/active snapshots that reference
it (directly or via dependsOn) are now automatically recompiled so their
rendered_env_json stays in sync.
Key changes:
- snapshotReferencesResource checks ResolvedResourcesJSON first (full
resolved set including dependsOn), falling back to
SelectedResourceIDsJSON for backward compatibility with older snapshots
- Repository-layer CAS via UPDATE ... WHERE updated_at = ? prevents
concurrent cascades from clobbering each other
- ListActiveSnapshots queries both 'compiled' and 'active' statuses
- Cascade runs in a background goroutine; errors are logged but never
fail the resource update itself
Summary
-------
Logging in as an admin and navigating to Workspace → My Instances
returned every instance in the cluster instead of the admin's own.
Role was overloaded to both widen the admin console surface AND
widen the self-scoped list endpoint, so the workspace view broke
the owner-isolation contract users expect.
Root cause
----------
`InstanceService.GetVisibleInstances(userID, role, ...)` branched
on role: admin callers fell through to `instanceRepo.GetAll`,
non-admin callers fell through to `GetByUserID`. The single
`GET /instances` handler passed the caller's role into that
function, so the same URL meant "my instances" for users and
"every instance in the system" for admins.
Fix
---
Split the two views at the API surface, not inside the service:
- `GET /instances` is now always caller-scoped. The handler calls
`GetByUserID` unconditionally and never reads the caller's role.
- `GET /admin/instances` is a new route, gated by the existing
admin middleware trio (`Auth` + `SetUserInfo` + `NewAdminAuth`).
It calls a new `InstanceService.GetAllInstances` that does not
look at any userID.
- `GetVisibleInstances` is removed; nothing else in the codebase
called it.
Frontend follows: `AdminDashboard` and `InstanceManagementPage`
switch to a new `adminInstanceService.getInstances()` that hits
`/admin/instances`. Per-instance admin actions (start, stop,
delete, proxy, etc.) are intentionally out of scope for this PR
and continue to use the existing endpoints.
Tests
-----
Two new service tests in `instance_visibility_test.go`:
- `TestGetByUserIDFiltersByCaller` pins the workspace contract:
regardless of how many admins or other users exist, a caller
only ever sees their own instances through this path.
- `TestGetAllInstancesReturnsEveryUser` covers the admin-console
path, including pagination.
`go build ./...`, `go test ./...`, and `npm run build` all pass.
Scope note
----------
This PR fixes the listing leak only. Per-instance handlers in
`instance_handler.go` retain their existing inline role checks
and will be audited separately.
Wrap c.Request.Body with http.MaxBytesReader(512 MiB) so oversize
uploads that bypass the edge (or when nginx is swapped for another
proxy) produce a JSON 413 with a clear message rather than a generic
500 or a parser-side surprise. Also guards against header-declared
sizes exceeding the cap.
Aligns backend and nginx limits on a single named constant.
Adds build-tagged (//go:build integration) tests that exercise the
Export and Import commands against a real OpenClaw pod:
- Export_RoundTrip: seeds a marker file under /config/.openclaw, runs
buildExportCommand(), asserts the marker is present in the streamed
tar.gz.
- Export_MissingWorkspace: moves .openclaw aside and asserts the
command exits with code 42 (maps to ErrOpenClawWorkspaceMissing).
- Import_OwnershipIsAbc: imports a minimal tar.gz and asserts the
restored file is owned by uid:gid 1000:1000, verifying the
`su abc` wrap.
Skipped by default (unit tests cover the command-string contract);
enable with `go test -tags integration` and OPENCLAW_TEST_POD set.
ExportOpenClaw now maps services.ErrOpenClawWorkspaceMissing to HTTP 404
with an explicit message, and rejects archives below a conservative
minimum-size threshold (100 bytes) as HTTP 500 rather than streaming an
empty body to the client with Content-Type: application/gzip.
Together with the service-side base-dir fix, this closes the
silent-failure path where downloading .openclaw from an instance without
a workspace produced a 200 OK with zero payload.
Previously Export/Import expanded ${HOME:-/home/user} to locate the
.openclaw workspace. kubectl exec spawns a fresh process as root with
HOME=/root, while the real workspace lives at /config/.openclaw (the
linuxserver entrypoint runs as user abc with HOME=/config). The result
was a silent failure: Export returned an empty 200 stream and Import
extracted into /root/ where OpenClaw never reads.
Resolve the workspace base via ${CLAWMANAGER_AGENT_PERSISTENT_DIR:-/config},
honoring the env var ClawManager already injects at pod creation. When
the workspace is missing, Export exits with code 42 and the service
layer maps it to ErrOpenClawWorkspaceMissing so handlers can surface
a proper HTTP status instead of streaming zero bytes.
Import now re-execs the extract as user abc via `exec su abc -s /bin/sh`
so restored files land owned by uid 1000, matching pod runtime ownership.
Pure refactor introducing buildExportCommand() and buildImportCommand()
helpers. Behavior unchanged; the buggy ${HOME:-/home/user} base dir is
preserved. Makes the commands testable without a cluster and sets up
the subsequent fix commit to change them in one place.
The agent session token issued during Register() has a fixed 24-hour
TTL that is never extended. After exactly 24 hours of continuous
operation, AuthenticateSession() rejects the token and every subsequent
heartbeat returns 401. Because the agent shim does not detect non-2xx
responses, it keeps sending heartbeats silently into the void and the
agent appears permanently offline — recoverable only by a full pod
restart.
Fix: roll the SessionExpiresAt forward by 24 hours on every successful
Heartbeat() call, turning the fixed deadline into a sliding window.
As long as the agent keeps heartbeating (every 30s by default), the
session never expires.
Discovered in production with 12 openclaw instances going offline
simultaneously after exactly 24 hours of uptime.
- Change cpu_cores from int to float64 in backend models, handlers, services
- Update frontend CreateInstancePage to accept decimal CPU values (step 0.1)
- Add DB migration 009_cpu_cores_decimal.sql for DECIMAL(10,2) columns
- Update deployment manifests (k8s, k3s, incluster) schema definitions
- Add GOFLAGS/GOPROXY/GOSUMDB build args to Dockerfile so that
--build-arg values passed by build scripts take effect during go mod download
litiantian03