* chore: ignore local agency and team snapshots
* feat: add team agent profile templates
* KANBAN
* template
* prototype
* Fix tenant runtime pool deployment
* Fix local team profiles runtime override
* Update local team profiles testing guide
* Adapt local team profiles test deployment
* Patch tenant NFS server to ClusterIP
* Use NFS export root for workspace mounts
* Tighten team task completion projection
* Fix team event body map comparison
* before modification
* temp_peer1
* leadermodel
* team_fix
* fix(team): harden workflow completion and workspace handling
* fix(team): harden redis workflow protocol
* discord link updated
* chore: checkpoint team workflow fixes
* fix: stabilize leader kanban ledger
* fix: coalesce leader mediated work items
* fix: confirm leader-mediated member results
* fix: harden leader mediated team state
* fix: stabilize team shared runtime state
* Fix team task reconciliation and create flow
* fix: stabilize leader mediated team orchestration
* fix: refine team workspace presentation
* checkpoint: persist team workflow ledger fixes
* fix: reconcile team workflow delivery state
* fix: preserve meaningful team chat events
* docs: add team workspace quick guides
* chore: ignore local team deployment files
* fix: preserve setgid on team shared directories
* fix(team): align workspace overlays and completion projection
* fix(team): preserve workflow identity recovery
* fix(team): protect terminal workflow projection
* fix(team): preserve collaboration flow after terminal results
* feat(skill-hub): add Skill Hub catalog, publish flow, and lite materialize pipeline
Introduce Skill Hub for browsing, importing, publishing, and installing skills, with lite instance package materialization and runtime sync support.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(skill-hub): remove token-governance hooks from Skill Hub PR
Strip validateManagedRuntimeEnvironmentOverrides, network lock policy sync,
and egress proxy audit wiring that belong to the upcoming token-usage work,
so the Skill Hub branch compiles independently.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(skill-hub): update migration number in materialize docs
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(skill-hub): make RuntimeAgentClient test stub and hub tests compile-safe
Add ResyncInstanceSkills to the runtime pool handler fake client, and harden
skill hub payload helpers/tests against nil storage/instance repos so go test passes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add Skill Hub hardening with session usage tracking and egress governance
Unify Skill Hub runtime sync improvements with session-token observability,
egress network policy, and admin/instance usage reporting for reopenable PR.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(skill-hub): repair CI tests and nested skill install
* fix(ci): restore release deployment configuration
---------
Co-authored-by: heshengran <heshengran@ieisystem.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(runtime): improve Lite gateway scheduling and proxy stability
- Expand the runtime gateway port range to match 100 instances per pod
- Scale runtime deployments based on pending backlog
- Serialize gateway creation per pod and handle starting/creating states
- Clean up missing gateway bindings from runtime agent reports
- Improve Lite gateway proxy token stripping and auth header injection
- Update deployment manifests, docs, and related tests
* fix(runtime): harden Hermes Lite gateway recovery
- Add hashed gateway token aliases for short-term token compatibility
- Fix /resume conversations failing when old sessions use mismatched gateway tokens
- Recover ports only from stale error bindings without affecting active gateways
* feat(runtime): enable bidirectional desktop clipboard and direct proxy
- Enable WebTop/Selkies/Kasm clipboard by default with per-instance overrides
- Document clipboard verification, configuration pitfalls, and IME troubleshooting
- Enable direct desktop proxy in the K8s and K3s manifests
---------
Co-authored-by: litiantian03 <litiantian03@ieisystem.com>
* chore: ignore local agency and team snapshots
* feat: add team agent profile templates
* KANBAN
* template
* prototype
* Fix tenant runtime pool deployment
* Fix local team profiles runtime override
* Update local team profiles testing guide
* Adapt local team profiles test deployment
* Patch tenant NFS server to ClusterIP
* Use NFS export root for workspace mounts
* Tighten team task completion projection
* Fix team event body map comparison
* before modification
* temp_peer1
* leadermodel
* team_fix
* fix(team): harden workflow completion and workspace handling
* fix(team): harden redis workflow protocol
* discord link updated
* chore: checkpoint team workflow fixes
* fix: stabilize leader kanban ledger
* fix: coalesce leader mediated work items
* fix: confirm leader-mediated member results
* fix: harden leader mediated team state
* fix: stabilize team shared runtime state
* Fix team task reconciliation and create flow
* fix: stabilize leader mediated team orchestration
* fix: refine team workspace presentation
* checkpoint: persist team workflow ledger fixes
* fix: reconcile team workflow delivery state
* fix: preserve meaningful team chat events
* docs: add team workspace quick guides
* chore: ignore local team deployment files
* fix: preserve setgid on team shared directories
* chore: ignore local agency and team snapshots
* feat: add team agent profile templates
* KANBAN
* template
* prototype
* Fix tenant runtime pool deployment
* Fix local team profiles runtime override
* Update local team profiles testing guide
* Adapt local team profiles test deployment
* Patch tenant NFS server to ClusterIP
* Use NFS export root for workspace mounts
* Tighten team task completion projection
* Fix team event body map comparison
* before modification
* temp_peer1
* leadermodel
* team_fix
* fix(team): harden workflow completion and workspace handling
* fix(team): harden redis workflow protocol
- Add APIs for batch creating and deleting Lite instances
- Support Lite batch create, selection, and delete flows in the instance list
- Fix Lite OpenClaw/Hermes workspace initial paths and skill directory paths
- Hide transient skill temp directories from workspace browsing
- Preserve runtime query tokens when stripping ClawManager desktop access tokens
- Add an available-skills endpoint for instances with actor-aware filtering
- Persist gateway token, agent bootstrap token, and OpenClaw config snapshots for Lite instances
- Inject agent and OpenClaw bootstrap env into gateway runtimes
- Materialize and remove Lite instance skills in the workspace filesystem
- Improve instance detail skill upload, bulk attachment, workspace refresh, and layout syncing
- Support multi-select skill attachment and show risk-policy filtering hints
- Mark instance skills as removed after successful uninstall commands
- Sync skill removal state when related workspace files are deleted
- Hide removed skills and avoid reactivating them from agent inventory reports
- Clear proxy-scoped OpenClaw control storage to prevent cross-instance state leakage
- Reuse desktop direct upstream resolution and include upstream in short external access tokens
- Default and normalize Selkies stream settings for desktop runtimes
- Remove legacy deployments with the same instance-id before ensuring the current deployment
- Preserve the unsaved desktop stream profile state in the frontend
- Add tests for stream profiles, pod envs, and deployment cleanup
- Add leader election to avoid duplicate sync and Team event consumers
- Proxy desktop traffic directly from nginx to instance Services to reduce load during concurrent viewers
- Add desktop stream profiles with low / standard / high options
- Limit hostPath PV fallback to manual storageClass only
Create instance workloads as single-replica Deployments instead of bare Pods so Kubernetes can recreate Pods after deletion, eviction, or node/runtime failures.
Update stop/delete and orphan cleanup to remove Deployments before residual Pods, and improve status sync to keep instances creating while a Deployment exists without a Pod and avoid stale Failed/Evicted Pods affecting instance state.
- Disable Webtop/KasmVNC bidirectional clipboard sync by default to reduce accidental paste during remote desktop input
- Derive desktop runtime /dev/shm defaults from instance memory: 1G for 4G, 2G for 8G, and 4G for 12G+
- Preserve explicit SHM_SIZE_GB overrides and keep shell runtime behavior unchanged
- Add tests for openclaw/hermes clipboard policy and /dev/shm default selection
- Support multiple OpenClaw channel resources and fix alias injection
Add cursor-based GET /teams/:id/tasks and GET /teams/:id/events APIs.
Team detail page loads older messages on scroll/click and deduplicates
collaboration chat messages with improved thread ordering.
- Add migrations plus Team/Member/Task/Event APIs; Redis Streams consumer projects inbox/events into DB as source of truth
- Inject Team Secret (Redis URL, team token) via envFrom; shared RWX PVC at /team; sync ConfigMap roster to /team/team.json
- Create member Pods through InstanceService; extend K8s (PVC/Secret/Pod/ConfigMap); stale-task sweep and Team/member deletion with cleanup
- Add /teams, /teams/new, /teams/🆔 creation wizard (roster, presets, shared env/OpenClaw plan), per-member desktops, collaboration timeline, debug dispatch (defaults to Leader when target omitted)
- add configurable /dev/shm mounts for instance pods with a bounded SHM_SIZE_GB override
- introduce pod security modes and use chromium-compat for OpenClaw instead of privileged
- keep privileged mode only as an explicit admin fallback
- remove the node-level clawmanager-node-tuner manifest to avoid changing host security defaults
- fix k3s HTTPS and API/proxy service port mappings
- add tests for SHM parsing and pod security mode behavior
BREAKING: PV hostPath prefix changed from /tmp/clawreef/ to /data/clawreef/
Problem:
- PV hostPath was hardcoded to /tmp/clawreef/user-{id}/instance-{id}
- /tmp/ is a volatile directory that may be cleaned by systemd-tmpfiles
or cleared on node reboot (depending on OS configuration)
- This caused data loss when worker-02 was rebooted by the hypervisor
Fix:
- Add configurable hostPathPrefix to RuntimePVCConfig (default: /data/clawreef)
- Support K8S_PV_HOST_PATH_PREFIX environment variable override
- Update deployment manifests to use /data/clawmanager/ for MySQL and MinIO
Migration:
- Existing deployments should move data from /tmp/clawreef/ to /data/clawreef/
and create a symlink for backward compatibility:
mkdir -p /data/clawreef
mv /tmp/clawreef/* /data/clawreef/
rm -rf /tmp/clawreef && ln -s /data/clawreef /tmp/clawreef
TestNormalizeOpenClawResourceContentPreservesUnknownChannelFields and
TestNormalizeOpenClawResourceContentPreservesFeishuSiblingAccounts were
declared twice causing build failure.
Every channel form editor (telegram, dingtalk-connector, slack, feishu)
silently drops fields on Save. The user flow that loses data:
1. Open a channel resource in the JSON tab, add any field not owned
by the form (e.g. webhook, custom capabilities, additional feishu
accounts).
2. Switch to the Form tab and change anything (e.g. rotate a secret).
3. Click Save.
4. The field from step 1 is gone from the saved config — and stays
gone after a reload.
The data loss occurs in two places; both must be fixed:
Frontend (OpenClawConfigCenterPage.tsx):
Each update*ChannelContentText rebuilt the output from a hard-coded
allowlist instead of merging form fields into the parsed config. Any
field not in the allowlist was dropped on Save. For feishu this
collapsed a multi-account config to a single `main` account.
Backend (openclaw_config_service.go):
normalizeOpenClawResourceContent calls normalize*ChannelConfigForEnv
on every create/update to validate and rebuild the stored config.
Those helpers use the same allowlist pattern, so even if the
frontend sends a complete config, the backend strips tenant-authored
keys before persisting.
Fix:
- Frontend: introduce a single mergeChannelConfig(existing, allowlisted)
helper and route every editor's output through it. Each handler
builds an allowlisted object containing only its known keys; the
helper overlays it onto the parsed existing config so unknown keys
survive. The merge point is now structural rather than a per-handler
convention, so a new editor (e.g. an MS Teams handler added in a
downstream patch) can't regress by forgetting the spread. Feishu's
accounts deep-merge is preserved by building the merged accounts
inside the allowlisted object before handing off to the helper.
- Backend: introduce mergeOpenClawChannelConfigForStorage, which
overlays the allowlist-normalized output onto the original parsed
payload at storage time. Normalized keys still win; unknown keys
pass through. For feishu the accounts map is merged member-wise.
Env-rendering semantics are unchanged. appendCompiledOpenClawEnvPayload
continues to call the *ForEnv helpers at render time against the
stored content, so the runtime env payload sees the same allowlist as
before.
Tests:
- Two existing ResourcePayloadFromModelNormalizesStored*JSON tests
asserted the pre-fix behavior (dropping `legacyField`) and are
updated to assert it survives.
- New TestNormalizeOpenClawResourceContentPreservesUnknownChannelFields
(table-driven, three editors) guards the preservation invariant.
- New TestNormalizeOpenClawResourceContentPreservesFeishuSiblingAccounts
guards the feishu deep-merge.
Manual verification against a live instance: add `webhook` via the
JSON tab, switch to Form tab, rotate a secret, Save, reload — the
webhook field is retained.
Kubernetes defaults imagePullPolicy to Always when the image tag is
:latest. This causes pod creation failures in air-gapped and enterprise
environments where nodes cannot reach external registries.
Changes:
- Add ImagePullPolicy field to PodConfig struct
- Default to IfNotPresent in CreatePod when not explicitly set
- Support IMAGE_PULL_POLICY env var for operator override
- Apply policy to both CreateInstance and StartInstance paths
Closes#94
Every channel form editor (telegram, dingtalk-connector, slack, feishu)
silently drops fields on Save. The user flow that loses data:
1. Open a channel resource in the JSON tab, add any field not owned
by the form (e.g. webhook, custom capabilities, additional feishu
accounts).
2. Switch to the Form tab and change anything (e.g. rotate a secret).
3. Click Save.
4. The field from step 1 is gone from the saved config — and stays
gone after a reload.
The data loss occurs in two places; both must be fixed:
Frontend (OpenClawConfigCenterPage.tsx):
Each update*ChannelContentText rebuilt the output from a hard-coded
allowlist instead of merging form fields into the parsed config. Any
field not in the allowlist was dropped on Save. For feishu this
collapsed a multi-account config to a single `main` account.
Backend (openclaw_config_service.go):
normalizeOpenClawResourceContent calls normalize*ChannelConfigForEnv
on every create/update to validate and rebuild the stored config.
Those helpers use the same allowlist pattern, so even if the
frontend sends a complete config, the backend strips tenant-authored
keys before persisting.
Fix:
- Frontend: introduce a single mergeChannelConfig(existing, allowlisted)
helper and route every editor's output through it. Each handler
builds an allowlisted object containing only its known keys; the
helper overlays it onto the parsed existing config so unknown keys
survive. The merge point is now structural rather than a per-handler
convention, so a new editor (e.g. an MS Teams handler added in a
downstream patch) can't regress by forgetting the spread. Feishu's
accounts deep-merge is preserved by building the merged accounts
inside the allowlisted object before handing off to the helper.
- Backend: introduce mergeOpenClawChannelConfigForStorage, which
overlays the allowlist-normalized output onto the original parsed
payload at storage time. Normalized keys still win; unknown keys
pass through. For feishu the accounts map is merged member-wise.
Env-rendering semantics are unchanged. appendCompiledOpenClawEnvPayload
continues to call the *ForEnv helpers at render time against the
stored content, so the runtime env payload sees the same allowlist as
before.
Tests:
- Two existing ResourcePayloadFromModelNormalizesStored*JSON tests
asserted the pre-fix behavior (dropping `legacyField`) and are
updated to assert it survives.
- New TestNormalizeOpenClawResourceContentPreservesUnknownChannelFields
(table-driven, three editors) guards the preservation invariant.
- New TestNormalizeOpenClawResourceContentPreservesFeishuSiblingAccounts
guards the feishu deep-merge.
Manual verification against a live instance: add `webhook` via the
JSON tab, switch to Form tab, rotate a secret, Save, reload — the
webhook field is retained.
Address three HIGH-severity issues from the memory-leak and bloat
analysis (issue #56).
1. Graceful shutdown (main.go)
- Replace gin's r.Run() with an explicit http.Server so the process
can intercept SIGINT / SIGTERM.
- On signal: drain active HTTP requests (10 s timeout), then stop
SyncService, WebSocket Hub, and InstanceAccessService cleanup
goroutine in order.
- Ensures database connections, K8s watchers, and background loops
are released cleanly on deploy or restart.
2. WebSocket Hub init race (websocket_service.go)
- GetHub() used a bare nil-check with no synchronisation; two
goroutines could each create a Hub and start a Run() loop.
- Replaced with sync.Once to guarantee exactly one Hub instance.
- Added a stop channel to Hub.Run() so the hub can be shut down
gracefully, closing all connected clients.
3. InstanceAccessService goroutine leak (instance_access_service.go)
- cleanupExpiredTokens() looped on ticker.C with no exit path,
leaking the goroutine for the lifetime of the process.
- Added a stopChan; cleanupExpiredTokens now selects on both the
ticker and the stop signal.
- Exposed Stop() on the service; InstanceHandler.Shutdown() calls
it during graceful shutdown.
Tests:
- TestGetHubReturnsSameInstance: singleton guarantee
- TestGetHubConcurrentAccess: 50-goroutine race test
- TestHubStopClosesClients: verifies client cleanup on Stop()
- TestInstanceAccessServiceStopTerminatesCleanup: Stop() is safe and
the service remains functional for token ops afterward
All existing tests continue to pass; full project build and regression
verified.
Ref: #56
When a resource is updated, all compiled/active snapshots that reference
it (directly or via dependsOn) are now automatically recompiled so their
rendered_env_json stays in sync.
Key changes:
- snapshotReferencesResource checks ResolvedResourcesJSON first (full
resolved set including dependsOn), falling back to
SelectedResourceIDsJSON for backward compatibility with older snapshots
- Repository-layer CAS via UPDATE ... WHERE updated_at = ? prevents
concurrent cascades from clobbering each other
- ListActiveSnapshots queries both 'compiled' and 'active' statuses
- Cascade runs in a background goroutine; errors are logged but never
fail the resource update itself
Stark-Han