# Probe: smoke # # Per-service L1+L2 health check across every showcase service on # Railway. Discovery-driven: the `railway-services` source enumerates # every `showcase-*` service in the orchestrator's project, and the # driver fans out one invocation per service. # # Each driver invocation issues THREE HTTP calls: # 1. GET `${publicUrl}/smoke` → primary `smoke:` tick # 2. GET `${publicUrl}/health` → side-emit `health:` # 3. POST `${publicUrl}/api/copilotkit/` → side-emit `agent:` # # (1) and (2) are L1: liveness + health. (3) is L2: the CopilotKit # runtime is mounted and answering — any non-404 response is green; # 404 or transport failure is red. Matches the `checkAgentEndpoint` # contract in the e2e helpers. # # `slug` is derived by stripping the `showcase-` prefix from the Railway # service name. So `showcase-ag2` → keys `smoke:ag2` / `health:ag2` / # `agent:ag2`. Each package service produces 3 independent probe # rows — each can flip green/red without blinding its siblings. # # Auto-discovery: adding a new `showcase-*` service on Railway # automatically picks it up on the next tick — no YAML edit required. # Only infra services (aimock/harness/pocketbase/shell*) are excluded, # because they don't expose a CopilotKit runtime or health endpoints. # If a new non-monitored service is created with the `showcase-` # prefix, extend `filter.nameExcludes` here. # # Schedule mirrors the pre-YAML cron cadence (every 15 minutes) — the # accompanying alert rule in config/alerts/ covers green→red / # red→green transitions and a cron-only heartbeat tick so silent # green runs still produce observable liveness. # # timeout_ms (30s) bounds BOTH the discovery enumerate phase AND each # per-target driver invocation at the invoker level. The driver also # re-applies the timeout inside each HTTP call so one hung endpoint # can't steal budget from its paired probes. # # This was originally 10s when the probe used a static target list. # After migrating to discovery-driven (railway-services), the same # timeout_ms governs the Railway enumerate call — a project-level # GraphQL query PLUS per-service variable lookups (bounded at # concurrency 8). With 35+ services, each variable round-trip ~1s, # enumerate alone takes ceil(35/8) × 1s ≈ 5s on a good tick, and # easily >10s on a slow one. The 10s cap caused persistent # discoveryFailed:true — enumerate timed out before reaching the # driver fan-out. 30s matches image-drift and qa (both Railway- # discovery probes that work reliably at this fleet size). # # max_concurrency (6) caps simultaneous per-tick driver invocations — # at 34 services the tick takes ≤ 6 pool-cycles and stays well under # any Railway edge rate limit. Per-tick socket bound: # max_concurrency × 3 endpoints = 18 inflight. kind: smoke id: smoke schedule: "*/5 * * * *" timeout_ms: 30000 max_concurrency: 6 discovery: source: railway-services filter: namePrefix: "showcase-" # Infra services that don't expose /smoke, /health, or the # CopilotKit runtime. Keep this list aligned with the excludes in # `src/probes/aimock-wiring.ts` — when a new infra service lands # on Railway under the `showcase-` prefix, add it in both places. nameExcludes: - "showcase-aimock" - "showcase-harness" - "showcase-pocketbase" - "showcase-shell" - "showcase-shell-dashboard" - "showcase-shell-docs" - "showcase-shell-dojo" # Decommissioned starters — Railway services stopped, deployments # halted (PR #4390). Excluded so probes don't scan dead endpoints. - "showcase-starter-ag2" - "showcase-starter-agno" - "showcase-starter-claude-sdk-python" - "showcase-starter-claude-sdk-typescript" - "showcase-starter-crewai-crews" - "showcase-starter-google-adk" - "showcase-starter-langgraph-fastapi" - "showcase-starter-langgraph-python" - "showcase-starter-langgraph-typescript" - "showcase-starter-langroid" - "showcase-starter-llamaindex" - "showcase-starter-mastra" - "showcase-starter-ms-agent-dotnet" - "showcase-starter-ms-agent-python" - "showcase-starter-pydantic-ai" - "showcase-starter-spring-ai" - "showcase-starter-strands" # key_template is interpolated against each RailwayServiceInfo record. # `${name}` is the full Railway service name (`showcase-ag2`). The # driver re-derives the slug by stripping the `showcase-` prefix, so # the writer sees `smoke:showcase-ag2` as the top-line key and # `health:ag2` / `agent:ag2` as the side-emit keys. We keep the # `smoke:` prefix carrying the full service name so existing alert # rules that dedupe on `smoke:*` continue to match. key_template: "smoke:${name}"