This PR completes the telemetry system by implementing the transport
layer for `ClearcutSender`. It enables actual HTTP communication with
the Clearcut backend, handling event batching, rate limiting, and
reliable delivery, including robust shutdown handling.
**Key Changes:**
* **HTTP Transport**: Implemented `fetch`-based transport sending `POST`
requests to the Clearcut HTTP server.
* **Event Batching**: Events are now buffered and flushed periodically
(default: 15 minutes) or on shutdown.
* **Reliability & Rate Limiting**:
* **Server-Side Backoff**: Respects `next_request_wait_millis` from
server responses to handle rate limiting dynamically.
* **Transient Error Retries**: Failed requests (5xx, 429) result in
events being requeued for the next flush.
* **Request Timeouts**: Enforced 30s timeout on requests to prevent
hanging processes.
* **Session Rotation**: Automatically rotates session IDs every 24
hours.
* **Safety & Stability**:
* **Buffer Overflow Protection**: Caps the buffer at 1000 events to
prevent memory leaks, dropping oldest events if necessary.
* **Optimistic Removal**: Prevents race conditions and duplicate events
during shutdown by optimistically removing events from the buffer before
sending.
* **Testing Improvements**:
* **E2E Robustness**: Updated E2E tests to use a mock web server instead
of relying on the logger to log specific lines.
**Implementation Roadmap:**
These changes finalize the planned telemetry architecture:
1. **CLI & Opt-out Mechanism
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/757))**
2. **Logger Scaffolding & Integration
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/758))**
3. **Persistence Layer
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/766))**
4. **Watchdog Process Architecture
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/769))**
5. **Transport, Batching & Retries (This PR):**
* Finalized `ClearcutSender` with HTTP transport, batching, and
server-directed backoff strategies.
---------
Co-authored-by: Alex Rudenko <alexrudenko@chromium.org>
This PR implements the watchdog process architecture for the telemetry
system. It moves the `ClearcutSender` execution to a dedicated child
process, ensuring that events—especially shutdown events—are reliably
transmitted even if the main server process terminates abruptly.
Added an e2e test that runs the server, checks the log file and confirms
the telemetry logs exist and that the watchdog process is correctly
killed after sending the shutdown event once the main process is killed.
**Implementation Roadmap:**
This is the fourth in a series of PRs designed to implement the
telemetry system:
1. **CLI & Opt-out Mechanism
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/757)):**
* Added `--usage-statistics` flag and transparency logging.
2. **Logger Scaffolding & Integration
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/758)):**
* **`ClearcutLogger`**: Implemented the main logging entry point.
* **One-way Data Flow**: Integrated `logToolInvocation` and
`logServerStart` hooks into `main.ts` to capture events.
* **`ClearcutSender`**: Introduced a transport abstraction.
* **Type Definitions**: Added TypeScript definitions for the telemetry
Protocol Buffer messages.
3. **Persistence Layer
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/766)):**
* **`FilePersistence`**: Implemented a local file-based state manager to
persist the `lastActive` timestamp.
* **Daily Active Logic**: Integrated persistence into `ClearcutLogger`
to automatically detect and log `daily_active` events (with
`days_since_last_active` calculation) via `logDailyActiveIfNeeded`.
4. **Watchdog Process Architecture (This PR):**
* **`WatchdogClient`**: Added a client-side wrapper to spawn and
communicate with the watchdog process via `stdin`.
* **`watchdog/main.ts`**: Created the entry point for the watchdog
process. It listens for IPC messages and uses `ClearcutSender` to
transmit events.
* **Reliable Shutdown**: The watchdog monitors the parent process and
guarantees a `shutdown` event is sent when the parent exits or crashes
(detecting `stdin` closure).
* **Refactoring**: Moved `ClearcutSender` to the `watchdog` directory
and updated `ClearcutLogger` to delegate event sending to the
`WatchdogClient`.
5. **Transport, Batching & Retries (Next):**
* Finalize `ClearcutSender` with actual HTTP transport logic, including
event batching and exponential backoff retries.
This PR implements the persistence layer for the telemetry system. It
introduces `FilePersistence` for local state management and integrates
it with `ClearcutLogger` to support "Daily Active" metric. I have
decided not to send `first_time_installation` events since we can deduce
them from `daily active` events where the `days_since_last_active` will
be `-1` for that case.
**Implementation Roadmap:**
This is the third in a series of PRs designed to implement the telemetry
system:
1. **CLI & Opt-out Mechanism
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/757)):**
* Added `--usage-statistics` flag and transparency logging.
2. **Logger Scaffolding & Integration
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/758)):**
* **`ClearcutLogger`**: Implemented the main logging entry point.
* **One-way Data Flow**: Integrated `logToolInvocation` and
`logServerStart` hooks into `main.ts` to capture events.
* **`ClearcutSender`**: Introduced a transport abstraction.
* **Type Definitions**: Added TypeScript definitions for the telemetry
Protocol Buffer messages.
3. **Persistence Layer (This PR):**
* **`FilePersistence`**: Implemented a local file-based state manager to
persist the `lastActive` timestamp.
* **Daily Active Logic**: Integrated persistence into `ClearcutLogger`
to automatically detect and log `daily_active` events (with
`days_since_last_active` calculation) via `logDailyActiveIfNeeded`.
4. **Watchdog Process Architecture (Next):**
* Move `ClearcutSender` execution to a dedicated watchdog process to
ensure reliable event transmission even during abrupt server shutdowns.
5. **Transport, Batching & Retries (Next):**
* Finalize `ClearcutSender` with actual HTTP transport logic, including
event batching and exponential backoff retries.
This PR implements the core logging infrastructure for the telemetry
system. It introduces the `ClearcutLogger` class, integrates it into the
server lifecycle events, and establishes the `ClearcutSender`
abstraction.
**Implementation Roadmap:**
This is the second in a series of PRs designed to implement a robust,
privacy-conscious telemetry system:
1. **CLI & Opt-out Mechanism
([Merged](https://github.com/ChromeDevTools/chrome-devtools-mcp/pull/757)):**
* Added `--usage-statistics` flag and transparency logging.
2. **Logger Scaffolding & Integration (This PR):**
* **`ClearcutLogger`**: Implemented the main logging entry point.
* **One-way Data Flow**: Integrated `logToolInvocation` and
`logServerStart` hooks into `main.ts` to capture events.
* **`ClearcutSender`**: Introduced a transport abstraction (currently a
dummy implementation) that will later handle HTTP requests, batching,
and retries.
* **Type Definitions**: Added TypeScript definitions for the telemetry
Protocol Buffer messages.
3. **Persistence Layer (Next):**
* Implement local state management to reliably track "First Time
Installation" and "Daily Active" metrics.
4. **Watchdog Process Architecture (Next):**
* Move `ClearcutSender` execution to a dedicated watchdog process to
ensure reliable event transmission even during abrupt server shutdowns.
5. **Transport, Batching & Retries (Next):**
* Finalize `ClearcutSender` with actual HTTP transport logic, including
event batching and exponential backoff retries.
Alex Rudenko