# Provider conformance

This harness keeps the services → agents → workflows lifecycle honest across the
supported AI providers. It runs the same end-to-end scenarios against each
configured provider and treats missing provider keys as an explicit skip, so the
suite is safe for local development, forks, and scheduled CI.

## What it exercises

`go run ./internal/harness/provider-conformance` fans out over the harnesses in
`internal/harness`:

- `universe` — service discovery plus agent tool calls over the real runtime.
- `agent-flow` — a workflow event that drives an agent to call services.
- `plan-delegate` — plan persistence plus agent-to-agent delegation and service
  calls.

The command also emits the registered provider capability matrix so the run shows
which providers advertise model, image, video, and streaming support.

## Local usage

Run the deterministic path with no secrets:

```sh
go run ./internal/harness/provider-conformance -providers mock
```

Run every live provider that has a key in the environment:

```sh
go run ./internal/harness/provider-conformance \
  -summary-json provider-conformance-summary.json \
  -summary-markdown provider-conformance-summary.md \
  -capabilities-markdown provider-capabilities.md
```

Provider keys are read from `MICRO_AI_API_KEY` or the provider-specific variable:

| Provider | Secret / environment variable |
| --- | --- |
| Anthropic | `ANTHROPIC_API_KEY` |
| OpenAI | `OPENAI_API_KEY` |
| Gemini | `GEMINI_API_KEY` |
| Groq | `GROQ_API_KEY` |
| Mistral | `MISTRAL_API_KEY` |
| Together | `TOGETHER_API_KEY` |
| AtlasCloud | `ATLASCLOUD_API_KEY` |

Use `-require-configured` when you want a selected provider without a key to fail
instead of skip:

```sh
go run ./internal/harness/provider-conformance \
  -providers anthropic,openai \
  -require-configured
```

## Scheduled CI behavior

The `Harness (E2E)` workflow runs on pushes and pull requests with deterministic
mock LLMs. On the daily schedule and manual dispatch it also runs the live
provider conformance job. That job:

1. reads the provider keys from repository secrets,
2. skips providers whose secrets are absent,
3. fails when any configured provider fails a harness, and
4. uploads JSON and Markdown coverage artifacts for the run.

The job also appends the Markdown summary and capability matrix to the GitHub
Actions step summary, making configured, skipped, and failed provider coverage
visible without downloading artifacts.

## Adding a provider

To bring a new provider into scheduled conformance:

1. register its `ai` provider implementation and capability metadata,
2. add the provider name and key variable to `providerEnv` in `main.go`,
3. import the provider package in `main.go`,
4. pass the matching repository secret through `.github/workflows/harness.yml`,
   and
5. run `go run ./internal/harness/provider-conformance -providers <name> \
   -require-configured` with a live key before opening the change.