项目文件夹

文件
Ali Khokhar af12e7b2bb Coordinate provider recovery under concurrent load (#1205)
## Problem

Concurrent transient failures could start independent retry, replay,
continuation, and repair loops while holding provider concurrency slots.
This multiplied upstream attempts and could delay or strand terminal
errors under fan-out.

## Changes

| Before | After |
| --- | --- |
| Retry paths owned separate attempt budgets. | One logical-execution
session caps all upstream work at five attempts. |
| Concurrent failures backed off independently. | One provider-owned
recovery episode elects a single half-open probe while followers
coalesce. |
| Backoff occupied stream concurrency. | Concurrency is held only while
an upstream operation or stream is active. |
| Provider catalog calls and stream creation used separate admission
paths. | Every upstream operation uses one provider-generation admission
controller. |
| Cancellation could leave recovery ownership or follower state
unresolved. | Cancellation releases permits, transfers probe ownership,
and unregisters waiting followers. |
| Late in-flight failures could cross an exhausted episode boundary. |
Every coalesced execution retains that generation's terminal outcome. |
| Replay tests allowed loose lifecycle assertions. | Exact SSE contracts
prove retries and continuations emit one unduplicated response. |
| Recovery wrappers could mask final diagnostics. | Final responses and
traces retain the raw provider failure and request ID. |

<!-- greptile_comment -->

<details open><summary><h3>Greptile Summary</h3></summary>

This PR coordinates provider recovery and retry work under concurrent
load. The main changes are:

- One five-attempt budget for each logical execution.
- Provider-wide recovery episodes with one elected probe.
- Shared admission for streams, catalog calls, rate limits, and
concurrency.
- Concurrency permits held only during active upstream work.
- Cancellation-safe probe ownership and preserved final diagnostics.
</details>

<h3>Confidence Score: 5/5</h3>

This looks safe to merge.

No blocking issues found in the changed code.

<details><summary><h3><a href="https://www.greptile.com/trex"><img
alt="T-Rex"
src="https://greptile-static-assets.s3.amazonaws.com/trex/trex_green.svg"
height="20" align="absmiddle"></a> T-Rex Logs</h3></summary>

**What T-Rex did**
- Reviewed the coordinated-recovery-01-before.log to understand how the
exhausted generation outcome was not preserved in a late in-flight
failure.
- Reviewed the coordinated-recovery-02-after.log to confirm that the
updated implementation preserves the exhausted generation outcome for
the same focused contract set.
- Validated that the provider-admission-full-current.log shows the
complete requested test file passed under Python 3.14 with uv run pytest
-n 0.

<a
href="https://app.greptile.com/trex/runs/15050270/artifacts"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifactsDark.svg?v=4"><source
media="(prefers-color-scheme: light)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifacts.svg?v=4"><img
alt="View all artifacts"
src="https://greptile-static-assets.s3.amazonaws.com/badges/ViewAllArtifacts.svg?v=4"></picture></a>

<sub><a href="https://www.greptile.com/trex"><img alt="T-Rex"
src="https://greptile-static-assets.s3.amazonaws.com/trex/trex_green.svg"
height="14" align="absmiddle"></a> Ran code and verified through
T-Rex</sub>
</details>

<details open><summary><h3>Important Files Changed</h3></summary>

| Filename | Overview |
|----------|----------|
| src/free_claude_code/providers/admission.py | Adds shared admission,
retry budgets, recovery episodes, probe election, and cancellation
handling. |
| src/free_claude_code/providers/openai_chat/provider.py | Moves stream
creation, replay, continuation, and repair onto one admission-owned
retry session. |
| src/free_claude_code/providers/stream_recovery.py | Selects replay,
continuation, repair, or final failure using the remaining shared
attempt budget. |
| src/free_claude_code/providers/failure_policy.py | Adds recovery
exhaustion handling and preserves the underlying provider error for
final classification. |
| src/free_claude_code/providers/runtime/factory.py | Creates one
admission controller per provider generation and passes it through
provider factories. |

</details>

<details open><summary><h3>Sequence Diagram</h3></summary>

<a href="#gh-light-mode-only">

```mermaid
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant E as Execution
participant A as Admission controller
participant P as Provider
participant F as Concurrent follower

E->>A: Open attempt
A->>P: Send upstream request
P-->>E: Retryable failure
E->>A: Open recovery episode
F->>A: Request admission
A-->>F: Coalesce and wait
E->>A: Claim probe
A->>P: Send half-open probe
alt Probe succeeds
    P-->>E: Valid response
    E->>A: Close recovery episode
    A-->>F: Release waiter
else Probe fails
    P-->>E: Retryable failure
    E->>A: Schedule next probe or finalize error
end
```

</a>
<a href="#gh-dark-mode-only">

```mermaid
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant E as Execution
participant A as Admission controller
participant P as Provider
participant F as Concurrent follower

E->>A: Open attempt
A->>P: Send upstream request
P-->>E: Retryable failure
E->>A: Open recovery episode
F->>A: Request admission
A-->>F: Coalesce and wait
E->>A: Claim probe
A->>P: Send half-open probe
alt Probe succeeds
    P-->>E: Valid response
    E->>A: Close recovery episode
    A-->>F: Release waiter
else Probe fails
    P-->>E: Retryable failure
    E->>A: Schedule next probe or finalize error
end
```

</a>
</details>

<sub>Reviews (2): Last reviewed commit: ["Harden coordinated retry
lifecycle
invar..."](https://github.com/alishahryar1/free-claude-code/commit/2e871c8649d148b5eb71d21f80bf870ae2d11708)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=45554917)</sub>

<!-- /greptile_comment -->
2026-07-19 22:43:29 -07:00

220 行
6.7 KiB
Python

import asyncio
import contextlib
import logging
import os
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from free_claude_code.config.settings import Settings
from tests.providers.support import immediate_admission
# Set mock environment BEFORE any imports that use Settings
os.environ.setdefault("NVIDIA_NIM_API_KEY", "test_key")
os.environ.setdefault("MODEL", "nvidia_nim/test-model")
os.environ["PTB_TIMEDELTA"] = "1"
# Ensure tests don't pick up a server API key from the repo .env
# (tests expect endpoints to be unauthenticated by default)
os.environ["ANTHROPIC_AUTH_TOKEN"] = ""
Settings.model_config = {**Settings.model_config, "env_file": None}
@pytest.fixture(autouse=True)
def _isolate_from_dotenv(monkeypatch):
"""Prevent Pydantic BaseSettings from reading the .env file during tests."""
monkeypatch.setattr(
Settings, "model_config", {**Settings.model_config, "env_file": None}
)
@pytest.fixture
def provider_config():
from free_claude_code.providers.base import ProviderConfig
return ProviderConfig(
api_key="test_key",
base_url="https://test.api.nvidia.com/v1",
rate_limit=10,
rate_window=60,
)
@pytest.fixture
def nim_provider(provider_config):
from free_claude_code.config.nim import NimSettings
from free_claude_code.providers.nvidia_nim import NvidiaNimProvider
return NvidiaNimProvider(
provider_config,
nim_settings=NimSettings(),
admission=immediate_admission(),
)
@pytest.fixture
def open_router_provider(provider_config):
from free_claude_code.providers.open_router import OpenRouterProvider
return OpenRouterProvider(provider_config, admission=immediate_admission())
@pytest.fixture
def lmstudio_provider(provider_config):
from free_claude_code.providers.base import ProviderConfig
from free_claude_code.providers.lmstudio import LMStudioProvider
lmstudio_config = ProviderConfig(
api_key="lm-studio",
base_url="http://localhost:1234/v1",
rate_limit=provider_config.rate_limit,
rate_window=provider_config.rate_window,
)
return LMStudioProvider(lmstudio_config, admission=immediate_admission())
@pytest.fixture
def llamacpp_provider(provider_config):
from free_claude_code.providers.base import ProviderConfig
from free_claude_code.providers.openai_chat import create_openai_chat_provider
llamacpp_config = ProviderConfig(
api_key="llamacpp",
base_url="http://localhost:8080/v1",
rate_limit=10,
rate_window=60,
)
return create_openai_chat_provider(
"llamacpp",
llamacpp_config,
immediate_admission(),
)
@pytest.fixture
def mock_cli_session():
from free_claude_code.messaging.managed_protocols import (
ManagedClaudeSessionProtocol,
)
session = MagicMock(spec=ManagedClaudeSessionProtocol)
session.start_task = MagicMock() # This will return an async generator
session.is_busy = False
return session
@pytest.fixture
def mock_cli_manager():
from free_claude_code.messaging.managed_protocols import (
ManagedClaudeSessionManagerProtocol,
)
manager = MagicMock(spec=ManagedClaudeSessionManagerProtocol)
manager.get_or_create_session = AsyncMock()
manager.register_real_session_id = AsyncMock(return_value=True)
manager.stop_all = AsyncMock()
manager.remove_session = AsyncMock(return_value=True)
manager.get_stats = MagicMock(return_value={"active_sessions": 0})
return manager
@pytest.fixture
def mock_platform():
from free_claude_code.messaging.platforms.ports import OutboundMessenger
platform = MagicMock(spec=OutboundMessenger)
platform.send_message = AsyncMock(return_value="msg_123")
platform.edit_message = AsyncMock()
platform.delete_message = AsyncMock()
platform.queue_send_message = AsyncMock(return_value="msg_123")
platform.queue_edit_message = AsyncMock()
platform.queue_delete_messages = AsyncMock()
platform.cancel_pending_voice = AsyncMock(return_value=None)
platform.cancel_all_pending_voices = AsyncMock(return_value=())
platform.cancel_pending_voices_in_scope = AsyncMock(return_value=())
def _fire_and_forget(task):
if asyncio.iscoroutine(task):
# Create a task to avoid "coroutine was never awaited" warning
return asyncio.create_task(task)
return None
platform.fire_and_forget = MagicMock(side_effect=_fire_and_forget)
return platform
@pytest.fixture
def mock_session_store():
from free_claude_code.messaging.session import SessionStore
store = MagicMock(spec=SessionStore)
store.save_tree = MagicMock()
store.get_tree = MagicMock(return_value=None)
store.register_node = MagicMock()
store.record_message_id = MagicMock()
store.get_tracked_message_ids_for_chat = MagicMock(return_value=[])
store.forget_tracked_message_ids = MagicMock()
store.clear_scope = MagicMock()
return store
@pytest.fixture
def incoming_message_factory():
_valid_keys = frozenset(
{
"text",
"chat_id",
"user_id",
"message_id",
"platform",
"reply_to_message_id",
"message_thread_id",
"username",
"timestamp",
"raw_event",
"status_message_id",
}
)
def _create(**kwargs):
from free_claude_code.messaging.models import IncomingMessage
defaults: dict[str, Any] = {
"text": "hello",
"chat_id": "chat_1",
"user_id": "user_1",
"message_id": "msg_1",
"platform": "telegram",
}
defaults.update(kwargs)
if "timestamp" in defaults and isinstance(defaults["timestamp"], str):
from datetime import datetime
defaults["timestamp"] = datetime.fromisoformat(defaults["timestamp"])
filtered = {k: v for k, v in defaults.items() if k in _valid_keys}
return IncomingMessage(**filtered)
return _create
@pytest.fixture(autouse=True)
def _propagate_loguru_to_caplog():
"""Route loguru logs to stdlib logging so pytest caplog captures them."""
from loguru import logger as loguru_logger
class _PropagateHandler:
def write(self, message):
record = message.record
level = record["level"].no
stdlib_level = min(level, logging.CRITICAL)
py_logger = logging.getLogger(record["name"])
py_logger.log(stdlib_level, record["message"])
handler_id = loguru_logger.add(_PropagateHandler(), format="{message}")
yield
with contextlib.suppress(ValueError):
loguru_logger.remove(
handler_id
) # Handler already removed (e.g. by test_logging_config)