* Initial plan
* Enhance cache rate limiting to protect against rolling deployment scenarios
Per @asim's feedback, rate limiting now applies to ALL cache refresh attempts
(not just error cases) to prevent registry overload during rolling deployments.
Changes:
- Rate limit ALL refresh attempts using lastRefreshAttempt tracking
- Always return stale cache if available (even if expired) when rate limiting
- Only return ErrNotFound when no cache exists during rate limit period
- Rate limiting happens inside singleflight to avoid race conditions
- Update test to reflect new behavior (no retry when stale cache + rate limit)
- Enhanced documentation with rolling deployment scenario examples
This addresses the scenario where all upstream services expire their cache
simultaneously during a downstream rolling deployment, which would previously
cause a stampede to the registry under high QPS.
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Remove unused failedAttempts and consecutiveFailures fields
Per @asim's feedback, these fields are no longer needed since the new
rate limiting strategy uses lastRefreshAttempt to track ALL refresh
attempts, not just failed ones.
Removed:
- failedAttempts map[string]time.Time
- consecutiveFailures int
- All code that sets these fields
The rate limiting logic now only uses lastRefreshAttempt, making these
fields redundant. All tests continue to pass.
Co-authored-by: asim <17530+asim@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Initial plan
* Add adaptive throttling to prevent cache penetration without stale cache
Implement rate limiting mechanism to protect etcd when:
- Cache is empty (no stale data available)
- Registry is failing/overloaded
- Multiple sequential requests arrive
Changes:
- Add MinimumRetryInterval option (default 5s)
- Track last failed attempt time per service
- Throttle requests within retry interval when no cache exists
- Clear throttling on successful lookup
- Add comprehensive tests validating throttling behavior
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Fix WaitGroup race condition in events stream tests
The WaitGroup.Add() calls were happening after goroutines started,
causing a race where Done() could be called before Add(), resulting
in "negative WaitGroup counter" panics.
Fixes:
- Move wg.Add(1) before goroutine in TestConsumeTopic
- Move wg.Add(2) before first goroutine in TestConsumeGroup
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Update registry cache documentation with adaptive throttling feature
Document the new adaptive throttling feature that prevents cache
penetration when registry is failing and no stale cache exists.
Includes:
- Feature overview
- Usage examples with configuration options
- Explanation of throttling behavior
- Example scenario demonstrating protection
Co-authored-by: asim <17530+asim@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Initial plan
* Replace KeepAliveOnce with KeepAlive to reduce etcd auth requests
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Add tests to verify cache penetration protection via singleflight
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Add etcd integration tests to CI workflow
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Fix race conditions and improve code quality based on review feedback
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Add workflow permissions to fix security scan finding
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Add comprehensive documentation for performance improvements
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Fix memory store limit/offset bug causing events test failure
Co-authored-by: asim <17530+asim@users.noreply.github.com>
* Fix memory store to filter before limiting in prefix/suffix reads
The previous fix had a logic error where limit/offset were applied before
prefix/suffix filtering. This could cause incorrect results when the first
N items in the unfiltered list don't match the search criteria.
Now filters first to get all matching keys, then applies limit/offset to
the filtered results. This ensures ReadLimit(1) always returns 1 matching
record if available, regardless of map iteration order.
Co-authored-by: asim <17530+asim@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: asim <17530+asim@users.noreply.github.com>
Copilot