/registry/etcd/PERFORMANCE.md

Etcd Registry Performance Improvements

This document describes the improvements made to address etcd authentication performance issues and cache penetration problems.

Problem Statement

Background

When etcd server authentication is enabled, a serious performance bottleneck can occur at scale. This was observed in production environments with 4000+ service pods.

Issues Identified

1. High Authentication QPS

• Root Cause: The etcd registry used KeepAliveOnce for lease renewal, which requires a new authentication request for each call
• Impact: With 4000+ pods registering every 30s (default RegisterInterval), this creates ~110 QPS of authentication requests
• Limitation: A typical 3-node etcd cluster (64C 256G HDD) can only handle ~100 QPS for authentication
• Result: Authentication requests overwhelm etcd, causing KeepAlive failures and service deregistrations

2. Cache Penetration

• Trigger: When KeepAlive fails, services deregister from etcd
• Chain Reaction:
  1. Registry watcher detects deletions
  2. Cache is cleared based on delete events
  3. All subsequent service lookups hit etcd directly (cache miss)
  4. Etcd is already overloaded, causing more failures
• Result: Cascading failure where all gRPC requests fail

Solution

1. Use Long-Lived KeepAlive Channels

Change: Replaced KeepAliveOnce with KeepAlive

Implementation:

• Added keepalive channel management to etcdRegistry struct
• Created startKeepAlive() method that establishes a long-lived keepalive stream
• Modified registerNode() to reuse existing keepalive channels
• Added stopKeepAlive() for proper cleanup on deregistration

Benefits:

• 97% reduction in auth requests: From ~110 QPS to ~3-4 QPS (4000 pods / TTL period)
• Single authentication per lease: KeepAlive authenticates once when establishing the stream
• Automatic renewal: Etcd sends keepalive responses automatically through the channel

Code Changes:

// Before: New auth request every heartbeat
if _, err := e.client.KeepAliveOnce(context.TODO(), leaseID); err != nil {
    // handle error
}

// After: Single auth request, reused channel
if err := e.startKeepAlive(s.Name+node.Id, leaseID); err != nil {
    // handle error
}

2. Verify Cache Penetration Protection

Existing Protection: The registry cache already uses singleflight pattern to prevent stampede

How it Works:

• When cache expires/is empty, first request triggers etcd query
• Concurrent requests for same service wait for the first request to complete
• All waiting requests receive the same result
• Only ONE etcd query happens regardless of concurrent request count

Additional Safety:

• Stale cache is returned when etcd fails (if cache data exists)
• Prevents cascading failures by avoiding repeated failed requests to etcd

Verification: Added comprehensive tests to confirm this behavior works correctly under load.

Performance Impact

Authentication Load Reduction

• Before: 4000 pods × (1 auth / 30s) = ~133 auth/sec
• After: 4000 pods × (1 auth / lease_ttl) ≈ 3-4 auth/sec (assuming 15min lease TTL)
• Reduction: ~97%

Cache Penetration Prevention

• Before: When cache clears, 1000s of concurrent requests → 1000s of etcd queries
• After: When cache clears, 1000s of concurrent requests → 1 etcd query (singleflight)
• Reduction: ~99.9%

Testing

Unit Tests

1. TestKeepAliveManagement: Validates keepalive lifecycle

   • Verifies channels are created on registration
   • Confirms channels are cleaned up on deregistration

2. TestKeepAliveReducesAuthRequests: Confirms channel reuse

   • Multiple re-registrations use the same keepalive channel
   • Validates auth request reduction

3. TestKeepAliveChannelReconnection: Tests error handling

   • Verifies proper cleanup when keepalive channel closes

4. TestSingleflightPreventsStampede: Validates cache behavior

   • 10 concurrent requests → 1 etcd query

5. TestStaleCacheOnError: Confirms graceful degradation

   • Returns stale cache when etcd fails

6. TestCachePenetrationPrevention: End-to-end validation

   • 50 concurrent requests during etcd failure → 1 etcd query
   • All requests receive stale cache

Integration Tests

• CI workflow runs tests against real etcd instance
• Validates behavior with actual etcd keepalive channels
• Tests run with race detector enabled

Migration Guide

For Library Users

No code changes required! The improvements are transparent:

• Existing applications automatically benefit from reduced auth load
• No API changes to registry.Registry interface

For Plugin Developers

If you maintain a custom registry plugin:

• Consider implementing long-lived keepalive channels
• Ensure your cache implementation uses singleflight pattern
• Add tests for concurrent access patterns

Monitoring Recommendations

Key Metrics to Track

1. Etcd Authentication Rate: Should drop by ~97%
2. Etcd Query Rate: Monitor for stampede prevention
3. Service Registration Success Rate: Should improve under load
4. Cache Hit Rate: Should remain high even during etcd issues

Expected Behavior

• Normal Operation: Low auth QPS, high cache hit rate
• During Etcd Issues: Stale cache served, limited etcd queries
• After Recovery: Cache refreshes gradually, no stampede

Related Issues

• Original Issue: [BUG] etcd authentication performance issue and registry cache penetration
• Etcd Documentation: https://etcd.io/docs/latest/learning/api/#lease-keepalive
• Singleflight Pattern: https://pkg.go.dev/golang.org/x/sync/singleflight

Security Considerations

• No Authentication Bypass: Changes only reduce frequency, not security
• Proper Cleanup: Keepalive channels properly closed on deregistration
• Race Condition Free: All map operations properly synchronized
• No Resource Leaks: Goroutines terminate when channels close

Future Enhancements

Potential improvements for consideration:

1. Adaptive TTL: Adjust keepalive frequency based on load
2. Circuit Breaker: Temporarily stop queries when etcd is degraded
3. Metrics: Expose keepalive channel count, auth rate, etc.
4. Backoff: Exponential backoff on keepalive failures