#π Note: Circuit Breaker Pattern
#Overview
The Circuit Breaker pattern prevents a system from repeatedly trying to execute an operation that is likely to fail, allowing it to recover gracefully without cascading failures across services.
#Core Idea
Inspired by electrical circuit breakers β when a downstream service is failing, "open the circuit" to stop sending requests, give it time to recover, and serve degraded/fallback responses instead.
#States
ββββββββββββ Failure threshold ββββββββββββ
β CLOSED β ββββββexceededβββββββββΆ β OPEN β
β (normal) β β (failing)β
ββββββββββββ βββββββ¬βββββ
β² β
β βββββββββββββ β
β Success β HALF-OPEN β Timeout β
ββββββββββββββ (testing) βββββββββββββββ
βββββββββββββ
β Failure
ββββββββββββΆ Back to OPEN- CLOSED: Requests flow normally. Failures are counted. If failures exceed threshold β switch to OPEN.
- OPEN: All requests immediately return fallback/error without calling downstream. After a timeout β switch to HALF-OPEN.
- HALF-OPEN: Allow a limited number of test requests through. If they succeed β CLOSED. If they fail β back to OPEN.
#Configuration Parameters
| Parameter | Typical Value | Purpose |
|---|---|---|
| Failure Threshold | 5 failures in 30s | When to trip the circuit |
| Open Timeout | 30β60 seconds | How long to wait before testing recovery |
| Half-Open Max Requests | 3β5 requests | How many test requests to allow |
| Success Threshold | 3 consecutive | How many successes needed to close circuit |
#When to Use
- Microservice Communication: Service A calls Service B; B goes down β A should degrade gracefully, not crash.
- External API Calls: Third-party APIs with unpredictable availability.
- Database Connections: Prevent connection pool exhaustion when DB is overloaded.
- ML Model Serving: If model server is slow/down, fallback to simpler logic (e.g., rules-only scoring).
#Fallback Strategies
- Cache: Serve stale cached data.
- Default Values: Return safe defaults (e.g., empty recommendations, approve low-risk transactions).
- Degraded Mode: Skip the failing component and proceed with partial functionality.
- Queue: Buffer requests for retry when the service recovers.
#Trade-offs
- Pros: Prevents cascade failures, faster failure detection, allows recovery time, improves overall system resilience.
- Cons: Adds complexity, fallback logic must be maintained, may mask underlying issues if not monitored.
#Used In This Repo
- E-Commerce (#11) β Search β Elasticsearch, Catalog β DB circuit breakers
- Recommendation Engine (#23) β Rec Service β Ranking Service fallback to cached/popular recs
- Fraud Detection (#24) β Scoring Service β ML Scorer fallback to rules-only scoring