📁 topics / 24-fraud-detection.md

#Design a Fraud Detection System (PayPal / Stripe / Banking Scale)

#1. Problem Statement & Clarifications

A Fraud Detection System is responsible for identifying and preventing fraudulent transactions in real-time across payment platforms, banking systems, and e-commerce. It must analyze every transaction as it occurs, assign a risk score, and decide whether to approve, flag for review, or block — all within milliseconds to avoid degrading the legitimate user experience.

#Functional Requirements

  1. Real-Time Transaction Scoring — Evaluate every transaction in < 100ms with a fraud risk score (0–1000) and a decision (APPROVE / REVIEW / BLOCK)
  2. Rule Engine — Support configurable business rules (e.g., "block transactions > $10K from new accounts", "flag if 5+ transactions in 1 minute")
  3. ML-Based Detection — Use trained ML models to detect complex fraud patterns (account takeover, synthetic identity, money laundering rings)
  4. Case Management — Flagged transactions routed to human analysts for manual review with all context
  5. Feedback Loop — Analyst decisions (confirmed fraud / false positive) feed back into model retraining
  6. Entity Profiling — Maintain real-time risk profiles for users, devices, IPs, merchants, and payment instruments

#Non-Functional Requirements

Requirement Target
Latency P99 < 100ms for real-time scoring (inline with payment flow)
Availability 99.999% — downtime means either fraud losses or blocked legitimate payments
Scale 50K transactions/sec (peak), 2B transactions/day
False Positive Rate < 1% (blocking legitimate users destroys trust)
Fraud Detection Rate > 95% of known fraud patterns caught
Consistency Strong consistency for blocklists; eventual for profile updates

#Out of Scope

#Assumptions

#2. Back-of-Envelope Estimation

#Traffic Estimates

Daily transactions:     2B
Avg TPS:                2B / 86400 ≈ 23K TPS
Peak TPS (3x):          ~70K TPS (Black Friday, holiday sales)
Flash peak (10x):       ~230K TPS (first seconds of mega-sale events)
Flagged for review:     ~0.5% = 10M/day
Confirmed fraud:        ~0.1% = 2M/day

#Storage Estimates

Transaction records:    2B/day × 2KB (enriched) = 4TB/day
Hot storage (90 days):  4TB × 90 = 360TB
Entity profiles:        500M users × 5KB = 2.5TB
Device fingerprints:    1B devices × 1KB = 1TB
Rule audit logs:        2B/day × 200B = 400GB/day
ML feature vectors:     2B/day × 500B = 1TB/day (retained 30 days = 30TB)
Total hot storage:      ~395TB

#Bandwidth

Ingress (transactions): 70K × 1KB = 70 MB/s (peak)
Scoring response:       70K × 200B = 14 MB/s
Feature enrichment:     70K × 5KB (profile lookups) = 350 MB/s internal

#Latency Budget Breakdown

Total budget:           100ms
├── Network + API GW:   5ms
├── Feature extraction: 15ms (parallel lookups to Redis/profile store)
├── Rule engine:        10ms (evaluate 500 rules)
├── ML scoring:         30ms (model inference)
├── Decision logic:     5ms (combine rule + ML scores)
├── Persist + respond:  10ms (async write, sync response)
└── Buffer:             25ms

#Cache Estimates

Entity profiles (hot):  50M active users × 5KB = 250GB (Redis cluster)
Blocklists:             10M entries × 100B = 1GB (replicated to all nodes)
Velocity counters:      50M users × 200B (count/sum/avg windows) = 10GB
Cache hit ratio target: 95%+ for profile lookups

#3. API Design

#Transaction Scoring API (Synchronous — inline with payment)

POST /api/v1/transactions/score
{
  "transaction_id": "TXN_abc123",
  "timestamp": "2025-05-29T12:00:00Z",
  "amount": 1499.99,
  "currency": "USD",
  "transaction_type": "PURCHASE",

  "payer": {
    "user_id": "U123",
    "email": "user@example.com",
    "phone": "+1234567890",
    "account_age_days": 45
  },

  "payee": {
    "merchant_id": "M456",
    "merchant_category": "electronics",
    "merchant_country": "US"
  },

  "payment_instrument": {
    "type": "CREDIT_CARD",
    "card_hash": "sha256_xxxx",   // tokenized, never raw PAN
    "issuer_bank": "Chase",
    "bin_country": "US"
  },

  "device": {
    "fingerprint_id": "DF_789",
    "ip_address": "203.0.113.42",
    "user_agent": "Mozilla/5.0...",
    "geo_location": {"lat": 37.77, "lng": -122.42}
  },

  "session": {
    "session_id": "SESS_xyz",
    "login_method": "password",
    "time_since_login_sec": 120
  }
}

Response 200 (< 100ms):
{
  "transaction_id": "TXN_abc123",
  "decision": "APPROVE",           // APPROVE | REVIEW | BLOCK
  "risk_score": 142,               // 0-1000 (higher = riskier)
  "risk_level": "LOW",             // LOW | MEDIUM | HIGH | CRITICAL
  "triggered_rules": [],
  "ml_scores": {
    "account_takeover": 0.02,
    "payment_fraud": 0.08,
    "money_laundering": 0.01
  },
  "requires_3ds": false,           // step-up authentication
  "request_id": "req_001"
}

#Report Fraud API (Analyst / User feedback)

POST /api/v1/transactions/{transaction_id}/feedback
{
  "reporter": "analyst_A1",
  "verdict": "CONFIRMED_FRAUD",    // CONFIRMED_FRAUD | FALSE_POSITIVE | SUSPICIOUS
  "fraud_type": "account_takeover",
  "notes": "Device fingerprint mismatch + geo anomaly",
  "evidence": ["screenshot_url_1"]
}

Response 200: { "status": "recorded", "case_id": "CASE_456" }

#Rule Management API (Internal — fraud ops team)

POST /api/v1/rules
{
  "rule_id": "R_100",
  "name": "high_value_new_account",
  "condition": "txn.amount > 5000 AND payer.account_age_days < 7",
  "action": "REVIEW",
  "priority": 1,
  "enabled": true,
  "expires_at": null
}

GET /api/v1/rules              → list all active rules
PUT /api/v1/rules/{rule_id}    → update rule
DELETE /api/v1/rules/{rule_id} → disable rule

#Entity Blocklist API

POST /api/v1/blocklist
{
  "entity_type": "DEVICE",        // USER | DEVICE | IP | CARD_HASH | EMAIL
  "entity_value": "DF_789",
  "reason": "confirmed_fraud_ring",
  "blocked_by": "analyst_A1",
  "ttl_hours": 8760              // 1 year; null = permanent
}

#4. Data Model

#Transaction Store (Cassandra — high write throughput, time-series access)

Partition Key: payer_user_id
Sort Key:      timestamp (descending)

{
  transaction_id:   "TXN_abc123",
  payer_user_id:    "U123",
  payee_merchant_id:"M456",
  amount:           1499.99,
  currency:         "USD",
  risk_score:       142,
  decision:         "APPROVE",
  triggered_rules:  ["R_100"],
  ml_scores:        {"ato":0.02, "payment":0.08},
  device_fp:        "DF_789",
  ip_address:       "203.0.113.42",
  geo:              {"lat":37.77,"lng":-122.42},
  timestamp:        1716984000,
  feedback:         null
}

#Entity Profile Store (Redis + PostgreSQL backing)

Redis Key: entity:user:{user_id}
{
  "lifetime_txn_count":     1240,
  "lifetime_txn_sum":       89432.50,
  "avg_txn_amount":         72.12,
  "txn_count_1h":           3,
  "txn_count_24h":          8,
  "txn_sum_24h":            540.00,
  "distinct_devices_7d":    2,
  "distinct_ips_24h":       1,
  "distinct_merchants_7d":  5,
  "last_txn_ts":            1716984000,
  "last_txn_geo":           {"lat":37.77,"lng":-122.42},
  "risk_tier":              "LOW",
  "fraud_history_count":    0
}
TTL: None (persistent, updated on every transaction)

Redis Key: entity:device:{fingerprint_id}
{
  "first_seen":             1710000000,
  "user_ids":               ["U123","U456"],  // multi-account detection
  "txn_count_24h":          5,
  "fraud_assoc_count":      0
}

Redis Key: entity:ip:{ip_hash}
{
  "geo_country":            "US",
  "is_vpn":                 false,
  "is_tor":                 false,
  "is_datacenter":          true,
  "user_ids_24h":           ["U123"],
  "txn_count_1h":           2
}

#Blocklist Store (Redis — replicated to all scoring nodes)

Redis Set: blocklist:device   → {"DF_789", "DF_012", ...}
Redis Set: blocklist:ip       → {"hash_ip1", "hash_ip2", ...}
Redis Set: blocklist:card     → {"sha256_card1", ...}
Redis Set: blocklist:email    → {"hash_email1", ...}
Redis Set: blocklist:user     → {"U999", ...}

#Rule Store (PostgreSQL + in-memory cache)

CREATE TABLE fraud_rules (
  rule_id       VARCHAR(50) PRIMARY KEY,
  name          VARCHAR(200),
  condition_dsl TEXT,                -- parsed DSL: "txn.amount > 5000 AND payer.account_age_days < 7"
  action        ENUM('APPROVE','REVIEW','BLOCK','STEP_UP'),
  priority      INT,                 -- lower = higher priority
  enabled       BOOLEAN DEFAULT true,
  created_by    VARCHAR(100),
  created_at    TIMESTAMP,
  updated_at    TIMESTAMP,
  expires_at    TIMESTAMP NULL
);

#Case Management Store (PostgreSQL)

CREATE TABLE fraud_cases (
  case_id         BIGSERIAL PRIMARY KEY,
  transaction_id  VARCHAR(50),
  user_id         VARCHAR(50),
  risk_score      INT,
  assigned_to     VARCHAR(100) NULL,
  status          ENUM('OPEN','IN_REVIEW','CONFIRMED_FRAUD','FALSE_POSITIVE','ESCALATED'),
  fraud_type      VARCHAR(50) NULL,
  notes           TEXT,
  created_at      TIMESTAMP,
  resolved_at     TIMESTAMP NULL
);
CREATE INDEX idx_cases_status ON fraud_cases(status, created_at);
CREATE INDEX idx_cases_analyst ON fraud_cases(assigned_to, status);

#ML Feature Store (Redis + offline in HDFS/S3)

Redis Key: ml:features:{transaction_id}
{
  // Pre-computed at scoring time, stored for model training feedback
  "user_velocity_1h":      3,
  "user_velocity_24h":     8,
  "amount_zscore":         1.2,       // how many std devs from user's avg
  "geo_distance_km":       0.5,       // from last known location
  "device_age_days":       120,
  "is_new_merchant":       false,
  "time_since_last_txn_s": 3600,
  "session_risk_score":    0.15
}
TTL: 30 days

#Access Patterns

Query Store Index/Key
Score a transaction (real-time) Redis entity profiles + blocklists Key: entity:{type}:{id}
Get user transaction history Cassandra PK: user_id, SK: timestamp DESC
Lookup blocklist membership Redis set SISMEMBER blocklist:{type}
Get active rules PostgreSQL + in-memory Cached in JVM/process memory
Query fraud cases for analyst PostgreSQL IX: status + created_at
Fetch ML features for training HDFS / S3 Batch export from Redis + Cassandra
Aggregate fraud metrics ClickHouse / Druid Time-series aggregation

#5. High-Level Design (HLD) & Scale Evolution

#Stage 1: MVP / Startup Scale (1K TPS, 10M transactions/month)

#Stage 2: Growth Scale (10K TPS, 500M transactions/month)

#Stage 3: PayPal/Stripe Scale (70K TPS, 2B transactions/day)

This is the target architecture — real-time scoring pipeline with parallel rule + ML evaluation.

#Architecture Diagram

                    ┌──────────────────┐
                    │   API Gateway     │
                    │ (Rate limit, mTLS)│
                    └────────┬─────────┘
                             │
                    ┌────────▼─────────┐
                    │  Scoring Service  │ ← Orchestrator (< 100ms total)
                    │  (Stateless pods) │
                    └────────┬─────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
              ▼              ▼              ▼
     ┌────────────┐  ┌─────────────┐  ┌──────────────┐
     │  Feature    │  │  Blocklist  │  │  (Parallel)  │
     │  Enrichment │  │  Check      │  │              │
     │  Service    │  │  (Redis)    │  │  ┌────────┐  │
     │             │  │             │  │  │ Rule   │  │
     │ • User prof │  │ • O(1) SET  │  │  │ Engine │  │
     │ • Device    │  │   lookups   │  │  └────────┘  │
     │ • IP/Geo    │  │ • Instant   │  │  ┌────────┐  │
     │ • Velocity  │  │   BLOCK if  │  │  │ ML     │  │
     └──────┬─────┘  │   matched   │  │  │ Scorer │  │
            │        └─────────────┘  │  └────────┘  │
            ▼                         └──────┬───────┘
     ┌─────────────┐                         │
     │  Redis       │                        ▼
     │  Profile +   │               ┌────────────────┐
     │  Velocity    │               │ Decision Engine │
     │  Store       │               │ (combine scores │
     └─────────────┘               │  → final action)│
                                    └────────┬───────┘
                                             │
                    ┌────────────────────────┼────────────────┐
                    ▼                        ▼                ▼
           ┌──────────────┐       ┌──────────────┐   ┌──────────────┐
           │  Response to  │       │  Kafka       │   │  Case Mgmt   │
           │  Payment Svc  │       │  Event Bus   │   │  (if REVIEW) │
           │  (sync)       │       │              │   │              │
           └──────────────┘       └──────┬───────┘   └──────────────┘
                                         │
                         ┌───────────────┼────────────────┐
                         ▼               ▼                ▼
                  ┌────────────┐  ┌────────────┐  ┌──────────────┐
                  │ Profile    │  │ Transaction│  │ ML Training  │
                  │ Updater    │  │ Store      │  │ Pipeline     │
                  │ (Flink)    │  │ (Cassandra)│  │ (Spark)      │
                  └────────────┘  └────────────┘  └──────────────┘

#Component Breakdown

Component Responsibility Tech Choice
API Gateway mTLS, rate limiting, request routing Kong / Envoy
Scoring Service Orchestrates enrichment → evaluation → decision Go microservice (low GC pause)
Feature Enrichment Parallel fetch of entity profiles + velocity counters Redis lookups (< 5ms)
Rule Engine Evaluate 500 business rules against enriched transaction In-process DSL evaluator (Rete algorithm)
ML Scorer Run fraud detection models (ATO, payment fraud, AML) TensorFlow Serving / ONNX Runtime
Decision Engine Combine rule + ML scores → final APPROVE/REVIEW/BLOCK In-process logic
Blocklist Service O(1) lookups for known bad entities Redis sets, replicated
Profile Updater Update entity profiles from transaction events Apache Flink consumer from Kafka
Case Management Route REVIEW transactions to analysts PostgreSQL + Web UI
ML Training Pipeline Retrain models daily with feedback labels Spark + XGBoost/PyTorch

#Data Flow

Real-Time Scoring Path (< 100ms):

Payment Service → API GW → Scoring Service
  → 1. Blocklist check (Redis SISMEMBER) → instant BLOCK if hit
  → 2. Feature Enrichment (parallel Redis reads):
       • User profile, device profile, IP profile
       • Velocity counters (txn count/sum in 1h, 24h, 7d windows)
       • Geo distance from last known location
  → 3. Parallel execution:
       a. Rule Engine: evaluate all active rules → list of triggered rules
       b. ML Scorer: compute fraud probability per fraud type
  → 4. Decision Engine: combine rule actions + ML scores → final decision
  → 5. Respond to Payment Service (sync)
  → 6. Publish event to Kafka (async) → profile update + persist + audit

Feedback Loop Path:

Analyst reviews case → marks CONFIRMED_FRAUD or FALSE_POSITIVE
  → Update case in PostgreSQL
  → Publish feedback event to Kafka
  → Consumer: update entity risk_tier in Redis
  → Consumer: label transaction in training dataset
  → Daily batch: retrain ML model → deploy via champion/challenger

#6. Deep Dive — Core Components

#Rule Engine — Detailed Design

Why a dedicated rule engine? Rules provide instant response to emerging fraud patterns (new attack vector → deploy rule in minutes, vs. days for ML model retrain).

Rule DSL Examples:

RULE high_value_new_account:
  WHEN txn.amount > 5000 AND payer.account_age_days < 7
  THEN REVIEW

RULE velocity_burst:
  WHEN velocity.user.txn_count_1h > 10 OR velocity.user.txn_sum_1h > 5000
  THEN BLOCK

RULE geo_impossible_travel:
  WHEN geo.distance_from_last_txn_km > 1000
       AND time.since_last_txn_sec < 3600
  THEN REVIEW

RULE device_account_hopping:
  WHEN device.distinct_users_24h > 3
  THEN BLOCK

RULE cross_border_mismatch:
  WHEN payment.bin_country != payer.country
       AND txn.amount > 1000
  THEN STEP_UP  // require 3DS authentication

Rule Evaluation with Rete Algorithm:

Rule Versioning & Rollback:

#ML Scoring Service — Detailed Design

Multi-Model Architecture:

┌─────────────────────────────────────────────────┐
│              ML Scoring Pipeline                 │
│                                                   │
│  Model 1: Account Takeover (ATO)                 │
│  → Features: login recency, device change,       │
│    IP anomaly, behavioral biometrics             │
│  → Output: P(ATO) = 0.02                        │
│                                                   │
│  Model 2: Payment Fraud                          │
│  → Features: amount z-score, merchant category,  │
│    velocity, card-not-present indicators         │
│  → Output: P(payment_fraud) = 0.08              │
│                                                   │
│  Model 3: Money Laundering (AML)                 │
│  → Features: transaction graph patterns,         │
│    structuring detection, beneficiary risk        │
│  → Output: P(AML) = 0.01                        │
│                                                   │
│  Ensemble: weighted_score = w1*ATO + w2*PF + w3*AML │
│  → Final risk_score = 142 / 1000                 │
└─────────────────────────────────────────────────┘

Champion/Challenger Deployment:

Traffic split:
  Champion model (v3.1): 95% of traffic → production decisions
  Challenger model (v3.2): 5% of traffic → shadow scoring (logged, not acted upon)

After 7 days:
  Compare precision, recall, F1 on labeled data
  If challenger outperforms → promote to champion
  If not → discard and iterate

Feature Engineering (Key Signals):

Feature Category Examples Signal
Velocity txn_count_1h, txn_sum_24h, distinct_merchants_7d Burst patterns indicate compromised account
Deviation amount_zscore, time_of_day_unusual, new_merchant_flag Anomalous behavior vs user's baseline
Graph shared_device_users, shared_ip_users, payment_ring_score Fraud rings share devices/IPs
Geo distance_from_last_txn, is_vpn, country_mismatch Impossible travel, anonymization
Device device_age_days, is_emulator, browser_fingerprint_change Spoofed or new devices
Behavioral typing_speed, mouse_movement_entropy, session_duration Bot vs human detection

#Velocity Counter Service — Detailed Design

Redis-Based Sliding Windows:

For each user, maintain windowed counters using Redis sorted sets:

ZADD velocity:txn_count:U123 <timestamp> <txn_id>
  → ZRANGEBYSCORE to count transactions in last 1h/24h/7d

HINCRBY velocity:txn_sum:U123:24h <amount>
  → HyperLogLog for distinct merchants/IPs

Alternatively: Redis Streams + Flink for exact windowed aggregations

Why not just SQL? At 70K TPS, issuing SELECT COUNT(*) queries against a transaction table would add 50ms+ latency. Redis sorted sets give O(log n) range queries in < 1ms.

#Scaling Strategy

Component Strategy
Scoring Service Stateless; horizontal scale behind LB; 200+ pods at peak
Redis Profiles Cluster mode, 250GB+ across 50+ nodes; read replicas for scoring reads
Redis Blocklists Replicated to every scoring node's local memory for O(1) latency
Rule Engine In-process (no network hop); rules loaded from PostgreSQL on startup + Kafka config updates
ML Scorer CPU-optimized models (XGBoost/ONNX); GPU only for deep learning models; batch scoring of concurrent requests
Kafka Partition by user_id; 1000+ partitions for parallel consumption
Cassandra Sharded by user_id; TTL for auto-cleanup; 90-day hot, cold archive to S3

#Caching Strategy

┌──────────────────────────────────────────────────────┐
│ Layer 1: Blocklists (Local in-process memory)         │
│ • Replicated from Redis on startup + Kafka updates   │
│ • < 0.1ms lookup; no network hop                     │
│ • Updated via pub/sub within seconds of blocklist add│
├──────────────────────────────────────────────────────┤
│ Layer 2: Entity Profiles (Redis Cluster)              │
│ • User, device, IP profiles (no TTL)                 │
│ • Updated on every transaction (read-modify-write)   │
│ • 95%+ hit rate; cold profiles loaded from PostgreSQL │
├──────────────────────────────────────────────────────┤
│ Layer 3: Rules (In-process, Rete network)             │
│ • Compiled on startup; hot-reloaded via Kafka topic  │
│ • Zero-latency evaluation after compilation          │
├──────────────────────────────────────────────────────┤
│ Layer 4: ML Models (In-process / sidecar)             │
│ • Models loaded into memory (50–200MB each)          │
│ • Swapped atomically on new version deployment       │
└──────────────────────────────────────────────────────┘

#Consistency Models

Data Model Rationale
Blocklists Strongly consistent (< 1s propagation) Blocking a fraudster must take effect immediately across all nodes
Entity profiles Eventually consistent (< 5s) Slight staleness acceptable; velocity counters are approximate
Rules Eventually consistent (< 10s) Rule updates propagated via Kafka; brief inconsistency across pods is acceptable
ML models Eventually consistent (minutes) Rolling deployment; brief model version skew during rollout is fine
Transaction records Eventually consistent Async write to Cassandra; scoring response doesn't depend on persistence

#7. Low-Level Design — Core Functionality

#Key Algorithms

1. Scoring Pipeline (Pseudocode)

def score_transaction(txn):
    # Step 1: Instant blocklist check
    if is_blocklisted(txn.payer.user_id, "user") or \
       is_blocklisted(txn.device.fingerprint_id, "device") or \
       is_blocklisted(txn.payment.card_hash, "card"):
        return Decision(action="BLOCK", score=1000, reason="blocklist_hit")

    # Step 2: Parallel feature enrichment
    futures = [
        async_fetch(f"entity:user:{txn.payer.user_id}"),
        async_fetch(f"entity:device:{txn.device.fingerprint_id}"),
        async_fetch(f"entity:ip:{hash(txn.device.ip_address)}"),
        async_fetch_velocity(txn.payer.user_id),
    ]
    user_prof, device_prof, ip_prof, velocity = await_all(futures, timeout=15ms)

    # Step 3: Build feature vector
    features = build_features(txn, user_prof, device_prof, ip_prof, velocity)

    # Step 4: Parallel rule + ML evaluation
    rule_result = rule_engine.evaluate(txn, features)    # < 10ms
    ml_scores = ml_scorer.predict(features)               # < 30ms

    # Step 5: Decision logic
    risk_score = compute_risk_score(rule_result, ml_scores)
    decision = decide(risk_score, rule_result)

    # Step 6: Async post-processing
    kafka.publish("txn.scored", {txn, features, decision})

    return decision

2. Geo-Velocity (Impossible Travel Detection)

def check_impossible_travel(current_txn, user_profile):
    last_geo = user_profile.last_txn_geo       # {"lat": 37.77, "lng": -122.42}
    last_ts = user_profile.last_txn_ts
    current_geo = current_txn.device.geo_location
    current_ts = current_txn.timestamp

    distance_km = haversine(last_geo, current_geo)
    time_diff_hours = (current_ts - last_ts) / 3600

    # Max plausible speed: 900 km/h (commercial flight)
    max_possible_km = time_diff_hours * 900

    if distance_km > max_possible_km and distance_km > 100:
        return RiskSignal(
            type="impossible_travel",
            severity=min(1.0, distance_km / max_possible_km),
            details=f"{distance_km:.0f}km in {time_diff_hours:.1f}h"
        )
    return None

3. Amount Anomaly Detection (Z-Score)

def amount_zscore(txn_amount, user_profile):
    mean = user_profile.avg_txn_amount
    std = user_profile.txn_amount_stddev
    if std == 0:
        return 0  # not enough history
    return (txn_amount - mean) / std

# z > 3.0 → 99.7% outside normal; highly suspicious
# z > 2.0 → 95.4% outside normal; worth flagging

#Design Patterns Used

Pattern Where Why
CQRS Real-time scoring (read) vs async persistence (write) Scoring path must be < 100ms; writes can be async
Event Sourcing All transactions + decisions logged to Kafka Immutable audit trail; replay for model retraining
Chain of Responsibility Blocklist → Rules → ML → Decision Each stage can short-circuit (blocklist hit → immediate BLOCK)
Strategy Pattern Multiple ML models plugged into scoring pipeline Swap models without changing orchestrator logic
Circuit Breaker Scoring Service → ML Scorer If ML is slow/down, fallback to rules-only scoring
Observer Kafka pub/sub for profile updates, blocklist sync Decouple scoring from profile maintenance

#8. Failure Handling & Edge Cases

#What Happens When X Fails?

Failure Impact Mitigation
Redis (profiles) down Can't enrich transactions Fallback to default risk profile (conservative); approve low-value txns, REVIEW high-value
ML Scorer down No ML scores Rules-only scoring (circuit breaker); ML contributes ~40% of detection — rules catch most common patterns
Kafka down Profiles not updating Scoring continues with stale profiles; queue events locally, replay when Kafka recovers
Cassandra down Transaction history unavailable Scoring unaffected (real-time path uses Redis); analysts lose case history temporarily
Rule engine bug Mass false positives/blocks Kill switch per rule; shadow-mode for new rules; auto-rollback if block rate exceeds threshold
Entire scoring service down Payments can't be risk-scored Payment service has fallback: approve txns < $100, REVIEW txns > $100 (fail-open with limits)

#Fail-Open vs Fail-Closed Trade-off

FAIL-OPEN  (approve when uncertain):
  ✅ No legitimate user impact
  ❌ Fraud slips through during outage
  → Use for: low-value transactions, established users

FAIL-CLOSED (block when uncertain):
  ✅ No fraud during outage
  ❌ Blocks all legitimate users
  → Use for: high-value transactions, new accounts, high-risk merchants

HYBRID (recommended):
  → amount < $100 AND account_age > 90d → APPROVE (fail-open)
  → amount > $1000 OR account_age < 7d  → REVIEW (conservative)
  → Everything else → APPROVE with enhanced post-transaction monitoring

#Edge Cases

#9. Monitoring & Observability

#Key Metrics

Metric Target Alert Threshold
Scoring P99 latency < 100ms > 200ms
Fraud detection rate (recall) > 95% < 90%
False positive rate < 1% > 2%
Block rate (% of transactions) ~0.3% > 1% (possible rule bug)
Review queue depth < 50K open cases > 100K (analyst backlog)
ML model precision > 85% < 75% (model drift)
Kafka consumer lag < 5s > 60s
Redis profile staleness < 5s > 30s
Blocklist propagation time < 1s > 5s
Event ingestion error rate < 0.01% > 0.1%

#Alerting Strategy

#SLAs / SLOs

Scoring API:        99.999% availability, P99 < 100ms
Event Ingestion:    99.99% availability, P99 < 50ms
Blocklist Sync:     99.99% propagation within 1s
Detection Rate:     > 95% of known fraud patterns (measured weekly)
False Positive:     < 1% of legitimate transactions blocked

#10. Trade-off Summary

Decision Chose Over Because
Scoring approach Rules + ML hybrid ML-only or rules-only Rules for fast response to new patterns; ML for complex detection. Together > either alone
Rule evaluation Rete algorithm (in-process) External rule engine (Drools) No network hop; sub-10ms for 500 rules; lower operational complexity
ML serving ONNX Runtime (CPU) TensorFlow Serving (GPU) XGBoost/LightGBM models are CPU-optimal; GPU adds latency for small batch sizes
Profile store Redis (primary) + PostgreSQL (backup) PostgreSQL only Sub-ms reads critical for 100ms budget; PostgreSQL as cold fallback
Transaction store Cassandra PostgreSQL, ClickHouse Write throughput at 70K TPS; time-series access pattern; auto-TTL for retention
Consistency for blocklists Strong (< 1s) Eventual Delayed blocklist = fraud window; worth the operational cost
Fail mode Hybrid (fail-open for low-risk, fail-closed for high-risk) Pure fail-open or fail-closed Balances fraud prevention with user experience based on risk context
Model deployment Champion/challenger shadow scoring Blue/green full switch Validates new model on real traffic without risk; 7-day evaluation period

#11. Extensions & Follow-ups

#What Would You Add With More Time?

  1. Graph Neural Networks for Ring Detection — Model device-IP-user-card relationships as a graph; detect fraud rings via community detection algorithms (e.g., Louvain)
  2. Behavioral Biometrics — Typing patterns, mouse movements, touch pressure as continuous authentication signals
  3. Explainable AI (XAI) — SHAP/LIME explanations for ML scores shown to analysts in case management UI
  4. Adaptive Thresholds — Automatically adjust REVIEW/BLOCK thresholds based on current fraud rate and analyst capacity
  5. Real-Time Model Updates — Online learning with reinforcement learning to adapt to new fraud patterns within hours instead of daily retrain
  6. Consortium Data — Cross-institution fraud signal sharing (anonymized) to detect fraudsters moving between platforms

#How Would This Change at 100x Scale?

#12. Cross-References

Topic Connection
E-Commerce (#11) Fraud scoring integrated into checkout flow for payment risk assessment
Payment System (#18) Fraud scoring is called inline during payment authorization
Rate Limiter (#2) Velocity-based fraud detection is essentially a specialized rate limiter
Notification System (#13) Fraud alerts sent to users via push/SMS when suspicious activity detected
Recommendation Engine (#23) Shares user behavior signals; adversarial attacks on recs can be fraud

#Building Blocks Used

#Concepts Used