📁 topics / 23-recommendation-engine.md

#Design a Recommendation Engine (Netflix / YouTube / Amazon Scale)

#1. Problem Statement & Clarifications

A Recommendation Engine is the system responsible for predicting and surfacing items (movies, products, songs, posts) that a user is most likely to engage with. It powers the "Recommended For You", "Because You Watched...", and "Customers Also Bought" sections seen across Netflix, YouTube, Amazon, and Spotify.

#Functional Requirements

  1. Personalized Recommendations — Generate ranked item lists tailored to each user based on their history, preferences, and behavior signals
  2. Real-Time Event Ingestion — Capture user interactions (views, clicks, purchases, ratings, dwell time) in real-time to update recommendation context
  3. Multiple Recommendation Types — Support different strategies: collaborative filtering (users like you), content-based (items like this), trending/popular, and "because you interacted with X"
  4. Candidate Generation + Ranking — Two-stage pipeline: generate hundreds of candidates, then rank them with a fine-grained ML model
  5. Explanation — Provide reasoning for recommendations ("Because you watched X", "Trending in your area")

#Non-Functional Requirements

Requirement Target
Read Latency P99 < 100ms for fetching personalized recommendations
Freshness New interactions reflected in recommendations within 5–30 minutes
Availability 99.99% — recommendations drive 35%+ of engagement
Scale 500M users, 100M items, 50B interaction events/day
Throughput 200K recommendation requests/sec at peak

#Out of Scope

#Assumptions

#2. Back-of-Envelope Estimation

#Traffic Estimates

DAU:                    200M users
Rec requests/day:       2B (avg 10 page loads with recs per user)
Rec QPS (avg):          2B / 86400 ≈ 23K QPS
Rec QPS (peak 3x):     ~70K QPS
Event ingestion:        50B events/day ≈ 580K events/sec
Event ingestion (peak): ~1.5M events/sec

#Storage Estimates

User profiles:          500M users × 2KB (demographics + prefs) = 1TB
User embeddings:        500M × 512B (256-dim float32) = 256GB
Item embeddings:        100M × 512B = 51GB
Interaction history:    50B events/day × 200B avg = 10TB/day (hot: 30 days = 300TB)
Feature store:          500M users × 1KB real-time features = 500GB
Item metadata:          100M × 5KB = 500GB
Total hot storage:      ~302TB (dominated by interaction logs)

#Bandwidth

Rec response:           ~5KB (50 items with metadata)
Peak rec bandwidth:     70K × 5KB = 350 MB/s
Event ingestion BW:     1.5M × 200B = 300 MB/s

#Cache Estimates

Top 5% items (popular):   5M items × 5KB = 25GB (item metadata cache)
Pre-computed recs:         50M active users × 2KB = 100GB (Redis cluster)
Cache hit ratio target:   90%+ for pre-computed, 95%+ for item metadata

#3. API Design

#Get Recommendations API

GET /api/v1/recommendations
  ?user_id=U123
  &context=homepage|post_play|product_page
  &item_id=I456              // optional: "related to this item"
  &rec_type=personalized|similar|trending|because_you_watched
  &limit=50
  &page_token=abc123

Response 200:
{
  "recommendations": [
    {
      "item_id": "I789",
      "score": 0.94,
      "title": "Stranger Things S4",
      "thumbnail_url": "https://cdn.example.com/...",
      "explanation": "Because you watched Dark",
      "rec_type": "because_you_watched",
      "source_item_id": "I456"
    }
  ],
  "request_id": "req_abc",          // for A/B tracking
  "model_version": "v3.2.1",
  "page_token": "def456"
}

#Record Event API (Client → Event Collector)

POST /api/v1/events
{
  "user_id": "U123",
  "event_type": "VIEW|CLICK|PURCHASE|RATE|SKIP|DWELL",
  "item_id": "I789",
  "timestamp": 1716982400,
  "context": {
    "page": "homepage",
    "position": 3,
    "rec_request_id": "req_abc",
    "device": "mobile_ios",
    "session_id": "sess_xyz"
  },
  "metadata": {
    "dwell_time_sec": 45,
    "rating": 4.5,
    "watch_pct": 0.72
  }
}

Response 202: { "status": "accepted" }

#Get Similar Items API

GET /api/v1/items/{item_id}/similar?limit=20

Response 200:
{
  "source_item": "I456",
  "similar_items": [
    { "item_id": "I789", "similarity_score": 0.91, "shared_attributes": ["sci-fi", "thriller"] }
  ]
}

#Internal: Fetch User Features (Feature Store)

GET /internal/features/user/{user_id}

Response 200:
{
  "user_id": "U123",
  "embedding": [0.12, -0.34, ...],         // 256-dim vector
  "real_time_features": {
    "last_5_items": ["I1","I2","I3","I4","I5"],
    "session_genre_dist": {"action":0.4, "comedy":0.3, "drama":0.3},
    "hours_since_last_visit": 2.5
  },
  "static_features": {
    "country": "US",
    "age_bucket": "25-34",
    "subscription_tier": "premium"
  }
}

#4. Data Model

#User Profile Store (PostgreSQL — sharded by user_id)

CREATE TABLE users (
  user_id       BIGINT PRIMARY KEY,
  country       VARCHAR(5),
  language      VARCHAR(10),
  age_bucket    VARCHAR(10),
  signup_date   TIMESTAMP,
  sub_tier      ENUM('free','basic','premium'),
  updated_at    TIMESTAMP
);

CREATE TABLE user_preferences (
  user_id       BIGINT REFERENCES users,
  category      VARCHAR(50),        -- e.g., "genre", "brand"
  value         VARCHAR(100),       -- e.g., "sci-fi", "Nike"
  weight        FLOAT,              -- learned or explicit preference
  PRIMARY KEY (user_id, category, value)
);

#Item Metadata Store (PostgreSQL)

CREATE TABLE items (
  item_id       BIGINT PRIMARY KEY,
  title         VARCHAR(500),
  description   TEXT,
  category_path VARCHAR(500),       -- "Movies > Sci-Fi > Thriller"
  attributes    JSONB,              -- {"genre":["sci-fi"],"director":"Nolan","year":2024}
  popularity    FLOAT,              -- normalized 0-1
  created_at    TIMESTAMP,
  updated_at    TIMESTAMP,
  status        ENUM('ACTIVE','INACTIVE')
);

#Embedding Store (Vector DB — Milvus / Pinecone / pgvector)

Collection: user_embeddings
  - user_id:    BIGINT (primary key)
  - vector:     FLOAT[256]          // 256-dim dense embedding
  - version:    INT                 // model version that produced this
  - updated_at: TIMESTAMP

Collection: item_embeddings
  - item_id:    BIGINT (primary key)
  - vector:     FLOAT[256]
  - version:    INT
  - updated_at: TIMESTAMP

Index: IVF_FLAT or HNSW for ANN (Approximate Nearest Neighbor) search

#Interaction Event Store (Kafka → Cassandra / ClickHouse)

Partition Key: user_id
Sort Key:      timestamp (descending)

{
  user_id:      "U123",
  item_id:      "I789",
  event_type:   "VIEW",
  timestamp:    1716982400,
  context:      {"page":"homepage","position":3},
  metadata:     {"dwell_time_sec":45,"watch_pct":0.72}
}

#Real-Time Feature Store (Redis Cluster)

Key:   user:features:{user_id}
Value: {
  "last_n_items":       ["I1","I2","I3","I4","I5"],
  "session_clicks":     12,
  "genre_dist_24h":     {"action":0.4,"comedy":0.3,"drama":0.3},
  "avg_dwell_time_7d":  35.2,
  "last_active_ts":     1716982400
}
TTL: 7 days (refreshed on every interaction)

#Access Patterns

Query Store Index/Key
Get user embedding Vector DB PK: user_id
ANN search (find similar items) Vector DB (Milvus) HNSW index on item_embeddings
Get user's recent interactions Cassandra PK: user_id, SK: timestamp DESC
Get real-time user features Redis Key: user:features:{user_id}
Get item metadata PostgreSQL + Redis cache PK: item_id
Get trending items by category Redis sorted set Key: trending:{category}
Batch user embeddings for training HDFS / S3 Batch export from Vector DB

#5. High-Level Design (HLD) & Scale Evolution

#Stage 1: MVP / Startup Scale (100K Users, 10K Items)

#Stage 2: Growth Scale (10M Users, 1M Items)

#Stage 3: Netflix/YouTube Scale (500M Users, 100M Items)

This is the target architecture — a fully decoupled, two-stage ML pipeline with real-time and batch components.

#Architecture Diagram

                    ┌──────────────┐
                    │   CDN / Edge │ ← Pre-computed recs for cold-start users
                    └──────┬───────┘
                           │
                ┌──────────▼──────────┐
                │    API Gateway       │
                │  (Rate limit, Auth)  │
                └──────────┬──────────┘
                           │
          ┌────────────────┼─────────────────┐
          ▼                ▼                  ▼
 ┌─────────────────┐ ┌──────────────┐ ┌───────────────┐
 │  Recommendation  │ │   Event      │ │  Similar Items│
 │  Service         │ │   Collector  │ │  Service      │
 │                  │ │              │ │               │
 │ • Orchestrates   │ │ • Validates  │ │ • Item-to-item│
 │   candidate gen  │ │ • Publishes  │ │   ANN lookup  │
 │   + ranking      │ │   to Kafka   │ │               │
 └────────┬─────────┘ └──────┬───────┘ └───────┬───────┘
          │                  │                  │
    ┌─────┴──────┐           ▼                  ▼
    ▼            ▼    ┌──────────────┐   ┌──────────────┐
┌────────┐ ┌────────┐│  Kafka       │   │  Vector DB   │
│Candidat│ │Ranking ││  Event Bus   │   │  (Milvus)    │
│  Gen   │ │Service ││              │   │  Item embeds │
│Service │ │        │└──────┬───────┘   └──────────────┘
└───┬────┘ └───┬────┘       │
    │          │      ┌─────┼──────────────┐
    ▼          ▼      ▼     ▼              ▼
┌────────┐ ┌──────┐ ┌────────────┐ ┌─────────────┐
│Vector  │ │Model │ │Feature     │ │Interaction  │
│  DB    │ │Server│ │Update Svc  │ │  Store      │
│(Milvus)│ │(TF   │ │(Flink/     │ │(Cassandra/  │
│User +  │ │Serve)│ │Spark Stream│ │ ClickHouse) │
│Item    │ │      │ │            │ │             │
│embeds  │ └──────┘ └─────┬──────┘ └─────────────┘
└────────┘                │
                   ┌──────▼───────┐
                   │  Redis        │
                   │  Feature Store│
                   │  + Rec Cache  │
                   └──────────────┘

   ┌─────────────────────────────────────────────────────┐
   │          Batch Pipeline (Every 4-6 hours)            │
   │  Spark/Flink → Train models → Update embeddings     │
   │  → Push to Vector DB → Warm Redis cache              │
   └─────────────────────────────────────────────────────┘

#Component Breakdown

Component Responsibility Tech Choice
API Gateway Rate limiting, auth, routing Kong / AWS API GW
Recommendation Service Orchestrates candidate gen → ranking → response Go / Java microservice
Candidate Generation Fast ANN retrieval of ~500 candidates Vector DB (Milvus) + HNSW
Ranking Service Re-rank candidates using fine-grained ML model TensorFlow Serving / Triton
Event Collector Validate & publish interaction events Go service → Kafka
Feature Update Service Real-time feature aggregation from events Apache Flink / Spark Streaming
Feature Store Serve user/item features at low latency Redis cluster
Vector DB Store & search embeddings via ANN Milvus / Pinecone
Model Server Serve trained ranking models TF Serving / Triton Inference
Batch Pipeline Retrain models, recompute embeddings Spark + TensorFlow on GPU cluster

#Data Flow

Recommendation Read Path:

User → API GW → Recommendation Service
  → 1. Fetch user features from Redis Feature Store
  → 2. Fetch user embedding from Vector DB
  → 3. Candidate Generation: ANN search in Vector DB → 500 candidates
  → 4. Fetch item features for candidates from Redis
  → 5. Ranking Service: ML model scores each candidate
  → 6. Post-processing: diversity filter, business rules, dedup
  → 7. Return top 50 ranked items with explanations

Event Write Path:

User interaction → Event Collector → Kafka (events topic)
  → Consumer 1: Feature Update Service → update Redis features
  → Consumer 2: Interaction Store → persist to Cassandra
  → Consumer 3: Near-real-time model update (optional)

Batch Path (Offline):

Interaction Store (Cassandra) → Spark batch job
  → Train collaborative filtering model (ALS / deep learning)
  → Generate new user & item embeddings
  → Push embeddings to Vector DB
  → Pre-compute top-N recs for active users → cache in Redis

#6. Deep Dive — Core Components

#Candidate Generation Service — Detailed Design

The two-stage architecture is critical: candidate generation narrows 100M items to ~500 cheaply, then ranking scores those 500 precisely.

Multiple Candidate Sources (Multi-Channel Retrieval):

┌──────────────────────────────────────────────────┐
│              Candidate Generation                 │
│                                                   │
│  Source 1: User-to-Item ANN (collaborative)       │
│  → User embedding → ANN search → 200 candidates  │
│                                                   │
│  Source 2: Item-to-Item ANN (content-based)       │
│  → Last 5 viewed items → ANN per item → 150 each │
│  → Deduplicate → 200 candidates                  │
│                                                   │
│  Source 3: Trending / Popular                     │
│  → Redis sorted set by category → 50 candidates  │
│                                                   │
│  Source 4: Editorial / Business Rules             │
│  → New releases, promoted content → 50 candidates │
│                                                   │
│  Total: ~500 unique candidates (after dedup)      │
└──────────────────────────────────────────────────┘

ANN Search with HNSW:

#Ranking Service — Detailed Design

Feature Vector for Ranking:

For each (user, candidate_item) pair, construct:
{
  // User features (from Feature Store)
  user_embedding:        [256 floats],
  user_genre_dist:       [20 floats],
  user_avg_dwell:        float,
  user_activity_level:   float,

  // Item features
  item_embedding:        [256 floats],
  item_popularity:       float,
  item_freshness:        float,    // days since release
  item_category_onehot:  [50 floats],

  // Cross features
  user_item_dot_product: float,    // embedding similarity
  user_category_affinity: float,   // user pref × item category match
  co_interaction_count:  int,      // how many similar users interacted

  // Context features
  time_of_day:           float,
  device_type:           onehot,
  page_context:          onehot
}

Ranking Model: Deep neural network (e.g., Wide & Deep, DCN, or two-tower with cross-attention) trained on click-through rate (CTR) and engagement signals. Served via TensorFlow Serving with GPU inference for batch scoring.

Post-Processing Pipeline:

Ranked candidates (500)
  → Remove already-watched items (interaction store lookup)
  → Diversity filter (max 3 items from same genre in top 20)
  → Business rules (boost new releases, contractual obligations)
  → Explanation generation (match rec_type to template)
  → Return top 50

#Feature Update Service — Detailed Design

Real-Time Feature Computation (Apache Flink):

Kafka events stream → Flink job:
  1. Sliding window: count clicks per genre in last 24h → genre_dist
  2. Tumbling window: avg dwell time in last 7 days → avg_dwell
  3. Session tracking: last N items interacted → last_n_items
  4. Write updated features to Redis Feature Store

Why Flink over Spark Streaming? True event-time processing, lower latency (sub-second vs micro-batch), better exactly-once semantics with Kafka.

#Scaling Strategy

Component Strategy
Vector DB Sharded by embedding ID; replicas per shard for read throughput. Milvus supports horizontal scaling natively
Redis Feature Store Cluster mode, 600GB+ across 100+ nodes; separate clusters for features vs rec cache
Ranking Service GPU-backed pods; horizontal scale behind LB; batch scoring (500 items per request) amortizes GPU cost
Kafka Partition events topic by user_id for ordered processing; 500+ partitions
Interaction Store Cassandra with TTL (keep 90 days hot); older data archived to S3/HDFS for training
Candidate Generation Stateless service; scale pods independently; cache popular item neighborhoods

#Caching Strategy

┌──────────────────────────────────────────────────────┐
│ Layer 1: Pre-computed Recommendations (Redis)         │
│ • Top 50 recs per active user (TTL: 30min)           │
│ • Refreshed on significant user activity             │
│ • Serves 60% of requests without hitting ML pipeline │
├──────────────────────────────────────────────────────┤
│ Layer 2: Item Metadata Cache (Redis)                  │
│ • Item title, thumbnail, attributes (TTL: 1h)        │
│ • 95% hit rate for popular items                     │
├──────────────────────────────────────────────────────┤
│ Layer 3: ANN Result Cache (Local in-process)          │
│ • Cache item-to-item neighbors (TTL: 4h)             │
│ • Item embeddings change only on batch retrain       │
├──────────────────────────────────────────────────────┤
│ Layer 4: Trending / Popular Lists (Redis Sorted Sets) │
│ • Pre-computed hourly; used for cold-start users     │
│ • Category-level and global trending                 │
└──────────────────────────────────────────────────────┘

#Consistency Models

Data Model Rationale
User embeddings Eventually consistent (4-6h) Updated on batch retrain; slight staleness is fine
Real-time features Eventually consistent (< 5s) Flink updates Redis within seconds of event
Item metadata Eventually consistent (< 30s) Stale title/image is briefly acceptable
Interaction events At-least-once delivery Kafka guarantees; dedup at consumer if needed
Pre-computed recs cache Eventually consistent (< 30min) Refreshed periodically; real-time path as fallback

#7. Low-Level Design — Core Functionality

#Key Algorithms

1. Two-Tower Embedding Model (Candidate Generation)

User Tower:                    Item Tower:
  user_id → Embedding            item_id → Embedding
  + user features                + item features
  → Dense layers                 → Dense layers
  → 256-dim user vector          → 256-dim item vector

Training: Maximize dot product for positive (user, item) pairs
          Minimize for negative samples (in-batch negatives)

Inference: Pre-compute all item vectors → build ANN index
           At request time: compute user vector → ANN search

2. Approximate Nearest Neighbor (HNSW)

HNSW Index Parameters:
  M = 16            (max connections per node per layer)
  ef_construction = 200  (search width during build)
  ef_search = 100        (search width during query)

Query: user_embedding → search HNSW index → top 200 items
Latency: < 5ms for 100M vectors at 95% recall
Memory: ~256B per vector × 100M = 25.6GB + graph overhead ≈ 40GB

3. Real-Time Feature Aggregation (Flink Pseudocode)

# Sliding window: genre distribution over last 24 hours
events_stream \
  .key_by("user_id") \
  .window(SlidingEventTimeWindows.of(hours(24), minutes(10))) \
  .aggregate(GenreDistributionAggregator()) \
  .add_sink(RedisSink("user:features:{user_id}", field="genre_dist_24h"))

# Per-event: update last-N items list
events_stream \
  .filter(lambda e: e.event_type in ["VIEW", "CLICK"]) \
  .key_by("user_id") \
  .process(LastNItemsProcessor(n=20)) \
  .add_sink(RedisSink("user:features:{user_id}", field="last_n_items"))

4. Diversity-Aware Re-ranking (MMR — Maximal Marginal Relevance)

def mmr_rerank(candidates, lambda_param=0.7, top_k=50):
    """Balance relevance and diversity using MMR."""
    selected = []
    remaining = list(candidates)

    while len(selected) < top_k and remaining:
        best_score = -inf
        best_item = None

        for item in remaining:
            relevance = item.ranking_score
            max_sim = max(cosine_sim(item.embedding, s.embedding)
                         for s in selected) if selected else 0
            mmr = lambda_param * relevance - (1 - lambda_param) * max_sim

            if mmr > best_score:
                best_score = mmr
                best_item = item

        selected.append(best_item)
        remaining.remove(best_item)

    return selected

#Design Patterns Used

Pattern Where Why
CQRS Event ingestion (write) vs Recommendation serving (read) Vastly different read/write patterns and scale
Two-Stage Pipeline Candidate Gen → Ranking Can't run expensive ML model on 100M items; narrow first, rank second
Event Sourcing Interaction events via Kafka Immutable event log enables replay for model retraining
Feature Store Pattern Redis-backed real-time features Decouple feature computation from model serving
Circuit Breaker Rec Service → Ranking Service Fallback to cached/popular recs if ML model is slow
Bulkhead Separate thread pools for each candidate source One slow source doesn't block others

#8. Failure Handling & Edge Cases

#What Happens When X Fails?

Failure Impact Mitigation
Vector DB down Can't generate ANN candidates Fallback to pre-computed recs from Redis cache; serve trending
Ranking Service (GPU) down Can't score candidates Return candidates sorted by embedding similarity (dot product); skip ML ranking
Redis Feature Store down Missing real-time features Use default/average features; model trained to handle missing features gracefully
Kafka lag Features become stale Alert on lag > 60s; recs still work but less personalized
Interaction Store down Can't filter already-seen items Serve recs anyway; minor UX degradation (user sees repeat recs)
Model Server OOM Ranking fails for batch Reduce batch size dynamically; circuit breaker activates fallback

#Cold Start Problem

This is the most critical edge case for recommendation engines:

NEW USER (no interaction history):
  1. Onboarding survey → explicit preferences → content-based recs
  2. Demographic-based recs (age, country, device)
  3. Global trending / popular items
  4. Explore/Exploit: show diverse items, learn quickly from clicks
  5. After 10-20 interactions → switch to collaborative filtering

NEW ITEM (no user interactions):
  1. Content-based features (genre, description, creator) → place in embedding space
  2. Boost factor for new items (exploration bonus in ranking score)
  3. Show to users who liked similar-attribute items
  4. After 100-500 interactions → collaborative signals kick in

#Edge Cases

#9. Monitoring & Observability

#Key Metrics

Metric Target Alert Threshold
Rec API P99 latency < 100ms > 250ms
Candidate gen P99 < 20ms > 50ms
Ranking P99 < 60ms > 150ms
Event ingestion lag (Kafka) < 5s > 60s
Pre-computed rec cache hit ratio > 60% < 40%
CTR on recommendations > 5% < 2% (model degradation)
Rec diversity score (ILS) > 0.4 < 0.2 (filter bubble)
Cold-start user coverage > 95% < 85%
Model serving error rate < 0.1% > 0.5%
Embedding freshness < 6h > 12h

#Alerting Strategy

#SLAs / SLOs

Recommendation API: 99.95% availability, P99 < 100ms
Event Ingestion:    99.99% availability, P99 < 50ms
Feature Freshness:  99% of events reflected in features within 30s
Model Freshness:    Embeddings updated every 6h with < 1h variance

#10. Trade-off Summary

Decision Chose Over Because
Candidate generation ANN (HNSW) on Vector DB Brute-force scoring all items Can't run ML model on 100M items in real-time; ANN narrows to 500 in < 10ms
Embedding index HNSW IVF_FLAT, LSH Better recall (95%+) at comparable latency; worth the extra memory
Feature store Redis (real-time) + HDFS (batch) Single unified store Real-time features need sub-ms reads; batch features need petabyte-scale storage
Event processing Apache Flink Spark Streaming True event-time semantics, sub-second latency vs micro-batch lag
Interaction store Cassandra PostgreSQL, ClickHouse High write throughput for 580K events/sec; time-series access pattern fits wide-column model
Ranking model serving TensorFlow Serving (GPU) CPU-only inference GPU batch scoring amortizes cost; 500 candidates scored in < 30ms on GPU vs 200ms on CPU
Rec staleness tolerance Eventually consistent (5-30 min) Real-time model updates Online model updates are complex and risky; batch retrain + real-time features is a good middle ground
Diversity approach MMR re-ranking post-hoc Diversity-aware training MMR is interpretable, tunable, and doesn't require model retraining to adjust

#11. Extensions & Follow-ups

#What Would You Add With More Time?

  1. Multi-Objective Ranking — Optimize for engagement, revenue, and user satisfaction simultaneously using Pareto-optimal ranking
  2. Contextual Bandits — Online learning for explore/exploit; adapt recs in real-time without full model retrain
  3. Cross-Domain Recommendations — "You watched X on Netflix, you might like Y on the e-commerce store" using shared user embeddings
  4. Graph Neural Networks — Model user-item-user interactions as a graph for richer collaborative signals (e.g., PinSage)
  5. Federated Learning — Train personalization models on-device for privacy-sensitive use cases
  6. Explanation Generation — Use LLMs to generate natural language explanations beyond template-based "Because you watched X"

#How Would This Change at 100x Scale?

#12. Cross-References

Topic Connection
E-Commerce (#11) Recommendation engine powers "related products" and "customers also bought" sections
Search Autocomplete (#14) Shares user behavior signals; search queries inform recommendation features
Notification System (#13) Delivers recommendation-based push notifications ("New release you might like")
Distributed Cache (#15) Redis cluster design underpins feature store and rec cache

#Building Blocks Used

#Concepts Used