📁 topics / 27-twitter-trends.md

#Design Twitter Trends (Twitter Scale)

Target Scale: 500M tweets/day, 350M MAU, 200K+ hashtags/hour, real-time global + regional + personalized trends.

#1. Problem Statement & Clarifications

#Functional Requirements

  1. Global Trending Topics: Compute and serve the top-30 trending hashtags/phrases globally, refreshed every 60 seconds.
  2. Regional Trends: Compute top-30 trends per geographic region (country, city), refreshed every 60 seconds.
  3. Personalized Trends: For a logged-in user, filter global/regional trends through their interest graph (exclude topics they always see, boost niche ones they care about).
  4. Trend Metadata: Each trend entry includes tweet volume, trend velocity (acceleration), and representative sample tweets.
  5. Trend Lifespan Tracking: Track when a topic started trending and estimate time-to-peak.
  6. Anti-Spam / Manipulation Filtering: Exclude coordinated inauthentic behavior — bot farms, purchased trend injection.

#Non-Functional Requirements

Requirement Target
Trend freshness ≤ 60 s from event to trend update
Trend API latency (P99) < 100 ms
Throughput 6K tweets/s avg; 30K/s peak (breaking news)
Availability 99.99%
Trend accuracy Top-K error < 5% for items in true top-30
Regional granularity Country-level (50 regions), City-level (500 cities)
Historical retention Trend snapshots retained for 90 days

#Out of Scope

#Assumptions

#2. Back-of-Envelope Estimation

#Traffic Estimates

Tweets/day          = 500M
Tweets/s (avg)      = 500M / 86400 ≈ 5.8K TPS
Peak TPS            = 5.8K × 5 (breaking news spike) ≈ 30K TPS
Hashtags per tweet  = avg 1.5 → 5.8K × 1.5 ≈ 8.7K hashtag events/s
Unique hashtags/hr  = ~200K (long tail; top 500 account for 80% of volume)

Trend API reads     = 350M MAU → assume 30% open app/hr → 105M req/hr
                    = 105M / 3600 ≈ 29K read QPS
                    Peak: 29K × 3 = 87K read QPS (during major events)

#Storage Estimates

Tweet event (Kafka payload):
  tweet_id (8B) + user_id (8B) + hashtags[] (avg 40B) + geo (8B) + ts (8B) + bot_score (4B)
  = ~76 B per tweet event
  = 500M × 76 B ≈ 38 GB/day (Kafka topic, 7-day retention = 266 GB)

Trend snapshot per region per minute:
  Top-30 entries × (hashtag 50B + count 8B + velocity 8B + sample_ids 3×8B) ≈ 2 KB
  Total regions = 1 global + 50 country + 500 city = 551 regions
  Per minute = 551 × 2 KB = 1.1 MB/min
  90-day retention = 1.1 MB × 60 × 24 × 90 ≈ 143 GB (PostgreSQL, compressed)

Count-Min Sketch (CMS) per region in-memory:
  w=3000, d=5, 32-bit counters → 3000 × 5 × 4 B = 60 KB per sketch
  551 regions × 60 KB = 33 MB total in-memory (trivial)

Redis ZSET (top-K per region):
  Top-500 per region × (50B hashtag + 8B score) ≈ 29 KB per region
  551 × 29 KB ≈ 16 MB total (negligible)

#Bandwidth

Kafka ingest         = 30K TPS peak × 200 B (full tweet payload) = 6 MB/s
Trend API response   = 87K QPS × 3 KB (30 trends + metadata) = 261 MB/s
  → Served from Redis/CDN cache, not app servers
Flink → Redis writes = 551 regions × 1 update/min = 9 writes/s (tiny)

#Cache Estimates

Trend results cached in Redis per region = 29 KB × 551 = 16 MB
Personalized trends = computed at edge from base trends + user interest vector
                    = no separate cache needed (computed in < 5ms)
CDN caches global trends for unauthenticated users (TTL = 60s)

#3. API Design

GET /v1/trends?woeid=1&limit=30&personalized=true
Authorization: Bearer <jwt>   (optional; omit for global unauthenticated)

woeid = Where On Earth ID (Yahoo WOEID standard)
  woeid=1       → worldwide
  woeid=23424977 → United States
  woeid=2459115  → New York City

Response 200:
{
  "as_of": "2026-06-02T00:16:00Z",
  "ttl_s": 60,
  "location": { "woeid": 1, "name": "Worldwide" },
  "trends": [
    {
      "rank": 1,
      "name": "#WorldCup2026",
      "query": "%23WorldCup2026",
      "tweet_volume": 1284000,
      "volume_24h": 8420000,
      "velocity": 3.4,           // acceleration: volume_now / volume_1h_ago
      "trending_since": "2026-06-02T00:00:00Z",
      "sample_tweets": ["tweet_id_1", "tweet_id_2", "tweet_id_3"]
    }
  ]
}
GET /v1/trends/available

Response 200:
[
  { "woeid": 1,         "name": "Worldwide",     "type": "Supername" },
  { "woeid": 23424977,  "name": "United States", "type": "Country"   },
  { "woeid": 2459115,   "name": "New York",      "type": "Town"      }
]

#POST /v1/internal/trends/refresh — Internal Trend Refresh (Flink → Trends Service)

POST /v1/internal/trends/refresh   [INTERNAL]
{
  "woeid": 1,
  "window_end": "2026-06-02T00:16:00Z",
  "top_k": [
    { "term": "#WorldCup2026", "count": 21400, "velocity": 3.4, "sample_ids": ["t1","t2","t3"] }
  ]
}
Response 204
GET /v1/trends/%23WorldCup2026/history?hours=24&resolution=5m

Response 200:
{
  "term": "#WorldCup2026",
  "data_points": [
    { "ts": "2026-06-01T00:00:00Z", "volume": 12000 },
    { "ts": "2026-06-01T00:05:00Z", "volume": 14500 }
  ]
}

#4. Data Model

#Tweet Events (Kafka — partitioned by user_id)

Topic: tweet-events
Partitions: 200 (keyed by user_id for ordering within user)
Retention: 7 days
Schema (Avro):
{
  tweet_id:   long,
  user_id:    long,
  text:       string,
  hashtags:   array<string>,
  phrases:    array<string>,    // extracted n-grams by NLP service
  lang:       string,           // ISO 639-1
  woeid:      int,              // resolved from user profile/IP
  bot_score:  float,            // 0.0 = human, 1.0 = bot
  created_at: long              // epoch millis
}

#Trend Snapshots (PostgreSQL — sharded by woeid)

CREATE TABLE trend_snapshots (
  woeid        INT          NOT NULL,
  window_end   TIMESTAMPTZ  NOT NULL,
  rank         SMALLINT     NOT NULL,
  term         VARCHAR(140) NOT NULL,
  tweet_volume BIGINT,
  velocity     FLOAT,
  sample_ids   BIGINT[],
  PRIMARY KEY (woeid, window_end, rank)
) PARTITION BY RANGE (window_end);

-- Monthly partitions, auto-dropped after 90 days
CREATE INDEX idx_snapshots_woeid_term ON trend_snapshots (woeid, term, window_end DESC);

#Real-Time Trend State (Redis — primary serving store)

# Top-K sorted set per region (score = weighted trend score)
Key:   trends:woeid:{woeid}
Type:  Redis ZSET (term → score)
TTL:   120 s (2× refresh cycle; prevents stale data serving)

# Current trend metadata (volume, velocity, samples)
Key:   trend_meta:{woeid}:{term}
Type:  Redis Hash
Fields: volume, velocity, trending_since, sample_ids (JSON)
TTL:   120 s

# Trend baseline (hourly historical avg for velocity calc)
Key:   baseline:{term}
Type:  Redis Hash (hour_of_week → avg_count)
TTL:   7 days (refreshed by nightly batch job)

#User Interest Profile (Cassandra — for personalized trends)

CREATE TABLE user_interest_profile (
  user_id    BIGINT PRIMARY KEY,
  topic_weights MAP<TEXT, FLOAT>,   -- {"#sports": 0.9, "#politics": 0.1}
  exclude_topics SET<TEXT>,          -- topics user has explicitly muted
  updated_at TIMESTAMP
);

#Access Patterns

Query Store Key / Index
Fetch top-30 trends for woeid Redis ZREVRANGE trends:woeid:{woeid} 0 29
Get trend metadata (volume, velocity) Redis HGETALL trend_meta:{woeid}:{term}
Persist trend snapshot for history PostgreSQL INSERT trend_snapshots
Query trend history for a term PostgreSQL idx_snapshots_woeid_term
Get user interest profile Cassandra PRIMARY KEY user_id
List available woeids Redis trends:available (preloaded set)

#5. High-Level Design (HLD) & Scale Evolution

#Stage 1: MVP / Startup Scale (1M DAU, 1 region, global trends only)

#Stage 2: Growth Scale (50M DAU, 50 country regions)

#Stage 3: Twitter Scale (350M MAU, 551 regions, personalized, ML-scored)

#Architecture Diagram

  Twitter Clients (Web/iOS/Android)
         │  GET /v1/trends
         ▼
  ┌─────────────────────────────────────────────────┐
  │            CDN (CloudFront)                     │
  │   Caches global/country trends TTL=60s          │
  │   Cache-miss → Trend API Service                │
  └──────────────────┬──────────────────────────────┘
                     │ (cache miss or personalized req)
                     ▼
  ┌──────────────────────────────────────────────────┐
  │           API Gateway + Load Balancer            │
  └───────┬──────────────────┬───────────────────────┘
          │                  │
          ▼                  ▼
  ┌──────────────┐   ┌──────────────────┐
  │ Trends API   │   │ Personalization  │
  │ Service      │   │ Service          │
  │ (read Redis) │   │ (rerank per user)│
  └──────┬───────┘   └────────┬─────────┘
         │                    │
         ▼                    ▼
  ┌──────────────────────────────────────────────┐
  │              Redis Cluster                   │
  │  trends:woeid:{id} ZSET (top-500 per region) │
  │  trend_meta:{woeid}:{term} Hash              │
  │  baseline:{term} Hash (historical avg)       │
  └──────────────────────────────────────────────┘
         ▲
         │  Writes every 60s per region
         │
  ┌──────────────────────────────────────────────────────────┐
  │                  Trend Computation Pipeline               │
  │                                                          │
  │   Tweet Events (Kafka, 200 partitions)                   │
  │         │                                                │
  │         ▼                                                │
  │   ┌──────────────────────────────────────────────┐      │
  │   │  Flink Streaming Job                         │      │
  │   │  ├── Ingest & Filter (bot_score > 0.7 → drop)│      │
  │   │  ├── Hashtag Extractor + NLP Phrase Extractor │      │
  │   │  ├── Geo Router (assign woeid)                │      │
  │   │  ├── Count-Min Sketch per region (60 buckets) │      │
  │   │  ├── Top-K Heap per region (Space-Saving)     │      │
  │   │  ├── Velocity Calculator (current/baseline)   │      │
  │   │  ├── Trend Scorer (count × velocity weight)   │      │
  │   │  └── Emit top-500 per region → Kafka output   │      │
  │   └──────────────────────────────────────────────┘      │
  │         │                                                │
  │         ▼                                                │
  │   ┌─────────────────────┐   ┌──────────────────────┐   │
  │   │  Trend Writer Svc   │   │  Snapshot Writer Svc  │   │
  │   │  → Redis ZSET update│   │  → PostgreSQL INSERT  │   │
  │   └─────────────────────┘   └──────────────────────┘   │
  └──────────────────────────────────────────────────────────┘

  Tweet Ingestion Path:
  Mobile → Tweet Service → Kafka (tweet-events)
                        └→ NLP Service (async phrase extraction → Kafka)

#Component Breakdown

Component Responsibility Tech Choice
CDN Cache global/country trends TTL=60s for anonymous users CloudFront / Fastly
Trends API Service Read Redis ZSET, format response Go (stateless, horizontally scaled)
Personalization Service Rerank trends using user interest profile Go + Cassandra user profiles
Flink Streaming Job Count-Min Sketch + top-K + velocity computation Apache Flink (parallelism=200)
Trend Writer Service Atomic Redis ZSET replacement per region Go
Snapshot Writer Service Persist trend snapshots to PostgreSQL Go
NLP Phrase Extractor Extract trending n-grams beyond hashtags Python microservice (spaCy)
Bot Filter Drop events with bot_score > threshold Inline Flink filter stage
Baseline Job Nightly batch — compute hourly avg per term Spark batch → Redis
Kafka Tweet event bus, trend-output topic Kafka (200 + 50 partitions)
Redis Cluster Serve trend ZSETs + metadata hashes Redis Cluster (6 shards)
PostgreSQL 90-day trend snapshot history PostgreSQL (range-partitioned)
Cassandra User interest profiles for personalization Cassandra

#Data Flow

Tweet Ingestion → Trend Update (Write Path):

User posts tweet
  → Tweet Service validates + stores tweet
  → Publishes to Kafka topic: tweet-events (partition = user_id % 200)
  → NLP Service (parallel consumer): extracts phrases → publishes enriched event

Flink Job (consumes tweet-events):
  1. Filter: drop if bot_score > 0.7
  2. Explode: for each hashtag/phrase in tweet → emit (woeid, term, count=1)
  3. Route: also emit to parent woeids (city → country → global fan-up)
  4. Aggregate: CMS update per (woeid, bucket_minute) + Space-Saving top-K heap
  5. Every 60s: emit snapshot per region → trend-output Kafka topic

Trend Writer (consumes trend-output):
  → Redis: ZADD trends:woeid:{id} {score} {term}  (atomic MULTI/EXEC)
  → Redis: HSET trend_meta:{woeid}:{term} volume velocity trending_since
  → Publishes to PostgreSQL snapshot writer

Trend API Read (Read Path):

Client → CDN (global/country, unauthenticated)
  HIT  → Return cached trend JSON (TTL=60s)
  MISS → Trends API Service

Trends API Service:
  → Redis ZREVRANGE trends:woeid:{woeid} 0 29 WITHSCORES   (~1ms)
  → For each term: Redis HGETALL trend_meta:{woeid}:{term}
  → If personalized=true: Personalization Service
      → Cassandra: fetch user_interest_profile
      → Rerank: boost terms matching high-weight topics; filter excluded topics
  → Return JSON to client (or CDN for caching)

#6. Deep Dive — Core Components

This is the algorithmic core. Each Flink task slot handles a subset of woeids.

Stage 1: Bucket Management (Sliding Window)

Window W = 60 min, Bucket size B = 1 min → 60 active buckets per (woeid, term)

Flink ProcessFunction maintains per-woeid state:
  - bucket_ring[60]: circular buffer of CMS tables (one per minute)
  - current_bucket_idx: int
  - top_k_heap: Space-Saving structure (capacity=500 terms)

On event (woeid, term, ts):
  bucket_idx = (ts.epochMinute) % 60
  bucket_ring[bucket_idx].update(term, 1)   # CMS update
  top_k_heap.update(term, 1)                # Space-Saving update

Every 60s (on watermark tick):
  Evict oldest bucket: bucket_ring[evict_idx].reset()
  Advance current_bucket_idx
  Recompute merged count for each term in top_k_heap:
    count(term) = min over all d rows of sum(bucket_ring[i].query(term)) for i in active_buckets
  Sort by trend_score = count × velocity_weight
  Emit top-500 → trend-output topic

Stage 2: Velocity Calculation

velocity(term, woeid) = count_current_window / max(1, baseline_count_this_hour_of_week)

baseline_count_this_hour_of_week fetched from Redis:
  Redis HGET baseline:{term} {day_of_week}_{hour_of_day}

If velocity > 3.0  → "Trending" label
If velocity > 10.0 → "Viral" label
New terms (no baseline) → velocity = count / global_avg_new_term_rate

Stage 3: Geo Fan-Up (Hierarchical Aggregation)

City woeid → Country woeid → Global woeid

Each tweet emits events for all levels:
  emit(city_woeid, term, 1)
  emit(country_woeid, term, 1)
  emit(GLOBAL_WOEID, term, 1)

Flink partitions state by woeid → each level processed independently.
Country and global trends naturally aggregate without explicit rollup joins.

#Personalization Service — Detailed Design

Personalization runs on the result set (top-500 per region), not on raw tweet events. This keeps the personalization service lightweight.

Input:  top-500 trends for user's woeid + user_interest_profile
Output: top-30 personalized trends

Algorithm:
  For each trend t in top-500:
    topic_boost = user_interest_profile.topic_weights.get(t.term, 0.5)
    if t.term in user_interest_profile.exclude_topics: skip
    personalized_score = t.trend_score × topic_boost

  Return top-30 by personalized_score

Why top-500 in Redis (not just top-30)? Personalization may boost rank-100 items above rank-5 for a specific user.

#Bot & Spam Filtering

Pre-filter in Flink (real-time):

Post-filter on trend output:

#Scaling Strategy

Component Strategy
Flink Job 200 task slots; partition by woeid hash; auto-scale on lag
Redis Cluster 6 shards (16 slots each); reads from replicas; 16 MB trend data — trivial
Trends API Stateless Go services; 50 replicas; behind CDN
Kafka tweet-events 200 partitions; RF=3; 7-day retention
PostgreSQL snapshots Monthly range partitions; 2 read replicas; auto-vacuum
CDN 60s TTL for global/country; no TTL for personalized (per-user, skip CDN)

#Caching Strategy

Layer 1 — CDN (CloudFront)
   Global + country trends for unauthenticated users
   TTL = 60s (matches refresh cycle)
   → Absorbs ~70% of all trend read traffic

Layer 2 — Redis ZSET (trends:woeid:{id})
   Top-500 terms per region, 551 regions, 16 MB total
   → Serves all cache-miss and personalized requests at ~1ms

Layer 3 — Redis Hash (trend_meta:{woeid}:{term})
   Metadata (volume, velocity, sample_ids) for top-500 terms per region
   → Fetched in pipeline with ZSET read

Layer 4 — Redis Hash (baseline:{term})
   7-day historical hourly averages for velocity computation
   → Updated nightly by Spark batch job
   → Fetched inline by Flink job (remote Redis call, cached locally in Flink state)

#Consistency Models

Data Model Rationale
Trend rankings (Redis ZSET) Eventual (Redis, 60s cycle) 60s lag acceptable; trends aren't real-time stock prices
Trend metadata (volume, velocity) Eventual Same 60s refresh cycle
Historical snapshots (PostgreSQL) Strong (PostgreSQL ACID) Snapshot history must be accurate for analytics
User interest profile Eventual (Cassandra QUORUM read) Slightly stale profile acceptable; personalization is best-effort
Bot filter decisions Eventual (async anomaly detector) Small window of manipulation exposure acceptable vs sync latency cost

#7. Low-Level Design — Core Functionality

#Key Algorithms

Algorithm 1: Count-Min Sketch Update & Query

import hashlib

class CountMinSketch:
    """
    Fixed-size 2D counter array for frequency estimation.
    ε=0.001, δ=0.01 → w=2718, d=5 → ~54 KB per sketch.
    """
    def __init__(self, width: int = 2718, depth: int = 5):
        self.w = width
        self.d = depth
        self.table = [[0] * width for _ in range(depth)]
        self._hash_seeds = [i * 2654435761 for i in range(1, depth + 1)]

    def _hash(self, item: str, seed: int) -> int:
        h = int(hashlib.md5(f"{seed}{item}".encode()).hexdigest(), 16)
        return h % self.w

    def update(self, item: str, count: int = 1) -> None:
        for i, seed in enumerate(self._hash_seeds):
            self.table[i][self._hash(item, seed)] += count

    def query(self, item: str) -> int:
        """Returns min-over-rows estimate. Always >= true count."""
        return min(
            self.table[i][self._hash(item, seed)]
            for i, seed in enumerate(self._hash_seeds)
        )

    def merge(self, other: "CountMinSketch") -> None:
        """Merge another sketch (element-wise add). Used for regional rollup."""
        assert self.w == other.w and self.d == other.d
        for i in range(self.d):
            for j in range(self.w):
                self.table[i][j] += other.table[i][j]

Algorithm 2: Bucketed Sliding Window with CMS

from collections import deque
import time

class SlidingWindowCMS:
    """
    60-minute sliding window using 60 × 1-minute CMS buckets.
    Old buckets are subtracted (or reset) as the window advances.
    """
    def __init__(self, window_minutes: int = 60, bucket_size_s: int = 60):
        self.bucket_size_s = bucket_size_s
        self.n_buckets = window_minutes
        self.buckets: deque[tuple[int, CountMinSketch]] = deque()  # (minute_epoch, CMS)
        self.current_bucket_ts = None
        self.current_cms = None

    def _get_bucket_ts(self, ts: float) -> int:
        return int(ts // self.bucket_size_s) * self.bucket_size_s

    def update(self, term: str, ts: float, count: int = 1) -> None:
        bucket_ts = self._get_bucket_ts(ts)

        # Rotate to new bucket if minute has advanced
        if bucket_ts != self.current_bucket_ts:
            if self.current_bucket_ts is not None:
                self.buckets.append((self.current_bucket_ts, self.current_cms))
            self.current_bucket_ts = bucket_ts
            self.current_cms = CountMinSketch()

        # Evict buckets outside the window
        cutoff = bucket_ts - (self.n_buckets * self.bucket_size_s)
        while self.buckets and self.buckets[0][0] < cutoff:
            self.buckets.popleft()

        self.current_cms.update(term, count)

    def query(self, term: str) -> int:
        """Sum estimates across all active buckets."""
        total = self.current_cms.query(term) if self.current_cms else 0
        for _, cms in self.buckets:
            total += cms.query(term)
        return total

Algorithm 3: Space-Saving Top-K

class SpaceSaving:
    """
    Metwally et al. Space-Saving algorithm for top-K heavy hitters.
    Guarantees: any item with true frequency > N/k is in the result.
    Stores exactly k (term, count, error) tuples.
    """
    def __init__(self, k: int = 500):
        self.k = k
        self.counts: dict[str, int] = {}   # term → estimated count
        self.errors: dict[str, int] = {}   # term → max overcount error

    def update(self, term: str, count: int = 1) -> None:
        if term in self.counts:
            self.counts[term] += count
        elif len(self.counts) < self.k:
            self.counts[term] = count
            self.errors[term] = 0
        else:
            # Evict minimum-count item; new item inherits its count
            min_term = min(self.counts, key=self.counts.get)
            min_count = self.counts.pop(min_term)
            self.errors.pop(min_term)
            self.counts[term] = min_count + count
            self.errors[term] = min_count   # inherited error

    def top_k(self, k: int = 30) -> list[tuple[str, int]]:
        return sorted(self.counts.items(), key=lambda x: -x[1])[:k]

#Design Patterns Used

Pattern Where Why
Count-Min Sketch Flink per-region frequency estimation O(1) memory regardless of unique hashtag count; ~54 KB vs GBs for exact HashMap
Sliding Window 60 × 1-min buckets per region Smooth trend curve; eliminates tumbling-window boundary resets visible to users
Fan-Out City → Country → Global hierarchical aggregation Each tweet event fans out to all ancestor woeids; no join needed at query time
CQRS Flink (write/compute) vs Trends API (read from Redis) Independent scaling; read path is pure Redis, zero Flink involvement
Event Sourcing Kafka tweet-events as immutable log Replay for debugging trend computation bugs; backfill historical snapshots
Circuit Breaker Trends API → Redis fallback to PostgreSQL snapshot If Redis unavailable, serve last persisted snapshot (slightly stale but correct)
Sharding PostgreSQL trend_snapshots partitioned by window_end Range partitions enable fast time-range queries; old partitions drop atomically
Bloom Filters Bot filter: track seen (user_id, hashtag) pairs per window Prevent single user from contributing more than N times without exact set storage

#8. Failure Handling & Edge Cases

#What Happens When X Fails?

Failure Impact Mitigation
Flink job crashes Trend refresh stops; Redis data goes stale after TTL Flink checkpointing every 30s; restarts from last checkpoint; Redis TTL=120s gives 2-cycle grace period
Redis cluster shard fails Trend reads fail for ~1/6 of woeids Redis Cluster auto-failover to replica (< 30s); Trends API falls back to PostgreSQL last snapshot for affected woeids
Kafka broker down Tweet events buffered in producers; Flink pauses Kafka RF=3; producer retries; Flink resumes from last offset on recovery; max ~30s gap in trend data
NLP Phrase Extractor down Only hashtag trends; no phrase trends Hashtag trends continue independently; NLP is separate consumer group; alert P1
PostgreSQL primary fails Snapshot writes fail; history reads on replicas OK Patroni failover < 30s; Flink snapshot writer buffers to local Flink state; replays on recovery
Bot filter model stale Stale bot scores → some bot events included Trend manipulation bounded by rate-limit (max 3 contributions/user/window); post-hoc suppression via anomaly detector
CDN cache poisoning Stale or corrupted trend returned globally CDN TTL=60s self-heals; Trends API endpoint has cache-control: max-age=60; purge API available for P0 incidents

#Domain-Specific Edge Cases

#9. Monitoring & Observability

#Key Metrics

Metric Target Alert Threshold
Flink job lag (tweet-events) < 30K msgs > 100K msgs for 3 min
Trend refresh latency (tweet → Redis) < 60 s P95 > 90 s for 5 min
Trends API P99 latency < 100 ms > 250 ms for 3 min
Redis hit rate for trends > 99.9% < 99% for 2 min
CDN cache hit rate (global trends) > 90% < 80% for 5 min
Top-K accuracy (sampled vs ground truth) < 5% rank error > 10% error in hourly check
Bot-filtered tweet % 20–40% (expected range) < 5% or > 60% (filter malfunction)
PostgreSQL snapshot write lag < 5 s > 30 s

#Alerting Strategy

P0 — Page immediately (any time)

P1 — Page during business hours

P2 — Ticket, no page

#SLAs / SLOs

Trend Freshness:
  SLO: 95% of trend refresh cycles complete within 60s of window close
  SLA: No trend data older than 3 minutes served to users at any time

API Latency:
  SLO: P50 < 20ms, P95 < 60ms, P99 < 100ms
  SLA: 99.9% of requests within 100ms over 30-day window

Availability:
  SLA: 99.99% uptime for Trends API = < 52 min downtime/year
  Acceptable degradation: Serve 5-min-old trends during Redis failover (< 60s window)

Accuracy:
  SLO: Items with true rank ≤ 30 appear in served top-30 with ≥ 95% probability
  (Space-Saving guarantee: any item with freq > N/500 is captured)

#10. Trade-off Summary

Decision Chose Over Because
Count-Min Sketch for frequency Probabilistic CMS (~54 KB/region) Exact HashMap (GBs for 200K unique terms) 200K unique hashtags/hr × 551 regions × 8B = 880 MB exact; CMS gives 54 KB × 551 = 30 MB; acceptable ε=0.1% error
Bucketed sliding window (60 × 1 min) Approximate (±1 min lag) Exact sliding window (store all events) Storing all 8.7K hashtag events/s in memory is infeasible; 1-min bucket lag is imperceptible to users
Space-Saving top-K (k=500) Probabilistic top-K Sort all 200K terms every 60s Sorting 200K terms every 60s × 551 regions = 110M sorts/min; Space-Saving is O(1) per event, O(k log k) per snapshot
Redis ZSET as serving store Pre-computed in Redis Query Flink state at read time Trend reads are 87K QPS; querying Flink state at read time would serialize all reads through Flink — impossible
Velocity over raw volume Trend score = count × velocity Raw volume ranking Raw volume favors permanently large topics (e.g., "#COVID"); velocity captures what's newly exciting — aligns with user expectation of "trending"
60s refresh cycle 60s Real-time (per-tweet update) Per-tweet Redis update = 8.7K writes/s to 551 ZSETs = 4.8M Redis ops/s; unnecessary given human perception of trends evolves over minutes
Fan-up (city → country → global) Each tweet emits to all ancestor woeids Rollup aggregation at query time Query-time rollup requires fetching all city ZSETs for a country — 500 Redis calls; fan-up pre-aggregates without joins
Personalization on top-500 result set Thin personalization layer Full personalized ranking from raw events Per-user Flink state for 350M users is infeasible; top-500 → top-30 rerank is a simple 5ms in-process operation

#11. Extensions & Follow-ups

#What Would You Add With More Time?

  1. Real-Time Trend Alerts via SSE: Push trend updates to open web clients via Server-Sent Events (SSE) so the trend panel updates without polling. Implemented via a Pub/Sub channel per woeid.
  2. Trend Lifecycle Visualization: Track each trend from emergence → peak → decay; expose a trend timeline chart via the history API. Requires storing velocity time-series in addition to volume.
  3. Multi-Language Phrase Clustering: #WorldCup and #CopaMundial are the same event. Cluster semantically equivalent terms using multilingual embeddings from a Vector DB; surface as a single unified trend.
  4. A/B Testing Trend Ranking Algorithms: Shadow-rank with alternative velocity functions (log-velocity, burst detection) and measure click-through rate (CTR) on trending topics.
  5. Trend Influence Attribution: For each trending topic, identify the "seed" tweet or account that first triggered the trend — useful for journalism and attribution.
  6. Predictive Trend Alerting: Use ARIMA or LSTM on historical trend baselines to predict which topics are likely to trend 10–15 minutes in advance, enabling proactive content moderation.

#How Would This Change at 100x Scale?

#12. Cross-References

Topic Connection
Rate Limiter (#2) Sliding window algorithm is shared; per-user contribution rate-limiting in Flink
Recommendation Engine (#23) Personalized trend reranking uses user interest vectors; similar to collaborative filtering signals
Fraud Detection (#24) Bot filtering and coordinated inauthentic behavior detection share feature patterns
Tinder Feed (#26) Fan-out pattern for candidate stack pre-computation mirrors trend pre-computation to Redis
Web Crawler (#12) Bloom filter deduplication pattern mirrors per-user tweet-contribution dedup

#Building Blocks Used

#Concepts Used