#🧱 Building Block: Apache Flink
#Overview
Apache Flink is a distributed stream processing framework designed for stateful computations over bounded (batch) and unbounded (stream) data with true event-time processing and exactly-once semantics.
#Key Concepts
- Event-Time Processing: Windows and triggers based on when events actually occurred, not when they arrive — handles out-of-order events correctly.
- Stateful Processing: Maintains state (counters, aggregates, windows) across events with checkpointing for fault tolerance.
- Exactly-Once Semantics: Guarantees each event is processed exactly once, even during failures, via distributed snapshots (Chandy-Lamport algorithm).
- Windowing: Tumbling, sliding, session, and global windows for time-based aggregations.
- Watermarks: Mechanism to track event-time progress and handle late-arriving data.
- Savepoints: Manual snapshots of application state for upgrades, migrations, and A/B testing.
#When to Use
- Real-Time Feature Engineering: Computing sliding-window aggregates (e.g., "clicks in last 1 hour per user") for ML feature stores.
- Stream ETL: Transforming and enriching event streams before writing to data stores.
- Complex Event Processing (CEP): Detecting patterns across event sequences (e.g., fraud detection velocity counters).
- Real-Time Analytics: Dashboards and metrics with sub-second latency.
#Flink vs Spark Streaming
| Dimension | Flink | Spark Streaming |
|---|---|---|
| Processing Model | True streaming (event-by-event) | Micro-batch (small batch intervals) |
| Latency | Sub-second | Seconds (batch interval) |
| Event-Time | First-class support | Added later, less mature |
| Exactly-Once | Built-in with Kafka | Requires careful config |
| State Management | Built-in, managed | External (e.g., Redis) |
#Trade-offs
- Pros: Lowest latency, true event-time, exactly-once, excellent Kafka integration, rich windowing API.
- Cons: Steeper learning curve than Spark, smaller ecosystem, operational complexity (managing state backends like RocksDB).
#Used In This Repo
- Recommendation Engine (#23) — Real-time feature computation (genre distributions, velocity counters)
- Fraud Detection (#24) — Entity profile aggregation from transaction events