Streaming Kappa Architecture

Replace {{PLACEHOLDERS}} with your environment values, then deploy to your stack.

# Streaming Lambda Architecture (Kappa Pattern)

> DE Architecture · {{ORGANIZATION_NAME}}

## Overview

Real-time analytics architecture using Kappa pattern: single stream processing path, no separate batch layer for speed-critical use cases.

## Architecture

```
 Producers                Stream Bus              Processing              Serving
┌──────────┐            ┌──────────┐            ┌──────────┐            ┌──────────┐
│ Apps/IoT │──events──▶ │ {{KAFKA}} │──consume──▶│ {{FLINK}} │──sink────▶│ {{OLAP}}  │
│ DB CDC   │            │ /Kinesis │            │ /Spark SS│            │ /Cache   │
└──────────┘            └──────────┘            └──────────┘            └──────────┘
                               │                       │
                               ▼                       ▼
                         Schema Registry         Dead Letter Queue
                         ({{REGISTRY}})          s3://{{DLQ_PATH}}
```

## Event Workflow

1. **Produce** - Events keyed by `{{PARTITION_KEY}}` with Avro/Protobuf schema
2. **Validate** - Schema registry enforces compatibility (BACKWARD)
3. **Process** - Stream job: parse → enrich → window → aggregate
4. **Serve** - Push to Pinot/Druid/Redis for sub-second queries
5. **Archive** - Optional compacted topic → S3 for replay/compliance

## Windowing & Aggregation Example

- Tumbling 5-minute windows for transaction counts
- Session windows for user journey analytics
- Late data handling: allowed lateness {{LATENESS_MINUTES}} min

## Operational Concerns

| Concern | Pattern |
|---------|---------|
| Exactly-once | Idempotent sinks + transactional offsets |
| Backfill | Reset offsets or replay from S3 archive |
| Schema change | Registry review + dual-write migration |
| Monitoring | Lag, throughput, error rate dashboards |

## When to Use Kappa vs Lambda

- **Kappa:** Low-latency dashboards, fraud detection, operational metrics
- **Lambda (batch+stream):** Complex historical corrections, heavy transforms

## {{ORGANIZATION_NAME}} Stack

Replace {{KAFKA}}, {{FLINK}}, {{OLAP}} with your streaming stack (MSK, Flink, ClickHouse, etc.).