Real-Time Streaming Data Platform Framework

Replace {{PLACEHOLDERS}} with your environment values, then deploy to your stack.

# Real-Time Streaming Platform Framework

**Version:** {{FRAMEWORK_VERSION}}  
**Owner:** {{STREAMING_PLATFORM_TEAM}}  
**Last Updated:** {{LAST_UPDATED_DATE}}  
**Organization:** {{ORGANIZATION_NAME}}

---

## Executive Summary

This framework defines the **real-time streaming data platform** for {{ORGANIZATION_NAME}}, supporting event-driven architectures using **Apache Kafka** and/or **Amazon Kinesis**. It covers ingestion, schema registry, stream processing, serving layers, and observability - enabling teams to build reliable, governable streaming pipelines from source to consumer.

**Platform goals:**

- End-to-end latency < {{PLATFORM_LATENCY_P99_MS}} ms p99 for Tier 1 streams
- Schema-safe evolution with backward/forward compatibility
- Exactly-once or at-least-once semantics - explicitly declared per pipeline
- Unified observability: lag, throughput, errors, consumer health

---

## 1. Reference Architecture

```
 PRODUCERS                    STREAMING PLATFORM                    CONSUMERS
┌─────────────┐         ┌─────────────────────────────┐         ┌─────────────┐
│ Apps / IoT  │──publish│  INGESTION LAYER            │         │ Real-time   │
│ OLTP CDC    │────────▶│  Kafka / Kinesis topics     │──subscribe▶ dashboards │
│ Web / Mobile│         │  + dead-letter topics       │         │ ML features │
└─────────────┘         └──────────────┬──────────────┘         │ Microservices│
                                       │                        │ Lake bronze │
                                       ▼                        └─────────────┘
                        ┌─────────────────────────────┐
                        │  SCHEMA REGISTRY            │
                        │  (Confluent / Glue Schema)  │
                        └──────────────┬──────────────┘
                                       │
                                       ▼
                        ┌─────────────────────────────┐
                        │  STREAM PROCESSING          │
                        │  Flink / Kafka Streams /    │
                        │  ksqlDB / Spark Structured  │
                        └──────────────┬──────────────┘
                                       │
                    ┌──────────────────┼──────────────────┐
                    ▼                  ▼                  ▼
             ┌───────────┐      ┌───────────┐      ┌───────────┐
             │ SERVING   │      │ MATERIALIZED│     │ SINK TO   │
             │ (API/Cache)│     │ VIEWS / KV  │     │ LAKE/DB   │
             └───────────┘      └───────────┘      └───────────┘
                                       │
                                       ▼
                        ┌─────────────────────────────┐
                        │  OBSERVABILITY              │
                        │  Metrics, logs, tracing, lag  │
                        └─────────────────────────────┘
```

---

## 2. Platform Selection Guide

| Criterion | Apache Kafka (MSK) | Amazon Kinesis |
|-----------|-------------------|----------------|
| Ecosystem | Rich (Connect, Streams, Flink) | AWS-native, simpler ops |
| Throughput | Very high with partitioning | Shard-based scaling |
| Retention | Long (days-weeks configurable) | Default 24h-365d |
| Schema Registry | Confluent / Apicurio | Glue Schema Registry |
| Multi-consumer replay | Excellent | Per-shard iterators |
| Operational model | {{KAFKA_OPS_MODEL}} | Fully managed |

**{{ORGANIZATION_NAME}} standard:** {{PRIMARY_STREAMING_PLATFORM}} for new workloads; {{SECONDARY_PLATFORM}} for AWS-native integrations only.

---

## 3. Ingestion Layer

### 3.1 Topic / Stream Naming Convention


// ... download full template for remaining code