Weather Kafka Pipeline
KafkaFlaskDockerSnowflakePower BI
An end-to-end streaming pipeline that ingests live weather from a Flask producer into Kafka topics. Consumers validate, enrich, and upsert into Snowflake. BI dashboards provide near real-time insight.
Highlights
- Idempotent consumers with retries and dead-letter topics.
- Schema evolution strategy, partitioning for scale.
- Dockerized local dev via Compose; CI on push to main.
- Snowflake staging → curated dimensional models.
- Structured logs, metrics hooks for observability.
Stack & Flow
Flask (producer) → Kafka → Python consumers → Snowflake → Power BI dashboards.