Data ArchitectureVER_4.02
Medallion Lakehouse Pipeline
PySparkdbtAzure DatabricksADLS Gen2Apache Airflow
System Architecture
Logic Flow & Infrastructure Analysis
01_The_Problem
Fragmented raw data and inconsistent business definitions made analytics unreliable and hard to evolve.
02_Architecture_Solution
Raw events landed in bronze on ADLS Gen2. PySpark jobs on Databricks refined records into conformed silver tables with quality gates. dbt models built curated gold layer marts (star schema) consumed by BI and analytics teams. Airflow orchestrated the full DAG with retries, SLAs, and lineage.
Execution Metrics
Throughput / Volume
TB-scale daily ingest
Performance / Latency
Hourly refresh cycles