Medallion Architecture on Azure Databricks: A Practical Guide

The medallion architecture is the de-facto standard for lakehouse pipelines. It organizes data into three layers — bronze, silver, and gold — each with a clear purpose.

The three layers

The point of this structure is incremental trust. Bronze captures everything exactly as it arrived. Silver is trustworthy for analysis. Gold is shaped for dashboards and reports.

Building bronze

Bronze should be a faithful copy — minimal transformation, just enough to land the data:

raw = spark.readStream.format("cloudFiles") \
    .option("cloudFiles.format", "json") \
    .load("/mnt/landing/orders")
 
raw.writeStream \
    .format("delta") \
    .option("checkpointLocation", "/mnt/checkpoints/orders") \
    .outputMode("append") \
    .start("/mnt/lakehouse/bronze/orders")

Building silver

Silver is where the real engineering happens — cleaning, typing, deduplication:

silver = (bronze
    .filter(col("amount") > 0)                       # drop junk
    .dropDuplicates(["order_id"])                     # dedupe
    .withColumn("order_date", to_date(col("ts")))     # type casting
    .withColumn("region", when(col("country").isin("IN", "US", "UK"), col("country")).otherwise("OTHER"))
)
 
silver.write.mode("overwrite") \
    .format("delta") \
    .saveAsTable("silver.orders")

Delta Lake is what makes this practical — ACID transactions, time travel, and MERGE for incremental upserts.

Building gold

Gold is shaped for consumption — one grain, one purpose:

gold = (silver
    .groupBy("order_date", "region")
    .agg(
        sum("amount").alias("revenue"),
        count("order_id").alias("orders"),
    )
)
 
gold.write.mode("overwrite") \
    .format("delta") \
    .saveAsTable("gold.daily_sales")

Lessons learned

• Keep bronze immutable — never rewrite history; reprocess forward instead
• Test at silver — that's the layer your consumers will trust
• Use MERGE for silver-upserts on large identity tables
• Partition gold by date — dashboards filter by date more than anything
• Monitor stream lag — a silent stream is a broken stream

The medallion pattern removed most of our "we need to rebuild the table" conversations. When data is wrong, we fix it at silver once and everything downstream heals.