What is a Delta table?

Microsoft Fabric uses Delta tables in its Lakehouses.

Delta Tables are a data management layer providing ACID transactions, schema enforcement, and versioning to big data workloads, and are part of the open-source storage framework called Delta Lake, originally developed by Databricks. Underlying the framework, data is stored in parquet files.

Core Features:

ACID Transactions 

Ensurea data integrity during concurrent reads and writes, just like a traditional database.

Schema Enforcement & Evolution

Schemas prevents corrupt or inconsistent data from being written. e.g. Like a SQL database, columns have a type, which prevents writing a string into a date or numeric column. 

The schema can evolve over time. e.g., delete or add new columns.

Time Travel

Query previous versions of the data using timestamps or version numbers. This is great for debugging or auditing.

Fast Reads with Indexing 

Delta tables use data skipping and Z-order indexing to speed up queries by avoiding unnecessary file scans.

Efficient Upserts and Deletes 

Delta tables support MERGE, UPDATE, and DELETE operations.

Upserts and versioning support use for Slowly Changing Dimensions (SCD) 

Storage

Delta tables are stored in Parquet files with transaction logs (_delta_log) that track changes.

Fabric contains utilities to manage these including OPTIMIZE and VACUUM to ensure efficiency and reduce size.

Accessibility with code (python)

  # Create a Delta table
  df.write
  	.format("delta")
    .save("/mnt/datalake/sales_data")

  # Read it
  df = spark.read
  	.format("delta")
    .load("/mnt/datalake/sales_data")

  # Time travel
  df_old = spark.read
  	.format("delta")
    .option("versionAsOf", 5)
  	.load("/mnt/datalake/sales_data")