Fabric - Data storage types

 

DRAFT DOCUMENT. DO NOT READ YET

WareHouse 

This is most suitable as a SQL Database replacement as it is T-SQL based, has good transactional support, handles large warehouses with ease, and great security options down to row and column level permissions.

T-SQL can be used to query and change the data, with support for stored procedures, views and functions, just like a SQL Database. Ideally, the data should be structured, and while support for semi-structured (JSON) is provided, features are limited in T-SQL.

LakeHouse

A LakeHouse is best suited for complex transformations due to its support for python notebooks, which not only provide flexibility in transformational computations, but also scalable processing through the Spark engine. The LakeHouse is the only option for unstructured batch data as it allows for file storage (and folders). Data can be read using notebooks, but also T-SQL, giving this storage type massive flexibility. You cannot use Stored procedures or Views and cannot make data changes with T-SQL.

A LakeHouse is useful for data analysis because it presents like a database to applications like Power BI or graphical charting plugins like HighCharts, but it also allows for many more connection types and data manipulation options, making it more flexible than a typical RDBMS like SQL Server.

EventHouse

An EventHouse is best suited to large volumes of near real-time streaming data, such as IoT, logs, and real time application feeds. Data is generally high volume semi-structured or unstructured. Once stored, the data is mostly read-only though the use of the Kusto Query Language (KQL), which is arguable simpler and more intuitive than T-SQL. T-SQL isn't supported. You can create canned queries (KQL querysets), and you can call these or issue ad-hoc queries through the API.