Fabric - Data ingestion

Getting data into a Lakehouse / Warehouse

Fabric offers many methods for getting data into Lakehouses and Warehouses. 

Method choice depends on the complexity of the source data, how easily you can connect to it from within notebook code, and how happy you are with Python vs no code processes.

Data Factory

Fabric contains a feature called Data Factory, derived from Azure Data Factory.

Data Factory is an orchestrator for managing pipelines of activities. An activity is a simple operation and the pipeline is the ordered process for running one or more activities.

Notebook

A notebook is an ordered script of comaands (usually Python).

You can call a notebook (or many notebooks) from Data Factory, but you can also schedule a notebook without using Data Factory, and as notebooks can contain multiple commands (cells), you can orchestrate many operations within a single notebook. A notebook can also call other notebooks, so the process can get quite involved.

You can run multiple notebooks in a batch using a Directed Acyclical Graph (DAG) whereby all notebooks run simultaneously or in a specific order (dependent on each other). If you run multiple notebooks in Data Factory, you will see that it creates a DAG in the pipeline, so these are different methods of doing the same thing (code vs no code)

External application

Fabric provides connection methods that can be used by external applications to push data into Fabric storage, so a bespoke application can populate fabric.

Stream data 

We can stream data from an application, log events etc. into Fabric using an EventStream.