Skip to main content

Loading data - General

Loading data is happening at multiple places with EasyFabric. In general data will move from source to bronze, to silver and will end at gold.

How it works​

The loading process is driven by metadata stored in the Files section of the Meta lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database.. The DAG_Loader_Bronze and DAG_Loader_Silver notebooks read these object YAMLA simple way to write configurations. It's basically a list that computers can read easily. configurations to determine what needs to be loaded, layer by layer. The DAG_Complete notebook chains the full run — Bronze, Silver, Gold and Tabular — in one go.

The load_meta_data.get_objects_by_folder function traverses the specified folder paths, reading each object configuration and its associated connection settings. This allows for centralized management of both source-specific and object-specific behaviors.

Concurrency and Load Order​

Managing source system load is critical. EasyFabric allows you to control this through:

  • Concurrency: Set on the connection (BronzeNotebookConcurrency / SilverNotebookConcurrency) to limit simultaneous connections to a source.
  • Load Order: Objects are processed in groups based on their BronzeLoadOrder / SilverLoadOrder setting (default is 0, settable on the connection or per object). You can use negative values (e.g., -10) to prioritize specific objects.

The loader notebooks identify all unique load orders and process them sequentially, creating a separate DAG for each order group. Object-level settings always take precedence over connection-level settings.

Source​

Loading data from the source can be very diverse. So there is not just one option that will handle all situations. Fabric has support for all kind of datasources and can also create shortcuts to sources. This can help by reducing the movement of data. If we look from a datawarehouse perspective, we commonly want to track data as much as possible, that's why a datawarehouse holds historized data, which means it can recall how the data looked like, when it was loaded into the datawarehouse. This will be explained further on at Bronze.

EasyFabric supports the following sources:

  • Azure Blob storage
  • Custom notebook connector
  • Fabric Files section
  • Fabric Shortcuts

EasyFabric supports the following sourcetypes

  • CSV
  • Excel (xlsx)
  • XML
  • JSON
  • Parquet
  • Notebook*
  • With a notebook it is possible to handle custom cases. Read more about this in the advanced topic about custom extensions.

Bronze​

Once data is retrieved from the source it is processed into the bronze layer. Based on the object configuration the data from the source is loaded into the bronze layer. At the deploymentThe process of "pushing a button" to make your configuration actual, working software in the cloud. stage, the tables are created in Bronze that will be loaded. If keep history is set on an object there will be 2 tables in the Bronze lakehouse: a landing table in the dbo schema for the rows as delivered by the source, and a history table in the his schema (Bronze.his). In this step EasyFabric will have to do a lot of things to make sure, the data is copied correctly into the tables, at this point the following things will be done:

  • Renaming the source columns into destination names
  • Ordering the columns in the exact position as the bronze table requires
  • Fill columns that are not delivered with blanks (optional)
  • Keep data in it's purest form possible. No checks at this point yet.
  • Process the delivered data also in the history

When processing the history, the data will be compared to the existing rows in the history table (Bronze.his), and if the row is new, updated or deleted, it will write a record to the table according to this method.

Now the data is available in Bronze. Let's move on to Silver.

Silver​

The silver tables are filled with the most recent data from the history tables in Bronze. For convenience the SYSTEMSTATETIMESTAMP is also copied as well as the last known deleted records. The latter can be easily skipped by adding a filter SYSTEMSTATETIMESTAMP > 0. The quality check that runs today is key integrity: duplicate keys are detected during loading and handled according to the configured violation action (BronzeKeyViolationAction: raise, keepone, or remove). Declarative quality checks such as mandatory/NOT NULL, range, or RegEx validation are not implemented — if you need them, add them in a custom extension notebook.

Gold​

The gold layer is based on the model that is supplied by the model.yaml file. It is possible to have multiple models in one Gold layer: each entry in the models: list of the environment configuration defines its own model (for example a base model plus a derived model), and the loaders select a model by its name. The gold layer tables have to be filled by data. This data is modelled in a certain way that it will suit the business needs. That's why there will be some transformation necessary between the data from silver, before loading it into the gold layer.

Read further here for more info on Gold loading.