Skip to main content

Lesson 7 — Loading data end-to-end

What you'll be able to do: trace a single row from a source file all the way to a report-ready Gold table, and name what orchestrates the trip.

Lesson 6 left you with empty tables. Now we fill them. Remember from Lesson 4: this is run time — a notebook running inside Fabric, using the wheel package, moving data through the layers you already deployed.

The idea​

Loading isn't one big script. It's a conveyor belt driven by the same metadata you wrote. A single orchestrator reads your object YAMLA simple way to write configurations. It's basically a list that computers can read easily. files, works out what needs loading and in what order, and calls the same standard loading logic for each table. Because it's metadata-driven, adding a table earlier changed nothing about how loading runs — the orchestrator just picks up one more object.

Follow one row​

Trace a customer record from customers.csv to a Gold report:

  1. Source → Bronze. The row is read from the file using the connection's parse settings and landed in the Bronze landing table (Bronze.dbo) as delivered — no cleaning. Because the object keeps history, the row is also compared against the history table in the same lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database. (Bronze.his) and recorded as new, updated, or unchanged.
  2. Bronze → Silver. The row is cleaned and typed per the object's column definitions. By default Silver keeps both the original and the converted value, and quality checks run here — this is where "trustworthy" happens.
  3. Silver → Gold. The row feeds the model (Lesson 8): joined with other Silver tables into facts and dimensions, ready for the semantic modelThe "brain" of your data that tells Power BI how different pieces of information relate to each other. and Power BI.

That's the whole journey: raw truth → clean and checked → business-shaped.

In EasyFabric — the orchestrator​

The notebook that drives this is DAG_Complete. It reads your object metadata from the Meta lakehouse and fans out into one child notebook per stage — DAG_Loader_Bronze, DAG_Loader_Silver, DAG_Gold, and DAG_Tabular — so it can load an entire workspace (all layers) or just part of it. In production it's scheduled so the platform refreshes on its own.

Two controls are worth knowing early, because both come straight from metadata:

  • Load order — objects load in groups by their BronzeLoadOrder / SilverLoadOrder (lower values go first). Use it when one table must exist before another. The loader groups objects by load order and processes each group in turn.
  • Concurrency — set on the connection (BronzeNotebookConcurrency / SilverNotebookConcurrency) to cap how many things hit a source at once, so you don't overwhelm it.

For settings that exist at both levels — like load order and parse settings — the object-level value always wins over the connection-level one, the same "lowest level wins" rule you saw in Lesson 6. (Concurrency exists only on the connection.)

A single run of the loader, at its simplest, is just choosing what to load:

folder_path = "Files/Objects"        # which objects, for Bronze and Silver
model_name = "DM"                    # which model, for Gold and Tabular
table_types = ["Fact", "Dim"]        # which model tables

Bronze runs first; Silver only runs if Bronze succeeded (or was skipped); Gold runs the model. You can load one layer, one source folder, or one object for testing — the shape is always the same.

Go deeper

The full run-book — filtering objects, per-run overrides like force_reload_bronze, running a notebook without touching the original — is in Loading data — hands-on. The conceptual walk of each layer is Loading data — General.

Check your understanding​

Table B is a lookup that Table A depends on. How do you make sure B loads first, and where does that setting live?

Give B a lower BronzeLoadOrder / SilverLoadOrder than A. It's an object setting in B's YAML (or on the connection, shared) — metadata, like everything else. The loader notebooks group objects by load order and run the lower group first.

The Silver step didn't run for a table. What's the most likely reason, given how the loader sequences layers?

Silver only runs if Bronze succeeded (or was explicitly skipped). If Bronze failed for that object, Silver is skipped by design — you don't clean data that didn't land. Check the Bronze step first.

You added a new object yesterday. What did you have to change in the orchestration to make it pick the new table up?

Nothing. The loader is metadata-driven — it reads whatever object YAML files are present. A new declared object is picked up automatically. That's the payoff of the whole declarative design: the orchestration never has to know about individual tables.

Recap​

  • Loading is run time: DAG_Complete fans out into the Bronze, Silver, Gold and Tabular loader notebooks — source → Bronze → Silver → Gold — calling the wheel package per table.
  • A row travels raw (+ history) → clean and checked → business-shaped.
  • Load order and concurrency are metadata controls; object-level beats connection-level; new objects are picked up automatically.

Next: the Gold layer and the model that shapes it — Modeling Gold for analytics →