Skip to main content

Lesson 7 — Loading data end-to-end

What you'll be able to do: trace a single row from a source file all the way to a report-ready Gold table, and name what orchestrates the trip.

Lesson 6 left you with empty tables. Now we fill them. Remember from Lesson 4: this is run time — a notebook running inside Fabric, using the wheel package, moving data through the layers you already deployed.

The idea​

Loading isn't one big script. It's a conveyor belt driven by the same metadata you wrote. A single orchestrator reads your object YAML files, works out what needs loading and in what order, and calls the same standard loading logic for each table. Because it's metadata-driven, adding a table earlier changed nothing about how loading runs — the orchestrator just picks up one more object.

Follow one row​

Trace a customer record from customers.csv to a Gold report:

  1. Source → Bronze. The row is read from the file using the connection's parse settings and landed in the Bronze table as delivered — no cleaning. Because the object keeps history, the row is also compared against the history table and recorded as new, updated, or unchanged.
  2. Bronze → Silver. The row is cleaned and typed per the object's column definitions. By default Silver keeps both the original and the converted value, and quality checks run here — this is where "trustworthy" happens.
  3. Silver → Gold. The row feeds the model (Lesson 8): joined with other Silver tables into facts and dimensions, ready for the semantic model and Power BI.

That's the whole journey: raw truth → clean and checked → business-shaped.

In EasyFabric — the orchestrator​

The notebook that drives this is DAG_LayerLoader. It reads your object metadata from the Meta lakehouse and can load an entire workspace (all layers) or just part of it. In production it's scheduled so the platform refreshes on its own.

Two controls are worth knowing early, because both come straight from metadata:

  • Load order — objects load in sequence by their load_order (default 0; negative values go first). Use it when one table must exist before another. The loader groups objects by load order and processes each group in turn.
  • Concurrency — set on the connection or object to cap how many things hit a source at once, so you don't overwhelm it.

Object-level settings always win over connection-level settings — the same "lowest level wins" rule you saw with parse settings in Lesson 6.

A single run of the loader, at its simplest, is just choosing what to load:

folder_path = "Files/Objects"                 # which objects
layers = ["Bronze", "Silver", "Gold"]         # which layers
model_file = "Files/Model/DM/model.yaml"      # which model, for Gold

Bronze runs first; Silver only runs if Bronze succeeded (or was skipped); Gold runs the model. You can load one layer, one source folder, or one object for testing — the shape is always the same.

Go deeper

The full run-book — filtering objects, per-run overrides like force_reload, running a notebook without touching the original — is in Loading data — hands-on. The conceptual walk of each layer is Loading data — General.

Check your understanding​

Table B is a lookup that Table A depends on. How do you make sure B loads first, and where does that setting live?

Give B a lower load_order than A (e.g. B at -10, A at the default 0). It's an object setting in B's YAML — metadata, like everything else. The DAG_LayerLoader groups objects by load order and runs the lower group first.

The Silver step didn't run for a table. What's the most likely reason, given how the loader sequences layers?

Silver only runs if Bronze succeeded (or was explicitly skipped). If Bronze failed for that object, Silver is skipped by design — you don't clean data that didn't land. Check the Bronze step first.

You added a new object yesterday. What did you have to change in the LayerLoader to make it pick the new table up?

Nothing. The loader is metadata-driven — it reads whatever object YAML files are present. A new declared object is picked up automatically. That's the payoff of the whole declarative design: the orchestration never has to know about individual tables.

Recap​

  • Loading is run time: DAG_LayerLoader orchestrates source → Bronze → Silver → Gold, calling the wheel package per table.
  • A row travels raw (+ history) → clean and checked → business-shaped.
  • Load order and concurrency are metadata controls; object-level beats connection-level; new objects are picked up automatically.

Next: the Gold layer and the model that shapes it — Modeling Gold for analytics →