Skip to main content

Lesson 3 — Metadata-driven & schema-on-write

What you'll be able to do: explain why you declare your platform in YAML, and why every table exists (empty) right after a deploy, before any data has loaded.

New to YAML?

YAML is just an indented list of key: value settings that both people and computers read easily. If it's unfamiliar, skim JSON & YAML first — this lesson is about why EasyFabric leans on it, not the syntax.

The idea, part 1 — metadata-driven​

There are two ways to build the same table.

  • Imperative: write the step-by-step code that creates it — CREATE TABLE ..., then the Spark job that loads it, then the view, then the history logic. You describe how.
  • Declarative: write down what you want — "a customers table, these columns, keep history, primary key CustomerID" — and let a tool figure out the how.

EasyFabric is fully declarative. The YAML is metadata: data about your data platform. From that one description, the framework derives the Bronze table, the history table, the Silver tables, the SQL views, and the loading logic. This is the framework's core principle in action:

Everything is driven by YAML. The model, the connections, the environments — all metadata.

Why this matters, concretely: because the description is the single source of truth, everything stays consistent (every table gets the same standard treatment), reproducible (regenerate the whole platform from the YAML any time), and reviewable (a change to your platform is a diff in a text file, reviewed in a pull request like any code).

The idea, part 2 — schema-on-write​

The second principle follows from the first. There are two moments a table's shape ("schema") can be decided:

  • Schema-on-read: dump raw files now, figure out the structure later, when someone queries. Flexible up front — painful forever after (inconsistent formats, slow queries, quality problems caught too late).
  • Schema-on-write: decide the structure up front, and validate data as it's written in.

EasyFabric is schema-on-write. This leads to the single most surprising thing for newcomers:

After a deploy, every Bronze, Silver, and Gold object already exists in Fabric — they are just empty. Loading data is a separate step that fills structures that are already there. It never creates them on the fly.

Deploy builds the shape; loading adds the contents. They are two different actions, and keeping them separate is deliberate.

A direct consequence worth internalizing: a table that isn't declared in YAML is not part of your platform. Deployment won't create it, so it gets none of the medallion treatment — no Bronze landing, no history, no Silver, no lineage. If you want it, you declare it.

In EasyFabric​

Here's a real object declaration (from the demo platform's customers table). Notice it says what, never how:

Table:
  Connection: adv-advworks
  SourceTable: customers.csv
  DataPlatformObjectname: customers
  KeepHistory: true
  Columns:
  - SourceColumn: CustomerID
    SourceDataType: int
    IsPrimaryKey: true
  - SourceColumn: FirstName
    SourceDataType: varchar(255)
  - SourceColumn: EmailAddress
    SourceDataType: varchar(500)

From this, EasyFabric knows enough to build a Bronze landing table, a Bronze history table (because KeepHistory: true and there's a primary key), and the Silver table — all before a single row of customers.csv is read.

Check your understanding​

You deploy a fresh platform, then open Fabric and query your customers Silver table. It returns zero rows. Bug, or expected?

Expected. Deploy creates the shape; the tables exist but are empty until you run the loading step. Schema-on-write separates "the object exists" from "the object has data" on purpose. Loading is Lesson 7.

A teammate created a table directly in the Fabric lakehouse by hand, and wonders why it has no history table and never shows up in lineage. What happened?

It was never declared in YAML, so EasyFabric doesn't know about it. Only declared objects get the medallion treatment (Bronze/history/Silver/lineage). Because the platform is metadata-driven, the YAML is the source of truth — anything created outside it is invisible to the framework and will be ignored (and potentially overwritten) on the next deploy.

Why is "change your platform = edit a text file in Git" such a big deal?

Because it makes platform changes reviewable and reproducible. A new table or column is a diff someone can review in a pull request; the whole platform can be regenerated from the YAML at any time; and there's a full history of who changed what and when. That's the payoff of being declarative and metadata-driven.

Recap​

  • Metadata-driven: you declare the platform in YAML (the what); the framework derives the how.
  • Schema-on-write: structure is decided up front; deploy creates every object empty, and loading fills them — two separate steps.
  • If it isn't in the YAML, it isn't part of your platform.

Next: the components that turn YAML into a running platform — The moving parts →