Skip to main content

The mid-silver seam & supplemental loads

MidSilverNotebook sits at a precise point in every history-enabled silver load: after Silver.his is written for this run, and before Silver.dbo is rebuilt from it. That seam is what makes it useful for backfilling rows a normal bronze load can't reach — rows added to Silver.his inside the hook are picked up automatically when the rebuild runs immediately afterwards.

This page covers the seam itself, and load_data_silver.supplement_history, the API built for feeding it.

The seam, precisely​

Every history-enabled silver load (keephistory: true, full or merge) follows the same two steps, in order:

  1. Write this run's changes into Silver.his.
  2. Rebuild Silver.dbo by reading all of Silver.his, keeping one row per primary key: the row with the highest abs(SYSTEMSTATETIMESTAMP) (ties broken by SYSTEMSTATETIMESTAMP desc, so a positive timestamp beats a negative one at the same magnitude), and only if that timestamp is > 0. A key whose winning row is negative — i.e. deleted — is dropped from Silver.dbo entirely. See History in Fabric Lakehouse for the full mechanics of that column.

MidSilverNotebook runs between those two steps. Anything the hook appends to Silver.his — including rows it added itself — is already there when step 2 reads "all of Silver.his", so it flows into Silver.dbo on the same run, governed by the same abs(SYSTEMSTATETIMESTAMP) desc / > 0 window as every other row. There is no separate hand-off; the hook doesn't write to Silver.dbo itself, it just has to land its rows in Silver.his before the rebuild reads it.

This is what distinguishes the mid-silver slot from its neighbours:

  • PreSilverNotebook runs before Silver.his is touched at all — it can't hand the load extra rows to merge in, only prepare things ahead of it.
  • PostSilverNotebook runs after Silver.dbo has already been rebuilt — anything it writes to Silver.his waits for the next load to appear in Silver.dbo.
  • MidSilverNotebook is the only slot that sits inside the seam between the two.

The supplemental load API​

def supplement_history(
    data_frame: DataFrame,
    table_config: TableConfig,
    config_manager: ConfigManager = None,
) -> str | None

load_data_silver.supplement_history is the entry point a mid-silver notebook uses to hand EasyFabric rows that are already in silver shape and destined straight for Silver.his, without routing them through bronze. table_config is always the target table's own config — the one whose Silver.his receives the rows; there is no separate config for the supplemental source.

Anti-join contract and its grain. Before appending, supplement_history anti-joins data_frame against the existing Silver.his on the pair (SYSTEMPRIMARYKEY, SYSTEMSTATETIMESTAMP) — the same primary-key grain the rest of his history is written at, not just the primary key alone. A row is skipped only when both the key and the exact timestamp are already present; the same key at a different timestamp is a new version and gets appended. That grain is what makes repeat calls with the same data_frame idempotent — re-running the notebook against an unchanged archive appends nothing the second time.

Schema requirements.

  • data_frame must already be in silver shape. silverexpression / converted-column transforms are not applied — those describe how the target's own bronze columns become silver columns, and this DataFrame never touches bronze.
  • It must contain the target's primary-key columns (table_config.get_keycolumn_array() — there is no per-call override).
  • SYSTEMSTATETIMESTAMP is mandatory. Supplemental rows carry their own historical effective time — the archive's own recorded date — and the framework will not invent one for them, because a supplemental source can itself hold history. A data_frame without the column is a hard failure.
  • SYSTEMSOURCETAG is optional; it defaults to "Not applicable" when absent.
  • data_frame need not carry every Silver.his column, the same as a bronze source file. Key columns are mandatory; any other column Silver.his has that data_frame lacks is logged as a column-mismatch warning and filled with NULL — unless it is non-nullable in Silver.his, which is a hard failure, since there is no value to fill it with. Extra columns data_frame carries that aren't part of Silver.his are dropped — only the target's own schema is written.

What the framework validates, all raising before anything is written:

  • The target has keephistory: true — supplement_history writes into Silver.his, and a table without history has none.
  • The target has primary-key columns configured.
  • data_frame contains every one of those primary-key columns.
  • data_frame has no duplicate rows for the same key at the same abs(SYSTEMSTATETIMESTAMP) — that combination must be unique within the batch you supply, the same constraint Silver.his itself upholds.
  • data_frame supplies SYSTEMSTATETIMESTAMP.
  • data_frame isn't missing any non-nullable column of Silver.his.

An empty data_frame is not an error — the call logs that there's nothing to append and returns without writing.

What appears in the run log: a start line naming the target table, a row count line for what was appended (or that everything was already present and skipped, keeping the call visibly idempotent), and — on failure — an error line naming supplement_history as the failing loader and the target table.

Only Silver.his is written. supplement_history itself never touches Silver.dbo — that's why calling it from MidSilverNotebook matters: it's what puts the new rows in Silver.his far enough ahead of the seam's rebuild step to be picked up automatically, on the same run.

A reference notebook​

The motivating case: rows recovered from another system, sitting in an archive table such as Bronze.his.<table>_archive, that never went through this table's own bronze ingest and need to land in Silver.his anyway. Configured as the table's MidSilverNotebook:

from easyfabric import load_data_silver, load_meta_data

obj = load_meta_data.get_object_by_file(object_yaml_file)

df_archive = spark.table("Bronze.his.customers_archive").select(
    "CustomerID", "Name", "Email", "SignupDate", "SYSTEMSTATETIMESTAMP"
)

load_data_silver.supplement_history(df_archive, obj)

No manual "which rows are missing" step is needed — supplement_history's own anti-join already limits the append to rows not already in Silver.his, so the notebook only has to shape the archive into silver columns plus the target's primary key and the archive's own SYSTEMSTATETIMESTAMP, which it must supply. Because the hook runs inside the seam, those rows are in Silver.dbo by the time this same load finishes — no separate rebuild step to remember.

When not to use it​

supplement_history bypasses bronze entirely: the rows it appends never land in Bronze.his, are never covered by bronze's own schema validation or business-key checks, and never go through silverexpression type conversion. That's the right trade-off for a one-off backfill from a source that no longer exists to be re-ingested from. It's the wrong tool if the rows belong in the table's regular delivery — if the source can still produce them, fix the bronze ingest instead (extend the source query, add a connection, adjust a filter) so they flow through the normal Bronze.dbo → Bronze.his → Silver.his → Silver.dbo pipelineAn automated "conveyor belt" that moves data from one place to another or performs a task automatically. with the rest of the table's data: validated, converted, and reconciled the same way on every run, instead of depending on a notebook someone has to remember to re-run.