Skip to main content

data.loadoverrides

LoadOverrides Objects​

@dataclass
class LoadOverrides()

Per-run overrides for the bronze/silver loaders.

These flip default validation/gating behaviour for a single run() call — typically during testing, a failed-load recovery, or an extra/manual run. Every field defaults to False meaning "use the YAMLA simple way to write configurations. It's basically a list that computers can read easily. / ConfigManager default"; no override changes behaviour unless explicitly set to True. Each field disables exactly one gate, and the fields are independent — they compose without overriding one another.

Passed via the overrides= argument of load_data_bronze.run() and load_data_silver.run(). The Load_bronze / Load_silver notebooks build one from their parameters, so the same overrides can be supplied from a runMultiple DAG args dict.

Notebook hooks (the data load itself still runs — only the named user-defined notebook is skipped):

Attributes:

  • skip_prebronze bool - Skip the pre-bronze notebook. Common case — the pre-bronze hook often re-fetches source data, which you don't want during a test or a manual reload.

  • skip_midbronze bool - Skip the mid-bronze notebook.

  • skip_postbronze bool - Skip the post-bronze notebook.

  • skip_presilver bool - Skip the pre-silver notebook.

  • skip_postsilver bool - Skip the post-silver notebook.

  • skip_stale_check bool - Skip Validation 1 (the bronzemaxfileagehours stale-file check). Use when a source file is older than the window but still the correct data (e.g. an extra run after the normal schedule). An INFO line is logged when skipped. Bronze only.

  • force_reload_bronze bool - Disable source-unchanged detection (bronzeskipifsourceunchanged) so the load proceeds even when the source files are byte-for-byte unchanged. Bronze only.

    INDEPENDENT of skip_stale_check: force_reload_bronze only disables the "source is identical, skip the load" optimisation. It does NOT disable the stale-file check. So with force_reload_bronze=True and skip_stale_check=False the load proceeds past the unchanged gate, but a too-old file can still abort it via Validation 1. To ignore both gates, set both flags.

  • force_reload_silver bool - Force a full silver rebuild from bronze, ignoring the "silver is already up to date" gate. Silver only; the bronze counterpart is force_reload_bronze and the two are independent (each run() call is one layer). Silver normally skips work when the highest SYSTEMSTATETIMESTAMP in Bronze.his equals the one in Silver.his (full load) or when the merge changes no rows.

    Use it when Bronze.his was rewritten retroactively — same timestamps, changed values — which the timestamp gate would otherwise treat as unchanged. It rebuilds Silver.his from all of Bronze.his (a full overwrite in the full-load path, a truncate + re-merge in the merge path), the flag-driven equivalent of manually truncating Silver.his before a run. No effect on the

  • ```keephistory` - false`` path, which already reloads every run.