loaders.file_tracker
get_tracker_file_path
def get_tracker_file_path(bronze_abfs_base_folder: str) -> str
Returns the ABFS path for the tracker file inside the table's Bronze folder.
The tracker always lives at::
<bronze_abfs_base_folder>/_tracking/tracker.json
For azblob (date-partitioned) pass the folder above the date partition,
e.g. abfss://…/Files/afas/Projecten.
For fabricfiles pass the folder that directly contains the files,
e.g. abfss://…/Files/afas/Projecten.
Arguments:
bronze_abfs_base_folder- Non-date-partitioned Bronze folder for this table.
Returns:
str- Full ABFS path to_tracking/tracker.json.
load_previous_snapshot
def load_previous_snapshot(tracker_path: str, shortcode: str) -> list[dict]
Loads this object's historical entries from the tracker (NDJSON — one entry per line).
One tracker file can hold more than one object: the fabricfiles path is derived from the Bronze folder and the source folder, so two objects reading the same source share it. Every entry records the object that wrote it, and reading returns only those — without the filter the second object compared against the first object's snapshot, concluded the source was unchanged and skipped its own load without an error (Bug #469).
Entries without a shortcode come from before the tracker recorded one and cannot be attributed, so they are ignored. That makes the first run after the upgrade reload once, which is recoverable; counting them would keep the silent skip alive, which is not.
Returns an empty list on first run (file does not exist yet).
Arguments:
tracker_path- ABFS path returned by :func:get_tracker_file_path.shortcode- Dataplatform object name (table_config.dataplatformobjectname), as written by :func:save_snapshot.
Returns:
List of entry dicts ordered oldest-first.
save_snapshot
def save_snapshot(tracker_path: str,
file_list: list[ObjectInfo],
shortcode: str,
existing: list[dict] | None = None,
max_entries: int = 1000) -> None
Appends the current run's file metadata to the tracker (NDJSON format) and
trims the log to max_entries total lines when the cap is exceeded.
Uses notebookutils.fs.append() for normal writes — the same approach as
OneLakeFileHandler — so no read-before-write is needed on the happy path.
put() is only used during a trim to rewrite the compacted file.
Arguments:
tracker_path- ABFS path from :func:get_tracker_file_path.file_list- Current :class:ObjectInfolist frompull_files.shortcode- Dataplatform object name (table_config.dataplatformobjectname).existing- Previously loaded snapshot (from :func:load_previous_snapshot). Used only for the trim count check. WhenNoneand a trim is needed, the file is re-read.max_entries- Maximum total entries to retain. Oldest are trimmed first.