Skip to main content

loaders.file_tracker

get_tracker_file_path​

def get_tracker_file_path(bronze_abfs_base_folder: str) -> str

Returns the ABFS path for the tracker file inside the table's Bronze folder.

The tracker always lives at::

<bronze_abfs_base_folder>/_tracking/tracker.json

For azblob (date-partitioned) pass the folder above the date partition, e.g. abfss://…/Files/afas/Projecten. For fabricfiles pass the folder that directly contains the files, e.g. abfss://…/Files/afas/Projecten.

Arguments:

  • bronze_abfs_base_folder - Non-date-partitioned Bronze folder for this table.

Returns:

  • str - Full ABFS path to _tracking/tracker.json.

load_previous_snapshot​

def load_previous_snapshot(tracker_path: str, shortcode: str) -> list[dict]

Loads this object's historical entries from the tracker (NDJSON — one entry per line).

One tracker file can hold more than one object: the fabricfiles path is derived from the Bronze folder and the source folder, so two objects reading the same source share it. Every entry records the object that wrote it, and reading returns only those — without the filter the second object compared against the first object's snapshot, concluded the source was unchanged and skipped its own load without an error (Bug #469).

Entries without a shortcode come from before the tracker recorded one and cannot be attributed, so they are ignored. That makes the first run after the upgrade reload once, which is recoverable; counting them would keep the silent skip alive, which is not.

Returns an empty list on first run (file does not exist yet).

Arguments:

  • tracker_path - ABFS path returned by :func:get_tracker_file_path.
  • shortcode - Dataplatform object name (table_config.dataplatformobjectname), as written by :func:save_snapshot.

Returns:

List of entry dicts ordered oldest-first.

save_snapshot​

def save_snapshot(tracker_path: str,
file_list: list[ObjectInfo],
shortcode: str,
existing: list[dict] | None = None,
max_entries: int = 1000) -> None

Appends the current run's file metadata to the tracker (NDJSON format) and trims the log to max_entries total lines when the cap is exceeded.

Uses notebookutils.fs.append() for normal writes — the same approach as OneLakeFileHandler — so no read-before-write is needed on the happy path. put() is only used during a trim to rewrite the compacted file.

Arguments:

  • tracker_path - ABFS path from :func:get_tracker_file_path.
  • file_list - Current :class:ObjectInfo list from pull_files.
  • shortcode - Dataplatform object name (table_config.dataplatformobjectname).
  • existing - Previously loaded snapshot (from :func:load_previous_snapshot). Used only for the trim count check. When None and a trim is needed, the file is re-read.
  • max_entries - Maximum total entries to retain. Oldest are trimmed first.