Skip to main content

Custom extensions

Every data platform has its unique characteristics. While many situations are similar, each company has its own specific requirements. Sometimes, it is necessary to perform actions that do not fit the standard workflow. In these scenarios, it is helpful to know that custom actions can be added to the flow.

The most logical place to implement these custom actions is in a notebook.

Between every layer, there is an option to run a custom notebook.

Pre notebook and post notebook​

To run a specific notebook before loading data into a layer, define a Pre notebook on the object yamlA simple way to write configurations. It's basically a list that computers can read easily. file. In this case before running bronze:

Table:
Connection: exa-example
SourceTable: example.json
DataPlatformObjectname: example
PreBronzeNotebook: 
    Notebook: Pull_example_file
    Param001: SomeValue

When the action to load bronze data is activated and if a prebronzenotebook is defined, this notebook will be run. The following parameters are supplied to it automatically:

  • sourcetable
  • sourcefilter
  • bronzefolder
  • keyvault
  • object_yaml_file — the path to this object's YAML file (not its contents). Use it to read the full object definition inside the notebook (see Reading the object).

A few internal context parameters are also passed (notebook_type, log_file_path, base_batch_id); you can ignore these unless you are integrating with EasyFabric's logging.

Optionally it is possible to supply 3 additional parameters (Param001, Param002, Param003) on the notebook node in the YAML. These arrive lower-cased as param001–param003. In this example there is one parameter Param001 supplied with a value of SomeValue, so the notebook receives param001="SomeValue".

The notebook that is called should have a parameter cell with the following:

sourcetable=""
sourcefilter=""
bronzefolder=""
keyvault=""
object_yaml_file=""
param001=""
param002=""
param003=""
Post notebook

Works equally as the pre notebook, but is run after the loading has finished.

The same contract for every hook​

These parameters are not specific to bronze or to the pre position. Every layer hook receives the identical set, so a notebook written for one slot works in any of them. The hooks are wired on the object YAML under:

Around bronzeAround silver
PreBronzeNotebookPreSilverNotebook
MidBronzeNotebookMidSilverNotebook
PostBronzeNotebookPostSilverNotebook

All six pass the same standard parameters (including object_yaml_file) plus their own Param001–Param003.

Reading the object inside the notebook​

Because object_yaml_file is a path, the notebook reads the object definition itself. The canonical way is load_meta_data.get_object_by_file, which returns a fully-initialised TableConfig:

from easyfabric import load_meta_data

# object_yaml_file is supplied by EasyFabric; guard for it so the cell is
# still runnable on its own during development.
if "object_yaml_file" in locals():
    obj = load_meta_data.get_object_by_file(object_yaml_file)   # -> TableConfig
    target = f"Files/{obj.bronzefolder}/{obj.sourcetable}"

get_object_by_file initialises the ConfigManager and applies the normal config-aware defaults. If you only want a raw parse with no config context, use TableConfig.from_yaml_file(object_yaml_file) instead. Do not use get_tableconfig_by_inputfile here — that resolves a data file back to its config by bronzefolder prefix, which is the opposite direction.

How the notebook reports a failure​

A custom notebook is part of the load, so a problem inside it has to reach EasyFabric — otherwise the loader records a successful run and the platform continues on missing or stale data. Two signals reach the platform, and they are handled identically.

Raise an exception — this is the recommended route. EasyFabric starts your notebook with notebookutils.notebook.run, so an exception inside the notebook travels straight back to the loader. The loader catches it, the object ends as a Failed segment in Meta.dbo.logging, and the run carries on with the next object. stop_at_error on the config manager is False by default; call cfg.set_stop_at_error() if you want the first failure to abort the whole run instead.

try:
    fetch_and_write()
except Exception as e:
    logging.error(f"ERROR: {e}")
    raise

Or exit with a value that starts with error. EasyFabric inspects the exit value of the notebook it ran and treats any string starting with error (case-insensitive) as a failure — same logging, same Failed segment.

except Exception as e:
    logging.error(f"ERROR: {e}")
    notebookutils.notebook.exit(f"ERROR: {e}")

This applies to every notebook EasyFabric runs for an object: the six Pre/Mid/Post hooks above and the BronzeNotebook / SilverNotebook full-layer overrides.

A plain return never reaches EasyFabric

A notebook's exit value is set only by notebookutils.notebook.exit(...). A return out of a function returns to the calling cell and nowhere else: notebook.run hands back None, EasyFabric sees a notebook that finished without complaint, and the load continues as if the data were there.

except Exception as e:
    logging.error(f"ERROR: {e}")
    return f"ERROR: {e}"      # invisible — the load continues

Raise instead, or exit with an ERROR: value. Logging alone is not a signal either: a log line does not fail the object.

Notebook as the source

The notebooks above run around an existing file or database source. When there is no such source — the data lives behind an API like Microsoft Graph / Entra and the notebook itself fetches it — see Connecting a source to bronze.