Logging

EasyFabric provides a structured logging system for Microsoft FabricAn all-in-one data and analytics platform from Microsoft. Think of it as a "digital warehouse" for all your company's information. notebooks. It produces one log file per pipelineAn automated "conveyor belt" that moves data from one place to another or performs a task automatically. run, stored in OneLake (Meta lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database.). All notebooks in the same execution tree share the same log file automatically.

Overview

• Each pipeline run gets a single log file, identified by Fabric's activityId
• Parent and child notebooks write to the same file — no manual wiring needed
• Log entries can be persisted to a SQL table (Meta.dbo.logging) for querying
• Logs include structured fields: source, object name, type, category, and message

Quick start — standalone notebook

The simplest case: a single notebook that initializes logging, does some work, and finalizes.

from easyfabric.fabric import init_logging, log_segment, save_log_file_to_table

init_logging(log_source="Silver", log_object="Customers")

with log_segment("Load", "Customers"):
    # your loading logic here
    pass

save_log_file_to_table()

init_logging creates a log file in the Meta lakehouse under Files/Logs/{date}/{activityId}.log. The log_segment context manager logs START and END entries around your code. save_log_file_to_table logs an END entry with duration and persists log entries to the database when called from the top-level notebook (or with end_log=True); in a child notebook it returns without emitting anything.

Quick start — DAG notebook

In a typical DAG, a parent notebook orchestrates multiple child notebooks (Bronze, Silver, Gold). Each child calls init_logging at the top — this is safe because init_logging is idempotent. It detects the shared activityId from Fabric's runtime context and reuses the existing log file.

Parent notebook (DAG):

from easyfabric.fabric import init_logging, log_segment, save_log_file_to_table

init_logging(log_source="Sys", log_object="DAG_Daily")

with log_segment("Orchestration", "Bronze"):
    notebookutils.notebook.run("Bronze_Load")

with log_segment("Orchestration", "Silver"):
    notebookutils.notebook.run("Silver_Load")

with log_segment("Orchestration", "Gold"):
    notebookutils.notebook.run("Gold_Load")

save_log_file_to_table()

Child notebook (e.g. Bronze_Load):

from easyfabric.fabric import init_logging, log_segment, save_log_file_to_table

# Safe to call again — reuses the parent's log file
init_logging(log_source="Bronze", log_object="Customers")

with log_segment("Load", "Customers"):
    # bronze loading logic
    pass

save_log_file_to_table()

Every notebook in the tree calls init_logging and save_log_file_to_table. Only the top-level notebook's call logs the END entry with duration and persists all log entries to the database; a child notebook's call returns without emitting. Top-level status is detected at runtime, so any notebook can be run independently as the entry point.

Parameters

init_logging(log_source, log_object, base_batch_id)

Initializes the logging session. Creates (or reuses) a log file on OneLake and configures the Python root logger.

Parameter	Type	Default	Description
log_source	str	"Sys"	The logical source layer (e.g., "Bronze", "Silver", "Gold", "Sys").
log_object	str	None	The object being processed (e.g., "Customers"). Defaults to the current notebook name if not provided.
base_batch_id	str	None	Keeps a whole logical run under one batch_id when a parent offloads children as separate Fabric jobs (each gets its own activityId). Pass the parent's id through every notebook boundary; when absent, the notebook's own activityId is used. The log file name always uses the own activityId, so concurrent jobs write distinct files.

Returns: The absolute OneLake path of the log file (str).

log_segment(type, name)

Context manager that logs START and END entries around a block of code. If an exception occurs, it logs the error and re-raises.

Parameter	Type	Description
type	str	The log category (e.g., "Load", "Transform", "Orchestration").
name	str	A descriptive name for the segment (e.g., "Customers", "Bronze Loading").

save_log_file_to_table(end_log)

Call at the end of every notebook. When called from the top-level notebook (or when end_log=True), it logs an END entry with total duration, persists all log entries to Meta.dbo.logging, clears handlers, and resets state. In a child notebook the call returns without emitting anything.

Parameter	Type	Default	Description
end_log	bool	False	Force log persistence and cleanup even if not the top-level notebook.

How it works — the notebook hierarchy

Fabric provides runtime context identifiers that EasyFabric uses to coordinate logging across notebooks:

Identifier	Scope	Used for
activityId	Shared across the entire execution tree	Log file name and batch_id in the logging table
currentRunId	Unique per notebook execution	Distinguishing log entries from different notebooks
parentRunId	References the parent notebook's run	Tracking the notebook call hierarchy
isReferenceRun	False/absent on the entry-point notebook, True on every child	Determining which notebook is the entry point (is_top_level_notebook())

Because all notebooks share the same activityId, they all write to the same log file — no configuration needed.

Error handling

log_segment catches exceptions, logs them as errors, and re-raises. You don't need to add try/except blocks for logging purposes.

with log_segment("Load", "Customers"):
    df = spark.read.csv("missing_file.csv")  # raises an exception
    # log_segment logs: "END: Customers (Failed): [error message]"
    # then re-raises the original exception

Errors are captured both in the OneLake log file and in the Meta.dbo.logging table (when save_log_file_to_table is called).

Important notes

• Call init_logging() at the top of every notebook — it's idempotent and safe to call multiple times. If the log file already exists for the current activityId, it reuses it.
• Call save_log_file_to_table() at the end of every notebook — it logs the END entry with duration and persists the log only when called from the top-level notebook (or with end_log=True); in a child notebook the call is a safe no-op.
• Interactive sessions — when running a notebook interactively (not via a pipeline), the log file is prefixed with usr_ (e.g., usr_{activityId}.log).
• Local development — without Fabric context (e.g., running locally), a UUID fallback is used as the activity ID.