Bronze

Overview

Pull raw data from configured sources (CSV/JSON/XML/Parquet/Excel) into bronze Delta tables. When keephistory: true on the per-table YAML, bronze also writes versioned change records to a Bronze.his history table — this is where row history lives. Both the Fabric item generator and the runtime loader consume the same per-table YAML.

What gets generated

Stage	Component	Output
Build	EasyFabric Generator (EFG)	`GenerateFabricObjects` → Fabric lakehouse tables (bronze)
Runtime	EasyFabric Runtime (EFR)	`easyfabric.load_data_bronze.run`, `easyfabric.load_data_bronze.dataframeloader`

Example YAML

Dataplatform/DP/Objects/AdvWorks/products.yaml

  Table:
  Connection: adv-advworks
  SourceTable: products.csv
  DataPlatformObjectname: products
  PreBronzeNotebook: 
    Notebook: Pull_Github
    Param001: products.csv
    Param002: products.csv
  KeepHistory: true
  Columns:
  - SourceColumn: ProductID
    SourceDataType: int
    IsPrimaryKey: true
  - SourceColumn: ProductName
    SourceDataType: varchar(255)        
  - SourceColumn: Category
    SourceDataType: varchar(255)    
  - SourceColumn: ListPrice
    SourceDataType: decimal(28,5)

Schema reference

Required fields marked *. Linked types are collapsible — click to expand inline.

Fabric Object

Name	Type	Description
Connection *	`String`	Connection to use for this object (as defined in connection)
Fields *	`List<FabricAttribute>`	Fields from the source object
SourceTable *	`String`	Name of the object in the source

Show optional fields (9)

Name	Type	Description
BronzeTableSettings	`FabricTableSettings`	Table Settings for Bronze
DataPlatformObjectName	`String`	To override the source name that is used in the dataplatform
Description	`String`	Description of the source
IsActive	`true/false`	Set to active for generating this object (default=true)
KeepHistory	`true/false`	Set to true if history is required in the silver layer (default=true). Primary key required.
Layers	`String[]`	Which layers to generate
Prefix	`String`	Set a prefix (default = '')
SilverTableSettings	`FabricTableSettings`	Table Settings for Silver
SourceSchema	`String`	Source schema in the source (if applicable)

Fabric Attribute

Name	Type	Description
SourceColumn *	`String`	Name of the attribute in the source system

Show optional fields (14)

Name	Type	Description
Classification	`ClassificationType`	Set the classification of a field (sensitive, restricted, internal, public (default = public)
Dataplatformname	`String`	To override the name used in the dataplatform (default is the sourcecolumn name used)
HstgType	`String`	Type used in HSTG. Will try to convert a value from the source to this datatype. It will also keep the original value in a separate column.
IsActive	`true/false`	Attribute is active (default=true)
IsNullable	`true/false`	n/a
IsPartition	`true/false`	Attribute is part of the partition column (default = false)
IsPrimaryKey	`true/false`	Attribute is part of the primary key (default = false)
IsSilverOnly	`true/false`	Attribute is only generated for Silver (default = false)
SilverConvertedColumn	`Nullable<true/false>`	If a SilverDataType is given, there is some kind of conversion happening. For the converted value a column is created that will have the converted value. Inherits FabricConfiguration.SilverConvertedColumn when unset. (Default=true)
SilverDataType	`String`	Datatype used in the Silver layer.
SilverExpression	`String`	The expression that is used to fill the Silver column with the changed datatype. Example to convert a datetime from string to datetime, use: "to_timestamp(your_string_column, 'dd-MM-yyyy HH:mm:ss')". If a SilverDataType is given, there is some kind of conversion happening.
SilverInitialColumn	`Nullable<true/false>`	Create a Silver column holding the original (pre-conversion) value when a Silver conversion is set. Inherits FabricConfiguration.SilverInitialColumn when unset. (Default=true)
SourceDataType	`String`	Datatype of the source (use source datatype, will be converted to dataplatform types automatically)
SourceExpression	`String`	The expression that is used to fill the column in the Bronze layer with a conversion. Prefer to use the SilverExpression option. Potentially obsolete.

Runtime-only fields

These fields are read by the EFR at runtime and have no EFG counterpart.

filetype
skipifsourceunchanged
bronzeloadskip
layers

EasyFabric Runtime

`load_data_bronze.run`

def run(tablefile: str,
        config_manager: ConfigManager = None,
        overrides: LoadOverrides = None) -> str | None

Runs the bronze loader process for a specified table configuration and pulls files from the source, processes them, and loads them into the bronze layer.

This function initializes the table configuration based on the specified table file and checks its active state for the bronze process. If the table's configuration specifies pre-processing or post-processing notebooks, they are executed accordingly. Files from the source are pulled, processed, and loaded into the bronze layer based on their specified file types. Supported file types include CSV, JSON, XML, Parquet, and Notebook. The function also handles exceptions and ensures logs are saved correctly.

Arguments:

tablefile str - Path to the YAML file representing a table's configuration.
config_manager ConfigManager - An instance of ConfigManager used for accessing the application's configuration settings.
overrides LoadOverrides - Optional per-run overrides that flip default validation/gating behaviour for this call only — skip notebook hooks, skip the stale-file check, or force a reload of unchanged source. Defaults to no overrides (use YAML / ConfigManager defaults).

Returns:

str - A message indicating the outcome of the loading process, such as the number of files loaded or an error message in case of failure.

Raises:

Exception - If the config_manager is not initialized, no active configuration can be found for the table, or the filetype is unsupported. @sidebar_position 3

`load_data_bronze.dataframeloader`

def dataframeloader(data_frame: DataFrame,
                    load_config: LoadConfig,
                    table_config: TableConfig,
                    config_manager: ConfigManager = None) -> str | None

Loads a DataFrame into a specified data platform table using the provided configuration and manager.

This function handles the loading operation by using detailed configurations for the DataFrame, table, and the application configuration manager. It sets up logging, ensures required parameters are initialized, and supports specific settings for different layers (e.g., bronze layer). The function handles exception logging and provides mechanisms to stop processing upon encountering errors based on configuration settings.

Arguments:

data_frame DataFrame - The data to be loaded into the specified table.
load_config LoadConfig - Contains configuration for the loading process, including destination table.
table_config TableConfig - Holds table-specific settings, e.g., table name identifiers and layers.
config_manager ConfigManager - Manages and validates application-level configurations.

LoadConfig fields

Runtime parameter bag — construct in code and pass to the loader. All fields are optional unless flagged below.

Field	Type	Description
`_layer`	`str`	Operational layer associated with the configuration. Defaults to "not set".
`dry_run`	`bool`	Indicates if the process should be executed in dry-run mode. Defaults to True.
`auto_null_column`	`bool`	Determines if null values should be automatically managed for columns. Defaults to True.
`load_type`	`LoadType`	Specifies the type of load operation. Defaults to LoadType.FULL.
`stop_at_error`	`bool`	Specifies whether the process should stop when an error occurs. Defaults to True.
`business_key_check`	`bool`	Indicates if business keys should be validated during the load. Defaults to True.
`log_row_count`	`bool`	Deprecated and ignored. Row counts are always logged; retained for backward compatibility. Defaults to False.
`key_violation_action`	`str`	Action to be taken when key violations occur. Defaults to "raise".
`load_history_table`	`bool`	When true, also load the object's history table alongside the main table. Defaults to False.
`model_object_name`	`Optional[str]`	Specifies the model object name used in operations. Defaults to None.

Returns:

str - Message indicating the result of the DataFrame loading process, including the target table name and error details if applicable.

Raises:

Exception - If the destination table name is missing from LoadConfig.
Exception - If the ConfigManager is not properly initialized.

easyfabric.data.TableConfig
easyfabric.data.Column
easyfabric.data.Connection

Overview​

What gets generated​

Example YAML​

Schema reference​

Fabric Object​

Runtime-only fields​

EasyFabric Runtime​

load_data_bronze.run​

load_data_bronze.dataframeloader​

Related data classes​