load_data_silver

The load_data_silver module manages the transition of data from the raw Bronze layer to the validated Silver layer. It is responsible for applying data quality checks, handling historical merging (SCD Type 1/2), and preparing the data for dimensional modeling in Gold.

Silver in 20 seconds: same flow as Bronze, only the notebook differs. The silver_dbo table is rebuilt on every run; silver_his loads incrementally from the last max timestamp.

run

def run(tablefile: str,
        config_manager: ConfigManager = None,
        overrides: LoadOverrides = None) -> str | None

Executes the silver extraction and transformation process for a table, using the configuration information provided in the ConfigManager. This involves loading, processing, and merging table data within the silver lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database. layer.

The function ensures the proper configuration of logging, verifies layer activity and table activation status, and invokes pre-silver and post-silver workflows if defined. It uses corresponding notebooks or direct data operations for the silver loading process.

Arguments:

• tablefile str - Path to the YAMLA simple way to write configurations. It's basically a list that computers can read easily. file containing table configuration details.
• config_manager ConfigManager - An instance of ConfigManager, pre-initialized with application configuration, connection, and lakehouse details.
• overrides LoadOverrides - Optional per-run overrides. Only skip_presilver / skip_postsilver and force_reload_silver apply to silver; the bronze hook flags, stale-check, and force_reload_bronze fields are bronze-only and ignored here. force_reload_silver rebuilds silver in full from bronze, bypassing the "silver is already up to date" gate.

Returns:

• Optional[str] - An error message containing the name of the failed table and the exception details, or None if the operation finishes successfully.

Raises:

• Exception - If ConfigManager is not initialized before invoking this function.
• Exception - If the bronze lakehouse configuration is not found.
• Exception - If the stop_at_error setting is enabled and an exception occurs in processing.

dataframeloader

def dataframeloader(data_frame: DataFrame,
                    table_config: TableConfig,
                    load_config: LoadConfig,
                    config_manager: ConfigManager = None) -> str | None

Loads data into a silver layer table in a lakehouse environment.

This function facilitates loading data from a given DataFrame into a table specified by a table configuration within a silver layer of the lakehouse architecture. It uses the provided configuration details to establish connections, manage runtime settings, and log relevant information during the operation. It validates critical configurations and raises appropriate exceptions in case of missing or invalid details.

Arguments:

• data_frame DataFrame - The input PySpark DataFrame containing data to load.
• table_config TableConfig - Configuration object specifying table details and related connection configurations.
• load_config LoadConfig - Configuration object holding load-specific settings, including the layer and runtime options.
• config_manager ConfigManager - Centralized configuration management object used to retrieve connection settings and maintain runtime parameters.

Returns:

• str - A string indicating the result of the load operation, either as success or an error message.

Raises:

• Exception - If load_config is None, if ConfigManager is not properly initialized, or if no silver lakehouse configuration is found.
• Exception - If there is an issue while performing the load operation and config_manager.stop_at_error is set to True.