Skip to main content

easyfabric.loaders.tableloader

dataframe_to_table​

def dataframe_to_table(df_source: DataFrame,
                       load_config: LoadConfig,
                       table_config: TableConfig,
                       config_manager: ConfigManager = None)

Loads a caller-supplied DataFrame into its target Delta table.

The target layer comes from load_config._layer, which the calling loader sets, so one entry point serves bronze and silver alike. This is the DataFrame path: for a bronze delivery read from files, use :func:prepare_bronze_dataframe with :func:overwrite_bronze_with_frames.

Arguments:

  • df_source - Source Spark DataFrame to load.
  • load_config - Configuration for the load process, including the layer.
  • table_config - Configuration for the target table.
  • config_manager - Global configuration manager.

Returns:

None

dataframe_to_gold​

def dataframe_to_gold(df_source: DataFrame,
                      load_config: LoadConfig,
                      model: Model,
                      config_manager: ConfigManager = None)

Loads a DataFrame into a Gold layer Delta table with surrogate key handling and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • load_config LoadConfig - Configuration for the load process
  • model Model - Data model containing table definitions
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If the table is not found in the model or if the source DataFrame does not match the target table schema
  • Exception - If an error occurs during the load process

Returns:

None

prepare_bronze_dataframe​

def prepare_bronze_dataframe(
        df_source: DataFrame,
        table_config: TableConfig,
        object_info: ObjectInfo,
        config_manager: ConfigManager = None) -> tuple[DataFrame, int] | None

Prepares a source DataFrame for the Bronze layer: expressions, defaults, system columns and alignment to the target schema. Writes nothing.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • table_config TableConfig - Configuration for the target table
  • object_info ObjectInfo - Source object information
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If source DataFrame is empty or invalid
  • Exception - If bronze lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database. configuration is missing

Returns:

The aligned frame together with its row count, or None when the source held no rows — the violation action decides whether that raises or warns.

dataframe_to_bronze​

def dataframe_to_bronze(df_source: DataFrame,
                        table_config: TableConfig,
                        object_info: ObjectInfo,
                        config_manager: ConfigManager = None)

Appends a single DataFrame to the Bronze landing table.

Kept for callers that load one frame at a time. load_data_bronze.run uses :func:prepare_bronze_dataframe and :func:overwrite_bronze_with_frames instead, so a whole delivery lands in one commit.

overwrite_bronze_with_frames​

def overwrite_bronze_with_frames(frames: list[DataFrame],
                                 row_count: int,
                                 table_config: TableConfig,
                                 config_manager: ConfigManager = None)

Replaces the Bronze landing table with the union of a delivery's frames.

One INSERT OVERWRITE is a single Delta commit, so a reader sees either the previous delivery or this one — never the empty table that a truncate-then-append left behind when a later file failed to parse.

Arguments:

  • frames - Prepared frames from :func:prepare_bronze_dataframe, in load order.
  • row_count - Total rows across frames, already counted while preparing.
  • table_config - Configuration for the target table.
  • config_manager - Global configuration manager.