Skip to main content

loaders.tableloader

The tableloader utility provides the core writing logic for the entire platform. Whether you are landing data in Bronze, merging in Silver, or publishing to Gold, these functions handle the physical storage interactions, schema enforcement, and Delta table operations.

View Functions

dataframe_to_table​

def dataframe_to_table(df_source: DataFrame,
                       load_config: LoadConfig,
                       table_config: TableConfig,
                       config_manager: ConfigManager = None)

Loads a caller-supplied DataFrame into its target Delta table.

The target layer comes from load_config._layer, which the calling loader sets, so one entry point serves bronze and silver alike. This is the DataFrame path: for a bronze delivery read from files, use prepare_bronze_dataframe with overwrite_bronze_with_frames.

Arguments:

  • df_source - Source Spark DataFrame to load.
  • load_config - Configuration for the load process, including the layer.
  • table_config - Configuration for the target table.
  • config_manager - Global configuration manager.

Returns:

None

dataframe_to_gold​

def dataframe_to_gold(df_source: DataFrame, load_config: LoadConfig,
                      model: Model, config_manager: ConfigManager = None)

Loads a DataFrame into a Gold layer Delta table with surrogate key handling and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • load_config LoadConfig - Configuration for the load process
  • model Model - Data model containing table definitions
  • config_manager ConfigManager - Global configuration manager. Defaults to the initialized global configuration.

Raises:

  • ValueError - If the table is not found in the model or if the source DataFrame does not match the target table schema
  • Exception - If an error occurs during the load process

Returns:

None

dataframe_to_bronze​

def dataframe_to_bronze(df_source: DataFrame, table_config: TableConfig,
                        object_info: ObjectInfo,
                        config_manager: ConfigManager = None)

Loads a DataFrame into a Bronze layer Delta table with system columns and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • table_config TableConfig - Configuration for the target table
  • object_info ObjectInfo - Source object information
  • config_manager ConfigManager - Global configuration manager. Defaults to the initialized global configuration.

Raises:

  • ValueError - If the source DataFrame is empty and the table's bronzeloadviolationaction is set to "stop" (otherwise a warning is logged and the load is skipped)
  • Exception - If writing to the bronze table fails

Returns:

None