Skip to main content

loaders.merge

merge_source_into_target​

def merge_source_into_target(table_config: TableConfig,
                             config_manager: ConfigManager,
                             columns_to_exclude: list[str] = None)

Merges source data from the bronze lakehouse into the silver lakehouse.

Arguments:

  • table_config TableConfig - Configuration for the table to be merged.
  • config_manager ConfigManager - Configuration manager instance.
  • columns_to_exclude list[str], optional - Columns to exclude from the merge. Defaults to None.

load_silver_table_from_bronze​

def load_silver_table_from_bronze(table_config: TableConfig,
                                  config_manager: ConfigManager,
                                  columns_to_exclude: list[str] = None)

Loads the silver table directly from the bronze lakehouse without merging.

Arguments:

  • table_config TableConfig - Configuration for the table to be loaded.
  • config_manager ConfigManager - Configuration manager instance.
  • columns_to_exclude list[str], optional - Columns to exclude from the load. Defaults to None.