Skip to main content

easyfabric.loaders.merge

full_load_silver​

def full_load_silver(table_config: TableConfig,
                     config_manager: ConfigManager = None,
                     columns_to_exclude: list[str] = None,
                     force_reload: bool = False)

Fully replace silver his table with source data from the bronze lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database. into the silver lakehouse.

Arguments:

  • table_config TableConfig - Configuration for the table to be merged.
  • config_manager ConfigManager - Configuration manager instance.
  • columns_to_exclude list[str], optional - Columns to exclude from the merge. Defaults to None.
  • force_reload bool, optional - Bypass the up-to-date gate and rebuild silver in full from bronze. Defaults to False.

merge_source_into_target​

def merge_source_into_target(table_config: TableConfig,
                             config_manager: ConfigManager = None,
                             columns_to_exclude: list[str] = None,
                             force_reload: bool = False)

Merges source data from the bronze lakehouse into the silver lakehouse.

Arguments:

  • table_config TableConfig - Configuration for the table to be merged.
  • config_manager ConfigManager - Configuration manager instance.
  • columns_to_exclude list[str], optional - Columns to exclude from the merge. Defaults to None.
  • force_reload bool, optional - Empty silver history first so the merge rebuilds it in full from bronze. Defaults to False.

bronze_into_silver​

def bronze_into_silver(table_config: TableConfig,
                       config_manager: ConfigManager = None,
                       columns_to_exclude: list[str] = None)

Merges source data from the bronze lakehouse into the silver lakehouse.

Arguments:

  • table_config TableConfig - Configuration for the table to be merged.
  • config_manager ConfigManager - Configuration manager instance.
  • columns_to_exclude list[str], optional - Columns to exclude from the merge. Defaults to None.

load_silver_table_from_bronze​

def load_silver_table_from_bronze(table_config: TableConfig,
                                  config_manager: ConfigManager = None,
                                  columns_to_exclude: list[str] = None)

Loads the silver table directly from the bronze lakehouse without merging.

Arguments:

  • table_config TableConfig - Configuration for the table to be loaded.
  • config_manager ConfigManager - Configuration manager instance.
  • columns_to_exclude list[str], optional - Columns to exclude from the load. Defaults to None.