Skip to main content

easyfabric.loaders.tableloader

json​

logging​

DeltaTable​

DataFrame​

Window​

coalesce​

col​

concat_ws​

expr​

lit​

row_number​

sha2​

when​

IntegerType​

LongType​

StringType​

MSG_TABLELOADER_COLUMN_MISMATCH​

MSG_TABLELOADER_DELETE_OP​

MSG_TABLELOADER_DRY_RUN​

MSG_TABLELOADER_EMPTY_SOURCE​

MSG_TABLELOADER_KEY_COLUMNS​

MSG_TABLELOADER_LOAD_DETAILS​

MSG_TABLELOADER_LOAD_START​

MSG_TABLELOADER_LOAD_TYPE​

MSG_TABLELOADER_LOADING_TABLE​

MSG_TABLELOADER_NO_ROWS_LOADED​

MSG_TABLELOADER_SAMPLE_ROWS​

MSG_TABLELOADER_SCHEMA_MISMATCH​

MSG_TABLELOADER_SCHEMA_UPDATE​

MSG_TABLELOADER_SK_INFO​

MSG_TABLELOADER_WRITING​

ConfigManager​

initialize_config​

LoadConfig​

LoadType​

Model​

ObjectInfo​

TableConfig​

align_dataframe_to_target_schema​

check_for_duplicate_keys_dataframe​

current_timestamp​

get_spark​

segment_record​

load_dataframe_silver​

merge_loadconfig_into_target​

spark_operation_with_retries​

spark​

dataframe_to_silver​

def dataframe_to_silver(df_source: DataFrame,
                        load_config: LoadConfig,
                        table_config: TableConfig,
                        config_manager: ConfigManager = None)

dataframe_to_bronze_old​

def dataframe_to_bronze_old(df_source: DataFrame,
                            load_config: LoadConfig,
                            table_config: TableConfig,
                            config_manager: ConfigManager = None)

dataframe_to_gold​

def dataframe_to_gold(df_source: DataFrame,
                      load_config: LoadConfig,
                      model: Model,
                      config_manager: ConfigManager = None)

Loads a DataFrame into a Gold layer Delta table with surrogate key handling and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • load_config LoadConfig - Configuration for the load process
  • model Model - Data model containing table definitions
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If the table is not found in the model or if the source DataFrame does not match the target table schema
  • Exception - If an error occurs during the load process

Returns:

None

handle_full_load​

def handle_full_load(load_config: LoadConfig, df_source, config_manager=None)

handle_append_load​

def handle_append_load(load_config: LoadConfig,
                       df_source,
                       config_manager=None)

handle_append_with_delete_load​

def handle_append_with_delete_load(load_config: LoadConfig,
                                   df_source,
                                   config_manager=None)

load_handlers​

dataframe_to_bronze​

def dataframe_to_bronze(df_source: DataFrame,
                        table_config: TableConfig,
                        object_info: ObjectInfo,
                        config_manager: ConfigManager = None)

Loads a DataFrame into a Bronze layer Delta table with system columns and validation.

Arguments:

  • df_source DataFrame - Source Spark DataFrame to load
  • table_config TableConfig - Configuration for the target table
  • object_info ObjectInfo - Source object information
  • config_manager ConfigManager - Global configuration manager

Raises:

  • ValueError - If source DataFrame is empty or invalid
  • Exception - If bronze lakehouse configuration is missing

Returns:

  • Dataframe - The loaded DataFrame after processing and validation