Skip to main content

easyfabric.data.loadconfig

LoadType Objects​

class LoadType(Enum)

Enum of supported data load types (Full, Append, Merge, etc.).

LoadConfig Objects​

@dataclass
class LoadConfig(DataClassFromDictMixin)

Configuration class for managing data load settings.

This class encapsulates the configuration settings for managing data load operations, enabling fine-grained control over various aspects such as load type, error handling, logging, and destination specifications. It is designed to adapt and configure data loading workflows seamlessly based on user-defined settings.

Attributes:

  • _layer str - Operational layer associated with the configuration. Defaults to "not set".
  • dry_run bool - Indicates if the process should be executed in dry-run mode. Defaults to True.
  • auto_null_column bool - Determines if null values should be automatically managed for columns. Defaults to True.
  • load_type LoadType - Specifies the type of load operation. Defaults to LoadType.FULL.
  • stop_at_error bool - Specifies whether the process should stop when an error occurs. Defaults to True.
  • business_key_check bool - Indicates if business keys should be validated during the load. Defaults to True.
  • log_row_count bool - Deprecated and ignored. Row counts are always logged; retained for backward compatibility. Defaults to False.
  • key_violation_action str - Action to be taken when key violations occur in bronze/silver loads. The gold loader ignores it: a dimension load always fails on duplicate business keys. Defaults to "raise".
  • load_history_table bool - When true, also load the object's history table alongside the main table. Defaults to False.
  • model_object_name Optional[str] - Specifies the model object name used in operations. Defaults to None.
  • merge_delete_query Optional[str] - The query for handling delete scenarios during merge operations. Defaults to None.
  • destination_full_tablename Optional[str] - Fully qualified destination table name (schema.table) the load writes to. Defaults to None.

get_destination_lakehouse​

@staticmethod
def get_destination_lakehouse(config_manager: ConfigManager)

Resolve the destination lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database. name for the bronze layer.

Arguments:

  • config_manager - The active ConfigManager instance.

Returns:

The lakehouse name string for the bronze layer.

get_destination_abfs​

@staticmethod
def get_destination_abfs(config_manager: ConfigManager)

Get the ABFS path for the bronze-layer destination.

Arguments:

  • config_manager - The active ConfigManager instance.

Returns:

The ABFS URI string for the bronze lakehouse.