Skip to main content

easyfabric.data.tableconfig

reset_dateformat_default_warnings​

def reset_dateformat_default_warnings() -> None

Vergeet welke objecten al gemeld zijn. Voor tests en een herstarte run.

Connection Objects​

@dataclass
class Connection(DataClassFromDictMixin)

Data source connection details.

Defines how to connect to and read from an external data source, including file type, delimiter, credentials, and load ordering.

from_yaml_file​

@classmethod
def from_yaml_file(cls, file_path: str)

Create a Connection instance from a YAMLA simple way to write configurations. It's basically a list that computers can read easily. file.

from_yaml_files​

@classmethod
def from_yaml_files(cls, file_paths: list[str])

Create a list of TableConfig instances from multiple YAML files.

NotebookConfig Objects​

@dataclass
class NotebookConfig(DataClassFromDictMixin)

Configuration for pre/post notebook execution hooks.

Specifies a notebook to run at a particular stage of the load pipelineAn automated "conveyor belt" that moves data from one place to another or performs a task automatically., along with an optional timeout and up to three parameters.

Trigger Objects​

@dataclass
class Trigger(DataClassFromDictMixin)

File trigger configuration for automatic loading.

Defines the notebook to execute and optional webhook to notify when a file event triggers a table load.

HistorySettings Objects​

@dataclass
class HistorySettings(DataClassFromDictMixin)

SCD Type 2 history tracking configuration.

Controls how historical changes are tracked at the bronze and silver layers, including filter queries for deletes and source rows.

Column Objects​

@dataclass
class Column(DataClassFromDictMixin)

Column definition with source mapping and data type.

Maps a source column to its destination name and data type, including optional silver-layer type conversion expressions and primary key / z-order flags.

TableConfig Objects​

@dataclass
class TableConfig(DataClassFromDictMixin)

Per-table configuration loaded from a YAML definition file.

Defines the source connection, column mappings, load settings, and data quality rules for a single table in the data pipeline.

bronzeloadviolationaction​

continue or stop

bronzechecksummismatchviolationaction​

continue or stop

silverdateconversionviolationaction​

continue or stop

validate_layer​

def validate_layer(layer) -> bool

Validates if the provided layer is supported for this table.

Arguments:

  • layer str - The layer to validate

Raises:

  • ValueError - If the layer is not supported for this table

get_object_name​

def get_object_name()

Get the logical object name for this table.

Returns dataplatformobjectname if set, otherwise derives the name from sourcetable.

get_abfs_table_path​

def get_abfs_table_path(config_manager: ConfigManager = None,
                        layer: str = "bronze",
                        history: bool = False) -> str

Get the full ABFS path for the table based on the layer. This includes the lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database. and schema.

get_full_table_name​

def get_full_table_name(config_manager: ConfigManager = None,
                        layer: str = "bronze",
                        history: bool = False) -> str

Get the full table name based on the layer. This includes the lakehouse and schema.

Examples:

History = false: Bronze.dbo.my_tablename Silver.dbo.my_tablename History = true: Bronze.his.my_tablename Silver.his.my_tablename

get_table_name​

def get_table_name(config_manager: ConfigManager = None,
                   layer: str = "bronze") -> str

Get the table name based on the layer.

get_zorder_column_names​

def get_zorder_column_names(config_manager: ConfigManager = None) -> list[str]

Returns list of ACTUAL column names (as they exist in the Delta table) that should be used in ZORDER BY, based on iszorder=True flag.

get_bronzeloadseverity​

def get_bronzeloadseverity() -> str

How loudly a stale source file (Validation 1) is reported.

Independent of bronzeloadviolationaction, which decides whether the load goes on.

get_bronzechecksummismatchseverity​

def get_bronzechecksummismatchseverity() -> str

How loudly a checksum/row-count mismatch (Validation 2) is reported.

get_bronzemaxfileagecriticalhours​

def get_bronzemaxfileagecriticalhours() -> Optional[int]

Age past which a stale file reports as critical instead of its base rung.

None — the default — never escalates.

get_bronzeunchangedseverity​

def get_bronzeunchangedseverity() -> str

How loudly an unchanged-for-too-long source is reported.

get_bronzemaxunchangedhours​

def get_bronzemaxunchangedhours() -> Optional[int]

Longest stretch the source may stay unchanged before a full load runs anyway.

Bounds the blind spot of a size-based comparison, which cannot tell apart two deliveries of equal byte length. None -- the default -- leaves it unbounded.

get_read_option​

def get_read_option(name: str)

Resolve a file read option, preferring the table value over the connection.

Used for CSV/JSON read settings (delimiter, header, quote, charset, ...) a table may override but otherwise inherits from its connection.

get_structtype_schema​

def get_structtype_schema() -> "tuple[StructType, dict[str, Any]]"

Convert column definitions to a PySpark StructType schema.

Returns:

A tuple of (StructType schema, dict mapping column names to their PySpark data types).

from_yaml​

@classmethod
def from_yaml(cls, yaml_string: str)

Create a Model instance from a YAML string.

from_yaml_file​

@classmethod
def from_yaml_file(cls, file_path: str)

Create a Model instance from a YAML file.

from_yaml_files​

@classmethod
def from_yaml_files(cls, file_paths: list[str])

Create a list of TableConfig instances from multiple YAML files.