Silver

Overview

Materialize a type-converted Delta table from bronze. With keephistory: false, silver is a flat typed copy of the latest bronze. With keephistory: true, silver maintains its own Silver.his history table populated from Bronze.his; silverloadtype selects either a full reload or an incremental merge into the existing silver table. Per-column type conversions and expressions live on the same per-table YAML as bronze.

What gets generated

Stage	Component	Output
Build	EasyFabric Generator (EFG)	GenerateFabricObjects → Fabric lakehouse tables (silver + history)
Runtime	EasyFabric Runtime (EFR)	easyfabric.load_data_silver.run, easyfabric.load_data_silver.dataframeloader

Example YAML

Dataplatform/DP/Objects/AdvWorks/orders.yaml

  Table:
  Connection: adv-advworks
  SourceTable: orders.csv
  DataPlatformObjectname: orders
  PreBronzeNotebook: 
    Notebook: Pull_Github
    Param001: orders.csv
    Param002: orders.csv
  KeepHistory: true
  Columns:
  - SourceColumn: SalesOrderID
    SourceDataType: varchar(255)
    IsPrimaryKey: true
  - SourceColumn: LineItem
    SourceDataType: int
    IsPrimaryKey: true
  - SourceColumn: OrderDate
    SourceDataType: datetime
  - SourceColumn: CustomerID
    SourceDataType: int
  - SourceColumn: ProductID
    SourceDataType: int  
  - SourceColumn: OrderQty
    SourceDataType: decimal(28,5)    
  - SourceColumn: LineItemTotal
    SourceDataType: decimal(28,5)

Schema reference

Required fields marked *. Linked types are collapsible — click to expand inline.

Fabric Object

Name	Type	Description
Connection *	String	Connection to use for this object (as defined in connection)
Fields *	List<FabricAttribute>	Fields from the source object
SourceTable *	String	Name of the object in the source

Show optional fields (9)

Name	Type	Description
BronzeTableSettings	FabricTableSettings	Table Settings for Bronze
DataPlatformObjectName	String	To override the source name that is used in the dataplatform
Description	String	Description of the source
IsActive	true/false	Set to active for generating this object (default=true)
KeepHistory	true/false	Set to true if history is required in the silver layer (default=true). Primary key required.
Layers	String[]	Which layers to generate
Prefix	String	Set a prefix (default = '')
SilverTableSettings	FabricTableSettings	Table Settings for Silver
SourceSchema	String	Source schema in the source (if applicable)

Fabric Attribute

Name	Type	Description
SourceColumn *	String	Name of the attribute in the source system

Show optional fields (14)

Name	Type	Description
Classification	ClassificationType	Set the classification of a field (sensitive, restricted, internal, public (default = public)
Dataplatformname	String	To override the name used in the dataplatform (default is the sourcecolumn name used)
HstgType	String	Type used in HSTG. Will try to convert a value from the source to this datatype. It will also keep the original value in a separate column.
IsActive	true/false	Attribute is active (default=true)
IsNullable	true/false	n/a
IsPartition	true/false	Attribute is part of the partition column (default = false)
IsPrimaryKey	true/false	Attribute is part of the primary key (default = false)
IsSilverOnly	true/false	Attribute is only generated for Silver (default = false)
SilverConvertedColumn	Nullable<true/false>	If a SilverDataType is given, there is some kind of conversion happening. For the converted value a column is created that will have the converted value. Inherits FabricConfiguration.SilverConvertedColumn when unset. (Default=true)
SilverDataType	String	Datatype used in the Silver layer.
SilverExpression	String	The expression that is used to fill the Silver column with the changed datatype. Example to convert a datetime from string to datetime, use: "to_timestamp(your_string_column, 'dd-MM-yyyy HH:mm:ss')". If a SilverDataType is given, there is some kind of conversion happening.
SilverInitialColumn	Nullable<true/false>	Create a Silver column holding the original (pre-conversion) value when a Silver conversion is set. Inherits FabricConfiguration.SilverInitialColumn when unset. (Default=true)
SourceDataType	String	Datatype of the source (use source datatype, will be converted to dataplatform types automatically)
SourceExpression	String	The expression that is used to fill the column in the Bronze layer with a conversion. Prefer to use the SilverExpression option. Potentially obsolete.

Runtime-only fields

These fields are read by the EFR at runtime and have no EFG counterpart.

• keephistory
• silverloadtype
• silverdatatype
• silverexpression
• issilveronly
• isdeletedidentifier

EasyFabric Runtime

load_data_silver.run

def run(tablefile: str,
        config_manager: ConfigManager = None,
        overrides: LoadOverrides = None) -> str | None

Executes the silver extraction and transformation process for a table, using the configuration information provided in the ConfigManager. This involves loading, processing, and merging table data within the silver lakehouse layer.

The function ensures the proper configuration of logging, verifies layer activity and table activation status, and invokes pre-silver and post-silver workflows if defined. It uses corresponding notebooks or direct data operations for the silver loading process.

Arguments:

• tablefile str - Path to the YAML file containing table configuration details.
• config_manager ConfigManager - An instance of ConfigManager, pre-initialized with application configuration, connection, and lakehouse details.
• overrides LoadOverrides - Optional per-run overrides. Only skip_presilver / skip_postsilver apply to silver; the bronze hook flags, stale-check, and force-reload fields are bronze-only and ignored here.

Returns:

• Optional[str] - An error message containing the name of the failed table and the exception details, or None if the operation finishes successfully.

Raises:

• Exception - If ConfigManager is not initialized before invoking this function.
• Exception - If the bronze lakehouse configuration is not found.
• Exception - If the stop_at_error setting is enabled and an exception occurs in processing.

load_data_silver.dataframeloader

def dataframeloader(data_frame: DataFrame,
                    table_config: TableConfig,
                    load_config: LoadConfig,
                    config_manager: ConfigManager = None) -> str | None

Loads data into a silver layer table in a lakehouse environment.

This function facilitates loading data from a given DataFrame into a table specified by a table configuration within a silver layer of the lakehouse architecture. It uses the provided configuration details to establish connections, manage runtime settings, and log relevant information during the operation. It validates critical configurations and raises appropriate exceptions in case of missing or invalid details.

Arguments:

• data_frame DataFrame - The input PySpark DataFrame containing data to load.
• table_config TableConfig - Configuration object specifying table details and related connection configurations.
• load_config LoadConfig - Configuration object holding load-specific settings, including the layer and runtime options.
• config_manager ConfigManager - Centralized configuration management object used to retrieve connection settings and maintain runtime parameters.

LoadConfig fields

Runtime parameter bag — construct in code and pass to the loader. All fields are optional unless flagged below.

Field	Type	Description
_layer	str	Operational layer associated with the configuration. Defaults to "not set".
dry_run	bool	Indicates if the process should be executed in dry-run mode. Defaults to True.
auto_null_column	bool	Determines if null values should be automatically managed for columns. Defaults to True.
load_type	LoadType	Specifies the type of load operation. Defaults to LoadType.FULL.
stop_at_error	bool	Specifies whether the process should stop when an error occurs. Defaults to True.
business_key_check	bool	Indicates if business keys should be validated during the load. Defaults to True.
log_row_count	bool	Deprecated and ignored. Row counts are always logged; retained for backward compatibility. Defaults to False.
key_violation_action	str	Action to be taken when key violations occur. Defaults to "raise".
load_history_table	bool	When true, also load the object's history table alongside the main table. Defaults to False.
model_object_name	Optional[str]	Specifies the model object name used in operations. Defaults to None.

Returns:

• str - A string indicating the result of the load operation, either as success or an error message.

Raises:

• Exception - If load_config is None, if ConfigManager is not properly initialized, or if no silver lakehouse configuration is found.
• Exception - If there is an issue while performing the load operation and config_manager.stop_at_error is set to True.

Related data classes

• easyfabric.data.TableConfig
• easyfabric.data.Column