Gold and Tabular

Overview

Load silver data into a gold analytical model (Power BI semantic model). The model YAML defines tables, columns, measures, partitions, and references — and feeds two generator outputs (Fabric semantic model item AND BIM file) plus the runtime loader.

What gets generated

Stage	Component	Output
Build	EasyFabric Generator (EFG)	GenerateFabricDatamart, GenerateModel → Fabric semantic model + BIM file
Runtime	EasyFabric Runtime (EFR)	easyfabric.load_data_gold.modelloader

Example YAML

Dataplatform/DP/Models/DM/model.yaml

- Table:
  Name: Verkopen
  Loadsettings:
    - Notebook: T_Verkopen
  SourceSchema: dbo
  SourceTable: f_verkopen
  TableType: Fact
  Description: "Fact / Feit: Verkopen"
  References:
    - Table: Product
    - Table: Kalender
  Columns:
    - SourceColumn: Quantity
      Type: decimal(21,5)
      IsHidden: true
  Measures: 
  - Name: Totaal Verkoop
    Expression: SUM(Verkopen[Quantity])
    Description: "Totaal Verkoop"
    FormatString: "#,##0.00"
    DisplayFolder: Omzet

- Table:
  Name: Product
  SourceSchema: dbo
  SourceTable: d_product
  Loadsettings:
    - Notebook: D_Product
  TableType: Dim
  Columns:
    - Name: Productnaam
      SourceColumn: ProductName
      Type: varchar(255)
    - Name: Category
      SourceColumn: Category
      Type: varchar(255)
   
- Table:
  Name: Kalender
  SourceSchema: dbo
  SourceTable: d_kalender
  Loadsettings:
    - Notebook: D_Kalender
  TableType: Dim
  DataCategory: Time
  KeyDataType: int
  Columns:
    - Name: BK_Kalender
      SourceColumn: BK_Kalender
      Type: int
      IsHidden: true
      IsKey: false
    - SourceColumn: Datum
      Type: date
      IsKey: true
    - SourceColumn: Jaar
      Type: int
    - SourceColumn: Kwartaal
      Type: varchar(10)
    - SourceColumn: Maandnummer
      Type: int
    - SourceColumn: Maandnaam
      OrderByColumn: Maandnummer
      Type: varchar(20)
  Measures:
    - Name: Jaar Filter
      Expression: |
        VAR TallyMin = Min(Tally[Tally Number integer]) + 0
        VAR TallyMax = Max(Tally[Tally Number integer]) + 0
        VAR MinYear = YEAR(NOW()) + TallyMin
        VAR MaxYear = YEAR(NOW()) + TallyMax
        VAR Years = CALCULATETABLE(FILTER('Kalender', 'Kalender'[Jaar] >= MinYear && 'Kalender'[Jaar] <= MaxYear))
        RETURN CALCULATE(IF(COUNTROWS(Years)>0,1,BLANK()))
      FormatString: GeheelGetal
    - Name: "# Dagen per jaar"
      Expression: |
        CALCULATE(MAX(Kalender[Dagnummer jaar]), ALLEXCEPT(Kalender, Kalender[Jaar]))
      FormatString: GeheelGetal

- Table:
  Name: Tally
  SourceSchema: dbo
  SourceTable: d_tally
  Loadsettings:
    - Notebook: D_Tally
  TableType: Dim
  Columns:
    - SourceColumn: TallyNumber
      Type: varchar(50)
    - Name: Tally Number integer
      SourceColumn: TallyNumberInt
      IsHidden: true
      Type: int
    - SourceColumn: Omgeving
      Type: varchar(50)

Schema reference

Required fields marked *. Linked types are collapsible — click to expand inline.

Tabular Model

Name	Type	Description
Columns *	List<TabularColumn>	List of columns in the model
Name *	String	Name of the table in the tabular model
TableType *	String	Type of the table in the model

Show optional fields (14)

Name	Type	Description
AIInstruction	String	AI-addressed instruction for the table. Surfaced in the wiki graph and never emitted to Power BI / Excel. When set it overrides Description as the graph instruction value; when both are empty the graph omits the field.
DataCategory	String	Classification of the table
Description	String	Description of the table
GoldTableSettings	FabricTableSettings	Table Settings for Gold
IsGenerated	true/false	Is the table included in the model
IsHidden	true/false	Is the table hidden in the model
Loadsettings	List<LoadSetting>	Notebook(s) that produce the gold table for this Tabular table. Consumed by the graph emitter to draw notebook -> gold-table lineage edges; ignored by SQL/TMSL generation.
Measures	List<DaxMeasure>	List of measures in the model
PermittedRoles	String[]	Roles that have permission to use this table
ProhibitedRoles	String[]	Roles that are not allowed to use this table
References	List<Reference>	Relations to other tables in the model
SourceSchema	String	Schema of the table in the source database (default=dm)
SourceTable	String	Name of the table in the source database
SurrogateKeyUnknownValue	String	Default value for surrogate key when no match is found, default also used for GUID fields. (default from Configuration)

Reference

Name	Type	Description
Table *	String	Name of the related table

Show optional fields (4)

Name	Type	Description
AIInstruction	String	AI-addressed instruction for this relationship. Surfaced in the wiki graph; omitted when empty. Reference has no Description, so there is no fallback.
Column	String	Columnname of the fact table. Dimension column will remain equal. Only necessary for multiple relationships to the same dimension. IsActiveRelation is mandatory when supplied.
IsActiveRelation	true/false	Is this relation active? (default=true)
MultiDirectionalFilter	true/false	Is this both way relation? Use with caution. (default=false)

Hierarchies

Name	Type	Description
Levels	String[]	A list of levels to add to the hierarchy (use the names from the tabular model
Name	String	Name of the hierarchy

Partition

Name	Type	Description
PartitionSource *	String	The (onelake) sourceobject, depends on datasourcetype that is used

Show optional fields (16)

Name	Type	Description
GroupName	String	The name of the partition. Should be unique per tabular table. Based on the config this name will be extended with the properties set elsewhere.
HasPartitionLeftOver	true/false	Create a leftover partition when using a partitiontype like (yyyyMM, yyyy, yyyyMMdd). It will create > then last partition (default=true)
IsGeneratedInDataplatform	true/false	Is this partition created in the dataplatform model. Set to false, for a view or a custom implementation. Make sure the requested field are available (default=true)
IsGeneratedInTabular	true/false	Is this partition created in the tabular model (default=true)
IsHistory	true/false	Is this partition used for history purposes. This can be used by the loading mechanism to skip this partition. (default=false)
Mode	String	The mode that is used for reading the data into tabular (import, directquery or default). Only import is supported (default=import)
Name	String	n/a
PartitionEnd	Nullable<Int32>	End - Start determines how many partitions that will be created. Also taking into account the size of every partition.
PartitionFilter	String	The filter that should occur on the sourceobject. Provide without WHERE, depends on datasourcetype that is used (example=MyCustomField='LoadThis'
PartitionFilterColumn	String	The filter is applied on the column, provided here and should exists in the underlying datasource.
PartitionIsHistory	Nullable<Int32>	The number of generated (oldest) date partitions that are categorised as history. History partitions are annotated so the loading mechanism can skip them except on a first or full load. (default=0)
PartitionProcessingEnd	String	Used as an instruction for the loading of the partition. Not used for generating the partition
PartitionSchema	String	The schema of the sourceobject, depends on datasourcetype that is used
PartitionSize	Nullable<Int32>	The size of every partition. In other words, how many to skip, before creating another partition. Default = 1
PartitionStart	Nullable<Int32>	The partition value to start with when creating the partition. Depending on PartitionType, Size and End it will help with generating the partitions.
PartitionType	PartitionTypes	The partitiontype is used to determine the kind of partition is created. More info in the tutorial. (default=All)

Tabular Column

Name	Type	Description
SourceColumn *	String	Sourcecolumn name, if name is not supplied, this name is used. Automatically camelcase script is applied.
Type *	String	Type of the field in the model

Show optional fields (18)

Name	Type	Description
AIInstruction	String	AI-addressed instruction for the field. Surfaced in the wiki graph and never emitted to Power BI / Excel. When set it overrides Description as the graph instruction value; when both are empty the graph omits the field.
Classification	String	Classification of this field. Set to sensitive for fields that should be handled with care.
ColumnSqlFunction	String	Before loading this column into tabular, use a scalar function on this column. Should be used for masking for example. Can make loading slow. Function should exist in database. Format is fn_YourFunction(0). The column name will be filled into 0
DataCategory	String	Data category of the column
DataType	String	Datatype of the column. Will be converted to Fabric datatype.
Description	String	Description of the field, used in BI-tools like Power BI and Excel
DisplayFolder	String	Folder for this field, used in BI-tools like Power BI and Excel
EncodingHint	String	Encoding Hint for columns (Default=Default)
FormatString	String	Format string of this field
IsAvailableInMdx	true/false	Set to false if not used in MDX clients for slicing and filtering
IsHidden	true/false	Field is hidden from the model (can still be queried)
IsKey	true/false	Field is used as the primary key for this table (default=false)
IsUnique	true/false	Field contains unique values (default=false)
Name	String	Name of the field in the model, use to override the automatic generated name based on sourcecolumn.
OrderByColumn	String	The field on which this field should order.
PermittedRoles	String[]	Roles that have permission to use this field
ProhibitedRoles	String[]	Roles that have no permission to use this field
SummarizeMeasure	String	Set the summarization for this field (sum, max, avg etcetera). Use measures instead.

Dax Measure

Name	Type	Description
AIInstruction	String	AI-addressed instruction for the measure. Surfaced in the wiki graph and never emitted to Power BI / Excel. When set it overrides Description as the graph instruction value; when both are empty the graph omits the field.
Annotations	List<Annotation>	Custom name/value annotations emitted onto the measure. The EF_ prefix is reserved for EasyFabric-internal annotations.
DataCategory	String	Data category for the measure
DerivedMeasures	List<DaxDerivedMeasure>	List of derived measures based on this measure
Description	String	Explanation of the measure, will be shown on the mouse over in Power BI. When not provided, the DAX is presented
DisplayFolder	String	Folder where the measure will be shown in the user interface of BI-tools like Power BI and Excel
Documentation	String	Place for documentation. Not used for generating anything at the moment.
Expression	String	DAX expression for the measure
ExpressionFile	String	DAX expression for the measure, based on a file
FormatString	String	Format string for the measure
FormatStringDefinition	String	Format string definition in case of Dynamic formatting for the measure
FormerName	String	For renaming, provide a formername and a copy with that will be created, so reports will remain working.
IsHidden	true/false	Hide the measure in the user interface of BI-tools like Power BI and Excel
IsSimpleMeasure	true/false	Is the measure a simple measure, like a column in a table, or a measure that is based on a DAX expression
Name	String	Name of the measure
Override	true/false	Only used in a derived model. When a measure with the same name already exists in the base model, set Override to true to replace it; otherwise generation fails so the clash is not silently ignored.

Dax Derived Measure

Name	Type	Description
DerivedMeasure *	String	Name of the derived measure

Show optional fields (9)

Name	Type	Description
DataCategory	String	Data category for the measure
Description	String	Override the description from the parent
DisplayFolder	String	Override the displayfolder from the parent
Documentation	String	Override the documentation from the parent
FormatString	String	Override the formatstring from the parent
FormatStringDefinition	String	Format string definition in case of Dynamic formatting for the measure
IsHidden	Nullable<true/false>	Override the ishidden property from the parent
Name	String	Name of the derived measure
Parameters	List<Dictionary<String, String>>	Parameters can be supplied by the calling measure

Tabular

Name	Type	Description
CompatibilityLevel *	Int32	Compatibility level of the tabular model (default=1600)
DatasourceName *	String	Name of the datasource (default=DefaultDS)
TabularName *	String	Name of the tabular model

Show optional fields (33)

Name	Type	Description
Account	String	n/a
AddMembers	true/false	Add members to the roles from role file (default=false)
BimOutputpath	String	Path where the bim file is saved
Connectionstring	String	n/a
Culture	String	n/a
DatabaseSchema	String	n/a
Dataset	String	Lakehouse dataset, for fabric connections
DatasourceType	String	Datasourcetype to set the source to Azure SQL or Fabric
DefaultDateFormatString	String	Format string applied to date columns (not datetime) that have no explicit FormatString. Default "dd-MM-yyyy".
DefaultEncodingHintKeyColumns	String	Default Encoding Hint for Key columns (Default=value)
DefaultEncodingHintRegularColumns	String	Default Encoding Hint for regular columns (Default=default)
DefaultPowerBiDataSourceVersion	String	n/a
DimBusinesskeyPostfix	String	Postfix of the businesskey column for the dimension
DimBusinesskeyPrefix	String	Prefix of the businesskey column for the dimension
DimkeyPostfix	String	Postfix of the keycolumn for the dimension. Will become obsolete. (Default=""
DimObjectPrefix	String	Dimension table prefix, for example D_.
DimSurrogatekeyPrefix	String	Prefix of the keycolumn for the dimension, will be replaced by DimSurrogatekeyPrefix
EffectiveDimTablePrefix	String	n/a
EffectiveFactTablePrefix	String	n/a
EffectiveSKPrefix	String	n/a
EffectiveTransformationObjectPrefix	String	n/a
FactObjectPrefix	String	Fact table prefix, for example F_. Will be replaced by FactTablePrefix
HideSurrogateKeys	true/false	Hide Surrogate keys in the model. Only disable when really needed. Will override other settings on the visibility! (default=True)
NormalizeAttributeNames	true/false	For special cases (like ontology) names should not contain certain characters. Will override other settings for labels of attributes! (default=False)
PlatformPowerUserRole	String	n/a
Rolespath	String	Relative path to the roles file
SetIsAvailableInMdx	true/false	When true, columns marked hidden or key are set NOT available in MDX. When false (default) every column is available in MDX.
SourceQueryCulture	String	n/a
SurrogateKeyUnknownValue	String	Default value for surrogate key when no match is found. (default='8C6BAD1B-71F9-FC31-EAB8-543E81F77571')
TabularHiddenFolder	String	Foldername for hidden items in the tabular model
TabularSystemFolder	String	Foldername for system items in the tabular model
TransformationObjectPrefix	String	Prefix for transformation objects, for example T_
TransformationPrefix	String	Prefix for transformation objects, will be replaced by TransformationObjectPrefix

Derived Model

Name	Type	Description
Measures	List<DaxMeasure>	n/a
Table	String	n/a

Format Strings

Name	Type	Description
FormatString	String	n/a
FriendlyName	String	n/a
IsDefault	true/false	n/a
Name	String	n/a

Measure From File

Name	Type	Description
Expression	String	n/a
FileName	String	n/a

Measure Template

Name	Type	Description
DataCategory	String	n/a
DerivedMeasure	String	n/a
Description	String	n/a
DisplayFolder	String	n/a
Expression	String	n/a
FormatString	String	n/a
IsHidden	Nullable<true/false>	n/a
MeasureName	String	n/a

Custom Roles

Name	Type	Description
Alias	String	n/a
Daxfilters	List<Daxfilter>	n/a
Groups	String[]	n/a
Members	String[]	n/a
Name	String	n/a

Daxfilter

Name	Type	Description
Expression	String	n/a
Table	String	n/a

EasyFabric Runtime

load_data_gold.modelloader

def modelloader(data_frame: DataFrame,
                load_config: LoadConfig,
                model_config: Model,
                config_manager: ConfigManager = None) -> str | None

Loads data into the gold layer of a data platform using the specified configurations and model.

This function handles loading a DataFrame into the data platform's "gold" layer. It configures logging, saves metadata, and uses the provided model configuration to apply necessary transformations and loading procedures. Proper initialization of LoadConfig, Model, and ConfigManager is required before calling this function.

Arguments:

• data_frame DataFrame - The data to be loaded into the platform.
• load_config LoadConfig - Configuration settings for the loading process.
• model_config Model - Model configuration required for transformations and validations.
• config_manager ConfigManager - Manages configuration details and ensures proper initialization.

LoadConfig fields

Runtime parameter bag — construct in code and pass to the loader. All fields are optional unless flagged below.

Field	Type	Description
_layer	str	Operational layer associated with the configuration. Defaults to "not set".
dry_run	bool	Indicates if the process should be executed in dry-run mode. Defaults to True.
auto_null_column	bool	Determines if null values should be automatically managed for columns. Defaults to True.
load_type	LoadType	Specifies the type of load operation. Defaults to LoadType.FULL.
stop_at_error	bool	Specifies whether the process should stop when an error occurs. Defaults to True.
business_key_check	bool	Indicates if business keys should be validated during the load. Defaults to True.
log_row_count	bool	Deprecated and ignored. Row counts are always logged; retained for backward compatibility. Defaults to False.
key_violation_action	str	Action to be taken when key violations occur. Defaults to "raise".
load_history_table	bool	When true, also load the object's history table alongside the main table. Defaults to False.
model_object_name	Optional[str]	Specifies the model object name used in operations. Defaults to None.

Returns:

• str - A string indicating the outcome of the loading process, or None if successful.

Raises:

• Exception - If load_config.model_object_name is missing.
• Exception - If config_manager is not properly initialized.
• Exception - If model_config is not provided.
• Exception - If an error occurs during processing and Model5 is True.

Related data classes

• easyfabric.data.Model
• easyfabric.data.Table
• easyfabric.data.Reference
• easyfabric.data.Partition