Gold and Tabular

Overview

Load silver data into a gold analytical model (Power BI semantic model). The model YAML defines tables, columns, measures, partitions, and references — and feeds two generator outputs (Fabric semantic model item AND BIM file) plus the runtime loader.

What gets generated

Stage	Component	Output
Build	EasyFabric Generator (EFG)	GenerateFabricDatamart, GenerateModel → Fabric semantic model + BIM file
Runtime	EasyFabric Runtime (EFR)	easyfabric.load_data_gold.modelloader

Example YAML

Dataplatform/DP/Models/DM/model.yaml

- Table:
  Name: Verkopen
  Loadsettings:
    - Notebook: T_Verkopen
  SourceSchema: dbo
  SourceTable: f_verkopen
  TableType: Fact
  Description: "Fact / Feit: Verkopen"
  References:
    - Table: Product
    - Table: Kalender
  Columns:
    - SourceColumn: Quantity
      Type: decimal(21,5)
      IsHidden: true
  Measures: 
  - Name: Totaal Verkoop
    Expression: SUM(Verkopen[Quantity])
    Description: "Totaal Verkoop"
    FormatString: "#,##0.00"
    DisplayFolder: Omzet

- Table:
  Name: Product
  SourceSchema: dbo
  SourceTable: d_product
  Loadsettings:
    - Notebook: D_Product
  TableType: Dim
  Columns:
    - Name: Productnaam
      SourceColumn: ProductName
      Type: varchar(255)
    - Name: Category
      SourceColumn: Category
      Type: varchar(255)
   
- Table:
  Name: Kalender
  SourceSchema: dbo
  SourceTable: d_kalender
  Loadsettings:
    - Notebook: D_Kalender
  TableType: Dim
  DataCategory: Time
  KeyDataType: int
  Columns:
    - Name: BK_Kalender
      SourceColumn: BK_Kalender
      Type: int
      IsHidden: true
      IsKey: false
    - SourceColumn: Datum
      Type: date
      IsKey: true
    - SourceColumn: Jaar
      Type: int
    - SourceColumn: Kwartaal
      Type: varchar(10)
    - SourceColumn: Maandnummer
      Type: int
    - SourceColumn: Maandnaam
      OrderByColumn: Maandnummer
      Type: varchar(20)
  Measures:
    - Name: Jaar Filter
      Expression: |
        VAR TallyMin = Min(Tally[Tally Number integer]) + 0
        VAR TallyMax = Max(Tally[Tally Number integer]) + 0
        VAR MinYear = YEAR(NOW()) + TallyMin
        VAR MaxYear = YEAR(NOW()) + TallyMax
        VAR Years = CALCULATETABLE(FILTER('Kalender', 'Kalender'[Jaar] >= MinYear && 'Kalender'[Jaar] <= MaxYear))
        RETURN CALCULATE(IF(COUNTROWS(Years)>0,1,BLANK()))
      FormatString: GeheelGetal
    - Name: "# Dagen per jaar"
      Expression: |
        CALCULATE(MAX(Kalender[Dagnummer jaar]), ALLEXCEPT(Kalender, Kalender[Jaar]))
      FormatString: GeheelGetal

- Table:
  Name: Tally
  SourceSchema: dbo
  SourceTable: d_tally
  Loadsettings:
    - Notebook: D_Tally
  TableType: Dim
  Columns:
    - SourceColumn: TallyNumber
      Type: varchar(50)
    - Name: Tally Number integer
      SourceColumn: TallyNumberInt
      IsHidden: true
      Type: int
    - SourceColumn: Omgeving
      Type: varchar(50)

Schema reference

Required fields marked *. Linked types are collapsible — click to expand inline.

Tabular Model

Name	Type	Description
Columns *	List<TabularColumn>	List of columns in the model
Name *	String	Name of the table in the tabular model
TableType *	String	Type of the table in the model

Show optional fields (10)

Name	Type	Description
DataCategory	String	Classification of the table
Description	String	Description of the table
IsGenerated	true/false	Is the table included in the model
IsHidden	true/false	Is the table hidden in the model
Measures	List<DaxMeasure>	List of measures in the model
PermittedRoles	String[]	Roles that have permission to use this table
ProhibitedRoles	String[]	Roles that are not allowed to use this table
References	List<Reference>	Relations to other tables in the model
SourceSchema	String	Schema of the table in the source database (default=dm)
SourceTable	String	Name of the table in the source database

Reference

Name	Type	Description
Table *	String	Name of the related table

Show optional fields (1)

Name	Type	Description
MultiDirectionalFilter	true/false	Is this both way relation? Use with caution. (default=false)

Hierarchies

Name	Type	Description
Levels	String[]	A list of levels to add to the hierarchy (use the names from the tabular model
Name	String	Name of the hierarchy

Partition

Name	Type	Description
PartitionSource *	String	The (onelake) sourceobject, depends on datasourcetype that is used

Show optional fields (16)

Name	Type	Description
GroupName	String	The name of the partition. Should be unique per tabular table. Based on the config this name will be extended with the properties set elsewhere.
HasPartitionLeftOver	true/false	Create a leftover partition when using a partitiontype like (yyyyMM, yyyy, yyyyMMdd). It will create > then last partition (default=true)
IsGeneratedInDataplatform	true/false	Is this partition created in the dataplatform model. Set to false, for a view or a custom implementation. Make sure the requested field are available (default=true)
IsGeneratedInTabular	true/false	Is this partition created in the tabular model (default=true)
IsHistory	true/false	Is this partition used for history purposes. This can be used by the loading mechanism to skip this partition. (default=false)
LoadLastPartitions	Nullable<Int32>	Used as an instruction for the loading of the partition. Not used for generating.
Mode	String	The mode that is used for reading the data into tabular (import, directquery or default). Only import is supported (default=import)
Name	String	n/a
PartitionEnd	Nullable<Int32>	End - Start determines how many partitions that will be created. Also taking into account the size of every partition.
PartitionFilter	String	The filter that should occur on the sourceobject. Provide without WHERE, depends on datasourcetype that is used (example=MyCustomField='LoadThis'
PartitionFilterColumn	String	The filter is applied on the column, provided here and should exists in the underlying datasource.
PartitionProcessingEnd	String	Used as an instruction for the loading of the partition. Not used for generating the partition
PartitionSchema	String	The schema of the sourceobject, depends on datasourcetype that is used
PartitionSize	Nullable<Int32>	The size of every partition. In other words, how many to skip, before creating another partition. Default = 1
PartitionStart	Nullable<Int32>	The partition value to start with when creating the partition. Depending on PartitionType, Size and End it will help with generating the partitions.
PartitionType	PartitionTypes	The partitiontype is used to determine the kind of partition is created. More info in the tutorial. (default=All)

Clustered Index

Name	Type	Description
Columns	String[]	Columns for the index
DataCompression	String	Compression type for the index. (default=none)
FillFactor	Int32	Fill factor for the index. (default = 90)
Online	String	Build/Rebuild the index online (default = off)
StatisticsNoRecompute	String	Statistics no recompute (default = on)

Tabular Column

Name	Type	Description
SourceColumn *	String	Sourcecolumn name, if name is not supplied, this name is used. Automatically camelcase script is applied.
Type *	String	Type of the field in the model

Show optional fields (15)

Name	Type	Description
Classification	String	Classification of this field. Set to sensitive for fields that should be handled with care.
ColumnSqlFunction	String	Before loading this column into tabular, use a scalar function on this column. Should be used for masking for example. Can make loading slow. Function should exist in database. Format is fn_YourFunction(0). The column name will be filled into 0
DataCategory	String	Data category of the column
DataType	String	Datatype of the column. Will be converted to Fabric datatype.
Description	String	Description of the field, used in BI-tools like Power BI and Excel
DisplayFolder	String	Folder for this field, used in BI-tools like Power BI and Excel
FormatString	String	Format string of this field
IsHidden	true/false	Field is hidden from the model (can still be queried)
IsKey	true/false	Field is used as the primary key for this table (default=false)
IsUnique	true/false	Field contains unique values (default=false)
Name	String	Name of the field in the model, use to override the automatic generated name based on sourcecolumn.
OrderByColumn	String	The field on which this field should order.
PermittedRoles	String[]	Roles that have permission to use this field
ProhibitedRoles	String[]	Roles that have no permission to use this field
SummarizeMeasure	String	Set the summarization for this field (sum, max, avg etcetera). Use measures instead.

Dax Measure

Name	Type	Description
DerivedMeasures	List<DaxDerivedMeasure>	List of derived measures based on this measure
Description	String	Explanation of the measure, will be shown on the mouse over in Power BI. When not provided, the DAX is presented
DisplayFolder	String	Folder where the measure will be shown in the user interface of BI-tools like Power BI and Excel
Documentation	String	Place for documentation. Not used for generating anything at the moment.
Expression	String	DAX expression for the measure
ExpressionFile	String	DAX expression for the measure, based on a file
FormatString	String	Format string for the measure
FormerName	String	For renaming, provide a formername and a copy with that will be created, so reports will remain working.
IsHidden	true/false	Hide the measure in the user interface of BI-tools like Power BI and Excel
IsSimpleMeasure	true/false	Is the measure a simple measure, like a column in a table, or a measure that is based on a DAX expression
Name	String	Name of the measure

Dax Derived Measure

Name	Type	Description
DerivedMeasure *	String	Name of the derived measure

Show optional fields (6)

Name	Type	Description
Description	String	Override the description from the parent
DisplayFolder	String	Override the displayfolder from the parent
Documentation	String	Override the documentation from the parent
FormatString	String	Override the formatstring from the parent
IsHidden	Nullable<true/false>	Override the ishidden property from the parent
Name	String	Name of the derived measure

Tabular

Name	Type	Description
CompatibilityLevel *	Int32	n/a
DatasourceName *	String	n/a
TabularName *	String	n/a

Show optional fields (19)

Name	Type	Description
Account	String	n/a
AddMembers	true/false	n/a
BimOutputpath	String	n/a
Connectionstring	String	n/a
Culture	String	n/a
DatabaseSchema	String	n/a
DatasourceType	String	n/a
DefaultDatasource	String	n/a
DefaultPowerBiDataSourceVersion	String	n/a
DimkeyPostfix	String	n/a
DimkeyPrefix	String	n/a
DimPrefix	String	n/a
FactPrefix	String	n/a
PlatformPowerUserRole	String	n/a
Rolespath	String	n/a
SourceQueryCulture	String	n/a
TabularHiddenFolder	String	n/a
TabularSystemFolder	String	n/a
TransformationPrefix	String	n/a

Derived Model

Name	Type	Description
Table	String	n/a

Format Strings

Name	Type	Description
FormatString	String	n/a
FriendlyName	String	n/a
IsDefault	true/false	n/a
Name	String	n/a

Measure From File

Name	Type	Description
Expression	String	n/a
FileName	String	n/a

Measure Template

Name	Type	Description
DerivedMeasure	String	n/a
Description	String	n/a
Expression	String	n/a
IsHidden	Nullable<true/false>	n/a
MeasureName	String	n/a

Custom Roles

Name	Type	Description
Alias	String	n/a
Daxfilters	List<Daxfilter>	n/a
Groups	String[]	n/a
Members	String[]	n/a
Name	String	n/a

Daxfilter

Name	Type	Description
Expression	String	n/a
Table	String	n/a

EasyFabric Runtime

load_data_gold.modelloader

def modelloader(data_frame: DataFrame, load_config: LoadConfig,
                model_config: Model, config_manager: ConfigManager=None)

Loads data into the gold layer of a data platform using the specified configurations and model.

This function handles loading a DataFrame into the data platform's "gold" layer. It configures logging, saves metadata, and uses the provided model configuration to apply necessary transformations and loading procedures. Proper initialization of LoadConfig, Model, and ConfigManager is required before calling this function.

Arguments:

• data_frame DataFrame - The data to be loaded into the platform.
• load_config LoadConfig - Configuration settings for the loading process.
• model_config Model - Model configuration required for transformations and validations.
• config_manager ConfigManager - Manages configuration details and ensures proper initialization.

LoadConfig fields

Runtime parameter bag — construct in code and pass to the loader. All fields are optional unless flagged below.

Field	Type	Description
_layer	str	Operational layer associated with the configuration. Defaults to "not set".
dry_run	bool	Indicates if the process should be executed in dry-run mode. Defaults to True.
auto_null_column	bool	Determines if null values should be automatically managed for columns. Defaults to True.
load_type	LoadType	Specifies the type of load operation. Defaults to LoadType.FULL.
stop_at_error	bool	Specifies whether the process should stop when an error occurs. Defaults to True.
business_key_check	bool	Indicates if business keys should be validated during the load. Defaults to True.
log_row_count	bool	Determines if row counts should be logged during the process. Defaults to False.
key_violation_action	str	Action to be taken when key violations occur. Defaults to "raise".
destination_schema	str	Schema of the destination table. Defaults to "dbo".
destination_table	Optional[str]	Name of the destination table. Defaults to None.

Returns:

• str - A string indicating the outcome of the loading process, or None if successful.

Raises:

• Exception - If load_config.model_object_name is missing.
• Exception - If config_manager is not properly initialized.
• Exception - If model_config is not provided.
• Exception - If any other error occurs during the loading process and Model5 is set to True.

Related data classes

• easyfabric.data.Model
• easyfabric.data.Table
• easyfabric.data.Reference
• easyfabric.data.Partition