Skip to main content

YAML Properties Reference

This page provides a comprehensive overview of all YAMLA simple way to write configurations. It's basically a list that computers can read easily. properties used across the EasyFabricCore system. It specifically identifies which properties are utilized by the major system components.

Component Overview​

  • API: Web API that receives JSON requests (converted from YAML) to generate Fabric objects.
  • Dataplatform: Desktop application (Generator.exe) that reads YAML files to generate notebooks, tables, and configurations.
  • Wheel Package (easyfabric-py): Python library used at runtime in Fabric notebooks to load and process data.

Required vs. optional properties​

Unless a property's description is marked (required), the property is optional and may be omitted.

  • A (default=…) annotation shows the value the system uses when you leave the property out.
  • A property with no default annotation is still optional unless marked (required); when omitted it is simply left unset, or inherited from a higher config level (see note 6).

Configuration Properties​

Defined in the main environment configuration file (e.g., dp.dev.yaml).

PropertyAPIDataplatformWheel PackageDescription
BronzeConnectionPrefix✅✅✅Use prefix from ConnectionPrefix for bronze objects (default=true)
BronzeKeepSourceColumnNames✅✅—Use source names even when overridden on bronze (default=false)
BronzeObjectPrefix✅✅✅Prefix for bronze objects (default='')
BronzeTableSettings✅✅—Table settings for Bronze layer (Delta properties)
ColumnNamesToLower✅✅—Generate all column names to lower (default=true)
ColumnNamesAutoFix✅✅—Auto fix columns with only numbers by adding 'z' prefix (default=true)
DefaultSeparator✅✅✅Separator used for object names (default='_')
DimBusinesskeyPrefix✅✅—Prefix of businesskey column for dimensions (default='BK_')
DimObjectPrefix✅✅—Prefix used for dimensions (default='D')
DimSurrogatekeyPrefix✅✅—Prefix used for dimension surrogate keys (default='SK_')
FactObjectPrefix✅✅—Prefix used for fact objects (default='F')
GoldTableSettings✅✅—Table settings for Gold layer (Delta properties)
HelpObjectPrefix✅✅—Prefix used for help objects (default='H'; a leading '_' in the Name is stripped before prefixing; empty means no prefix)
HistoryTablePostfix✅✅✅Postfix used for objects with history (default='')
IsBronzeStringTypes✅✅✅Create fields in bronze layer with datatype string (default=true)
Lakehouses✅✅✅List of lakehouses used in medallion layer
ObjectPartitionColumn✅✅—Name of partition column (default='SYSTEMPARTITION')
ObjectSourceTagColumn✅✅—Name of system source tag column (default='SYSTEMSOURCETAG')
RemoveNewLineFromScripts✅✅—Remove newline from generated scripts (default=false)
SchemaHistory✅✅✅Schema name for history tables (default='his')
SilverConnectionPrefix✅✅✅Use connection prefix for object names on silver (default=true)
SilverConvertedColumn✅✅—Create Silver column with converted value (default=true)
SilverConvertedPostfix✅✅✅Postfix for converted column in silver (default='_convert')
SilverInitialColumn✅✅—Create Silver column with original value (default=true)
SilverObjectPrefix✅✅—Prefix for silver objects (default='')
SilverPrimarykeySeparator✅✅—Separator for SYSTEMPRIMARYKEY field (default='
SilverTableSettings✅✅—Table settings for Silver layer (Delta properties)
SurrogateKeyUnknownValue✅✅—Default value for surrogate key when no match (default='-1')
TransformationBusinessKeysNotNull✅✅—Business keys in transformation objects mandatory (default=false)
TransformationObjectPrefix✅✅—Prefix used for transformation objects (default='T')
DefaultEncodingHintKeyColumns✅✅—Default encoding hint for key columns (default='Value')
DefaultEncodingHintRegularColumns✅✅—Default encoding hint for regular columns (default='Default')

Connection Properties​

Defined in connection YAML files (e.g., adv-advworks.yaml).

PropertyAPIDataplatformWheel PackageDescription
ConnectionName✅✅✅Name of the connection (required)
ConnectionPrefix✅✅✅Prefix used when connection prefix is enabled (required)
ConnectionType——✅Type of connection (e.g., azblob, fabricfiles, customperfile, noaction)
Description———Description of the connection. Read by no component; a note for whoever opens the file next
BronzeCleanupActive——✅Enable automatic cleanup of expired bronze history rows (default: unset — cleanup off)
BronzeChecksumMismatchViolationAction——✅Response when a file's checksum/size changed but no new rows landed in Bronze.his (Validation 2): continue warns, stop raises, silent does neither. Independent of BronzeLoadViolationAction (default: unset — falls back to continue on the table)
BronzeFolder——✅Folder path for bronze files
BronzeKeyViolationAction——✅Action on key violations (raise/keepone/remove)
BronzeLoadOrder——✅Load order for bronze notebooks (default=0)
BronzeLoadSkip——✅Skip bronze loading (default=false)
BronzeLoadViolationAction——✅Fallback for tables that don't set their own BronzeLoadViolationAction: continue warns, stop raises, silent does neither. An empty delivery is not a delete: when a source file arrives with zero usable rows, the bronze history writer skips delete detection entirely, so the existing history keeps its active rows whatever this setting says. (default: unset — falls back to stop on the table)
BronzeMaxRetentionHours——✅Retention window in hours for bronze history cleanup when BronzeCleanupActive is on (default: unset)
BronzeNotebookConcurrency——✅Max object notebooks of this connection running at the same time within one load-order group, on top of the loader's own capacity ceiling (default=0 — unlimited)
BronzeTableSettings✅✅—Table settings for Bronze at connection level
Charset——✅Character set for file reading
Container——✅Storage container name
Dateformat——✅Default date format for parsing, for every object on this connection that sets none itself. With nothing set anywhere the loader falls back to M/d/yyyy h:mm:ss a and says so in the log
Delimiter——✅CSV delimiter (default=',')
Escape——✅Escape character for CSV
FileExtension——✅File extension filter
FileType——✅Type of file (csv, json, xml, parquet, etc.)
Header——✅CSV has header row (default=true)
IgnoreNullKey——✅Ignore records with null keys (default=true)
InferSchema——✅Infer schema from data (default=false)
IsBronzeStringTypes✅✅✅Override config setting for this connection
IsTriggerable——✅Connection can be triggered (default=true)
JsonPath——✅Path to the nested array of records, e.g. $.rows. Omit for files that are themselves a JSON array (top-level records load directly)
KeyvaultSecretConnectionString——✅Keyvault secret name for connection string
BronzeMaxFileAgeHours——✅Max age in hours of a source file; older files are reported per BronzeLoadViolationAction (and caught even when the source is unchanged) (default=2). Was MaxFileAgeHours, which still works and carries its value over
BronzeMaxFileAgeCriticalHours——✅Age past which a stale source file is reported as critical instead of its base severity. Must be above BronzeMaxFileAgeHours (default: unset — a stale file reports the same however long it stays stale)
BronzeMaxUnchangedHours——✅Longest stretch the source may stay unchanged before one full load runs anyway and the expiry is reported. Bounds the blind spot of a size-based comparison (default: unset — unbounded; see Skip-if-Unchanged)
Mode——✅Spark read mode (default='PERMISSIVE')
Multiline——✅Support multiline records (default=false)
NotebookTimeout——✅Default timeout for notebooks (default=1800)
ProcessType——✅Type of processing
Quote——✅Quote character for CSV
QuoteMode——✅Quote mode for CSV
SilverDateConversionViolationAction——✅Response when a date pattern empties a typed column completely — every filled source value became NULL in silver: continue warns with column, pattern and an example value, stop raises, silent skips the measurement (default: unset — falls back to continue on the table)
SilverLoadOrder——✅Load order for silver notebooks (default=0)
SilverNotebookConcurrency——✅Max object notebooks of this connection running at the same time within one load-order group, on top of the loader's own capacity ceiling (default=0 — unlimited)
SilverTableSettings✅✅—Table settings for Silver at connection level
BronzeSkipIfSourceUnchanged——✅Skip the load when source files are unchanged since the last run (default: unset — inherited; see Skip-if-Unchanged). Was SkipIfSourceUnchanged, which still works and carries its value over
BronzeLoadSeverity——✅How loudly a stale source file is reported: none, info, warning, critical. Independent of BronzeLoadViolationAction, which decides whether the load goes on (default=warning)
BronzeChecksumMismatchSeverity——✅How loudly a checksum/row-count mismatch (Validation 2) is reported: none, info, warning, critical (default=warning)
BronzeUnchangedSeverity——✅How loudly an expired BronzeMaxUnchangedHours window is reported, on the same four rungs (default=warning)
SourceFolder——✅Source folder path

Object/Table Properties​

Defined in object YAML files (e.g., customers.yaml).

PropertyAPIDataplatformWheel PackageDescription
Connection✅✅✅Connection to use for this object (required)
SourceTable✅✅✅Name of the object in the source
SourceSchema✅✅—Source schema (if applicable)
DataPlatformObjectname✅✅✅Override source name in dataplatform (required)
Description✅✅—Description of the source
IsActive✅✅✅Set to active for generating (default=true)
KeepHistory✅✅✅Enable history tracking (default=true — see note 9)
KeepOneDayRecords——✅Keep only one day of records (default=false)
Layers✅✅✅Which layers to generate (ALL/Bronze/Silver/Gold, default=ALL)
Prefix✅✅—Set a prefix for this object
Columns✅✅✅List of columns/fields from source
Fields✅✅—Alternative spelling of Columns, read by the API and Generator only
BronzeTableSettings✅✅—Table settings for Bronze at object level
SilverTableSettings✅✅—Table settings for Silver at object level
SourceFilter——✅Filter query for source data
SourceFolder——✅Override source folder
SourceOrder——✅Order by clause for source
BronzeFolder——✅Override bronze folder
BronzeKeyViolationAction——✅Override connection setting
BronzeLoadOrder——✅Override connection load order
BronzeLoadSkip——✅Skip bronze load for this table (default=false)
BronzeLoadViolationAction——✅Response to load violations including stale files (bronzemaxfileagehours): continue warns, stop raises, silent does neither. An empty delivery is not a delete: when a source file arrives with zero usable rows, the bronze history writer skips delete detection entirely, so the existing history keeps its active rows whatever this setting says. (default=stop, falls back to the connection)
BronzeChecksumMismatchViolationAction——✅Response when a file's checksum/size changed but no new rows landed in Bronze.his (Validation 2): continue warns, stop raises, silent does neither. Separate from BronzeLoadViolationAction, so a stale-file stop policy doesn't also turn this heuristic into a hard error (default=continue, falls back to the connection)
BronzeMaxExpectedDuration——✅Max expected duration for bronze (default=1800)
BronzeCleanupActive——✅Override connection: enable cleanup of expired bronze history rows
BronzeMaxRetentionHours——✅Override connection: retention window in hours for bronze history cleanup
BronzeSkipIfSourceUnchanged——✅Skip the load when source files are unchanged since the last run
BronzeMaxUnchangedHours——✅Override connection: longest stretch this source may stay unchanged before a full load runs anyway
BronzeMaxFileAgeHours——✅Override connection: max age in hours of a source file before it is reported per BronzeLoadViolationAction
BronzeMaxFileAgeCriticalHours——✅Override connection: age past which a stale file for this object reports as critical
BronzeLoadSeverity——✅Override connection: how loudly a stale source file for this object is reported
BronzeChecksumMismatchSeverity——✅Override connection: how loudly a checksum/row-count mismatch for this object is reported
BronzeUnchangedSeverity——✅Override connection: how loudly an expired unchanged-window for this object is reported
DateFormat——✅Override date format for this table. It applies to every date column of the object that sets none itself, so an object whose columns are delivered in different formats needs the format per column
FileExtension——✅Override file extension
FileType——✅Override file type
ConnectionType——✅Override connection type (e.g., azblob, fabricfiles, customperfile, noaction)
Delimiter——✅Override connection CSV delimiter
Header——✅Override connection: CSV has header row
Multiline——✅Override connection: support multiline records
Escape——✅Override connection CSV escape character
Quote——✅Override connection CSV quote character
QuoteMode——✅Override connection CSV quote mode
Charset——✅Override connection character set for file reading
JsonPath——✅Override connection JSON path for extracting data
HistorySettings——✅History-specific settings
IgnoreNullKey——✅Override ignore null key setting (default=true)
PreBronzeNotebook——✅Notebook to run before bronze
MidBronzeNotebook——✅Notebook to run during bronze
PostBronzeNotebook——✅Notebook to run after bronze
PreSilverNotebook——✅Notebook to run before silver
MidSilverNotebook——✅Notebook to run during silver
PostSilverNotebook——✅Notebook to run after silver
BronzeNotebook——✅Custom bronze notebook
SilverNotebook——✅Custom silver notebook
SheetName——✅Excel sheet name
SilverLoadOrder——✅Override silver load order
SilverDateConversionViolationAction——✅Response when a date pattern empties a typed column completely — every filled source value became NULL in silver: continue warns with column, pattern and an example value, stop raises, silent skips the measurement (default=continue, falls back to the connection)
SilverLoadType——✅Load type (full/merge/omit, default='full')
SilverMaxExpectedDuration——✅Max expected duration for silver (default=1800)
Trigger——✅Trigger configuration
XmlRowPath——✅XML row path for parsing

Column/Field Properties​

Defined within the Columns or Fields array in object files.

PropertyAPIDataplatformWheel PackageDescription
SourceColumn✅✅✅Name of the attribute in source system
SourceDataType✅✅✅Datatype of the source
DataPlatformName✅✅✅Override name used in dataplatform
IsPrimaryKey✅✅✅Attribute is part of primary key (default=false)
IsNullable✅✅✅Attribute can be null (default=true for non-PK)
IsActive✅✅—Attribute is active (default=true). Applied by the Generator; an inactive column is never generated, so the wheel never sees it
IsSilverOnly✅✅✅Attribute only generated for Silver (default=false)
IsPartition✅✅—Attribute is part of partition column (default=false)
SilverDataType✅✅✅Datatype used in Silver layer
SilverExpression✅✅✅Expression to fill Silver column
SilverConvertedColumn✅✅✅Create converted column (default=true)
SourceExpression✅✅✅Expression for Bronze layer conversion
Classification✅✅—Field classification (sensitive/restricted/internal/public)
HstgType✅✅—Type used in HSTG (historization)
DefaultValue——✅Default value for column
DateFormat——✅Date format for this column
DestinationColumn——✅Destination column name (legacy)
ConversionExpression——✅Conversion expression (legacy)
IsZOrder——✅Use column for Z-ordering (default=false)
IsDeletedIdentifier——✅Column identifies deleted records (default=false)

History Settings Properties​

Nested within HistorySettings in table configuration.

PropertyAPIDataplatformWheel PackageDescription
BronzeDeleteFilterQuery——✅Filter query for bronze delete operations (cannot be combined with BronzeSkipDelete)
BronzeSkipDelete——✅Skip delete operations in bronze (default=false)
SourceFilterQuery——✅Filter query for source data; may be combined with BronzeDeleteFilterQuery or BronzeSkipDelete
SilverFilterQuery——✅Filter query for silver data

Notebook Configuration Properties​

Nested within notebook configuration objects (e.g., PreBronzeNotebook).

PropertyAPIDataplatformWheel PackageDescription
Notebook——✅Notebook name to execute
Timeout——✅Notebook execution timeout in seconds (optional; default=3600 / 1 hour when omitted)
Param001——✅First parameter to pass to notebook
Param002——✅Second parameter to pass to notebook
Param003——✅Third parameter to pass to notebook

Trigger Properties​

Nested within Trigger configuration.

PropertyAPIDataplatformWheel PackageDescription
Notebook——✅Notebook to trigger
NotifyWebhookUrl——✅Webhook URL for notifications
NotifyWebhookType——✅Type of webhook notification

Lakehouse Properties​

Defined within the Lakehouses array in environment configuration.

PropertyAPIDataplatformWheel PackageDescription
Lakehouse✅✅✅Name of the lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database.
Layer✅✅✅Layer (Bronze/Silver/Gold/Meta)
Workspace✅✅✅Fabric workspace name
UseSchema✅✅—Use schema for this lakehouse (default=true)
DefaultSchema✅✅—Schema to use (default='dbo' or 'his' for history)

Table Settings Properties​

Nested within BronzeTableSettings, SilverTableSettings or GoldTableSettings. These accept any valid Delta Lake table properties. The gold settings can also be set per table in the tabular model (GoldTableSettings on a table entry), which overrides the configuration-level value.

PropertyAPIDataplatformWheel PackageDescription
delta.enableChangeDataFeed✅✅—Enable change data feed
delta.autoOptimize.optimizeWrite✅✅—Enable optimize write
delta.autoOptimize.autoCompact✅✅—Enable auto compaction
delta.logRetentionDuration✅✅—Log retention duration
delta.deletedFileRetentionDuration✅✅—Deleted file retention
spark.sql.parquet.vorder.default✅✅—Enable V-Order
delta.stats.extended.collect✅✅—Collect extended statistics
delta.stats.extended.inject✅✅—Inject extended statistics
(Any Delta property)✅✅—FabricTableSettings accepts arbitrary Delta properties

Notes​

  1. API receives YAML converted to JSON in web requests and uses C# model classes to deserialize
  2. Dataplatform reads YAML files directly from disk and uses the same C# model classes
  3. Wheel Package reads YAML files at runtime in notebooks using Python dataclasses
  4. Properties marked with ✅ are actively used by that component
  5. Properties marked with — are not used by that component
  6. Table Settings can be defined at three levels (Configuration → Connection → Object) with object-level overriding connection-level, which overrides configuration-level
  7. The Dataplatform and API share the same model classes, so they support identical properties for object/field/configuration definitions
  8. The Wheel Package has additional runtime properties for orchestration, notebook execution, and data loading that are not needed during generation
  9. KeepHistory defaults to true everywhere. Both the generator (API / Dataplatform) and the runtime Wheel Package treat an omitted KeepHistory as true: generated objects get history tables and the runtime writes history rows by default. Set KeepHistory: false on the object to opt out of history tracking (for example when the source has no primary key).