Splitting the model with Include
Overview
A large model.yaml can be split into several files with Include. The EasyFabric Generator resolves every Include while it
builds: it reads the included files, splices their contents in, and sends one
complete document to the generation API. The included files are a way to
organise your source; nothing downstream ever sees them.
Include works in every file of a model folder
(Dataplatform/DP/Models/<model>/): model.yaml, the derived model, roles,
format strings and measure templates.
Include forms
One file per list item
An - Include: item in a list is replaced by the contents of the file. When
the included file holds a list, its items are spliced in at that position, so
Include items mix freely with inline entries.
Dataplatform/DP/Models/DM/model.yaml
- Table:
Name: Calendar
TableType: Dimension
- Include: model_dims.yaml
- Include: model_facts.yaml
Dataplatform/DP/Models/DM/model_dims.yaml
- Table:
Name: Customer
TableType: Dimension
- Table:
Name: Product
TableType: Dimension
Several files under one Include
- Include:
- model_dims.yaml
- model_facts.yaml
The files are spliced in the order they are listed.
Include inside an entry
An Include key next to other keys merges the included file into that entry.
Lists from both sides are combined; for a single value defined on both sides,
the included file wins. Use this to share a block, such as a set of columns,
between tables.
- Table:
Name: Customer
Include: shared/audit_columns.yaml
Columns:
- SourceColumn: CustomerName
shared/audit_columns.yaml
Columns:
- SourceColumn: CreatedAt
- SourceColumn: ModifiedAt
Nested includes
An included file may itself contain Include. Nesting has no fixed depth.
Paths
A path in Include is relative to the folder of the file that contains the
Include, not to the model folder or the repository root. In a nested
include, the inner path is resolved from the inner file's own folder.
Subfolders are allowed, which keeps the main file short:
Dataplatform/DP/Models/DM/
├── model.yaml # - Include: base/model_dims.yaml
└── base/
├── model_dims.yaml # - Include: model_dims_customer.yaml (resolved inside base/)
├── model_dims_customer.yaml
└── model_facts.yaml
Included files may be YAMLA simple way to write configurations. It's basically a list that computers can read easily. (.yaml, .yml) or JSON.
Only the named files of a model folder (model.yaml, roles, and so on) are
read on their own, so fragments can sit next to them or in a subfolder. Do
not call a fragment model.yaml, not even in a subfolder: the Generator
validates every model.yaml under Models/ as a model of its own.
Rules
- A missing file fails the build. When an
Includepoints to a file that does not exist, the Generator stops with an error that names the include, the file that contains it and the path it looked for:Include 'model_facts.yaml' in …/model.yaml points to a file that does not exist: …/model_facts.yaml. An incomplete model never reaches the API or Fabric. - Fabric reads the resolved build output, never the source. On every build
the Generator writes a fully resolved copy of the metadata YAML to
FabricBuilds/Metadata(setmetadataOutputFolderin the configuration file to use another folder; the pipelineAn automated "conveyor belt" that moves data from one place to another or performs a task automatically. reads the same setting). The pipeline uploads that copy to the same folders in the Meta lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database.Filesas before, under the same file names, soconfig.yamland the notebooks need no change. After you edit a source file, run the Generator before uploading; a manual upload only works fromFabricBuilds/Metadata.FabricBuilds/Metadatais build output and is not committed to git. - The EasyFabric Runtime does not resolve
Include. A YAML file in Fabric that still contains anIncludekey is rejected withUnresolved Include in '<file>': the wheel does not resolve Include. Upload the Generator build output (FabricBuilds/Metadata) instead of the source YAML.Seeing it means raw source was uploaded instead of the build output. - Comments do not survive. The resolved copy is rewritten from the parsed YAML, so comments, key order and formatting from the source are gone. The source files stay the place to edit; never edit the resolved copy.
- A platform without
Includeworks as before. Its resolved copy has the same content as the source.
Upgrading an existing platform
The Generator client and the pipeline templates in Yml change together:
the new client writes FabricBuilds/Metadata, and the new pipeline uploads
from it. Roll them out in the same change.
- A new pipeline with an old client finds no resolved files. The build stops right after the Generator step with an error that points at the Generator version, and a deploy of an older build artifact stops before the upload with the same message.
- An old pipeline with a new client still uploads the raw source. That works
as long as no file uses
Include; as soon as one does, the EasyFabric Runtime rejects it in Fabric.