Skip to main content

Splitting the model with Include

Overview​

A large model.yaml can be split into several files with Include. The EasyFabric Generator resolves every Include while it builds: it reads the included files, splices their contents in, and sends one complete document to the generation API. The included files are a way to organise your source; nothing downstream ever sees them.

Include works in every file of a model folder (Dataplatform/DP/Models/<model>/): model.yaml, the derived model, roles, format strings and measure templates.

Include forms​

One file per list item​

An - Include: item in a list is replaced by the contents of the file. When the included file holds a list, its items are spliced in at that position, so Include items mix freely with inline entries.

Dataplatform/DP/Models/DM/model.yaml

- Table:
Name: Calendar
TableType: Dimension
- Include: model_dims.yaml
- Include: model_facts.yaml

Dataplatform/DP/Models/DM/model_dims.yaml

- Table:
Name: Customer
TableType: Dimension
- Table:
Name: Product
TableType: Dimension

Several files under one Include​

- Include:
- model_dims.yaml
- model_facts.yaml

The files are spliced in the order they are listed.

Include inside an entry​

An Include key next to other keys merges the included file into that entry. Lists from both sides are combined; for a single value defined on both sides, the included file wins. Use this to share a block, such as a set of columns, between tables.

- Table:
Name: Customer
Include: shared/audit_columns.yaml
Columns:
- SourceColumn: CustomerName

shared/audit_columns.yaml

Columns:
- SourceColumn: CreatedAt
- SourceColumn: ModifiedAt

Nested includes​

An included file may itself contain Include. Nesting has no fixed depth.

Paths​

A path in Include is relative to the folder of the file that contains the Include, not to the model folder or the repository root. In a nested include, the inner path is resolved from the inner file's own folder.

Subfolders are allowed, which keeps the main file short:

Dataplatform/DP/Models/DM/
├── model.yaml # - Include: base/model_dims.yaml
└── base/
├── model_dims.yaml # - Include: model_dims_customer.yaml (resolved inside base/)
├── model_dims_customer.yaml
└── model_facts.yaml

Included files may be YAMLA simple way to write configurations. It's basically a list that computers can read easily. (.yaml, .yml) or JSON.

Only the named files of a model folder (model.yaml, roles, and so on) are read on their own, so fragments can sit next to them or in a subfolder. Do not call a fragment model.yaml, not even in a subfolder: the Generator validates every model.yaml under Models/ as a model of its own.

Rules​

  • A missing file fails the build. When an Include points to a file that does not exist, the Generator stops with an error that names the include, the file that contains it and the path it looked for: Include 'model_facts.yaml' in …/model.yaml points to a file that does not exist: …/model_facts.yaml. An incomplete model never reaches the API or Fabric.
  • Fabric reads the resolved build output, never the source. On every build the Generator writes a fully resolved copy of the metadata YAML to FabricBuilds/Metadata (set metadataOutputFolder in the configuration file to use another folder; the pipelineAn automated "conveyor belt" that moves data from one place to another or performs a task automatically. reads the same setting). The pipeline uploads that copy to the same folders in the Meta lakehouseA place where you store both "raw" data (like files) and "organized" data (like tables). It combines the best of a File Cabinet and a Database. Files as before, under the same file names, so config.yaml and the notebooks need no change. After you edit a source file, run the Generator before uploading; a manual upload only works from FabricBuilds/Metadata. FabricBuilds/Metadata is build output and is not committed to git.
  • The EasyFabric Runtime does not resolve Include. A YAML file in Fabric that still contains an Include key is rejected with Unresolved Include in '<file>': the wheel does not resolve Include. Upload the Generator build output (FabricBuilds/Metadata) instead of the source YAML. Seeing it means raw source was uploaded instead of the build output.
  • Comments do not survive. The resolved copy is rewritten from the parsed YAML, so comments, key order and formatting from the source are gone. The source files stay the place to edit; never edit the resolved copy.
  • A platform without Include works as before. Its resolved copy has the same content as the source.

Upgrading an existing platform​

The Generator client and the pipeline templates in Yml change together: the new client writes FabricBuilds/Metadata, and the new pipeline uploads from it. Roll them out in the same change.

  • A new pipeline with an old client finds no resolved files. The build stops right after the Generator step with an error that points at the Generator version, and a deploy of an older build artifact stops before the upload with the same message.
  • An old pipeline with a new client still uploads the raw source. That works as long as no file uses Include; as soon as one does, the EasyFabric Runtime rejects it in Fabric.