Skip to content

TOML Configuration

The fundamentals of PIPES are about configurations.

Configuration files make up the foundation of the metadata management of your PIPES projects. Submitting configuration TOML files into the CLI is the bases of most PIPES user interactions. Tutorials on creating and submitting configurations to PIPES are located here.

PIPES manages 5 different types of configuration files:

  1. Project config: Initializes the project, defines project, project run, and model requirements/scenarios/assumptions/scheduling.
  2. Model Run Config: Model run config defines the model run activity configurations, including expected datasets, handoffs, and tasks that PIPES should track.
  3. Dataset Config: Used to checkin a dataset.
  4. Task Creation Config: Used to submit completed tasks.
  5. Task Planning Config: Used as an ad-hoc way to add a task to a Model Run that is in-progress. The specific schema used depends on the type of task you want to create.

Three pre-defined task types:

  • Transformation Task Creation
  • QAQC Task Creation
  • Visualization Task Creation

Each PIPES config schemas is designed with sensible defaults, but are also flexible to allow users to capture the complexities of their projects. We use Pydantic to validate PIPES input configurations against a defined schemas.

Warning

Currently PIPES does not save submitted config files. It is planed to add this feature in future iterations. We request our early users save their config files so we can reference them if you run in to bugs.

Get config template

Use the get template commands to get the template config for each schema type. Commands for generating templates from PIPES are listed in CLI section.