Skip to content

Datasets

Datasets represent output data products of a model run. Examples include:

  • Handoff datasets
  • Datasets that will become handoff datasets
  • Datasets needed to complete any PIPES tasks (e.g. visualization)

Datasets and dataset task flows can span across model runs and project runs.

Note

Dataset submission is currently only availble via the CLI. Future development plans to support UI submissions.

For PIPES MVP, supported storage system types are one of:

  • ESIFRepoAPI
  • AmazonS3
  • HPCStorage
  • DataFoundry

Additional storage options will be added with future releases. If you have a specific storage type that you need, please reach out to the PIPES development team.

See the schema requirements for more information on the metadata fields in the Dataset Config file.