Skip to content

Create tasks

Get a Task Config Template

You can use the command below to generate a Task template. Pick one of Transformation, QAQC, or Viusalization task types

$ pipes task template -t task-creation  --task-type {Transformation/QAQC/Visualization}-o my-transformation-task-creation.toml

To log progress in PIPES, users should use the subtask_ids list to indicate which tasks planned in the Model Run were performed.

Note

Task creation can include more than one planned task id. For example, it might make sense to perform two transformations (like changing the model and weather years) at the same time. When this is the case, users should include both of these task ids in subtask_ids. Tasks that are different types (e.g., Transformation and QAQC) cannot be created together.

Any checked-in datasets that are inputs to the task should be indicated in the dataset_ids list.

Artifacts of tasks that are not datasets, such as images or video files, can be listed under outputs where a location should be specified and any other pieces of metadata a user wishes to log can also be added.

  • For more description of tasks, please see the Task Reference.

  • For more specifics on the metadata keys and their types in the Dataset template, check out the Task Config.

Submit a Task

  1. Edit the information provided in the template.

  2. Submit the config.

The CLI method for submitting the config to PIPES is pipes handoff create-tasks. An example call is:

$ pipes task create -p test1 -r 1 -m dsgrid -x model-run-1 -f project/tasks/test_qaqc.toml --{task-fail/task-pass}

Users should pick either task-pass or task-fail depending on the outcome of their work. Failed tasks are not required to be submitted to PIPES, this is most useful for capturing the outcomes of automated tasks.

See defining your context for more information on the rest of the flags.

Transformations

Transformation tasks requires additional information to be submitted about the relevant output datasets. They follow the same structure defined above, but they also require a Dataset Config (using the -d or --dataset CLI flag) to be provided during the CLI task creation process. As an example:

$ pipes task create -p test1 -r 1 -m dsgrid -x model-run-1 -f project/tasks/test_transform.toml -d project/tasks/transformed_dataset.toml --task-pass

Referencing datasets in other model runs

Transformations might need to reference datasets outside of the Model Run or Project (but still in the PIPES universe). To add links from these datasets to the transformed dataset, use the vertex_ids option. This takes a list of vertex IDs for any dataset and links them as an input to the transformed dataset. This can be useful for weather data or other static GIS data that is used across many projects but wasn’t specifically created during the current Model Run.