Tasks

What is a task?

Tasks in PIPES represent units of work performed on datasets that occur often in integrated modeling projects, especially when exchanging data. Tasks are typically planned at Model Run initialization, but can also be added after Model Run initialization if needed. When a task is completed, users can indicate this to PIPES via a Task Creation. Tasks help track the progress and needs of a project.

Currently, tasks are not automated processes within PIPES, though we are exploring integrating with workflow management tools to allow for task automation.

Note

Additional tasks that were not defined in the Model Run initialization process can be added to Model Runs, but these tasks won’t count towards the progress of the Model Run, but can be useful for tracking ad hoc analysis or visualization steps. They can also be incorporated into future Model Runs if they become a critical part of the workflow.

There are three types of tasks:

1. Transformation
2. QAQC
3. Visualization

Note

As defined in the roadmap, we plan to add support for a fourth type of task: Knowledge Exchanges

Transformation

Transformation tasks describe how a dataset is altered to meet metadata requirements. PIPES expects users to indicate which piece of metadata is being changed and what the change is (i.e., the to and from values). For example, if a dataset has temporal_info.resolution = hourly but a down stream model expects temporal_info.resolution = 1 minute then a transformation that achieves this change in resolution must be performed. PIPES can automatically determine transformations that need to occur based on differences in requirements across models, and this is discussed further in Validation. In general, PIPES assumes that transformations will need to be performed on handoff datasets, but it is not required.

QAQC

QAQC tasks capture dataset validation and acceptance criteria which are crucial for ensuring successful integration of two models. For example, QAQC tasks could ensure that data is within certain bounds or adheres to various metrics. The exact nature of the criteria is totally customizable for users. See the Task Planning for more information.

Visualization

Visualization tasks capture which datasets should be visualized and in what manner. When creating a visualization task, it may be useful to indicate the outputs of the task (such as images or videos) in the outputs field in the Task Creation.