Skip to content

Welcome to PIPES

Description

What is PIPES

PIPES (Pipeline for Integrated Projects in Energy Systems) is a project management, data management, and a workflow management layer for integrated modeling teams.

Inspired by the complex data management challenges of the LA100 Study, PIPES was designed to help scale large integrated modeling projects at the laboratory (and beyond).

Metadata management

In its simplest form, PIPES manages metadata associated with an integrated modeling project. This metadata management includes metadata about scenarios, requirements, and assumptions across different levels of the pipeline (e.g., Project, Model, Dataset, Task, etc.); metadata about the underlying pipeline topology (e.g., how things are connected and what data handoffs are happening); and metadata around managing the project (e.g., who is the responsible party, gantt scheduling information, milestones, pass/fail statuses, modeling progress, etc.).

In coordination with the larger project team, PIs/data managers will plan out the project structure, expected outputs, and the modeling pipeline at the beginning of the project. As the project kicks off, modeling teams will register model runs, datasets, and tasks such as transformations, QAQC, and visualizations. This provides a rich set of metadata about how the project is progressing, the modeling assumptions made at each step and metadata for individual datasets and tasks. Importantly, PIPES tracks requirements and acceptance criteria throughout the pipeline. All registered activities—especially handoffs between models—require validation checkpoints and may trigger event notifications to users.

Key features

  • Robust, multi-level, and multi-dimensional metadata management

  • Storage agnostic

  • Graph pipeline representation

  • Task management (QAQC, transformations, visualization, and knowledge exchanges)

  • Human-in-the-loop QAQC checkpoints (pass/fail)

  • Automated data and metadata validation

  • Required transforms for dataset handoffs

  • Event triggers and notifications

  • Command-line interface (CLI)

  • Web-based User Interface (UI) for pipeline visibility and project coordination / management

  • API for model integration

Key improvements

  • Better manage integrated modeling projects and business logic

  • Help teams find and share data

  • Provide visibility into the pipeline and track progress

  • Help teams validate models and find errors early

  • Improve integration speed (reduced coordination time)

  • Improve ability to serve many communities simultaneously

Architecture diagram

The architecture diagram below shows various components of our PIPES system.

Architecture-Diagram

  • At the core is the Server, which provides PIPES services through REST API endpoints built with FastAPI.

  • Behind the server are two data stores: a metadata store using AWS DocumentDB and a graph store powered by AWS Neptune.

  • On the frontend, users can interact with PIPES via the CLI or Web UI, which communicate with the API endpoints.

These components work together as a system to provide services for data management and collaboration to help with integrated modeling programs.

Target users

Role Interval/External User Type Access Methods Permissions
Developer Internal PIPES Developer, ITS PIPES code Sudo - Develops and manages PIPES infrastructure
Manager Internal, External PI/PM, Data Manager, SCRUM Master, Technical Lead CLI & Web UI+ Read & write at highest level – Initializes project. Manages pipeline expectations, schedule, requirements
Modeler Internal, External Team Member, Modelers (ESI and ARIES), External Modelers/Partners* CLI & Web UI+ Read & write at lower level – Execute models, check in/out data, perform tasks (QAQC, transform, visualization). Cannot modify higher level project nodes.
Client External DOE, Community Partners (Local Gov’t, Utilities, CBOs, etc.) Web UI (controlled view) Limited & read-only – Permission to see certain granted visualizations and results

Our vision

The long-term vision behind PIPES is that it be used in any integrated modeling project at the laboratory (and across DOE’s National Laboratories). PIPES is agnostic and independent of modeling work and implementations because it primarily captures model inputs and model outputs. This allows PIPES to be composable with many different workflows and applications, including hybrid cloud/HPC workflows, HERO, the ARIES Virutal Emulation Environment (VEE), etc.