Developer reference

Technical stacks

PIPES API

This is the core of the PIPES server, defining the schema for all types of metadata related to project activities and exposing REST APIs for PIPES operations. The PIPES API is built using the FastAPI web framework, providing a RESTful API with JWT-based authentication via the authentication header.

There are two databases behind for data storage:

• Metadata Storage: Uses MongoDB[https://www.mongodb.com/] for local development and Amazon DocumentDB (with MongoDB compatibility) in production to store metadata. ODM (Object Document Mapper): Built with Beanie, an asynchronous ODM for MongoDB.
• Graph Storage: Utilizes Amazon Neptune, supporting Gremlin queries via the python-gremlin package.

Note

PIPES uses AWS Neptune for the graph database backend solution. Neptune is a fully managed graph database solution hosted on AWS. Neptune supports both gremlin and SPARQL graph query languages, however the PIPES code utilizes the gremlin variant. It was decided to use a graph database backend as the data being modeled (projects, runs, models, datasets, etc.) are highly interconnected and a graph allows for the relationships between different components to be expressed naturally. The graph database also allows for queries to evolve as relationships are added or expanded.

PIPES Web

The web interface is a ReactJS-based application that serves as the primary user interface. Key libraries are used here:

• amazon-cognito-identity-js – Handles authentication with Amazon Cognito.
• reactflow – Enables interactive graph flow creation based on modeling integration.

The web UI interacts with the PIPES API for PIPES project operations.

PIPES CLI

The CLI is a command-line interface built with Python Click, a widely used framework for CLI applications.

• It provides an efficient way to interact with PIPES API server via restful API calls.
• Enables metadata configuration and project management directly from the terminal console.

At this stage, the PIPES CLI is the primary tool to interact with PIPES API for data ingestion.

GitHub repositories

This project has open-sourced three GitHub repositories, all distributed under the BSD 3-Clause license.

• PIPES API: https://github.com/nrel-pipes/pipes-api

• PIPES Web: https://github.com/nrel-pipes/pipes-web

• PIPES CLI: https://github.com/nrel-pipes/nrel-pipes

We welcome feedback and contributions! Feel free to suggest improvements, report issues, or submit pull requests. 🚀