DCNiOS

DCNiOS, Data Connector through Apache NiFi for OSCAR, is an open-source command-line tool that facilitates the creation of event-driven processes connecting a Storage System to a scalable OSCAR cluster by employing predefined dataflows that are processed by Apache NiFi like: - dCache - Amazon S3 - Kafka as input of an event streaming platform. - Rucio

Apache NiFi is a reliable system to process and distribute data through powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

OSCAR is an open-source platform for serverless event-driven data processing of containerized applications across the computing continuum.

The DCNiOS command-line interface allows you to create a NiFi dataflow though a YAML file (see figure).

After creating a dataflow using DCNiOS, when an event is captured (e.g., Server-sent event by dCache), Apache NiFi stores it in an internal queue and then sends it for processing to an OSCAR cluster at a customized rate. In the OSCAR cluster, a user-defined service creates a job to process the stored data.

Therefore, DCNiOS has been made to interact with NiFi and deploy a complete dataflow. It uses HTTP calls to communicate with a Nifi cluster.

Apache NiFi can be automatically deployed using the Infrastructure Manager (IM) on a dynamically provisioned Kubernetes Cluster. Note that a NiFi registry is not required.

Achievements

This software has received a silver badge according to the Software Quality Baseline criteria defined by the EOSC-Synergy project.

Getting Started

Prerequisites

• OSCAR cluster with services deployed.
• NiFi Cluster deployed.
• A Python distribution such as Anaconda.

Installation

1. Download the repository in a local folder.

   bash git clone git@github.com:interTwin-eu/dcnios.git cd dcnios

2. Set up a virtual environment (optional but recommended):

   You can choose between Conda or Venv for managing Python dependencies.

   Using Conda

   Create a new Conda environment and activate it:

   bash conda create --name dcnios python=3.7.6 conda activate dcnios

   Using Venv

   Create a virtual environment with Venv and activate it:

   bash python -m venv dcniosenv source dcniosenv/bin/activate

3. Install Dependencies

   Install all the required dependencies listed in requirements.txt:

   bash pip install -r requirements.txt

   Alternatively, you can install the minimum required dependencies for DCNiOS:

   bash pip install pyyaml==6.0 requests==2.28.2 oscar_python==1.0.3

4. Verify Installation

   Once you've installed the dependencies. You can verify this by running a simple Python script. Try the following Python command to check if the packages are installed correctly:

   bash python -c "import yaml, requests, oscar_python; print('All dependencies are correctly installed.')"

Authors

• Germán Moltó mailto:gmolto@dsic.upv.es
• Estíbaliz Parcero mailto:esparig@i3m.upv.es
• Sergio Langarita mailto:slangarita@i3m.upv.es

Instituto de Instrumentación para Imagen Molecular (I3M), Centro Mixto CSIC — Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, España.

Versions and Maintenance

There is only one version in maintenance: - The main branch in the source code repository maintains a working state version of the software component. - Documentation is updated with the new software versions involving any substantial or minimal change in the application's behavior. Any issue can be reported in the Issues section of the GitHub project. - Documentation is updated whenever reported as inaccurate or unclear.

Licensing

DCNiOS is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Acknowledgements

This work was supported by the project “An interdisciplinary Digital Twin Engine for science’’ (interTwin), which has received funding from the European Union’s Horizon Europe Programme under Grant 101058386.

More information

You can find more information in the OSCAR's blog.