FAIR Data Pipeline C++ API

NOTE: Currently this API implementation consists of components, assembly of these will depend on the final structure of the API itself.

Contents

• Outline
• API Backend Components
  • API
  • LocalFileSystem
    • Reading an Estimate
    • Reading an Array
    • Reading a Data Table Column
    • Reading a Distribution
    • Creating an Estimate
    • Creating a Distribution
    • Creating an Array
• Installation
• Unit Tests

Outline

The main class the user will interact with is DataPipeline which has only the required methods such as read_estimate etc. This class has a member which is a pointer to an underlying DataPipelineImpl_ class which performs the various procedures required to handle the data. A logger has been used to give as much feedback to the user as possible, the verbosity being handled by a log level argument.

Development has been made mainly to this internal class, the idea being that the top layer API will be constructed once the schema/workflow for FAIR Data Pipeline is finalised.

API Backend Components

The included unit tests show the API in action testing various methods for DataPipelineImpl_, LocalFileSystem and the extra utility methods required.

API

The core of the framework is the API class which sends and receives information to the RestAPI via CURL. It has no specific knowledge about the pipeline itself.

```c++

include "fdp/registry/api.hxx"

include "json/json.h"

include

include

using namespace FDP;

void run_api() { API* api = new API("https://data.scrc.uk/api/");

Json::Value data;
data["website"] = "www.nosuchsite.uk";
data["name"] = "test_data";
data["abbreviation"] = "nss";

const std::string api_token = "APITOKEN";

api->request("object/1234");
api->post("source", data, api_token);

} ```

LocalFileSystem

This largely handles the translation of statements in the configuration file into the retrieval of or submission of objects in the registry. It includes most of the HDF5 reading methods.

Reading an Estimate

```c++

include "fdp/registry/file_system.hxx"

include

using namespace FDP;

void readparamdemo() { const std::filesystem::path tomlfile = "/path/to/estimate/0.1.0.toml"; const double value = readpointestimatefromtoml(tomlfile); } ```

Reading an Array

```c++

include "fdp/registry/file_system.hxx"

include "fdp/objects/data.hxx"

include

using namespace FDP;

void readarraydemo() { const std::filesystem::path arrayfile = "path/to/array/0.20210427.0.h5"; const std::filesystem::path component = "testarray"; const ArrayObject* readarray(arrayfile, component); } ```

Reading a Data Table Column

```c++

include "fdp/registry/file_system.hxx"

include "fdp/objects/data.hxx"

include

include

using namespace FDP;

void readtablecoldemo() { const std::filesystem::path arrayfile = "path/to/array/0.20210427.0.h5"; const std::filesystem::path component = "testtable"; const std::string column = "col1"; const DataTableColumn* readtablecolumn(array_file, component, column); } ```

Reading a Distribution

```c++

include "fdp/registry/file_system.hxx"

include "fdp/objects/distributions.hxx"

include

include

using namespace FDP;

void readdistributiondemo() { const std::filesystem::path tomlfile = "/path/to/distribution/0.1.0.toml"; const Distribution* dis = readdistributionfromtoml(toml_file);

std::random_device rd;
std::mt19937* gen = new std::mt19937(rd());
const double random_val = dis->generate(gen);

} ```

Creating an Estimate

```c++

include "fdp/registry/file_system.hxx"

include "fdp/utilities/semver.hxx"

include

include

using namespace FDP;

void createparamdemo() { const std::filesystem::path configfile = "path/to/config.yaml"; LocalFileSystem* filesystem = new LocalFileSystem(config_file);

const version v1(0, 2, 0);
const double value = 7;
const std::string label = "my_params/param_a";

create_estimate(value, label, v1, file_system);

} ```

Creating a Distribution

```c++

include "fdp/registry/file_system.hxx"

include "fdp/utilities/semver.hxx"

include

include

using namespace FDP;

void createdisdemo() { const std::filesystem::path configfile = "path/to/config.yaml"; LocalFileSystem* filesystem = new LocalFileSystem(config_file);

const version v1(0, 2, 0);
const std::string label = "my_distributions/dis_a";

const double mu = 5.2;
const double sigma = 1.2;

Normal* dis = new Normal(mu, sigma);

create_distribution(dis, label, v1, file_system);

} ```

Creating an Array

```c++

include "fdp/registry/file_system.hxx"

include "fdp/objects/data.hxx"

include

include

using namespace FDP;

void createarraydemo() { const std::filesystem::path configfile = "path/to/config.yaml"; LocalFileSystem* filesystem = new LocalFileSystem(config_file);

const std::filesystem::path data_product = "test_data/test_arrays";
const std::filesystem::path component = "demo_1";

const ArrayObject<float>* arr = new ArrayObject<float>(
    {"dimension_1", "dimension_2"},             // Dimension titles
    {{"a1", "a2", "a3"}, {"b1", "b2", "b3"}},   // Dimension names
    {3, 3},                                     // Dimension sizes
    {1., 2., 3., 4., 5., 6., 7., 8., 9.}
);

create_array(arr, data_product, component, file_system);

} ```

Installation

You can build and test the library using CMake, this implementation requires C++17 or later as it makes use of the std::filesystem library. All dependencies are externally fetched however it is recommended that you install the following in advance:

• HDF5
• CURL

compile the library and tests by running:

$ cmake -Bbuild -DFDPAPI_BUILD_TESTS=ON $ cmake --build build

Unit Tests

The unit tests connect to the FDP remote registry API, and so will not work if this service is no longer available.

Before running the tests you will need to download the test data sets by running:

$ bash test/test_setup.sh you can then launch the GTest binary created during compilation: $ ./build/test/fdpapi-tests