Data

Tools

Indexes

Applications

Experiments

Open Source Science

boilerdata

Data processing pipeline for a nucleate pool boiling apparatus

https://github.com/softboiler/boilerdata

Science Score: 54.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (15.9%) to scientific vocabulary
Last synced: 10 months ago · JSON representation ·

Repository

Data processing pipeline for a nucleate pool boiling apparatus

Basic Info
Statistics
  • Stars: 0
  • Watchers: 1
  • Forks: 0
  • Open Issues: 8
  • Releases: 4
Created over 4 years ago · Last pushed 11 months ago
Metadata Files
Readme Changelog Contributing License Code of conduct Citation Security

README.md

boilerdata

DOI All Contributors

Data processing pipeline for a nucleate pool boiling apparatus.

Overview

The data processing approach taken in this repository started over at pdpipewrench. It was initially conceptualized as a way to outfit pdpipe pipelines from configuration files, allowing for Pandas pipeline orchestration with minimal code. I have since adopted a less aggressive tact, where I still separate configuration out into YAML files (constants, file paths, pipeline function arguments, etc.), but pipeline logic is handled in pipeline.py. I have also done away with using pdpipe in this approach, as it doesn't lend itself particularly well to ETL. Besides, my data processing need is not quite the "flavor" of statistical data science type approaches supported by pdpipe.

This new approach maintains the benefits of writing logic in Python, while allowing configuration in files. I am using Pydantic as the interface between my configs and my logic, which allows me to specify allowable values with Enums and other typing constructs. Expressing allowable configurations with Pydantic allows for generation of schema for your config files, raising errors on typos or missing keys, for example. I also specify the "shape" of my input and output data in configs, and validate my dataframes with pandera. Once these components are in place, it is easy to implement new functionality in the pipeline.

Usage

If you would like to adopt this approach to processing your own data, you may clone this repository and begin swapping configs and logic for your own, or use a similar architecture for your data processing. To run a working example with some actual data from this study, perform the following steps:

  1. Clone this repository and open it in your terminal or IDE (e.g. git clone https://github.com/blakeNaccarato/boilerdata.git boilerdata).
  2. Navigate to the clone directory in a terminal window (e.g. cd boilerdata).
  3. Create a Python 3.10 virtual environment (e.g. py -3.10 -m venv .venv on Windows w/ Python 3.10 installed from python.org).
  4. Activate the virtual environment (e.g. .venv/scripts/activate on Windows).
  5. Run pip install --editable . to install boilerdata package in an editable fashion. This step may take awhile.
  6. Delete the top-level data and config directories, then copy the config and data folders inside of tests/data to the root directory.
  7. Copy the .propshop folder in tests/data/.propshop to your user-folder (e.g. C:/Users/<you>/.propshop on Windows).
  8. Run dvc repro metrics to execute the data process up to that stage.

The data process should run the following stages: axes, modelfun, runs, parse_benchmarsk, pipeline, and metrics. Some stages are skipped because we specified to run just the necessary stages up to metrics (the example data doesn't currently include the literature data). You may inspect the pipeline stages of the same name in src/boilerdata/stages, such as pipeline.py to see the logic that runs during that stage. This example happens to use Python scripts, but you could define a stage in dvc.yaml that instead runs Matlab scripts, or any arbitrary action. This approach allows for the data process to be reliably reproduced over time, and for the process to be easily modified and extended in a collabroative effort.

There are other details of this process, such as the hosting of data in the data folder in a Google Cloud Bucket (alternatively it can be hosted on Google Drive), and more. This has to do with the need to store data (especially large datasets) outside of the repository, and access it in an authenticated fashion.

Project information

Contributors

Blake Naccarato
Blake Naccarato
💻

Owner

  • Name: softboiler
  • Login: softboiler
  • Kind: organization
  • Location: United States of America

Softboiled software for boiling and thermal science.

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
  - family-names: "Naccarato"
    given-names: "Blake"
    orcid: "https://orcid.org/0000-0003-4851-4724"
  - family-names: "Kim"
    given-names: "Kwang"
    orcid: "https://orcid.org/0000-0003-2134-4964"
title: "boilerdata"
doi: "10.5281/zenodo.7826604"
date-released: 2023-04-13
url: "https://github.com/blakeNaccarato/boilerdata"

GitHub Events

Total
  • Issue comment event: 1
  • Push event: 106
  • Pull request event: 6
Last Year
  • Issue comment event: 1
  • Push event: 106
  • Pull request event: 6

Issues and Pull Requests

Last synced: 10 months ago

All Time
  • Total issues: 0
  • Total pull requests: 4
  • Average time to close issues: N/A
  • Average time to close pull requests: 13 days
  • Total issue authors: 0
  • Total pull request authors: 2
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 4
Past Year
  • Issues: 0
  • Pull requests: 4
  • Average time to close issues: N/A
  • Average time to close pull requests: 13 days
  • Issue authors: 0
  • Pull request authors: 2
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 4
View more stats
Top Authors
Issue Authors
  • blakeNaccarato (2)
Pull Request Authors
  • renovate[bot] (6)
  • pre-commit-ci[bot] (4)
Top Labels
Issue Labels
internal (2) bug (1)
Pull Request Labels
dependencies (6)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 35 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 4
  • Total maintainers: 1
pypi.org: boilerdata

Data processing pipeline for a nucleate pool boiling apparatus

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 35 Last month
Rankings
Dependent packages count: 7.1%
Average: 18.7%
Dependent repos count: 30.3%
Maintainers (1)
blakeNaccarato
Last synced: 11 months ago

Dependencies

.github/workflows/changerelease.yml actions
  • dropseed/changerelease v1 composite
.github/workflows/main.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
  • blakeNaccarato/copier-python-workflow-setup v0.0.9 composite
  • codecov/codecov-action v3.1.1 composite
  • dorny/paths-filter v2 composite
  • stefanzweifel/git-auto-commit-action v4 composite
.github/workflows/sphinx.yml actions
  • actions/deploy-pages v1 composite
  • actions/upload-pages-artifact v1 composite
  • blakeNaccarato/copier-python-workflow-setup v0.0.9 composite
.tools/requirements/requirements_both.txt pypi
  • IPython *
  • flake8 *
  • flake8-absolute-import *
  • flake8-bandit *
  • flake8-black *
  • flake8-bugbear *
  • flake8-builtins *
  • flake8-comprehensions *
  • flake8-fixme *
  • flake8-noqa *
  • flit *
  • pep8-naming *
  • pyright ==1.1.287
  • pytest *
  • pytest-cov *
  • toml *
.tools/requirements/requirements_dev.txt pypi
  • autoflake * development
  • black * development
  • debugpy * development
  • flake8-codes * development
  • ipympl * development
  • jupyter * development
  • keyring * development
  • pre-commit * development
  • rich * development
  • snakeviz * development
  • sourcery-cli * development
.tools/requirements/requirements_docs.txt pypi
  • myst-parser *
  • sphinx *
  • sphinx-autobuild ==2021.03.14
  • sphinx-book-theme *
  • sphinx-design *
pyproject.toml pypi
  • dill ~=0.3.6
  • dvc [gs]~=2.41.1
  • matplotlib ~=3.6.0
  • originpro ~=1.1.2
  • pandas ~=1.5.0
  • pandera ~=0.13.1
  • papermill ~=2.4.0
  • pyXSteam ~=0.4.9
  • pyarrow ~=10.0.0
  • pydantic ~=1.10.1
  • pyjanitor ~=0.24.0
  • pyyaml ~=6.0
  • ruamel.yaml ~=0.17.21
  • scipy ~=1.10.0
  • seaborn ~=0.12.1
  • sympy ~=1.11.1
  • uncertainties ~=3.1.7
.github/workflows/release.yml actions
  • blakeNaccarato/copier-python-workflow-setup v0.2.3 composite
  • dropseed/changerelease v1.6.0 composite
.github/workflows/codeql.yml actions
  • actions/checkout v4.1.1 composite
  • github/codeql-action/analyze v2.22.5 composite
  • github/codeql-action/autobuild v2.22.5 composite
  • github/codeql-action/init v2.22.5 composite
.github/workflows/minimum.yml actions
  • blakeNaccarato/copier-python-workflow-setup v0.2.3 composite
  • codecov/codecov-action v3.1.4 composite
.tools/requirements/requirements.txt pypi
  • IPython ==8.17.2
  • matplotlib ==3.8.2
  • nbqa ==1.7.0
  • numpy ==1.26.2
  • pandas ==2.1.1
  • pandas-stubs *
  • pandera ==0.17.2
  • propshop ==0.1.1
  • pyXSteam ==0.4.9
  • pyarrow ==14.0.1
  • pydantic ==1.10.13
  • scipy ==1.11.3
  • seaborn ==0.13.0
.tools/requirements/requirements_ci.txt pypi
.tools/requirements/requirements_core.txt pypi
  • copier ==8.1.0
  • dulwich ==0.21.6
  • flit_core ==3.9.0
  • wheel ==0.41.3
.tools/requirements/requirements_nodeps.txt pypi
tests/pyproject.toml pypi