lstosa

On-site data processing pipeline for CTAO LST-1

https://github.com/cta-observatory/lstosa

Science Score: 49.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
✓
DOI references
Found 3 DOI reference(s) in README
✓
Academic publication links
Links to: zenodo.org
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (15.7%) to scientific vocabulary

Last synced: 6 months ago · JSON representation

Repository

On-site data processing pipeline for CTAO LST-1

Basic Info

Host: GitHub
Owner: cta-observatory
License: bsd-3-clause
Language: Python
Default Branch: main
Homepage: https://lstosa.readthedocs.io/
Size: 3 MB

Statistics

Stars: 5
Watchers: 9
Forks: 5
Open Issues: 24
Releases: 31

Created over 4 years ago · Last pushed 7 months ago

Metadata Files

Readme License Codemeta

lstosa

Onsite processing pipeline for the Large-Sized Telescope prototype (LST-1) of CTAO (Cherenkov Telescope Array Observatory) based on cta-lstchain running on the LST-1 IT onsite data center at Observatorio Roque de los Muchachos (La Palma, Spain). It automatically carries out the next-day analysis of observed data using cron jobs, parallelizing the processing using the job scheduler SLURM. It provides data quality monitoring and tracking of analysis products' provenance. Moreover, it also massively reprocesses the entire LST-1 dataset with each cta-lstchain major release.

Code: https://github.com/cta-observatory/lstosa
Docs: https://lstosa.readthedocs.io/
License: BSD-3-Clause

Install

We recommend using an isolated conda environment. - Install mamba/miniconda first. - Clone the repository, create and activate the conda environment using the environment.yml file:

```bash
git clone https://github.com/cta-observatory/lstosa.git
cd lstosa
conda env create -n osa -f environment.yml
conda activate osa
```

Then install lstosa as a user with: pip install lstosa, or as a developer with: pip install -e .. To install test, docs dependencies use pip install -e .[test], pip install -e .[doc] or simply pip install -e .[all]

In case you want to install the lstchain development version instead of a fixed tag, you can run inside the osa environment:

bash pip install git+https://github.com/cta-observatory/cta-lstchain

To update the environment (provided dependencies get updated), use:

bash conda env update -n osa -f environment.yml

Note to developers: to enforce a unique code convention, please install pre-commit (pre-commit install) after cloning the repository and creating the conda environment. This will black the committed files automatically.

Workflow management

lstosa workflow is handled daily by the sequencer script, which identifies which observations are to be processed, generates the analysis workflow, and submits the jobs. A first calibration job produces the daily calibration coefficients. Subsequently, data reconstruction jobs are scheduled on a subrun-wise basis (1 job corresponds to around 10 seconds of observed data, and its processing up to DL2 takes about 30-40 mins).

```mermaid

flowchart LR

daq --> osa_seq
osa_seq --> slurm --> osa_closer
daq[DAQ]


subgraph osa_seq [sequencer]
    direction TB
A(Daily observation summary)
B(Generate workflow)
C(Submit jobs)
A --> B --> C
end

subgraph slurm [SLURM parallel processing]
    direction TB
H(Calibration sequence)
I(Reconstruction sequences)
H --> I
end

subgraph osa_closer [autocloser]
    direction TB
D(Check job completion)
E(Move files to final directories)
F(Merge files)
G(Parse provenance logs)
D --> E --> F --> G
end

```

Usage

To use lstosa, you will first need to symlink some auxiliary files in a similar directory tree structure to the standard data production and set the paths correctly in your lstosa configuration file. Then to process all the runs from a given date, you can run the following command (use first the --simulate option to dry-run without actually submitting jobs):

bash sequencer --config your_osa_config.cfg --date YYYY-MM-DD LST1

Once all jobs finish, the autocloser script checks job completion, merges files, moves them to their final directories, and parses provenance logs.

bash autocloser --config your_osa_config.cfg --date YYYY-MM-DD LST1

Dataflow

```mermaid graph LR

subgraph DAQ
D1[R0]
D2[DRS4 calib run]
D3[Pedestal calib run]
D4[Pointing log]
end

D2 --> C1
D3 --> C2

subgraph Calibration
C1[DRS4 baseline correction]
C2[Calibration charge coeffitiens]
C1 --> C2
end

subgraph lstMCpipe
M1[gamma DL2 MC]
M2[RF models]
end

subgraph Sky-data reconstruction
S1[DL1a]
S2[DL1b]
S3[muons]
S4[DL1 check]
S5[DL2]
S6[DL3]
S7[IRF]
D1 & D4 & C1 & C2 --> S1
S1 --> S3
S1 --> S2
S2 & S3 --> S4
S2 ---> S5
S5 --> S6
S7 --> S6
M2 --> S5
M1 --> S7
end

subgraph High-level Gammapy
DL4
DL5
S6 --> DL4
DL4 --> DL5 --> ...
end

```

Warning: standard production of DL3 data and higher-level results is still under development.

Owner

Name: Cherenkov Telescope Array Consortium
Login: cta-observatory
Kind: organization

Website: www.cta-observatory.org
Repositories: 54
Profile: https://github.com/cta-observatory

open-source software for the CTA Consortium.

CodeMeta (codemeta.json)

{
  "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
  "@type": "SoftwareSourceCode",
  "license": "https://spdx.org/licenses/BSD-3-Clause",
  "codeRepository": "https://github.com/cta-observatory/lstosa",
  "contIntegration": "https://github.com/cta-observatory/lstosa/actions",
  "dateCreated": "2020-01-29",
  "datePublished": "2022-05-20",
  "downloadUrl": "https://github.com/cta-observatory/lstosa/releases/tag/v0.9.4",
  "issueTracker": "https://github.com/cta-observatory/lstosa/issues",
  "name": "lstosa",
  "version": "0.9.4",
  "identifier": "10.5281/zenodo.6567234",
  "description": "Workflow management framework for the onsite analysis of LST-1 data.",
  "applicationCategory": "CTA, LST",
  "developmentStatus": "active",
  "isPartOf": "https://www.cta-observatory.org/",
  "keywords": [
    "data analysis pipeline",
    "analysis workflow management",
    "Large Size Telescope",
    "LST",
    "Cherenkov Telescope Array",
    "CTA"
  ],
  "programmingLanguage": [
    "Python 3"
  ],
  "author": [
    {
      "@id": "https://orcid.org/0000-0001-9400-0922",
      "@type": "Person",
      "affiliation": {
        "@type": "Organization",
        "name": "IPARCOS Institute and EMFTEL Department, Universidad Complutense de Madrid, E-28040 Madrid, Spain"
      },
      "familyName": "Morcuende",
      "givenName": "Daniel"
    },
    {
      "@id": "https://orcid.org/0000-0002-3171-5039",
      "@type": "Person",
      "affiliation": {
        "@type": "Organization",
        "name": "Center for Astrophysics | Harvard & Smithsonian, CfA, 60 Garden St., 02138 Cambridge MA, US"
      },
      "familyName": "Saha",
      "givenName": "Lab"
    },
    {
      "@id": "https://orcid.org/0000-0003-3274-4445",
      "@type": "Person",
      "affiliation": {
        "@type": "Organization",
        "name": "Instituto de Astrofísica de Andalucía - CSIC, Gta. de la Astronomía, 18008 Granada, Spain"
      },
      "familyName": "Ruiz",
      "givenName": "José Enrique"
    },
    {
      "@id": "https://orcid.org/0000-0001-7282-2394",
      "@type": "Person",
      "affiliation": {
        "@type": "Organization",
        "name": "IPARCOS Institute and EMFTEL Department, Universidad Complutense de Madrid, E-28040 Madrid, Spain"
      },
      "familyName": "Contreras",
      "givenName": "José Luis"
    },
    {
      "@id": "https://orcid.org/0000-0002-1757-5826",
      "@type": "Person",
      "affiliation": {
        "@type": "Organization",
        "name": "IPARCOS Institute and EMFTEL Department, Universidad Complutense de Madrid, E-28040 Madrid, Spain"
      },
      "familyName": "Baquero",
      "givenName": "Andrés"
    },
    {
      "@id": "https://orcid.org/0000-0003-3848-922X",
      "@type": "Person",
      "affiliation": {
        "@type": "Organization",
        "name": "IPARCOS Institute and EMFTEL Department, Universidad Complutense de Madrid, E-28040 Madrid, Spain"
      },
      "familyName": "Láinez",
      "givenName": "María"
    },
    {
      "@id": "https://orcid.org/0000-0001-7993-8189",
      "@type": "Person",
      "affiliation": {
        "@type": "Organization",
        "name": "TU Dortmund University, Otto-Hahn-Str. 4a 44227 Dortmund, Germany"
      },
      "familyName": "Nöthe",
      "givenName": "Maximilian"
    }
  ],
  "maintainer": {
    "@type": "Person",
    "@id": "https://orcid.org/0000-0001-9400-0922",
    "givenName": "Morcuende",
    "familyName": "Daniel",
    "email": "dmorcuen@ucm.es",
    "affiliation": {
      "@type": "Organization",
      "name": "IPARCOS Institute and EMFTEL Department, Universidad Complutense de Madrid, E-28040 Madrid, Spain"
    }
  }
}

GitHub Events

Total

Create event: 36
Issues event: 21
Release event: 9
Watch event: 1
Delete event: 16
Issue comment event: 37
Push event: 113
Gollum event: 1
Pull request event: 59
Pull request review event: 32
Pull request review comment event: 15

Last Year

Create event: 36
Issues event: 21
Release event: 9
Watch event: 1
Delete event: 16
Issue comment event: 37
Push event: 113
Gollum event: 1
Pull request event: 59
Pull request review event: 32
Pull request review comment event: 15

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 61
Total pull requests: 170
Average time to close issues: 4 months
Average time to close pull requests: 17 days
Total issue authors: 4
Total pull request authors: 6
Average comments per issue: 0.77
Average comments per pull request: 1.11
Merged pull requests: 160
Bot issues: 0
Bot pull requests: 5

Past Year

Issues: 11
Pull requests: 45
Average time to close issues: 28 days
Average time to close pull requests: 10 days
Issue authors: 3
Pull request authors: 5
Average comments per issue: 0.55
Average comments per pull request: 0.69
Merged pull requests: 39
Bot issues: 0
Bot pull requests: 5

View more stats

Top Authors

Issue Authors

morcuended (47)
marialainez (9)
contrera (2)
moralejo (1)

Pull Request Authors

marialainez (118)
morcuended (64)
rcervinoucm (7)
dependabot[bot] (6)
contrera (2)
noeliacastrejon (2)

Top Labels

Issue Labels

enhancement (7) bug (6) help wanted (5) question (2) postDL1 (2) lstchain v0.10 (1) invalid (1)

Pull Request Labels

ready to review (6) dependencies (6) github_actions (1)

Packages

Total packages: 1
Total downloads:
- pypi 302 last-month

Total dependent packages: 0
Total dependent repositories: 1
Total versions: 28
Total maintainers: 1

pypi.org: lstosa

Onsite analysis pipeline for the CTA LST-1

Documentation: https://lstosa.readthedocs.io/
License: BSD-3-Clause
Latest release: 0.11.3
published 7 months ago

Versions: 28
Dependent Packages: 0
Dependent Repositories: 1
Downloads: 302 Last month

Rankings

Dependent packages count: 7.4%

Forks count: 15.4%

Average: 18.6%

Dependent repos count: 22.2%

Downloads: 22.7%

Stargazers count: 25.1%

Maintainers (1)

morcuende

Last synced: 7 months ago

Dependencies

.github/workflows/ci.yml actions

actions/checkout v2 composite
actions/checkout v3 composite
actions/setup-python v2 composite
codecov/codecov-action v3 composite
conda-incubator/setup-miniconda v2.1.1 composite

.github/workflows/deploy.yml actions

actions/checkout v3 composite
actions/setup-python v3 composite
pypa/gh-action-pypi-publish master composite

pyproject.toml pypi

environment.yml conda

astropy ~=5.0
black
click
coverage
ctapipe ~=0.19.2
ctapipe_io_lst 0.22.*
ctaplot ~=0.6.4
freezegun
gammapy 1.1.*
h5py
joblib ~=1.2.0
lstchain ~=0.10.0
matplotlib 3.7.*
numpy
numpydoc
pip
pre-commit
protobuf 3.20.*
prov
psutil
pydata-sphinx-theme
pydot
pydotplus
pyirf 0.8.*
pymongo
pyparsing
pytest
pytest-cov
python 3.11.*
pyyaml
ruff
scikit-learn 1.2.*
sphinx
sphinx-argparse
sphinx-automodapi
tenacity

src/osa/provenance/config/environment.yaml pypi

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science

lstosa

Science Score: 49.0%

Repository

Basic Info

Statistics

Metadata Files

README.md

lstosa

Install

Workflow management

Usage

Dataflow

Owner

CodeMeta (codemeta.json)

GitHub Events

Total

Last Year

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: lstosa

Rankings

Maintainers (1)

Dependencies