Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/crc-fonda/fonda-airflow-dags

FONDA Airflow Dags

https://github.com/crc-fonda/fonda-airflow-dags

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (9.4%) to scientific vocabulary
Last synced: 6 months ago · JSON representation

Repository

FONDA Airflow Dags

Basic Info
  • Host: GitHub
  • Owner: CRC-FONDA
  • Language: Python
  • Default Branch: main
  • Size: 166 KB
Statistics
  • Stars: 0
  • Watchers: 1
  • Forks: 0
  • Open Issues: 2
  • Releases: 0
Created about 5 years ago · Last pushed 8 months ago
Metadata Files
Readme

README.md

FONDA Airflow Dags

This repository contains Apache Airflow DAGs used in the FONDA project. The DAGs are located in ./dags, currently organized in subfolders based on FONDA subprojects and regulary synchronized with the Airflow instance running in the reference stack.

To add a DAG, create a branch and use the development environment as described below. When you're done writing your DAG, you can create a Pull Request and merge to the main branch.

TODO: - [ ] integrate repository into Jenkins CI and apply testing/static-code-analysis before merging - [ ] synchronize other DAG repositories with reference stack

Development environment

We provide scripts to create a local development environment based on Kubernetes in Docker which can be used to develop Airflow DAGs in a similar environment to the FONDA reference stack. Kubernetes will be deployed in Docker containers on your local machine and you can use the KubernetesPodOperator to create DAGs.

Requirements
Installation
  • ./kind/start-dev-env.sh
  • ./kind/create-admin-user.sh

Open http://localhost:8888 in your browser. The dags folder is available in Airflow and you can start developing your DAG.

Delete development environment
  • ./kind/stop-dev-env.sh

Owner

  • Name: CRC-FONDA
  • Login: CRC-FONDA
  • Kind: organization

GitHub Events

Total
  • Push event: 19
Last Year
  • Push event: 19

Dependencies

requirements.txt pypi
  • attrs ==21.4.0
  • iniconfig ==1.1.1
  • packaging ==21.3
  • pluggy ==1.0.0
  • py ==1.11.0
  • pyparsing ==3.0.7
  • pytest ==7.0.1
  • tomli ==2.0.1