Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/broadinstitute/reaper

Cloud workflow post-processor

https://github.com/broadinstitute/reaper

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (8.0%) to scientific vocabulary
Last synced: 6 months ago · JSON representation

Repository

Cloud workflow post-processor

Basic Info
  • Host: GitHub
  • Owner: broadinstitute
  • License: bsd-3-clause
  • Language: Scala
  • Default Branch: develop
  • Size: 24.4 KB
Statistics
  • Stars: 0
  • Watchers: 26
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Created over 7 years ago · Last pushed over 7 years ago
Metadata Files
Readme License

README.md

Reaper

Post-processor for Cromwell workflow outputs.

Overview

Reaper will be a long-running web service responsible for "post-processing" the outputs of production workflows in the cloud. It will monitor Cromwell for successful workflows, and move the outputs of each workflow it finds into long-term storage while recording the files in Clio. It will also parse metrics outputs and persist them to the cloud DB.

Motivation

The idea for Reaper was born out of pain points we experienced when running post-processing-style tasks in WDL for Arrays analysis and delivery in the cloud:

  • Implicit dependencies on external systems make WDLs more complicated, more difficult to test reproducibly, and more dangerous to call-cache
  • Injecting access credentials for Broad-internal systems into our cloud pipelines makes those pipelines harder to test, deploy, and share
  • Transient network errors between systems cause spurious failures in Cromwell and burn operations time on diagnosis / resubmission

By taking ownership of tasks which require interacting with external systems, Reaper aims to enable writing cloud pipelines which are simple to write, test, and operate.

Getting Started

Install Mill

http://www.lihaoyi.com/mill/#installation

Build

mill server.compile

Generate Intellij .idea files

mill mill.scalalib.GenIdea/idea

Quickly run a server

mill server

Run the server while watching for source changes

mill -w server

Run the tests

mill server.test

Run the integration tests

mill integration.test

Format the code with scalafmt

mill server.reformat

Format all the code with scalafmt

mill mill.scalalib.scalafmt.ScalafmtModule/reformatAll __.sources

More information

Internal documentation is hosted in Confluence here.

Owner

  • Name: Broad Institute
  • Login: broadinstitute
  • Kind: organization
  • Location: Cambridge, MA

Broad Institute of MIT and Harvard

GitHub Events

Total
Last Year

Issues and Pull Requests

Last synced: 9 months ago

All Time
  • Total issues: 0
  • Total pull requests: 12
  • Average time to close issues: N/A
  • Average time to close pull requests: 7 days
  • Total issue authors: 0
  • Total pull request authors: 2
  • Average comments per issue: 0
  • Average comments per pull request: 0.5
  • Merged pull requests: 12
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • jacarey (5)
  • danxmoran (3)
Top Labels
Issue Labels
Pull Request Labels