https://github.com/broadinstitute/dagster-utils
Utilities and reusable patterns for using Dagster for Broad pipeline orchestration
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
2 of 6 committers (33.3%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (13.7%) to scientific vocabulary
Repository
Utilities and reusable patterns for using Dagster for Broad pipeline orchestration
Basic Info
- Host: GitHub
- Owner: broadinstitute
- License: bsd-3-clause
- Language: Python
- Default Branch: main
- Size: 94.7 KB
Statistics
- Stars: 4
- Watchers: 6
- Forks: 0
- Open Issues: 1
- Releases: 30
Metadata Files
README.md
Dagster Utils
A collection of common utilities used by the Monster team to develop Dagster pipelines. Each subsection has its own readme explaining its contents.
How to use this library
This library is hosted on PyPI and will work with most Python package managers out of the box. You can reference this package as broad_dagster_utils, e.g. for Poetry:
```
pyproject.toml
[tool.poetry.dependencies] broaddagsterutils = "^0.1.0" ```
Note that, despite this name, you'll need to import the package as dagster_utils, e.g.:
python
from dagster_utils.typing import DagsterConfigDict
Local Testing
For development against a local checkout in another project (i.e., a project with a dependency on dagster_utils), make the following adjustment to the project's pyproject.toml:
broad-dagster-utils = {path = "<relative path to your dagster_utils checkout>", develop = true}
Requirements
Please DO NOT use requirements.txt to install dependencies. These are generated automatically using Github Actions and used exclusively for security scanning.
Instead, use Poetry. poetry install Poetry will automatically install dependencies for you, and it will also generate a lock file that will ensure that all developers are using the same versions of dependencies.
Versioning
This library is versioned semantically.
- Major versions (e.g. 1.4.6 -> 2.0.0) represent significant shifts in functionality and may alter the call signatures of core features of the library. You may need to follow a migration plan to upgrade to a new major version.
- Minor versions (e.g. 1.4.6 -> 1.5.0) represent the removal or alteration of specific library features. They will not change core functionality, but the changelog should be reviewed before upgrading to avoid unexpected feature removals.
- Patch versions (e.g. 1.4.6 -> 1.4.7) represent internal improvements or new features that do not alter existing functionality or call signatures. They may introduce deprecations, but they will never remove deprecated functions. You can always safely upgrade to a new patch version.
- Prerelease versions (e.g. 1.4.6 -> 1.4.7-alpha.1) represent changes that have not yet been made part of an official release. See "Releasing a new version" below for more info.
Describing changes
When describing changes made in a commit message, we want to be more thorough than usual, since bugs in dependencies are harder to diagnose. Break down the changes into these categories (omitting any categories that don't apply):
- New features are changes that add functionality, such as new optional arguments for existing functions or entire new classes/functions. These changes should never require that existing code be changed.
- Bugfixes are fairly self-explanatory - bugs that were identified and addressed. If fixing a bug required removing or altering existing features, make sure to list those under "breaking changes" as well.
- Deprecations are features that are still usable, but have been marked as deprecated (and thus trigger a warning when used). They are planned to be removed in a future version. Always try to deprecate functionality before it's removed.
- Breaking changes are changes that may break existing code using this library, such as renaming, removing, or reordering arguments to a function, deleting functionality (including deprecated functionality), or otherwise altering the library in ways that users will need to account for. Users should be able to use this section as a complete guide to upgrading their applications to be compatible with the new version.
Releasing a new version
NOTE tag, release & publish to PyPi are currently manual, due to the deprecation of Vault for secrets. \ Documentation for these tasks are in the Dev Playbook
To release a new version, determine what type of version increase your changes constitute (see the above guide) and update the version listed in pyproject.toml accordingly. Poetry has several version bump commands to help with this. You can update the version in a dedicated PR or as part of another change. When a PR that updates the version number lands on main, an action will run to create a new tag for that version number, followed by cutting a Git release and publishing the new version to PyPI.
When making changes to this repository you must first set up Git Secrets - see Setup Git Secrets
Owner
- Name: Broad Institute
- Login: broadinstitute
- Kind: organization
- Location: Cambridge, MA
- Website: http://www.broadinstitute.org/
- Twitter: broadinstitute
- Repositories: 1,083
- Profile: https://github.com/broadinstitute
Broad Institute of MIT and Harvard
GitHub Events
Total
- Release event: 6
- Delete event: 13
- Issue comment event: 3
- Push event: 37
- Pull request review event: 4
- Pull request event: 17
- Create event: 15
Last Year
- Release event: 6
- Delete event: 13
- Issue comment event: 3
- Push event: 37
- Pull request review event: 4
- Pull request event: 17
- Create event: 15
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| aherbst-broad | 7****d | 22 |
| Steve Pletcher | s****e@b****g | 14 |
| Barbara Hill | b****l@b****g | 10 |
| John Kitonyo | j****i@o****m | 3 |
| Steve Pletcher | s****e@s****m | 1 |
| Quazi | 5****d | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 0
- Total pull requests: 54
- Average time to close issues: N/A
- Average time to close pull requests: 3 days
- Total issue authors: 0
- Total pull request authors: 5
- Average comments per issue: 0
- Average comments per pull request: 0.15
- Merged pull requests: 50
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 19
- Average time to close issues: N/A
- Average time to close pull requests: 4 days
- Issue authors: 0
- Pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.26
- Merged pull requests: 16
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
- aherbst-broad (21)
- bahill (19)
- ThePletch (9)
- kilonzi (4)
- quazi-h (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 47 last-month
- Total dependent packages: 0
- Total dependent repositories: 2
- Total versions: 32
- Total maintainers: 2
pypi.org: broad-dagster-utils
Common utilities and objects for building Dagster pipelines
- Homepage: https://github.com/broadinstitute/dagster-utils
- Documentation: https://broad-dagster-utils.readthedocs.io/
- License: BSD-3-Clause
-
Latest release: 0.6.7
published over 4 years ago