Data

Tools

Indexes

Applications

Experiments

Open Source Science

refget

Python tools for identification and distribution of reference sequences and sequence collections

https://github.com/refgenie/refget

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.4%) to scientific vocabulary

Keywords from Contributors

interactive serializer packaging network-simulation hacking autograding observability embedded optim standardization
Last synced: 11 months ago · JSON representation

Repository

Python tools for identification and distribution of reference sequences and sequence collections

Basic Info
  • Host: GitHub
  • Owner: refgenie
  • License: bsd-2-clause
  • Language: Python
  • Default Branch: master
  • Homepage: https://refgenie.org/refget/
  • Size: 1.01 MB
Statistics
  • Stars: 9
  • Watchers: 1
  • Forks: 1
  • Open Issues: 7
  • Releases: 12
Created about 6 years ago · Last pushed 12 months ago
Metadata Files
Readme License

README.md

Refget

Run pytests

User-facing documentation is hosted at refgenie.org/refget.

This repository includes:

  1. /refget: The refget Python package, which provides a Python interface to both remote and local use of refget standards. It has clients and functions for both refget sequences and refget sequence collections (seqcol).
  2. /seqcolapi: Sequence collections API software, a FastAPI wrapper built on top of the refget package. It provides a bare-bones Sequence Collections API service.
  3. /deployment: Server configurations for demo instances and public deployed instances. There are also github workflows (in .github/workflows) that deploy the demo server instance from this repository.
  4. /test_fasta and /test_api: Dummy data and a compliance test, to test external implementations of the Refget Sequence Collections API.
  5. /frontend: a React seqcolapi front-end.

Testing

Local unit tests of refget package

  • pytest to test refget package, local unit tests

Development and deployment: Backend

Easy-peasy way

In a moment I'll show you how to do these steps individually, but if you're in a hurry, the easy way get a development API running for testing is to just use my very simple shell script like this (no data persistence, just loads demo data):

console bash deployment/demo_up.sh

This will: - populate env vars - launch postgres container with docker - run the refget service with uvicorn - load up the demo data - block the terminal until you press Ctrl+C, which will shut down all services.

Step-by-step process

Alternatively, if you want to run each step separately to see what's really going on, start here.

Setting up a database connection

First configure a database connection through environment variables. Choose one of these:

source deployment/local_demo/local_demo.env # local demo (see below to create the database using docker) source deployment/seqcolapi.databio.org/production.env # connect to production database

If you're using the local_demo, then use docker to launch a local postgres database service like this:

docker run --rm --name refget-postgres -p 127.0.0.1:5432:5432 \ -e POSTGRES_PASSWORD \ -e POSTGRES_USER \ -e POSTGRES_DB \ -e POSTGRES_HOST \ postgres:17.0

If you need to load test data into your server, then you have to install gtars (with pip install gtars), a Python package for computing GA4GH digests. You can then load test data like this:

python data_loaders/load_demo_seqcols.py

or:

refget add-fasta -p test_fasta/test_fasta_metadata.csv -r test_fasta

Running the seqcolapi API backend

Run the demo seqcolapi service like this:

uvicorn seqcolapi.main:app --reload --port 8100

Running with docker

To build the docker file, first build the image from the root of this repository:

docker build -f deployment/dockerhub/Dockerfile -t databio/seqcolapi seqcolapi

To run in container:

source deployment/seqcolapi.databio.org/production.env docker run --rm -p 8000:80 --name seqcolapi \ --env "POSTGRES_USER" \ --env "POSTGRES_DB" \ --env "POSTGRES_PASSWORD" \ --env "POSTGRES_HOST" \ databio/seqcolapi

Deploying container to dockerhub

Use the github action in this repo which deploys on release, or through manual dispatch.

Running the frontend

Once you have a backend running, you can run a frontend to interact with it

Local client with local server

cd frontend npm i VITE_API_BASE="http://localhost:8100" npm run dev

Local client with production server

cd frontend npm i VITE_API_BASE="https://seqcolapi.databio.org" npm run dev

Deploy to AWS ECS

  • Test locally first, using 1. native test; 2. local docker test.

Deploying

  1. Ensure the refget package master branch is as you want it.
  2. Deploy the updated secqolapi app to dockerhub (using manual dispatch, or deploy on github release).
  3. Finally, deploy the instance with manual dispatch using the included GitHub action.

Developer notes

Models

The objects and attributes are represented as SQLModel objects in refget/models.py. To add a new attribute:

  1. create a new model. This will create a table for that model, etc.
  2. change the function that creates the objects, to populate the new attribute.

Example of loading reference fasta datasets:

refget add-fasta -p ref_fasta.csv -r $BRICKYARD/datasets_downloaded/pangenome_fasta/reference_fasta

Owner

  • Name: Refgenie
  • Login: refgenie
  • Kind: organization

Standardized assets for reference genomes

GitHub Events

Total
  • Create event: 17
  • Issues event: 3
  • Release event: 5
  • Watch event: 4
  • Delete event: 9
  • Issue comment event: 52
  • Push event: 100
  • Pull request review event: 13
  • Pull request review comment event: 16
  • Pull request event: 39
Last Year
  • Create event: 17
  • Issues event: 3
  • Release event: 5
  • Watch event: 4
  • Delete event: 9
  • Issue comment event: 52
  • Push event: 100
  • Pull request review event: 13
  • Pull request review comment event: 16
  • Pull request event: 39

Committers

Last synced: about 1 year ago

All Time
  • Total Commits: 193
  • Total Committers: 4
  • Avg Commits per committer: 48.25
  • Development Distribution Score (DDS): 0.036
Past Year
  • Commits: 101
  • Committers: 3
  • Avg Commits per committer: 33.667
  • Development Distribution Score (DDS): 0.05
Top Committers
Name Email Commits
nsheff n****f 186
dependabot[bot] 4****] 3
Michal Stolarczyk s****3@g****m 3
Andrew Yates a****z 1

Issues and Pull Requests

Last synced: 11 months ago

All Time
  • Total issues: 6
  • Total pull requests: 52
  • Average time to close issues: N/A
  • Average time to close pull requests: 16 days
  • Total issue authors: 2
  • Total pull request authors: 5
  • Average comments per issue: 0.17
  • Average comments per pull request: 1.06
  • Merged pull requests: 33
  • Bot issues: 0
  • Bot pull requests: 17
Past Year
  • Issues: 3
  • Pull requests: 41
  • Average time to close issues: N/A
  • Average time to close pull requests: 21 days
  • Issue authors: 2
  • Pull request authors: 4
  • Average comments per issue: 0.0
  • Average comments per pull request: 1.32
  • Merged pull requests: 22
  • Bot issues: 0
  • Bot pull requests: 17
View more stats
Top Authors
Issue Authors
  • nsheff (4)
  • donaldcampbelljr (2)
Pull Request Authors
  • nsheff (29)
  • dependabot[bot] (17)
  • donaldcampbelljr (7)
  • sanghoonio (2)
  • andrewyatz (1)
Top Labels
Issue Labels
enhancement (1)
Pull Request Labels
dependencies (17) javascript (13)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 125 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 13
  • Total maintainers: 1
pypi.org: refget

Python client for refget

  • Versions: 13
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 125 Last month
Rankings
Dependent packages count: 10.0%
Stargazers count: 21.5%
Dependent repos count: 21.7%
Forks count: 22.6%
Average: 27.9%
Downloads: 63.7%
Maintainers (1)
databio
Last synced: 11 months ago

Dependencies

requirements/requirements-all.txt pypi
  • henge ==0.1.1
  • pyfaidx *
  • pyyaml *
  • requests *
  • yacman >=0.6.7
requirements/requirements-test.txt pypi
  • coveralls >=1.1 test
  • oyaml * test
  • pytest-cov ==2.6.1 test
.github/workflows/black.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
  • psf/black stable composite
.github/workflows/python-publish.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v5 composite
  • pypa/gh-action-pypi-publish release/v1 composite
.github/workflows/run-codecov.yml actions
  • actions/checkout v2 composite
  • codecov/codecov-action v2 composite
.github/workflows/run-pytest.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v5 composite
pyproject.toml pypi
requirements/requirements-dev.txt pypi
requirements/requirements-docs.txt pypi
setup.py pypi
deployment/dockerhub/Dockerfile docker
  • tiangolo/uvicorn-gunicorn python3.11-slim build
deployment/seqcolapi.databio.org/Dockerfile docker
  • databio/seqcolapi latest build
seqcolapi/requirements/requirements-seqcolapi.txt pypi
  • aiofiles *
  • fastapi *
  • henge >=0.2.1
  • jinja2 *
  • pipestat *
  • psycopg2-binary *
  • refget *
  • ubiquerg >=0.6.1
  • uvicorn >=0.7.1
  • yacman >=0.9.0