https://github.com/cdcgov/tostadas - v8.0.0

What's Changed

• Updating github actions workflow by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/344
• Ick4 update submission by @jessicarowell in https://github.com/CDCgov/tostadas/pull/345

NCBI allows UI-less updating of BioSample submissions, and TOSTADAS v8.0.0 can do this using the --workflow update_submission workflow option.

To submit updated metadata to biosample, use the following command:

nextflow run main.nf -profile <docker|singularity> --workflow update_submission --dry_run false --species mpxv --submission_config <path/to/submission_config.yaml> --original_submission_dir <results/mpxv_test_metadata/submission_outputs> --meta_path <path/to/updated/metadata/file>

Please make sure your updated metadata Excel file has a biosample_accession column that contains accurate accession IDs. TOSTADAS does not check these for accuracy. Please make sure they are correct.

Note: TOSTADAS uses the ncbi-spuid field to match samples in the metadata file and the original submission.xml. The sample_name field is not preserved in the submission.xml, so it cannot be used as an identifier for this workflow.

Full Changelog: https://github.com/CDCgov/tostadas/compare/v7.1.0...v8.0.0

- Python
Published by jessicarowell 10 months ago

https://github.com/cdcgov/tostadas - v7.1.0

What's Changed

• Adds updated nf-tests for v7.0.0 workflows, subworkflows, and modules (nf-test changes by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/343)
• Updates to docs/developer_notes.md

Full Changelog: https://github.com/CDCgov/tostadas/compare/v7.0.2...v7.1.0

What's Changed

• nf-test changes by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/343

Full Changelog: https://github.com/CDCgov/tostadas/compare/v7.0.2...v7.1.0

- Python
Published by jessicarowell 11 months ago

https://github.com/cdcgov/tostadas - v7.0.2

What's Changed

• Move the updatedmetapath default assignment to the workflow because global reassignment isn't being recognized by @jessicarowell in https://github.com/CDCgov/tostadas/pull/336
• Clean up the individual database flags by @jessicarowell in https://github.com/CDCgov/tostadas/pull/338
• output_dir -> outdir replacements for all output sudirectories by @jessicarowell in https://github.com/CDCgov/tostadas/pull/340
• Custom fields function needs to ignore the test_field columns by @jessicarowell in https://github.com/CDCgov/tostadas/pull/342

Full Changelog: https://github.com/CDCgov/tostadas/compare/v7.0.1...v7.0.2

- Python
Published by jessicarowell 11 months ago

https://github.com/cdcgov/tostadas - v7.0.1

What's Changed

• Patch: Update will ensure fastq files are not stored in output dir by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/332

Full Changelog: https://github.com/CDCgov/tostadas/compare/v7.0.0...v7.0.1

- Python
Published by jessicarowell 11 months ago

https://github.com/cdcgov/tostadas - v7.0.0

What's Changed

• Pipeline split into two separate workflows: biosampleandsra.nf and genbank.nf. This was necessary because genbank requires assigned biosample accession IDs.
• Added fasta validation for the Genbank workflow
• Added a check to exit gracefully if metadata validation log contains ERRORs (i.e., would fail NCBI submission)
• Added and corrected routing for several log files (mostly in submission subworkflow)
• Added a new subworkflow to handle fetching reports and aggregating the data
• Added a process to produce an updated metadata Excel file, with validated contents and accession IDs for BioSample and SRA, if assigned
• Re-wrote and simplified the wait process (that waits between submitting and fetching reports)
• Applied many fixes to the source files for the GenBank sqn file (source.src, comment.cmt, authorset.sbt) which were not pulling the correct metadata

• Refactored to support wastewater data which has a completely different submission.xml file. Use this by setting wastewater flag in nextflow.config to true and by using the wastewater metadata template (ssets/sample_metadata/wastewater_biosample_template.xlsx) and the custom fields file (--custom_fields_file assets/custom_meta_fields/ww_biosample_pkg.json)

• Many bug fixes

Full Changelog: https://github.com/CDCgov/tostadas/compare/v6.0.1...v7.0.0

- Python
Published by jessicarowell 11 months ago

https://github.com/cdcgov/tostadas - v6.0.1

What's Changed

• README and documentation updates

Full Changelog: https://github.com/CDCgov/tostadas/compare/v6.0.0...v6.0.1

- Python
Published by jessicarowell 11 months ago

https://github.com/cdcgov/tostadas - v6.0.0

What's Changed

*Perform submission in batches using batchsize param *Run a submission dry run using dryrun param (creates a submission.log file in the submission outdir) *Use true, unmodified nf-core versions of nf-core bakta modules *Add a registry param for quay.io, and modify dockerhub container prefixes to accommodate *Change output_dir to outdir everywhere *Integrate "nanopore, illumina, or illumina & nanopore" for SRA submissions *Fixes a lot of dperecated/incorrect Nextflow syntax stuff *Incorporates basic nf-tests for all modules and subworkflows (full pipeline nf-test coming in a future update) *Includes re-writes and/or restructuring of most workflows/subworkflows *For AWS -> will fetch the files from AWS, download to local, then upload to ftp (cloud-to-cloud coming in a future update) *Probably a lot of other stuff I'm forgetting... @jessicarowell in https://github.com/CDCgov/tostadas/pull/323

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.2.6...v6.0.0

- Python
Published by jessicarowell 12 months ago

https://github.com/cdcgov/tostadas - v5.2.6

What's Changed

• Correct Prod to Production by @jessicarowell in https://github.com/CDCgov/tostadas/pull/307
• Add libraryname (default "Not Provided"), drop testfields from validated tsvs, update for illumina/nanopore by @jessicarowell in https://github.com/CDCgov/tostadas/pull/317
• bug fix for validate_metadata.py by @jessicarowell in https://github.com/CDCgov/tostadas/pull/319

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.2.5...v5.2.6

- Python
Published by jessicarowell about 1 year ago

https://github.com/cdcgov/tostadas - v5.2.5

What's Changed

*Patch to change ftp folder Prod to Production by @jessicarowell in https://github.com/CDCgov/tostadas/pull/305

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.2.3...v5.2.5

- Python
Published by jessicarowell about 1 year ago

https://github.com/cdcgov/tostadas - v5.2.4

What's Changed

*Patch by @jessicarowell in https://github.com/CDCgov/tostadas/pull/305 *Don't throw error for "at least one required" metadata parameters if empty, just insert "Not Provided" and alert user *Only throw an error if both "at least one required" columns are missing from metadata Excel file

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.2.3...v5.2.4

- Python
Published by jessicarowell about 1 year ago

https://github.com/cdcgov/tostadas - v5.2.3

What's Changed

• Bakta troubleshooting by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/303

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.2.2...v5.2.3

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v5.2.2

What's Changed

• Merging markdown pages for Mkdocs documentation update by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/299
• update default submission_config path by @jessicarowell in https://github.com/CDCgov/tostadas/pull/301
• docs: create mkdocs yml by @slsevilla in https://github.com/CDCgov/tostadas/pull/290

New Contributors

• @slsevilla made their first contribution in https://github.com/CDCgov/tostadas/pull/290

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.1.2...v5.2.2

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v5.1.2

What's Changed

• Help message addition by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/278
• View a list of TOSTADAS parameters by running nextflow run main.nf --help (to see the most commonly user-configured parameters) or nextflow run main.nf --help --showHIdden to show all available parameters.

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.0.2...v5.1.2

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v5.0.2

What's Changed

• Test command update and relative path change by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/298
• Patch to make Nextflow accept a relative path for submission_config.yaml
• Simply test_params.config by removing params redundant to nextflow.config
• Corrects test commands in the README

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.0.1...v5.0.2

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v5.0.1

What's Changed

• 294 internal bug fetch reports only by @jessicarowell in https://github.com/CDCgov/tostadas/pull/295
• Fixes an error in the `--fetchreportsonly" workflow

Full Changelog: https://github.com/CDCgov/tostadas/compare/v5.0.0...v5.0.1

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v5.0.0

What's Changed

• In the metadata file, users should indicate local or cloud for "file_location".
• Users can transfer files located on s3, Azure Blob Storage, or GCP storage to NBCI via ftp.
• In this release, cloud-to-local (ftp) transfer support has been implemented.
• File checking for SRA submissions are removed from submission and validation python scripts and incorporated directly into Nextflow
• Adds support for cloud-to-local SRA file transfer by @jessicarowell in https://github.com/CDCgov/tostadas/pull/273

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.3.4...v5.0.0

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.3.4

What's Changed

• Patch to fix 287 internal bug bacteria by @jessicarowell in https://github.com/CDCgov/tostadas/pull/288

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.3.3...v4.3.4

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.3.3

What's Changed

• fasta header changes in bacterial sample by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/279

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.3.2...v4.3.3

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.3.2

What's Changed

• Patch: uncomment paramiko by @jessicarowell in https://github.com/CDCgov/tostadas/pull/283

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.3.1...v4.3.2

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.3.1

What's Changed

• Patch: change sequencename to samplename to prevent submission from being skipped by @jessicarowell in https://github.com/CDCgov/tostadas/pull/281

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.3.0...v4.3.1

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.3.0

What's Changed

• Minor update to dynamically handle required parameters for BioSample packages
• Checks metadata columns against a yaml file of NCBI's BioSample package requirements. If a required column is missing, an error is generated. If a required column is present but sample data is empty for that column, it will be populated with "Not Provided".

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.2.6...v4.3.0

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.2.6

What's Changed

• Patch: validate if sample_name is present in the metadata file

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.2.5...v4.2.6

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.2.5

What's Changed

• Adding troubleshooting section to README to address 2 common issues: 1) forcing an update of an older, cached tostadas image 2) duplicate IDs in FASTA causes unexpected behavior in table2asn

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.2.4...v4.2.5

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.2.4

What's Changed

• Conda profile will now default to mamba if available, else it will use conda

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.2.3...v4.2.4

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.2.3

What's Changed

• Allowing relative paths for submission config by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/269

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.2.2...v4.2.3

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.2.2

What's Changed

• Update README.md by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/261
• For verbose (v) and short (s) flags, set valid MM DD dates by @jessicarowell in https://github.com/CDCgov/tostadas/pull/264
• Removed old unused general_utility.py script by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/266

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.2.1...v4.2.2

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.2.1

What's Changed

• Schema and config parameter updates + removed enable_conda parameter by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/260

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.2.0...v4.2.1

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.2.0

What's Changed

• Added --fetch_reports_only which looks on the NCBI server for report.xml files for the samples in the metadata file the user has indicated, for the databases the user has indicated.
• Redefined --update_submission so it should be used for updating records on BioSample or SRA. It requires valid accession IDs in the submission_report.csv.
• So TOSTADAS now has 3 different submission workflows: --submission runs an initial submission to BioSample, SRA, or Genbank. --fetch_reports_only will look for the report.xml files, and --update_submission should be used to update existing records with valid accession IDs.
• Rework update submissions to allow re-submissions by @jessicarowell in https://github.com/CDCgov/tostadas/pull/256

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.1.4...v4.2.0

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.1.4

What's Changed

• Patch to fully integrate the keepdemographicinfo and date_format flags by @jessicarowell in https://github.com/CDCgov/tostadas/pull/249

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.1.3...v4.1.4

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.1.3

What's Changed

• Patch to allow relative file paths by @RamiyapriyaS in https://github.com/CDCgov/tostadas/pull/257

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.1.2...v4.1.3

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.1.2

Changes from Previous Release (v4.1.1):

• Patch to allow for "dots" in sample names in the metadata Excel file

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.1.1...v4.1.2

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.1.1

## Changes from Previous Release (v4.1.0):

• Removed legacy code in the VALIDATE_PARAMS process and moved validation plugin to main workflow
• Updated the nextflow_schema.json file to search for file patterns in input parameters
• Updated README to indicate nf-schema warning message in Nextflow version 24 or later.

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.1.0...v4.1.1

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - v4.1.0

Changes from Previous Release (v4.0.0):

(1) New Nextflow validation schema

• Added support for nf-schema@2.1.1 to validate Nextflow parameters in config files
• This plug-in does not work by default on Nextflow version 24.10.0. It will need to be installed first, and this will be tested and added to the README in a future minor update.

(2) Added basic support for custom metadata and BioSample packages

• Users can add custom fields to their metadata file and the custom_fields json template to have those fields added as Attributes in the BioSample submission.xml file
• Users can specify a BioSample package in the submission config file

(4) Environment changes and upgrades

• Fixes a conda envionment issue

(3) Bug fixes

• Patch to required fields in metadata validation (to allow valid strings)
• Fixed a broken while loop
• Allow submission to run without fasta file (updated in both submission modules)
• Patch that fixed several bugs in metadata validation and submission

New Contributors

• @Alan-Collins made their first contribution in https://github.com/CDCgov/tostadas/pull/240

Full Changelog: https://github.com/CDCgov/tostadas/compare/v4.0.0...v4.1.0

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - Release 4.0.0

Changes from Previous Release (v3.1.0):

(1) New, streamlined submission code

• Supports ftp submission to GenBank (SARS-CoV-2, influenza, and bacteria)
• GenBank ftp submission uses table2asn (v1.28.1179) to prepare an ASN.1 file, authorset.sbt file, and comment.cmt file and uploads these to NCBI along with a submission.xml file.
• Still support non-ftp GenBank submission, either manually (send the email yourself) or automatically send submission to user and NCBI GenBank submission team. This mode uses table2asn to prepare an ASN.1 file, authorset.sbt, comment.cmt, and source.src file. IT also creates a zip file of these extra table2asn outputs.
• Change name of raw fastq files to _R{1,2}.fastq.gz (important: code assumes your fastq files are gzipped) before ftp upload to SRA repository.
• Added support for SRA publication hold (delayed release)
• Note: Submissions are currently completed one sample at a time. Batch submissions coming soon.
• SFTP support added (still in beta testing)

(2) Minor corrections to metadata files

• Two columns adjusted to clarify input and to match NCBI attribute name (age and sex --> hostage and hostsex)
• Extraneous "description" column removed from metadata files
• gff_path added to some test metadata files where it was missing
• "Cdiphtheriae" changed to "bacteria" in test data, for added clarity

(4) Environment changes and upgrades

• table2asn added to environment.yml file
• nameparser (v1.0.5) python library added to environment.yml file
• paramiko (v3.4.0) python library added to environment.yml file

(3) Bug fixes

• Several bug fixes were made along the way...

- Python
Published by jessicarowell over 1 year ago

https://github.com/cdcgov/tostadas - βeta3

Changes from Previous Release (βeta2):

(1) New gene annotation pipeline (VADR) integrated into Nextflow wrapper:

• Can be accessed through both the main workflow (using the run_vadr parameter) and as a separate entrypoint
  • VADR is restricted to only running through Docker, as the docker container is being pulled from the StaphB DockerHub (VADR Container). The actual docker container is specified via a new parameter: docker_container_vadr
• If both Liftoff and VADR gene annotation are toggled on, for the main workflow, they will run in parallel during their respective processes and through the submission portion of the pipeline. VADR will output files in the directory defined by the vadr_output_dir parameter
• Necessary MPXV model files for reference have been added to the repository, therefore no additional setup is required. A new parameter named vadr_models_dir has been added to encode the location of these model files

(2) New templates for opening an issue have been added:

• bug related and opened by a maintainer/collaborator
• feature related and opened by a maintainer/collaborator
• bug related and opened by a user
• feature related and opened by a user

The various options can be found here: Issues Tab

(3) Updates to the pull-request template:

• custom checklist
• CDC specific criteria (for maintainers/collaborators)

(4) README.md has had multiple changes including:

• all information about VADR including where to find accompanying documentation for this package and the parameters mentioned above
• general ordering with certain sections have been switched or change
• information added in the setup section to include steps for provisioning a Scicomp (HPC Cluster at the CDC) environment to run the pipeline
• certain pieces of information have been made present or more obvious within the quick-start and running the pipeline sections (wait time, setting up your submission configuration file, etc.)
• more information has been added to explain required parameters/inputs for the submission entrypoints

(5) Test files (.yaml and .json) have been created within the nf_params directory

Autogenerated Release Notes:

• Template for Git Issues by @ankushkgupta2 in https://github.com/CDCgov/tostadas/pull/12
• New Issue Templates + Default Labels and Titles - Issue #18 by @ankushkgupta2 in https://github.com/CDCgov/tostadas/pull/19
• README.md Improvements (Issue #26) by @ankushkgupta2 in https://github.com/CDCgov/tostadas/pull/30
• Residual README.md Changes From PR #30 (Root: Issue #26) by @ankushkgupta2 in https://github.com/CDCgov/tostadas/pull/31
• VADR Annotation Integration by @ankushkgupta2 in https://github.com/CDCgov/tostadas/pull/28
• Updates to Issue / PR Temps by @ankushkgupta2 in https://github.com/CDCgov/tostadas/pull/24
• Maintainer Contact + Issue Request Information by @ankushkgupta2 in https://github.com/CDCgov/tostadas/pull/23
• Patch for VADR Enhancement and Parameter Check Changes by @ankushkgupta2 in https://github.com/CDCgov/tostadas/pull/32

Full Changelog: https://github.com/CDCgov/tostadas/compare/v1.0.0-beta...v2.0.0-beta

- Python
Published by ankushkgupta2 about 3 years ago

https://github.com/cdcgov/tostadas - βeta2

Added Functionality:

(1) Ability to easily specify a database for sample submission through the Nextflow wrapper (submission_database)

(2) Pre-existing email functionality, for notifying user during a Genbank submission with genbank_submission_type set to table2asn, is now configurable within the Nextflow wrapper * Functionality is toggled on/off with send_submission_email * Email recipient is specified as a new field within your submission config under 'general' as 'notifemailrecipient'

(3) Git submodule integration of internal CDC submission configs with the repository * Configs exist in a remote GitLab repository with permissions limited to CDC personnel with access to the Monkeypox group * Documentation outlining steps on how to initialize the submodule and pull these config files from the remote location to your local version of the repo is specified within docs/cdcconfigsaccess.md

(4) Fully Nextflow Tower compatible * Docker container is now hosted DockerHub and can automatically be pulled in (assuming you have the proper permissions) * Rewriting/refactoring of the Python scripts handling database submission via FTP * Enhancements of Nextflow wrapper structure/code to now scale across various environments

(5) Updates to README.md to include these latest changes (new/removed parameters, general restructuring etc.), as well as updates to PULLREQUESTTEMPLATE.md

- Python
Published by ankushkgupta2 over 3 years ago

https://github.com/cdcgov/tostadas - βeta

First release for Beta Testing.

- Python
Published by ankushkgupta2 over 3 years ago