https://github.com/broadinstitute/viral-pipelines - v2.4.1.1

What's Changed

• Fix GH Actions builds by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/595
• metagenomic denovo optimizations and fixes by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/594

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.4.1.0...v2.4.1.1

- WDL
Published by dpark01 about 1 year ago

https://github.com/broadinstitute/viral-pipelines - v2.4.1.0

What's Changed

New workflows or functionality

• add workflow "unpackarchivetobucket"; add task "unpackarchivetobucket_path" by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/573
• assembly and genbank updates by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/579
• NCBI submission workflow updates, new workflow assemble_denovo_metagenomic, submit_biosample, submit_sra, submit_genbank. Prune vestigal workflows demux_metag, taxid_to_nextclade, contigs. by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/583

Workflow edits and fixes

• s/Array[File]+/Array[File?]+/g for tsvfiles input in terratsvtotable.wdl by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/563
• bugfix: demux_deplete with biosample attributes by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/572
• scaffoldandrefine_multitaxa: add more output columns by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/574
• update workflows for genbank submission by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/576
• genbank bug fixes by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/581
• demux and nextclade updates by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/582
• genbank_single: made mapped bam input optional by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/584
• Optimizations for Genbank submission of coinfections by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/587
• hotfix by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/590

New options, inputs, or outputs

• demux_deplete: allow multiple biosample attribute files by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/561
• add contextual fasta input for augurfromassemblies by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/558
• add deduplication of sequences by ID in augurfromassemblies by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/564
• Expose --timetree parameter in refineaugurtree by @lakras in https://github.com/broadinstitute/viral-pipelines/pull/570
• Exposed --root-sequence parameter of augur ancestral by @lakras in https://github.com/broadinstitute/viral-pipelines/pull/585
• report pcttophitoftotalreads as one of the outputs of alignand_… by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/577

Infrastructural updates

• update terra introspection to support GCP Batch backend by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/575
• fix checkterraenv by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/589
• add task: downloadfromurl() to download a URL to a file by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/562

Dependency changes

• update viral-core 2.3.6 -> 2.4.0 by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/566
• bump viral core 2.4.0 -> 2.4.1 by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/569
• bump pangolin and nextclade by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/565
• bump viral-classify 2.2.4.2 -> 2.2.5 by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/568
• remove exemption from global version pin for viral-core image used in checkterraenv; update viral-core spec of the task to 2.4.0 by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/571

Build changes

• limit GH build concurrency by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/549
• cleanup .dockstore.yml by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/578

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.3.6.0...v2.4.1.0

- WDL
Published by dpark01 over 1 year ago

https://github.com/broadinstitute/viral-pipelines - v2.3.6.0

What's Changed

enterics related

• update enterics visualization docker image by @jpramosbarroso in https://github.com/broadinstitute/viral-pipelines/pull/516
• Make createentericsqc_viz whitespace tolerant by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/517
• update personal branch by @jpramosbarroso in https://github.com/broadinstitute/viral-pipelines/pull/520
• createentericsqc_viz simplify and improve portability by @jpramosbarroso in https://github.com/broadinstitute/viral-pipelines/pull/521
• update docker for try/except block with new error message by @jpramosbarroso in https://github.com/broadinstitute/viral-pipelines/pull/522

assembly scaffolding & reference selection improvements

• scaffoldandrefine_multitaxa -- more thorough assembly outputs by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/524
• scaffolding and reference selection based on ANI by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/528
• Reference selection improvements and nextclade generalization by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/529
• scaffolding regression fixes plus docker updates by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/547
• scaffold multitaxa: drop empty assembly output rows by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/554
• report reference genome length in alignreads output; use this value for assemblerefbased. referencegenomelength by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/538

demux_deplete and terra table data model updates

• demux_deplete: make depletion optional, load biosample metadata to tables by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/551
• demux_deplete small updates by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/552
• demux_deplete fixes/updates by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/555

miscellaneous fixes and edge cases

• allow additional options for picard in FastqToUBAM by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/518
• WDL code/style cleanups by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/523
• expanded parameterization of alignandcount and additional output metrics by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/525
• guard against shell pipefails in alignandcount in instances where empty inputs would result in div-by-zero conditions by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/535
• classify_single: parameterize taxa to deplete by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/536
• bugfix fetchsrato_bam by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/542
• Update polyphonia input parameters and defaults by @lakras in https://github.com/broadinstitute/viral-pipelines/pull/545

docker container updates

• build and container updates by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/526
• update viral-core 2.3.0 -> 2.3.1 by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/527
• increase default instance memory for kraken2 task 72 -> 90GB by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/540
• nextclade updates by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/557
• bump upstream docker images by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/556

build & CI related

• add build environment spec to .readthedocs.yml config by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/541
• documentation build fixes by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/546
• bugfix dx deploy failures by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/548

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.2.4.0...v2.3.6.0

- WDL
Published by dpark01 almost 2 years ago

https://github.com/broadinstitute/viral-pipelines - v2.2.4.0

What's Changed

New workflows

• Enterics QC visualizations
  • WDL for creating Enterics Visualizations by @schaluva in #500
  • create generalized version of enterics visualization WDL by @schaluva in #501
  • Visualization HTML PR by @jpramosbarroso in https://github.com/broadinstitute/viral-pipelines/pull/513
  • Fixing duplicate inputs bug by @jpramosbarroso in https://github.com/broadinstitute/viral-pipelines/pull/514
  • integrating threshold file by @jpramosbarroso in https://github.com/broadinstitute/viral-pipelines/pull/515
• Terra-on-GCP-specific features
  • add checkterraenv task by @tomkinsc in #497
  • add workflows to add to or create library and sample tables based on demultiplexing output by @tomkinsc in #507
  • mark checkterraenv task as volatile by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/510
  • include user email among outputs of checkterraenv task by @tomkinsc in https://github.com/broadinstitute/viral-pipelines/pull/508
• transmission reconstruction from @ispecht
  • add reconstructR workflow by @dpark01 in #504
• mass-scale metagenomic denovo assembly
  • add new workflow scaffoldandrefine_multitaxa by @dpark01 in https://github.com/broadinstitute/viral-pipelines/pull/506
  • more scaffolding updates by @dpark01 in #511

Updates to existing workflows

• pass through sbt template if optional defaultsyaml is not provided and authorlist string is empty by @tomkinsc in #492
• mark downsample_bams task as volatile (nondeterministic) by @tomkinsc in #494
• expose numberOfNegativeControls in illumina_demux by @tomkinsc in #498
• add maskedpositions and maskedpositionsfile arguments to task polyphoniadetectcrosscontamination by @lakras in #499
• fix mem high watermark reporting by @tomkinsc in #502
• mass update of upstream docker tags by @dpark01 in #505
  • broadinstitute/viral-core: 2.1.33 -> 2.2.4
  • broadinstitute/viral-classify: 2.1.33.0 -> 2.2.3.0 (later bumped to 2.2.4.0 in #511)
  • broadinstitute/viral-baseimage: 0.1.20 -> 0.2.0
  • staphb/vadr: 1.5.1 -> 1.6.3
  • staphb/pangolin: 4.3.1-pdata-1.22 -> 4.3.1-pdata-1.23.1
  • nextstrain/nextclade: 2.12.0 -> 2.14.0

Build and CI

• enable merge_group builds in github actions CI by @dpark01 in #503
• allow docker image pinning to be skipped with #skip-global-version-pin annotation by @tomkinsc in #493
• dockstore updates by @dpark01 in #512
• clean up disk space in Github Actions VMs #505

New Contributors

• @jpramosbarroso made their first contribution in #513
• @ispecht 's code is utilized in a new workflow for #504

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.17...v2.2.4.0

- WDL
Published by dpark01 over 2 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.17

What's Changed

new features or functionality

• extend beast task and workflow to glob and output *.ops, *.rates, and *.root files by @tomkinsc in #478
• Subsampler and augur workflow by @dpark01 in #483

bug fixes and small changes

• syntax bugfixes by @dpark01 in #462
• tweaks to demuxonly and demuxmetag by @dpark01 in #464
• omit --dirs from the MultiQC invocation to only show sample names in report by @tomkinsc in #467
• hotfix for bug in output path by @dpark01 in #480
• sc2 nextstrain: reduce temp file sizes by @dpark01 in #484
• FetchSRAto_BAM - tolerate no isolate defined by @dpark01 in #489
• Update genbank.WDL so author_list input is optional by @tomkinsc in #473

documentation updates

• Enhancing documentation for viral-pipelines by @golu099 in #461
• move transposed parameter_meta block outside command block scope to where it belongs by @tomkinsc in #477
• Update LICENSE copyright year by @tomkinsc in #476

parameterization and default VM shapes

• de novo assembly & scaffold mem default 31 -> 63 GB by @tomkinsc in #465
• parameterize disk size for illumina_demux task by @tomkinsc in #470
• maxrecordsin_ram param 2000000 -> 10000000 for Nova S2 by @tomkinsc in #468

upstream docker tags / version dependencies

• bump pangolin by @dpark01 in #482

build updates and fixes

• delete the .travis directory by @tomkinsc in #448
• dnanexus CI build fixes by @dpark01 in #485 #486 #487 #488
• GitHub actions update by @tomkinsc in #474

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.16...v2.1.33.17

- WDL
Published by dpark01 almost 3 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.16

What's Changed

functionality changes and bug fixes

• Convert amplicon16S_analysis workflow (QIIME) from single-sample to multi-sample by @golu099 in #452
• samtools ampliconstats needs bed in certain order by @dpark01 in #454

parameterization and defaults

• parameterize and update defaults for VM shapes in nextstrain tasks by @dpark01 in #455
• parameterize more assembly knobs by @dpark01 in #457

update docker tags / version dependencies

• update to pangolin 4.2 / pdata 1.18 by @dpark01 in #451
• Update pangolin, nextclade, and vadr docker images by @dpark01 in #456

build updates and fixes

• documentation build fixes for ReadTheDocs by @tomkinsc in #450
• update comment about DXAPITOKEN by @tomkinsc in #449

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.15...v2.1.33.16

- WDL
Published by dpark01 over 3 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.15

What's Changed

new and changed workflows

• Add workflow for QIIME2 classification of 16S reads by @golu099 in #439
• simplify augurfromassemblies workflow by @dpark01 in #440
• denovo assembly SPAdes updates by @dpark01 in #441
• genbank prep and assemble_denovo updates by @dpark01 in #443
• add new workflow nextclade_single by @dpark01 in #444
• update classify_single to accept multiple bams by @dpark01 in #447

smaller docker, VM, and mechanical updates

• more disk for nextstrain; dockstore API fix by @dpark01 in #437
• Make Pipelines Azure compatible Closes #143 by @vsmalladi in #438
• set default GPU to K80->P4 in BEAST task; parameterize beagle_order by @tomkinsc in #442
• pangolin 4.1.3 / pdata 1.17 by @dpark01 in #445

New Contributors

• @vsmalladi made their first contribution in #438
• @golu099 made their first contribution in #439

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.14...v2.1.33.15

- WDL
Published by dpark01 over 3 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.14

What's Changed

Feature changes

• task alignment_metrics: more resilient to large amplicon designs by @dpark01 in #435

Docker image updates

• update nextclade and vadr docker images by @dpark01 in #433
• bump pangolin to 4.1.2-1.14 and nextclade to 2.5.0 by @dpark01 in #434

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.13...v2.1.33.14

- WDL
Published by dpark01 almost 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.13

What's Changed

Feature changes

• CDC SC2 viral pipeline updates (parameters, reporting, QC) by @dpark01 in #431
• expose exclude/drop list input for nextstrainbuildsubsample task by @tomkinsc in #428

Docker image updates

• nextclade 2.0 updates by @dpark01 in #430
• update pangolin docker to 4.1.1-pdata-1.11 by @dpark01 in #432

VM shape tuning

• parameterize machinememgb for ancestral_traits task by @tomkinsc in #427
• in exportauspicejson parameterize task mem, raise default to 64GB; increase AUGURRECURSIONLIMIT from 10k to 15k by @tomkinsc in #429

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.12...v2.1.33.13

- WDL
Published by dpark01 about 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.12

What's Changed

• lofreq task portability, misc other edits by @dpark01 in #426

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.11...v2.1.33.12

- WDL
Published by dpark01 about 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.11

What's Changed

• incorporate assemble_refbased as refine/polish for denovo by @dpark01 in #422
• new metagenomic_denovo workflow by @dpark01 in #423
• consensus calling major_cutoff knobs and defaults by @dpark01 in #424
• update pangolin to pdata 1.9 by @dpark01 in #425

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.10...v2.1.33.11

- WDL
Published by dpark01 about 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.10

What's Changed

Minor functional updates

• CDC contractual updates to sarscov2_illumina_full (min coverage, genome name default prefix) - #419
• pangolin tasks: updatedbsnow invokes update data only (not pangolin software) - #420

Docker image updates only:

• pangolin docker: several routine docker image updates - #414 #416 #417 #421

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.9...v2.1.33.10

- WDL
Published by dpark01 about 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.9

This release fixes a consistent major failure mode (which only emerged recently) when executing most assembly, demux, or phylogenetic workflows on Google Cloud (e.g., via Terra).

What's Changed

• make memory telemetry fail gracefully by @dpark01 in #413

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.8...v2.1.33.9

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.8

See release notes for v2.1.33.7, which failed to sync to Dockstore. v2.1.33.8 has no non-whitespace changes from v2.1.33.7, and is only meant to fix the sync.

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.6...v2.1.33.8

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.7

What's Changed

• sc2 sequencing report ram and pangolin update fault tolerance by @dpark01 in #410
• pangolin v4 by @dpark01 in #411

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.6...v2.1.33.7

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.6

What's Changed

• update pangolin and nextclade docker images by @dpark01 in #409

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.5...v2.1.33.6

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.5

What's Changed

• post CDC contract: genome ID renaming by @dpark01 in #408

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.4...v2.1.33.5

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.4

What's Changed

• increase tsv_join default mem 7->15 GB by @tomkinsc in #406
• update pangolin container by @dpark01 in #407

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.3...v2.1.33.4

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.3

What's Changed

docker image updates

• nextclade 1.7.0 to 1.10.0 by @dpark01 in #404
• pangolin container update by @dpark01 in #405

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.2...v2.1.33.3

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.2

What's Changed

new features / behavior

• new read_depths workflow by @lakras in #393
• tsv_join task: add option to control preference of input order (left or right join) by @dpark01 in #400

optimizations / vm tuning

• parameterize memory for tasksnextstrain.tipfrequencies task; increase default to 30GB by @tomkinsc in #398

docker image updates

• bump panoglin 3.1.16 -> 3.1.17 and nextclade 1.4.0 -> 1.7.0 by @dpark01 in #399
• update pangoLEARN, pango-designation, scorpio, constellations by @dpark01 in #401
• update vadr 1.3 -> 1.4 by @dpark01 in #402
• update pangoLEARN, pango-designation, constellations; update vadr 1.4 -> 1.4.1 by @dpark01 in #403

Full Changelog

https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.1...v2.1.33.2

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.1

What's Changed

new features

• robustness: allow one retry for gisaid uploader [#395]

optimizations

• slight speed optimizations to assemblerefbased and demuxdeplete [#397]

vm/image updates

• bump pangolearn docker [#394]
• bump vadr docker (new models for sc2) [#396]

contributors

@dpark01

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.33.0...v2.1.33.1

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.33.0

What's Changed

new features

• turn on allowNestedInputs for subworkflows: this exposes to the end-user all task-level optional inputs from subworkflows [#387]
• expose --print-all-iSNVs option for cross-contamination detection [#386]
• add updatedbsnow option to pangolin tasks [#392]
• add generic uploadentitiestsv task [#391]
• Filter CRSP samples -> biosample registration to only clinical tests (not pooled samples) [#374]
• CRSP meta ETL: add new body part value [#390]
• add sample ID list to outputs of nextclade/pangolin many sample tasks [#373]

bug fixes

• use WDL value for submittinglabname in gisaidmetaprep python body [#376]

optimizations

• allow maxRetries=2 for most WDL tasks, increasing robustness of execution on Terra [#379, #383]
• sarscov2batchrelineage: speed up localization of input data [#375]
• descatter CDC AWS delivery: copy all raw reads in a single aws S3 copy task instead of hundreds--faster, more reliable, etc [#389]

vm/image updates

• bump viral-core image and remove in-WDL workaround [#377]
• update pangolin docker [#380, #385, #388]
• update nextstrain docker to most recent tag (20211012 ) and update ncov commit to v9 release [#378]
• increase mem of utils.tsv_join task 7GB->15GB, allow maxRetries=2 [#379]
• increase default memory for tsv_join 7->32gb [#382]
• more ram and disk for sc2 reports [#384]
• VM tuning of nextclade & pangolin many-sample tasks based on measured usage of a typical 768-sample flowcell [#373]

build updates

• github actions updates [#381]
• dnanexus build fixes [#372]

Contributors

@dpark01 @tomkinsc @lakras

Full Changelog: https://github.com/broadinstitute/viral-pipelines/compare/v2.1.32.4...v2.1.33.0

- WDL
Published by dpark01 over 4 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.32.4

new features: - nextcladeversion output string now includes nextclade datasets "tag" (version/date) [#371] - implement nextclademultisample and pangolinmultisample with Map task outputs, switch sarscov2batchrelineage and sarscov2illuminafull to use multisample pangolin and nextclade tasks to increase compute efficiency and reduce shard counts [#368]

bug fixes: - rename detectcrosscontamination task wdl to be distinct from workflow name to fix dxWDL builds [#370]

vm/image updates: - update pangolin 3.1.11 to 3.1.14, update pangolearn 2021-09-17 to 2021-09-28, update nextclade 1.2.3 to 1.4.0 [#371]

- WDL
Published by dpark01 almost 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.32.3

improvements: - sarscov2_biosample_load workflow: stop using today's date in constructing ftp directory path for NCBI BioSample submissions in order to allow call caching for jobs run on different days [#366]

bug fixes: - fix for Array[Array[String]] alerts output variable from vadr task (update to new vadr output format) [#363] - edge case bug fix for nextstrain subsampling keep_list (was always mangling the first entry of a user-specified keep list) [#364]

Broad-specific: - add more external lab names to task crsp_meta_etl [#367]

minor VM/docker changes: - VM shape changes in nextstrain pipeline [#362] - pangolearn image update [#365]

- WDL
Published by dpark01 almost 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.32.2

• bugfix: Broad dashboard output file should be txt not tsv [#360]
• docker update to pangolearn 2021-08-24 [#359]
• docker update to sc2-rmd:0.1.25 [#361]

- WDL
Published by dpark01 almost 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.32.1

bugfixes: - workflow sarscov2_nextstrain and sarscov2_nextstrain_aligned_input: bug fix to DAG -- ensure that treetime and ancestral inference are using masked alignments, not unmasked alignments [#358] - task crspmetaetl: add more possible values to the controlled vocabulary options for body_part [#357]

- WDL
Published by dpark01 almost 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.32.0

new features: - most task runtime blocks now support cromwell auto memory scaling/retry - automated data release and delivery sarscov2_data_release - batch recalling of pango/nextclade lineages sarscov2_batch_relineage - improved automated BioSample registration and metadata handling from Broad CRSP samples and external non-Broad samples via GP pipeline - add sarscov2_biosample_load as optional subworkflow call at the beginning of sarscov2_illumina_full for fully automated use by Terra workflow launcher - updated/improved Picard-based illumina demux - move state public health reporting from sarscov2_illumina_full to sarscov2_data_release

bug fixes:

minor updates to docker images and vm shapes: - pangolin 3.1.11 / pangolearn 2021-08-09 - nextclade 1.2.3 - vadr 1.3 - nextstrain 20210413T201712Z - sc2-rmd, viral-core

build changes: - GitHub Actions CI added, now primary. Travis CI still active at the moment

- WDL
Published by tomkinsc almost 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.28.0

new features: - new workflow sarscov2nextstrain [#204, #208, #219] - updates to genbank submission [#201, #207] - update vadr alert criteria based on NCBI recommendations [#234, #249] - add nextclade tree outputs to sarscov2illuminafull [#233] - add sequencing reports via rmarkdown (sarscov2illuminafull) [#222, #226, #228, #235, #236, #244, #245, #248, #265] - ivar trim updates: emit ivar trim stats (assemblerefbased) and compute summary stats (sarscov2illuminafull) [#237] - terra table upload and download [#206, #241] - add picard wgs metrics, alignment metrics, and insert size metrics to assemblerefbased and sarscov2illuminafull [#239, #282] - add bucket delivery of data for CDC, SRA, and GP reporting to sarscov2illumina_full [#258, #263, #278] - add tasks and workflows for NCBI BioSample registration and metadata retrieval [#279] - automated filtering of libraries from failed NTC controls [#266]

bug fixes: - bugfix whitespace handling in gzcat task [#230] - deduplicate output rows from srametaprep [#220] - GISAID metadata output should be CSV not TSV [#273] - derive Illumina run ID from XML instead of tarball filename [#275]

minor updates to docker images and vm shapes: - vm shape updates on augur steps [#205, #224, #225, #229, #232] - bump viral-core docker [#242, #243, #268, #270, #274, #276, #281, #284] - bump ivar docker [#209] - bump pangolin/pangolearn [#203, #205, #210, #213, #214, #215, #216, #217, #218, #240, #250, #254, #267, #271, #285] - bump vadr docker [#216, #264] - bump nextstrain/base [#238] - bump sc2-rmd docker [#269, #277] - update nextmeta tsv output behavior to match new nextmeta spellings [#231]

build changes: - bump cromwell and womtool 54 to 61 [#272] - temporarily drop dnanexus builds from Travis until we clean up the dnanexus CI project [#280, #283]

- WDL
Published by dpark01 about 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.19.0

Added new workflow: sarscov2_sra_to_genbank -- this takes sequencing reads from INSDC (via NCBI SRA), assembles, annotates, and QCs genomes, and produces Genbank and GISAID submission bundles based on the metadata in NCBI (SRA and BioSample). The Genbank submission will be tied to the same source BioProject and BioSamples that the reads were linked to in SRA. This workflow is able to merge together multiple read sets (SRA records) from the same BioSample and produce one assembly per BioSample. It will automatically detect sequencing platform (only Illumina and Oxford Nanopore currently supported) as well as amplicon vs metagenomic library designs based on the SRA metadata, and assemble appropriately. This has been tested on Illumina reads, ONT reads, amplicon libraries, metagenomic libraries, reads submitted to NCBI SRA, and reads originally submitted to ENA and synced with NCBI. [#197, #200]

Minor changes and fixes to sarscov2_illumina_full: - filter genbank/gisaid submission packages to only sequences present in biosample attributes file [#200] - relax minimum genome unambig bp cutoff from 20kb to 15kb [#200] - allow for merging multiple biosample attributes tsvs together in sarscov2_illumina_full [#200] - add "Sequencing Technology" column to both genbank and gisaid submission packages [#200] - greatly simplify the final assembly metrics metadata output from both workflows (single tsv instead of compound array structures) [#200] - makes filename outputs a bit more organized [#200] - exposes cleanedbamuris text file output for easy SRA submission [#200] - replace the first several steps with an invocation of demux_deplete as a subworkflow to reduce code duplication [#197]

Other minor changes: - sarscov2_lineages and sarscov2_illumina_full: rename output variable pangolin_clade to pango_lineage to stay in line with the nomenclature of the PANGOLIN authors. [#197] - increase default RAM for GATK UG consensus calling in assemble_refbased from 7GB to 15GB. [#200] - bump nextclade image and pangoLEARN database to latest [#198]. nextclade update improves deletion variant naming. pangolin update keeps up with latest lineage assignments. - bump viral-core docker 2.1.18 to 2.1.19 to fix demux scenario with single-index/paired-reads [#199]

- WDL
Published by dpark01 over 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.18.0

New general workflows: - new workflow demux_deplete. This sits between demux_only (demux and fastqc) and demux_plus (which adds kraken, spades, etc) and just does demux, fastqc, and depletion. If optionally given "augmented" samplesheets and NCBI BioSample mappings, it will produce SRA submission bundles as well. [#191] - new workflow mafft_and_snp_annotated, which adds snpEff annotation to the snp-sites output [#194]

New SARS-CoV-2 specific workflows. Up until this release, all included workflows were generally applicable to most viral taxa. This release includes a number of single-taxon workflows exclusively for SARS-CoV-2 in order to increase efficiency for high throughput work on this one virus. - new workflow sarscov2_illumina_full is a full end-to-end workflow from Illumina BCL tarball through Genbank, SRA, and GISAID submission bundles. It wraps together demux_deplete, assemble_refbased, sarscov2_lineages, sarscov2_genbank. It requires the user to pre-register NCBI BioSample entries and to provide an "augmented" samplesheet for demux. [#191, #196] - new workflow sarscov2_genbank. Prepares single-segmented genome assemblies for submission to NCBI Genbank using their new SARS-CoV-2 submission mechanism (which may become more mainstream for other viruses as well). Incorporates the new VADR (Viral Annotation DefineR) tool from NCBI to annotate (produce tbl files) and QC (flag frameshift and other problems) using the same settings that Genbank uses for QC -- this filters out genomes from submission that fail VADR QC and should result in Genbank submissions with no rejections. [#191] - new workflow sarscov2_lineages and sarscov2_nextclade_multi. Runs Nextclade and Pangolin to do lineage/clade classification on SARS-CoV-2 genomes. [#184, #185, #186]

• nextstrain/augur workflow improvements and bugfixes to allow for merging of multiple metadata tsv files. This simplifies the process of regular builds where some data is changing frequently [#189, #181, #191]

• docker image updates: [#195, #190, #187, #182, #191, #193]

• VM shape updates [#188]

• README update with diagram [#183, @llangit-broad]

- WDL
Published by dpark01 over 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.12.0

Fixes: * remove recursion limit for finding the RunInfo.xml file when unpacking sequencing run tarballs in illumina_demux [#179]

Updated: * bump viral docker base layers to images based on viral-core 2.1.12 [#180]

- WDL
Published by tomkinsc over 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.10.0

New or changed WDL workflows: - new workflow: subsamplebymetadatawithfocal [#161]

Changes: - rename workflow augurfromnewick to augurexportonly, add new workflow augurfrommltree [#151] - drop support for trinity (pinned version) assembler [#168] - bump upstream docker images [#169, #163] - README changes [#162]

Fixes: - bugfix: isnvspersample when specifying optional parameters [#160] - bugfixes for use of set -o pipefail [#175, #173] - bugfix: optional input handling in filterbamto_taxa [#171]

VM shape parameterization or changes to defaults: - mafft parameterization [#170] - memory increase to task filtersequencestolist [#159] - parameterize BEAST GPU settings [#158] - memory increase to assemblerefbased (specifically to task alignreads) [#157] - memory increase to task multialignmafft [#155] - memory increase & cpu decrease to task refineaugurtree and ancestraltree (the timetree invocations) [#153] - parameterize CPU count for task draftaugurtree (iqtree) [#150]

DNAnexus: - update DNAnexus demux_launcher to v2 instance types [#156] - update instance types to v2 [#167]

- WDL
Published by dpark01 over 5 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.8.0

Update to Dockstore 1.9 API - direct github apps based integration for all branch commits and tagged releases to viral-pipelines to Dockstore (skips viral-ngs-staging intermediary) using their latest API released a few days ago [#139, #140]

Workflow updates: - fixes to NCBI FetchSRAtoBAM [#138] - new workflow classifysingle (a single-sample version of classifymulti) [#133] - new mafftandsnp [#132, #133] - replace all augur align calls with our own mafft invocation that is much faster [#132] - augurfrommsa can take multiple keeplist inputs [#134] - allow kraken2 task to take fasta input, not just bam (to allow for running on contigs) [#127]

Misc: - various upstream docker image version updates [#126, #136, #137, #138] - VM shape tuning [#128, #129] - dxWDL update [#131]

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.4.1

• assemblerefbased: additional stats output (`numreadgroups,numlibraries,disttorefsnps,disttorefindels) and more stringent default coverage requirements forrun_discordance` stats [#125]
• correct semantic versioning to match viral-core base version 2.1.4

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.0.4

assemblerefbased workflow: - default to minimap2 aligner instead of novoalign [#123, #110] - add rundiscordance task and output metrics for measuring discordance between sequencing runs within a sample [#124] - disable ivar trim quality trimming due to unidirectional trim behavior (only run ivar trim for PCR primers) [#122] - remove MultiQC from refbased workflow [#82]

metagenomics: - bugfixes for kraken2 build [#120, #117] - bugfixes for kaiju task [#113] - remove blastx (optional) tasks from classify_multi [#102, #106]

BEAST (GPU): - increase dnanexus runtime limit and specify dx_instance type [#116, #118] - update docker image [#111]

nextstrain / phylogenetics: - add snp-sites task to emit vcf files [#115] - add augur mask step [#99] - VM shape tuning [#112] - rename and reorganize workflows a bit so they are all called augur_from_xxx [#121, #101]

miscellaneous: - demux workflows: append lane number to bam filenames [#98, #100] - update viral-core [#105, #124] - update dxWDL [#103]

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.0.2

• update nextstrain/base docker tag to build-20200529T044753Z [#95]
• add sequence counts (pre/post) as Int WDL task outputs for filter_subsample_sequences [#96]

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.0.1

• update ncbi-tools docker image to tagged version with sra-tools v2.10.7
  • bugfix: correct description parameter name for auger export

- WDL
Published by tomkinsc about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.1.0.0

• assemble_refbased changes [#83, #84]:
  • bugfix: workflows should never fail when no genome is produced. Empty output files are now properly emitted with no errors.
  • increase default min_coverage for consensus calling from 2 major allele reads to 3
  • bump ivar docker images to 1.2.2
  • update documentation
• buildaugurtree changes [#85, #90]
  • add augur clades task and integrate with main workflow
  • integrate existing augur filter task with main workflow
  • increase AUGUR_RECURSION_LIMIT to handle very deep/large trees
  • update documentation
• filterbamtotaxa bugfix for `taxonomicids` optional input [#92]
• add align_and_count_multiple_report for improved/bugfixed spike in count handling [#91]
• VM runtime reshaping [#88, #89, #92]
  • disable GCP preemption for certain long running tasks
  • expose disk size requirements for certain tasks and increase their default values
  • increase default RAM for rmdup_ubam task
• update viral-core (and downstream docker images) to v2.1.0

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.21.5

• added vcf merging workflow and tasks (with snpEff annotation) [#61, #77]
• update dnanexus default kraken2 database [#80]
• Travis: bugfix for dockstore & viral-ngs-staging for tagged releases failing to sync [#78]
• Travis: reduce the number of CI tests for cromwell-on-local-travis instead of running the full set (for build speedups)
• update to viral-phylo 2.0.21.5
• genbank workflow updates [#79, #81]:
  • switch from mafft MSA-based alignment before annotation transfer to a scattered pairwise alignment prior to annotation transfer[#76]
  • add travis test coverage for both single-segment and multi-segment genomes to cover both code paths
  • RTD documentation updates

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.21.4

• new workflows for beast: beast_gpu, beasttoauspice [#66, #74]
• new workflows: subsamplebymetadata and newicktoauspice [#68]
• Genbank: improved documentation of submission process and slight workflow updates [#51, #71, #75]
• Travis build: github staging / dockstore -- no longer tag non-master branch commits (and instead rely on unversioned branch labels) [#72]
• Travis build: update dxWDL 1.46.4 to 1.47 to reduce dxda file localization instability [#70]

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.21.3

Travis CI fixes and tweaks

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.21.2

Major changes: - travis CI changes and cleanups, including new flattened-wdl deploy to github.com/broadinstitute/viral-ngs-staging [#62, #63] - updates to nextstrain workflows: adds tasks for augur traits, augur filter, and inserts augur traits optionally into the main workflow [#59] - metagenomics: addition of new workflows for classifykraken2, classifymulti (which demonstrates kraken2-based human read depletion) as well as tasks for kraken2-based classification and database building, as well as blastx-based classification. Main workflows such as demuxplus or assembledenovo have not yet changed default behavior until we get comfortable with the new workflows over various data sets. [#60]

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.21.1

This new release: - collapses a number of redundant workflow wdl files into a singular workflow with branching logic depending on user options -- in particular, all of assemble_denovo_*.wdl has been collapsed into assemble_denovo.wdl with options defining whether to deplete, call isnvs, etc. - upgrade viral-core to 2.0.21, which removes unaligned read deduplication (via MVicuna) as a default pass from the taxonomic depletion step. A separate WDL task (taskreadutils.rmdupubam) is now provided to perform this function and is added into to various workflows (such as assembledenovo*) where this is useful, but omitted from others (assemblerefbased, demuxplus) where it is not. - adds a fastqtoubam workflow that converts paired or single-end fastq reads to an unmapped bam file with appropriate read group headers [#57] - bug fixes to multiqc task to allow for multiple input files that share the same basename (but different directory paths) -- this is a common scenario for nextseq flowcells or any other scenario where the exact same libraries are sequenced on multiple lanes of a flowcell. [#58] - slight updates to buildaugurtree workflow [#48]

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.21.0

Updates: - Convert all remaining WDL to v1.0 of language spec [#50, #53] - Add WDL memory parameterization to all tasks [#43] - Add multiqc in demuxplus and bugfix for execution on PAPI [#42, #54] - Add new trimal task and workflows [#44] - Rename spikein tasks and workflows to alignand_count [#55, #1] - Bump viral-core to 2.0.21 -- includes illumina demux barcode counting optimization [#52] - auto doc build on readthedocs works again [#47] - Travis CI: re-enable cromwell tests and reorganize build stages a bit

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.20.3

PR #41 -- adds continuous deploy to GCS, with updated documentation to reflect that.

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.20.2

add merged krona reports (#40) to krakenuniq task and demux_plus workflow

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.20.1

This release includes some updates to the CI build, to reference based assembly workflows, and adds nextstrain workflows, along with other smaller tweaks. - add nextstrain augur workflow (#39) - add bams_multiqc workflow (#34) - improvements and updates to refbased assembly workflow, including accepting multiple bams per sample at input, scattering alignment jobs by bam, and merging and reheadering as needed (#33, #36) - VM tuning (#37, #31, #29) - update viral-core to 2.0.20.1 - update viral-assemble to 2.0.17.0 - changes to Travis CI build structure, with increased reliance on miniwdl (#38)

- WDL
Published by dpark01 about 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.17.0

• Bump viral-core and viral-assemble to v2.0.17.0, updates Picard and Novoalign, among other things
• Modernize assemblerefbased workflow and incorporate iVar primer trimming
• Add MultiQC into WDL workflows
• Tune VM/instance shapes for demux
• Parallelize post-demux fastqc runs

- WDL
Published by dpark01 over 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.15.0

- WDL
Published by dpark01 over 6 years ago

https://github.com/broadinstitute/viral-pipelines - v2.0.11.0

Very first working version post-refactor!

- WDL
Published by dpark01 over 6 years ago