Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of https://github.com/broadinstitute/long-read-pipelines

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.56

Updated to GATK 4.6.1.0 and added new parameter for extra memory for joint calling (#505)

  • Updated to GATK 4.6.1.0 for joint genotyping.

  • Updated the following tasks to use GATK 4.6.1.0:

    • GnarlyGenotypeGVCFs
    • GenotypeGVCFs
    • ImportGVCFs

  • Added parameter for extra memory to ImportGVCFs

- Jupyter Notebook
Published by github-actions[bot] 11 months ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.55

Added workaround for batch python issue. (#504)

- Jupyter Notebook
Published by github-actions[bot] about 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.54

Fix to HRP2/3 deletion detection for GCP batch backend. (#503)

  • Updated HRP2/3 workflow to fix Batch backend issues.

  • Now uses File for the bam file input.

- Jupyter Notebook
Published by github-actions[bot] about 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.53

Fixes to runtime attributes for workflows for new GCP Batch engine (#501)

  • Fixed all bootDiskSizeGb entries to be ints.

- Jupyter Notebook
Published by github-actions[bot] about 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.52

Set Conda py version for miniwdl (#502)

  • set to use latest miniwdl version

  • set python version to 3.12 in CI/CD

  • set python version to 3.13 in CI/CD, set conda environment to 3.12

  • Added composite action.yml

  • added checkout before setup env action

  • new line in setup env yml

  • setting lfs set to true

  • setting lfs set to true to cd and ci_nightly

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.51

Added base_bucket parameter to reports to fix issue. (#499)

- Jupyter Notebook
Published by github-actions[bot] about 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.50

Create stand-alone workflow for short read malaria QC (#498)

  • Added QCAssessment workflow to run stand-alone QC on ILMN samples. (Primarily for malaria work)
  • Added QC WDL to dockstore.

- Jupyter Notebook
Published by github-actions[bot] about 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.49

Added stand-alone workflow for barcode/fingerprint extraction (#497)

  • Added workflow to extract SNP barcodes / fingerprints from a VCF (ExtractSnpBarcodeFromVcfFile)
  • Added ExtractSnpBarcodeFromVcfFile to dockstore.

- Jupyter Notebook
Published by github-actions[bot] about 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.48

Changed docker image for mosdepth to use gcr instead of quay (#496)

- Jupyter Notebook
Published by github-actions[bot] about 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.47

Improvement: Add updated bubble plots to summary reports (#495)

  • fix reading drug resistance text file and change page title

  • fix another spot parsing the DR text file

  • add new bubble plot with support for hom/het/missing variants

  • update docker image tag to latest

  • change source of drug resistances to workspace table, not text file

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.46

Fixed drug resistance calculations for missing data; added marker output to data table (#494)

  • Fixed bug where functionally annotated output VCF would only be set if the gcs_output dir was specified.

  • Removed all finalization steps from PfalciparumTypeDrugResistanceMarkers.

  • Fixed drug res analysis to handle missing vs hom_ref

  • Updated CreateDrugResistanceSummary to use missing and hom_ref

  • Added individual marker outputs.

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.45

Add WDL Workflow: Generate summary reports for malaria samples (#493)

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.44

Added minimum for miniwdl version to use (#492)

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.43

Updated pip cache step to version 4 (#491)

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.42

Added DeepTrio Workflow (#490)

  • Added DeepTrio task to DeepVariant.wdl

  • Added DeepTrio wdl with workflow to call DeepTrio.

  • Added DeepTrio workflow to .dockstore.yml

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.41

set a single python version in git_pages (#489)

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.40

Added system deps in nightly test (#488)

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.39

Made MarkDuplicates optional to enable Amplicon data processing. (#486)

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.38

Updated python version in git actions from 3.7 to 3.13 (#487)

  • Updated python version in git actions from 3.7 to 3.13

  • Update python setup action version from 2 to 4

  • Update python setup action version from 4 to 5

  • Install libbz2-dev in deps

  • Added additional deps in "Install dependencies"

  • update tox py version from py37 to 313

  • created separate step for system deps, added step to make sure miniconda is installed, added cache for pip deps

  • removed miniconda step, replaced the system dep installation step

  • Added system deps and pip caching to cd.yml

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.37

Fixed typo in zarr output file. (#484)

  • Made zarr output file optional

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.36

Made zarr conversion optional in SRJointCallGVCFsWithGenomicsDB.wdl (#483)

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.35

Added snpeffdbidentifier for Functional annotation. (#482)

  • Removed hard-coded plasmodium snpeff database specification.
  • Added new input snpeff_db_identifier for a snpeff database specification to SRJointCallGVCFsWithGenomicsDB and FunctionalAnnotation`

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.34

Reserved more non-heap memory for the JVM when calling HaplotypeCaller (#481)

  • Updated HaplotypeCaller task to reserve 20% or 4gb of memory for off-heap use (whichever is larger). This prevents out of memory errors occurring in native libraries that HaplotypeCaller calls into (i.e. the GKL).

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.33

SRJointCallGVCFsWithGenomicsDB.wdl now prefers dbsnp_vcf to known_sites_vcf for genotyping (#480)

  • Added ResolveMapKeysInPriorityOrder to resolve keys in a WDL Map object in priority order. This enables fallbacks / preferences for different entries in a map (such as the refmap).
  • For genotyping with the gnarly genotyper, the dbsnp resource now prefers dbsnpvcf from the RefMap. If this does not exist, it will fall back to knownsitesvcf. This allows different resources to be used for BQSR (which prefers knownsitesvcf in SRFlowcell) and the gnarly genotyper, which now prefers dbsnpvcf.

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.32

Updated SRUtils.BamToFq and SRBamToFq with cram support and better memory / cpu settings (#479)

  • Made finalization optional in SRBamToFq.
  • Now BamToFq sorts and converts in 2 separate commands.
  • Now BamToFq utilizes all available processors.
  • BamToFq cpu and ram now 8 cores and 16gb respectively.

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.31

Fixed unbound variable error in some cases of QC failure. (#478)

  • When $mosdepthqcstatus was true and $callable_status was false, there was a bug that caused an unbound variable exception (message had not been defined in that code path).

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.30

Fixed bug when workflow had no inputs/outputs. (#477)

  • Fixed the bug where if a workflow had no inputs or outputs the backup process would fail due to a python KeyError.

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.29

Increased minimum boot disk size to 25Gb for all tasks. (#476)

Fixed critical issue with boot disk size in Terra. Google likely changed their image and the boot disk is no longer big enough. Updated boot disk sizes to be 25 gb (or larger) by default.

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.28

Added more QC metrics and a pass / fail assessment to SRFlowcell (#475)

  • Changed read quality stratification numbers to better suit ILMN data in SRFlowcell.
  • Added QC pass / fail assessment to SRFlowcell. Defaults are set up for P. falciparum short read data, but can be adapted to other use cases.

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.27

Added support for giving background samples to joint calling workflow. (#474)

This adds in the ability to specify an Array[Array[File]] containing GVCFs to include as background samples in a joint call in SRJointCallGVCFsWithGenomicsDB. This input is optional.

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.26

Updates for Malaria pipelines: human read decontamination and WDLs for on-prem Malaria pipelines (#473)

  • Added WDLS for Broad on-prem Malaria Pipelines
  • Updated dockstore with Broad on prem malaria pipelines
  • Finished implementing bowtie2 based read decontamination
  • Enabled read decontamination in SRFlowcell
  • Updated Utils.SortSam to use samtools

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.25

Updated interval extraction for short read joint calling for sharding. (#471)

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.24

Fixed JVM memory spec for VariantUtils::AnnotateVcfWithBedRegions (#469)

- Jupyter Notebook
Published by github-actions[bot] over 1 year ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.23

Updated drug resistance marker code to account for het genotypes. (#467)

  • Updated drug resistance marker code.

  • Now PfalciparumTypeDrugResistanceMarkers::CallDrugResistanceMutations pulls info directly from the VCF file using pysam, rather than a series of shell commands.

  • Now PfalciparumTypeDrugResistanceMarkers::CallDrugResistanceMutations preserves het/hom genotype information in the raw output file.

  • PfalciparumDrugResistanceSummary.wdl now calls into PfalciparumTypeDrugResistanceMarkers for CreateDrugResistanceSummary.

  • Minor refactoring of names.

  • Added VCF Index as input to PfalciparumTypeDrugResistanceMarkers.wdl

  • Fixed output names in PfalciparumDrugResistanceSummary.wdl.

- Jupyter Notebook
Published by github-actions[bot] almost 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.22

Updates to ONT workflows to bring them closer to current conventions (#466)

  • Documentation updates for SRWholeGenome.wdl and ONTWholeGenome.wdl
  • VariantUtils::MergeAndSortVCFs now has a fall-back to bcftools concat with normal compression if --naive fails.
  • Fixed VariantUtils::MergeAndSortVCFs to use lr-basic:0.1.2
  • CallVariantsONT.wdl now takes a ref_map instead of reference fasta/fai/dict.
  • CallVariantsONT.wdl now filters out the mt_chr_name if call_small_vars_on_mitochondria is set to false.

- Jupyter Notebook
Published by github-actions[bot] almost 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.21

Hotfix for BackupWorkspace.wdl and minor change to the hrp2/hrp3 detection wdl. (#464)

  • Added volatile: true to `BackupWorkspace.wdl task.

  • Increased disk space for PfalciparumHrp2Hrp3Status.wdl

  • Fixed disk size to depend on all inputs for PfalciparumHrp2Hrp3Status.wdl

- Jupyter Notebook
Published by github-actions[bot] almost 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.20

Changed output names of drug resistances in Pf drug resistance WDLs. (#463)

  • Names of drug resistance outputs in PfalciparumTypeDrugResistanceMarkers.wdl have been changed to the form: drug_status_DRUGNAME (e.g. drug_status_chloroquine)

- Jupyter Notebook
Published by github-actions[bot] almost 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.19

Cleaned up short read workflows, minor feature additions (#461)

  • Added docker image and WDL to backup workspaces.
  • Added util-malaria-coi docker image.
  • Fixed bug in SRFlowcell that preferentially pulled the existing aligned bam rather than the merged aligned bam.
  • Cleaned up redundant / unnecessary inputs and outputs from SRWholeGenome.wdl
  • Enabled SRWholeGenome.wdl and HaplotypeCaller.wdl to use an interval_list to call variants in subsets of the genome.
  • Renamed ONTPfTypeDrugResistanceMarkers.wdl to PfalciparumTypeDrugResistanceMarkers.wdl.
  • Renamed ONTPfHrp2Hrp3Status.wdl to PfalciparumHrp2Hrp3Status.wdl
  • PfalciparumDrugResistanceSummary.wdl now produces individual drug resistances/sensitivites as additional outputs.
  • PfalciparumDrugResistanceSummary.wdl is now deprecated.
  • PfalciparumDrugResistanceMarkers.wdl now produces drug resistances/sensitivites as well as the summary file. This deprecates PfalciparumDrugResistanceSummary.wdl.
  • Added PfalciparumPolygenomicityBarcodeEstimate.wdl to estimate CoI based on the 24 SNP molecular barcode.
  • Cleaned up some outputs in SRJointCallGVCFsWithGenomicsDB.wdl.
  • Added flag for dangling end recovery to HaplotypeCaller.wdl.
  • Minor fixes for MiniWDL style suggestions (TrainCnnFilters.wdl, LRCNVs.wdl, AlignedMetrics.wdl, FastQC.wdl, SRUtils.wdl, Utils.wdl, VariantUtils.wdl, SRJointGenotyping.wdl, NanoPlot.wdl, Pf_Niare_HaplotypeCaller.wdl)

- Jupyter Notebook
Published by github-actions[bot] almost 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.18

Reworked SRJointGenotyping sharding to be configurable at runtime. (#455)

  • Sharding now splits the VCF files into chunks by number of bp. This allows for wider sharding / better scalability for large callsets. By default the behavior is to shard by contig - this is controlled by setting the number of bp per shard to be very large (999999999). If the number of bp per shard is larger than the contig size, the whole contig is used.

  • Changed MakeSitesOnlyVCF task to use bcftools.

  • Updated joint call WDL to accept interval lists.

- Jupyter Notebook
Published by github-actions[bot] almost 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.17

Made the finalize steps optional for workflows in the srmalaria workspace. (#453)

  • Made gcs_root_out_dir optional in SRFlowcell
  • Made gcs_out_root_dir optional in SRWholeGenome
  • Made gcs_out_root_dir optional for SRJointCallGVCFsWithGenomicsDB

- Jupyter Notebook
Published by github-actions[bot] about 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.16

Updated short read variant calling and joint calling pipelines to run at scale (#450)

Single-sample variant calling:

  • Fixied critical typo in HaplotypeCaller disk spec.
  • Fixed wdl-computed divide by zero error in SRFlowcell outputs.
  • Updated MergeVCFs to have an option to name an output as a GVCF.
  • Removed deprecated GC logging flags from GATK commands.
  • Fixed issue in FastQC if no read qualities are in the file.
  • Added missing annotation groups to ReblockGVCFs.
  • HaplotypeCaller WDL now returns the reblocked GVCF.

Joint Calling: * Added option to use gnarly genotyper * Added het inputs to joint genotyping * Fixed java memory allocation in joint genotyping to be based on memory of the VM, not hard-coded * Fixed name of outdir in ConvertToZarrStore to be correct for this workflow. * Updated the zarr conversion to use parallel Dask processes and to log to stdout. * Upped default zarr conversion memory to 32gb.

VETS: * Added stack trace logging for errors in ExtractVariantAnnotations, TrainVariantAnnotationsModel, and ScoreVariantAnnotations. * Removed HAPCOMP, HAPDOM, and HEC from default annotations for SNP and INDEL VETS filtration on joint-called VCF files. Need to do more testing / debugging to include these in joint calling.

Misc: * Fixed the name of the workflow in ExpandedDrugResistanceMarkerAggregation to reflect the actual name. * Miscellaneous updates for debugging

- Jupyter Notebook
Published by github-actions[bot] about 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.15

make submissions folder deletion better (#445)

  • make submissions folder deletion better

- Jupyter Notebook
Published by github-actions[bot] about 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.14

Several updates for making Malaria Joint Calling easier. (#449)

  • Reintroduced ReblockGVCF step.

  • Updated HaplotypeCaller::ReblockGVCF to point to latest Docker image with fix for ReblockGVCF annotations.

  • Updated HaplotypeCaller::ReblockGVCF to use 2 cores.

  • Disabled DeepVariant/Pepper calling in SRWholeGenome by default.

  • Removed SRJointGenotyping::ReblockGVCF - it should only be defined in one place.

  • Removed hard filtered output file.

  • Updating all GATK 4.3 tasks to GATK 4.5

  • Disabled QC when running on a singe bam file input.

  • Added a runtime_attr override for HaplotypeCaller subworkflow.

  • Update Utils.wdl

Modified the disks from " LOCAL" to " SSD"

  • Updating tasks to use SSD rather than LOCAL disk.

  • Moved HaplotypeCaller and ReblockGVCF to SSD from HDD

  • Fixed a bug in compute_sr_stats.py that allowed nans

  • Updated sr-utils docker to use mamba conda env solver.

  • Updates to sr-utils docker image.

  • Updated conda solver in sr-utils to mamba.

  • Fixed minor deprecation warning in compute_sr_stats.py

  • Fixed issue in compute_sr_stats.py that caused certain inputs to fail due to missing base qualities or nan values.

  • Updated version of sr-utils docker image to 0.2.2.

  • Updated sr-utils docker image to version 0.2.2

  • Added note for updating nightly GATK build.

Co-authored-by: Shadi Zaheri 74751641+shadizaheri@users.noreply.github.com

- Jupyter Notebook
Published by github-actions[bot] about 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.13

update dockstore cli in git action yml (#447)

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.12

Add WDL Workflow: Plots metrics for SV Calls for Multiple Samples (#438)

  • SvQcplots.wdl draft

  • Added concat sv stats

  • added compile sv stats by sample

  • added runtime in compileSVstats

  • added addcoverage task

  • added plotting task

  • adding docker for plotting

Co-authored-by: Steve Huang shuang@broadinstitute.org

  • Update wdl/pipelines/TechAgnostic/Visualization/SvQCPlots.wdl

specified bcfQuerySV task docker

Co-authored-by: Steve Huang shuang@broadinstitute.org

  • add meta section to SvQCPlots.wdl

  • add option to name output plot notebook SvQCPlots.wdl

  • Minor additions (#444)

Co-authored-by: Steve Huang shuang@broadinstitute.org

  • update docker to ubuntu for addCoverageToSVstats

  • containerized compilesvstats

  • added a full path to compilesvstats.py

  • fix compilesvstats.py svtypes

  • updated papermillbase docker version build

Co-authored-by: bshifaw bshifaw@broadinstitute.com Co-authored-by: Steve Huang shuang@broadinstitute.org

- Jupyter Notebook
Published by github-actions[bot] about 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.11

Adding Base Image for Papermill Docker (#437)

  • adding docker for papermill

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] over 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.10

Adding in all the work from the long-term Special Project Malaria branch (#440)

  • Adding in all the work from the long-term Special Project Malaria branch

- Jupyter Notebook
Published by github-actions[bot] over 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.9

Added section on external contribution to development Readme (#435)

  • Added section on external contribution

  • Added links to repo site to main repo README.md

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] over 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.8

Merge pull request #428 from broadinstitute/tws_cleanup

repo cleanup

- Jupyter Notebook
Published by github-actions[bot] over 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.7

trivial update to .gitignore

- Jupyter Notebook
Published by github-actions[bot] over 2 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.6

Fix dockstore validator step in yml to insure action fails if step fails (#415)

  • Updated java version

  • manually checking dockstore validation results

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] almost 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.5

fix a missing space in dockstore.yml (#413)

- Jupyter Notebook
Published by github-actions[bot] almost 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.4

Send nightly status message to #lrma-repos, rather than #lrma (#411)

- Jupyter Notebook
Published by github-actions[bot] almost 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.3

add CleanupIntermediate to dockstore.yml (#409)

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.2

deprecate sniffles-1 and CLR WG workflow (#408)

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.1

finalize merged FASTQ in PBAssembleWithHifiasm (#405)

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_4.0.0

Major changes from the last major release include

  • WDL directory structure change which organizes .wdls into specific analysis folders
  • Actions to generate a repository Git page to list WDL workflow documentation and Developer Guide

- Jupyter Notebook
Published by bshifaw about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.76

Update Continuous integration to use bumpversion tool (#404)

  • Updated cd.yml to use bumpversion tool action

  • Updated .bumpversion.cfg to specify tag_name and have version match current repo version

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.75

Add bumpversion cfg (#403)

  • add .bumpversion.cfg

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.74

Update task descriptions (#402)

  • updated description to wdl files in task Alignment, Assembly, and Preprocessing

  • updated description to wdl files in task Transcriptomics

  • updated description to wdl files in task QC

  • updated description to wdl files in task Utility

  • updated description to wdl files in task VariantCalling

  • fixes to pass miniwdl validation

  • fixes to pass womtool validation

  • Update wdl/tasks/QC/Fingerprinting.wdl

  • Update wdl/tasks/Utility/ONTUtils.wdl

Co-authored-by: Steve Huang shuang@broadinstitute.org

Co-authored-by: bshifaw bshifaw@broadinstitute.com Co-authored-by: Steve Huang shuang@broadinstitute.org

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.73

development docs draft (#396)

  • development docs draft

  • added intro for dockerstyleguide.md

  • add to dockerstyleguide.md

  • add to wdlstyleguide.md

  • renamed devguide to developmentguide

  • rewrote overview, added visuals for directory structure

  • Minor fixes, Added Workflow Deployment in README.md

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.72

Fix for Repo Git Page action yml (#398)

  • added make docs dir if not exist

  • add graphviz as an installation requirement

  • switching to mac environment

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.71

Automatically Generate Repository Git Page (#391)

  • add git page action

  • Add scripts to create dot and png files for workflows

  • Add scripts to add png link to md

  • Add mkdocs.yml file

  • changed png link to be relative path instead of absolute.

  • Added script to add header to wdl markdown, and remove task info from wdl workflow markdown

  • edited wdlviz.py to add the option to name the output file by a specified name, edited createwdlvisual.py to use wdl basename for dot file

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.70

New workflows to deal with ONT flowcells that have multiple basecall directories (#325)

  • one workflow that essentially copies the current ONTFlowcell, but aims at a basecall directory
  • one workflow that plays the role of ONTFlowcell, but actually aggregates results from the WF above

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.69

add miniwdl and shellcheck installation to cd.yml (#397)

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.68

reheader FP vcfs from full hg38 to no_alt (#344)

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.67

Move wdl files to central directory called wdl (#393)

  • moved wdls to wdl folder, updated testing location for wdl validation

  • updated wdl paths in dockstore.yml

  • renamed 'deprecated_wdl' to 'deprecated', updated docstore.yml accordingly

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.66

Standardize wdl description (#392)

  • standardized pipeline/ONT workflow meta and parameter_meta bocks

  • standardized pipeline/pacbio workflow meta and parameter_meta bocks

  • standardized pipeline/techAgnostic workflow meta and parameter_meta bocks

Co-authored-by: bshifaw bshifaw@broadinstitute.com

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.65

Update hifiasm assembly (only) workflow (#390)

  • MAJOR CHANGES:

    • HiFiasm
    • update hifiasm docker
    • formulated resource allocation
    • As requested, pipeline generates alternate contigs as well. Note this essentially forces two assemblies on one set of FASTQs, as haplotig-generating assemblies and alternate-generating assemblies are generated with different options.
    • QUAST
    • update and shrink Quast docker
    • bigger disk for Quast task
    • now supports multiple assemblies being evaluated at the same time
    • It's almost reference-free now:
    • variant calling is no longer offered
    • only depends on a reference (e.g. T2T) for assembly evaluation (Quast)

  • MINOR CHANGES:

    • update various tasks
    • optionally compress files when finalizing
    • merge MergeFastqs and MergeFastqGzs into one task
    • more memory for MergeFastqs; uses LOCAL SSD when files are not too large

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.64

Kvg smrtlink updates (#389)

  • Fix to handle a change in where SMRTLink keeps the run date field

  • Updated docker image and WDL tasks to make use of updated PacBio metadata parser

  • Specify version number for lr-basic docker image in MergeAndSortVCFs

  • Fixed an issue where Sniffles2 wasn't always run when it should be

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.63

Kvg update downloaders (#379)

  • Updated cloud downloaders

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.62

Rev to Guppy 6.4.6 (#386)

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.61

Adding WDL validation with miniwdl to nightly and push tests (#388)

  • added test for miniwdl validation

  • added shellcheck to push and nightly actions so that miniwdl can validate command in tasks

  • Longbow.wdl and PBFlowcell.wdl changes to satisfy miniwdl validation

  • Postprocessing_Tasks.wdl, Longbow.wdl, and MASSeq.wdl changes to satisfy miniwdl validation

  • Assigning ConvertToHailMT call an alias to avoid having similar names as wdl workflow name

  • miniwdl 1.9.0 does not allow concatenating optional variables with none-optional Strings. Using select_first to set optional to none-optional.

  • fix womtool validation error for ONTFlowcell.wdl regarding "Cannot dereference a Map[String, String]? value using a String key"

  • updated wdls such that they user separate instead of sep

  • add end of file line Structs.wdl

Co-authored-by: bshifaw bshifaw@broadinstitute.com Co-authored-by: Steve Huang shuang@broadinstitute.org

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.60

WDL Reorganisation (#384)

  • Add pipeline, tasks, and structs folder to root dir.

  • update pbccs.wdl, AlignAndCheckFingerprintCCS.wdl path in dockstore.yml

  • updated several import paths for wdl workflows

  • added dockstore validation action, updated dockstore.yml

  • "git moved" wdl workflows to pipeline folder

  • "git moved" wdl workflows to task folder

  • Renamed wdl to deprecated_wdl

  • Added ReadMe.md deprecated, updated imported for deprecated wdls

  • removed PBCCSNoSharding.wdl and ExtractAouData.wdl from dockstore.yml

  • updated wdl validate py script to point to new dir

  • added getreversewdl_deps.py

  • added draftreleasenotes.py, script to get wdls edited between two branches

  • removed MalariaAssembly.wdl from .dockstore.yml

  • add ArgumentParser for draftreleasenotes.py

  • minor change to raising error for missing file in testwdlvalidity.py

Co-authored-by: bshifaw bshifaw@broadinstitute.com Co-authored-by: Steve Huang shuang@broadinstitute.org

- Jupyter Notebook
Published by github-actions[bot] about 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.59

Merge pull request #377 from broadinstitute/kvg_glnexus

Added GLNexus GVCF joint calling.

- Jupyter Notebook
Published by github-actions[bot] over 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.58

Terra changed where submissions go (now at gs:///submissions/) Some small optimizations too.

- Jupyter Notebook
Published by github-actions[bot] over 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.57

update help message with -g

- Jupyter Notebook
Published by github-actions[bot] over 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.56

fix an error in the step checking multiple bams sharing the same sample name

- Jupyter Notebook
Published by github-actions[bot] over 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.55

add sample_id to workflow (#369)

- Jupyter Notebook
Published by github-actions[bot] over 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.54

Create Sniffles2 Docker and workflow. Update variant calling pipeline with Sniffles2. (#365)

- Jupyter Notebook
Published by github-actions[bot] over 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.53

Merge pull request #368 from broadinstitute/jtsupdatelongbowfor3pumifix

Updated longbow to v0.5.40 for 3' UMI length fix.

- Jupyter Notebook
Published by github-actions[bot] over 3 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.52

Significantly reduce NanoPlotFromBam wall-clock time * use a new docker which supports parallelization * drop supp alns

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.51

Merge pull request #360 from broadinstitute/jtsupdatemasseqlongbowversion

Updating to longbow v0.5.39 to fix pomegranate heap overflow.

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.50

Merge pull request #357 from broadinstitute/jtsfixtagsformasseqv0

Added single-cell tag cleanup for v0 of the MAS-ISO-seq pipeline

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.49

Merge pull request #356 from broadinstitute/jtsmas-seqoptimizations

  • Reduced cost of pbflowcell for MAS-ISO-seq samples.
  • Added step to fix longbow tags that have position information for reads that are reverse-complemented.

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.48

Merge pull request #340 from broadinstitute/kvgpbmasseq

Bring the MAS-ISO-seq workflows into our production pipelines

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.47

New workflow that follows up to process the result of the demultiplexing workflow

  • performs alignment to specified reference (practically avoid sharding BAMs as we've seen issues of sharding demultiplexed BAMs)

  • performs fingerprint verification by default (offers option to turn off FP check)

  • collects alignment metrics

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.46

New unified workflow for handling barcoded & on-instrument CCSed SMRTCells

  • summary on raw input PBI
  • smrttools-based metrics generation
  • demultiplexing and demultiplexing 2ndary metrics generation and check all expected barcodes are detected
  • default to extract Hifi reads from the input CCS bam, and offers choice if demultiplexing with CCS bam or Hifi bam
  • printout processing date on success

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.45

A quick fix of pb align task (#341)

  • use tab explicitly, this avoids errors when lab enters spaces in the RG line

  • ignore vscode stuff

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.44

randomly assign zones to resource-demanding tasks (#328)

  • DV-Pepper-Margin subworkflows
  • Clair
  • PBSV

random zone generation happens only once per genome, attempting to balance wall-clock time and cross-zone egress

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.43

Kvg pf drug resistance (#337)

  • Added a simple workflow to type some common drug resistance markers in P. falciparum.

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.42

Kvg lr malaria cigass ucad (#336)

  • Workflow to test HRP2/HRP3 presence/absence in P. falciparum data

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.41

Parse on-instrument CCS report (#334)

  • test HA test sample

  • found better way to check and re-org a bit

- Jupyter Notebook
Published by github-actions[bot] almost 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.40

Fix PB run info (#332)

  • tiny fix for detectruninfo.py, update dockers accordingly

  • fix how to detech if CCS'ed

- Jupyter Notebook
Published by github-actions[bot] about 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.39

default resource allocation for DV-pepper pipeline was off (#333)

- Jupyter Notebook
Published by github-actions[bot] about 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.38

minimum disk 1 (#331)

- Jupyter Notebook
Published by github-actions[bot] about 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.37

Small fixes to PBFlowcell (#326)

  • handle updated ccs report
  • new task for computing appropriate # of LOCAL ssd disks due to new PAPI change
  • to increase throughput, use randomly assigned zones for ShardLongReads (avoiding issue 324)

- Jupyter Notebook
Published by github-actions[bot] about 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.36

Update FP workflow (#318)

  • allow users to specify explicitly which gt VCF to use

  • fix a bug where VCF filtering may filter away all contig lines in the header

- Jupyter Notebook
Published by github-actions[bot] about 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.35

Some cleanup (#320)

  • drop longshot from pipeline calls (not removing it all together yet)

  • check on sample names when merging

  • avoid pipefail when there're multiple read groups

- Jupyter Notebook
Published by github-actions[bot] about 4 years ago

https://github.com/broadinstitute/long-read-pipelines - lrp_3.0.34

Misc updates to the WGS pipelines (#302)

  • Separate small, SV call, and:

    • FixSnifflesVCF guard against when only a subset of issues exist in the VCF
    • Drop Clair3 from CLR data, as that's not supported

  • FOUR SMALL FIXES:

    • guard against spaces in LB field in input BAM....
    • optionally check PacBio ShardLongreads outputs (sometimes EOF mark missing)
    • use DV 1.3.0, bundled in custom docker with resource monitoring script
    • improved PBSV resource allocation based on scatter scheme choice, as PAPI 10 observed for high coverage data when not proecssing per chr

  • provide default values to WGS pipelines

- Jupyter Notebook
Published by github-actions[bot] about 4 years ago