Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of multiqc

multiqc - v1.31

MultiQC v1.31 - 2025-09-05

Adding new module for Xenium analysis, 10x Genomics Xenium spatial transcriptomics quality control report.

The parquet format is stable since 1.29, renaming the output file from BETA-multiqc.parquet to multiqc.parquet.

New modules

Feature updates and improvements

  • Add return_html parameter for programmatic HTML access (#3304)
  • File search optimization: avoid reading files when contents exclusion patterns are not provided (#3312)
  • Rename BETA-multiqc.parquet to multiqc.parquet (#3332)

Fixes

  • Scatter plot: fix hiding dots by legend click (#3321)
  • Plots: set unique series label for each plot type (#3330)
  • Fix bulk sample renaming buttons (#3300)
  • Fix config flag types in schema (#3318)
  • Ignore pyc files when copying html files (#3320)

Module updates

  • Picard tools: enhance QualityByCycleMetrics to support original quality scores (#3307)
  • STAR: improve module color scheme for better accessibility (#3305, #3306)
  • Sequali: support insert size metrics (#3303)
  • Missing modules to general stats: Busco, CheckM, CheckM2, GTDB-Tk (#3289)
  • cells2stats: add support for optical pooled screening output (#3277)
  • fastp: add before-filtering mean r1/r2 length to general stats (#3280)
  • Cutadapt: improve sample name extraction for stdin input (#3333)

Module fixes

  • bases2fastq: fix index error (#3328)
  • Picard: fix VariantCallingMetrics to support sample renaming (#3298)
  • fastp: fix sample naming for paired-end reads (#3302)
  • bcftools: fix singleton count calculation to include indels (#3295)
  • CheckM2: column spelling fix (#3283)
  • Bulk replace deprecated hide_empty with hide_zero_cats (#3296)

Infrastructure and packaging

  • WASM workaround: if write_parquet not supported by polars, write a CSV file (#3309)
  • Add Claude instructions (#3301)
  • Add Claude review action (#3299)

- JavaScript
Published by vladsavelyev 9 months ago

multiqc - v1.30

MultiQC v1.30 - 2025-07-09

Minor improvements and fixes.

Feature updates

  • Support markdown in custom content (#3204)
  • Support reasoning models in AI summaries (#3265)
  • Always save AI prompt to file as llms-full.txt (#3257)
  • Allow Plotly theme configuration with config.plot_theme (#3250)
  • Use polars-lts-cpu library for polars for better CPU compatibility across Docker containers (#3240)
  • Allow adding any metric from multiqc_data.json to general stats with config.general_stats_columns (#3245)
  • Box plot: dynamically determine whether to show interactive points for outliers (similar to violin plots). Add boxplot_boxpoints config option to override it explicitly (#3244, #3248)
  • Detect AI provider for summaries from environment variables (#3254)
  • Scatter plot: allow custom marker symbol (#3234)
  • Use universal config.use_filename_as_sample_name config flag for modules, and deprecate module-specific flags (3241)
  • Allow multi-line info sections in module sections (#3225)
  • AI summaries: update the model defaults (default to Claude 4) (#3265)
  • Color schemes: lighten colours with alpha instead of admixing white (#3252)
  • Performance: use OrderedDict for saved_raw_data_keys (#3235)

Fixes

  • Custom content line plots: allow passing data as a list of tuples in JSON to avoid ordering issues (#3246)
  • Fix merging modules from multiple parquet files (#3267)
  • Fix series labels in heatmap subtitle (#3260)
  • Fix custom content description when grouping section with parent_id (#3242)
  • Fix timezone issue in parquet export (#3224)

Module updates and fixes

  • Elembio bases2fastq: fix PE bug, add % bases below PF plot (#3233)
  • Picard: support sentieon for WgsMetrics (#3200)
  • Picard: new OxoGMetrics search pattern (#3212)
  • VEP: support version 114 (#3209)
  • samtools: support ampliconclip stats (#3214)
  • Percolator: handle comments in parsing (#3217)
  • RSeQC: normalise junction saturation plot (#3216)
  • Somalier: fix sex plot description (#3261)
  • FastQC: fix subtitle in status check heatmap and adapters content plot (#3260)
  • Cutadapt: fix extracting sample name from command line params (#3262)
  • UMItools: bump up search pattern checked number of lines (#3263)
  • Samtools: allow configuring metrics to add to general stats with config.general_stats_columns (#3245)

- JavaScript
Published by vladsavelyev 11 months ago

multiqc - v1.29

MultiQC v1.29 - 2025-05-22

Highlights

BETA: MultiQC now writes out an Apache Parquet file multiqc_data/BETA-multiqc.parquet for each run containing plot and table data and metadata. This outputs is useful to:

  • Reproduce MultiQC runs without access to the original data.
  • Combine multiple MultiQC runs across different sampels or modules.
  • Populate an analytics database for e.g. time series analysis.

See the full documentation on using parquet output here.

Note that the format is unstable may change in the following release as we are still exploring the best format for the OLAP integration.

Feature updates and improvements

  • Write plot and table data in a Parquet file (#3118, #3194, #3171, #3175)
  • Support Markdown in custom content (#3204)
  • AI summaries:
    • AWS Bedrock support for AI summaries (#3115) (#3115)
    • Allow customizing prompt sent to LLM (#3131)
    • Ignore formatting errors when generating AI summary from violin plot (#3129)
    • Headers for AzureAI: add config.ai_auth_type (#3122)
  • New CLI command: multiqc --check-config (#3120, #3147)
  • Allow both table ID and table anchor in table custom config (#3190)
  • Add option --only-samples - opposite of --ignore-samples (#3174)
  • Generalize customizing general stats columns (#3164)
  • Use killable processes for kaleido plot export (#3157)
  • Add timeout to plot export (#3111)
  • Customize plot subtitle ("N samples") (#3160)
  • Sort custom-content by section name by default (#3137)

New modules

  • New module: Elembio bases2fastq (#3123)
  • New module: Elembio cells2stats (#3176)

Fixes

  • Avoid crash if failed to remove the tmp dir (#3159)
  • Make sure multiqc.log has full debug log (#3173)
  • MegaQC config attribute fix (#3172)
  • Custom content: fix when section id and plot anchor are identical (#3184)
  • Fix false auto-opening toolbox when SeqeraAI is selected (#3193)
  • Fix applying data_labels extra parameters in plot config (#3181)
  • Handle timezone locale error (#3149)
  • Document multiqc.parse_logs("multiqc.parquet") (#3169)
  • Fix barplot CSV in-browser export (#3148)
  • Fix setting custom_table_header_config for custom table (#3135)
  • Coerce x-values to float when setting x-range on linegraph (#3130)
  • Fix pre-configured table sort defaultsort (#3140)
  • Fix barplot CSV in-browser export (#3148)
  • AI summaries:
    • Fix bug generating AI table from violin plot (#3128)

Module updates and fixes

  • NanoStat: allow customizing general stats columns (#3163)
  • bcftools stats: enable log switch for indel plot (#3196)
  • Bases2Fastq: support handling of single end sequencing data (#3179)
  • Bases2Fastq: support handling of single-end sequencing data (#3189)
  • samtools: generalize customizing general stats columns (#3164)
  • samtools markdup: handle input with absolute paths (#3150)
  • NanoStat: allow customizing general stats columns (#3163)
  • Bases2Fastq: update links and definitions (#3158)
  • PRINSEQ++: fix logger warnings (#3198)
  • fastp: don't display decimal points on filtering result bar chart (#3132)
  • kraken: hide read-count decimals for taxa plot (#3143)

- JavaScript
Published by vladsavelyev about 1 year ago

multiqc - v1.28

MultiQC v1.28 - 2025-03-21

JSON schema for MultiQC config & AI summaries fixes and optimizations.

Feature updates and improvements

  • Add JSON schema for config (#3100)
  • Allow configure dynamic toolbox settings in YAML config (#3102)
  • AI prompt optimizations (#3109)
  • Round table values for AI prompt (#3124)
  • Custom content: support .gif, .webp and .tiff (#3058)
  • Ignore *.jar files for file search (#3097)

Module updates

  • QUAST: add ANI column (#3091)
  • Trimmomatic: use more specific search pattern (#3098)

Fixes

  • AI in-browser summary: fix prompt for string-type cells in tables (#3096)
  • Fix generating AI summaries from custom content table with empty cells (#3095)
  • Bargraph: respect pconfig.use_legend if defined (#3112)
  • Fix warnings when read_count_multiplier: 1 (#3094)

Module fixes

  • VEP: small fix when data in category is missing (#3116)
  • BCLConvert: fix index parsing (#3101)

Optimization, refactoring and typing

  • Move special case modules into core (#3114)
  • Refactor plotting code to save and load input data (#3069)

- JavaScript
Published by vladsavelyev about 1 year ago

multiqc - v1.27.1

MultiQC v1.27.1 - 2025-02-07

Some fixes and improvements to AI summaries generation:

  • Supporting other LLM providers with a custom endpoint passed through the command line (docs)
  • Summaries should now work with bioconda installation without additional dependencies.
  • Option to anonymize sample names.

Feature updates and improvements

  • AI summaries:

    • Add "Chat with Seqera AI" buttons to all summaries (#3067)
    • Add options to configure custom OpenAI endpoint (#3083)
    • Support anonymizing sample names (#3074)
    • Use tiktoken to estimate token counts for all providers (#3064)
    • Remove the dependency on langchain, openai, and anthropic packages (#3066)
    • Always save prompt to file (#3078)
    • Box plot: use statistics instead of raw values (#3077)
    • Fix for in-browser summaries: when "Clear summary" clicked on global, clear also the generated extended summary (#3068)
    • Make token count warning a debug (#3071)

  • Improving table small scatter plots:

    • Support hide, rename, highlight (#3082)
    • Remove legend to make more space (#3081)

  • Configuring from env: resolve empty to None (#3072)

  • Do not print stack trace on KeyboardInterrupt unless verbose (#3063)

  • Custom content:

    • Allow leading empty columns when parsing TSV (#3059)
    • Handle empty input files (#3086)

Module updates

  • Sequali: change DOI to article link (#3084)

Module fixes

  • FastQC: fix "Per Base Sequence Content" heatmap (#3075)

- JavaScript
Published by vladsavelyev over 1 year ago

multiqc - v1.27

MultiQC v1.27 - 2025-01-22

Highlights

You can now generate AI summaries of reports with --ai, or dynamically in existing reports. See documentation for more details.

Feature updates and improvements

  • AI summaries (#2915)
  • Add "Clustered" view to heatmaps (#3035)
  • Config option to control when table "Configure Column" button is disabled (#3045)

Fixes

  • Preserve order of saved_raw_data_keys in multiqc_data.json (#3044)

Module updates

  • Checkm2: support v1.0.1, be less strict to header columns (#3046)
  • Bustools: support 0.44.1 (#3053)

Infrastructure and packaging

  • Update ruff and pre-commit (#3048)

- JavaScript
Published by vladsavelyev over 1 year ago

multiqc - v1.26

MultiQC v1.26 - 2024-12-17

Christmas release! 10 new modules, 11 improved modules, and a few core improvements.

Highlights

  • Custom content: allow quoting strings in CSV to prevent interpreting them as numbers (#3020):
  • Naturally sort samples in tables and bar plots (#2999)
  • Auto-save dynamic toolbox settings (#3001)
  • report.saved_raw_data is not preserved by default to save on memory footprint (#3010). You can enable it with preserve_module_raw_data: true in your config file.

New modules

  • MosaiCatcher - tool for classifying strand states of each chromosome in each cell using a Hidden Markov Model (#1910)
  • ataqv - tool for assessing the quality of a genome using a broader set of marker genes (#3013)
  • telseq - a software that estimates telomere length from whole genome sequencing data (BAMs) (#2910)
  • Cell Ranger ARC - analysis of single cell expression or VDJ data produced by 10X Genomics (#2925)
  • GTDB-Tk - toolkit for assigning objective taxonomic classifications to bacterial and archaeal genomes (#2970)
  • Checkm - genome completeness and contamination based on the presence or absence of marker genes (#2990)
  • Checkm2: rapid, scalable and accurate tool for assessing microbial genome quality using machine learning (#2978)
  • Haplocheck - detects in-sample contamination in mtDNA or WGS sequencing studies by analyzing the mitchondrial content (#2933)
  • mgikit - demultiplexes FASTQ files from an MGI sequencing instrument (#3023)
  • Percolator - semi-supervised learning for peptide identification from shotgun proteomics datasets (#3011)

Module updates

  • GATK BQSR: support Sentieon QualCal output (#3008)
  • Pychopper: add section with UMI barpot (#3024)
  • fastp: add overrepresented sequences plot and table (#3000)
  • RNA-SeqC: more metrics in general statistics table (#3017)
  • phantompeakqualtools: adjust minimum value to 1 for NSC coeff (#3021)
  • QualiMap RNASeq: add gen stats columns for exonic, intronic, intergenic, overlapping (#3018)
  • QualiMap BamQC: add general stats columns for dup rate, number of Ns, mapped paired reads (#2989)
  • DRAGEN: RNA quant metrics: support forward and reverse fragments in barplot (#3012)
  • Interop: remove "%" suffix for nan values (#2995)
  • Dedup: use a more specific search pattern (#2988)
  • Bclconvert: group samples and lanes, refactor (#2872)
  • Nanostat: new format: fix parsing sample names (#3007)

Feature updates and improvements

  • Custom content: allow quoting strings in CSV (#3020)
  • Naturally sort samples in tables and bar plots (#2999)
  • Auto-save dynamic toolbox settings to session and to a file (#3001)
  • Support Python 3.13, drop 3.8 (#2862)
  • Add table tooltips about merged values in groupped samples (#3002)

Fixes

  • Align order of lineplot categories across samples (#3033)

Optimization, refactoring and typing

  • Memory optimization: do not preserve saved_raw_data in modules, write directly to files instead (#3010)
  • Violin plot: add dt into Dataset objects instead of main_table_dt (#2968)

Infrastructure and packaging

  • Add Cursor rules for module creation (#3025)
  • Fix docker build: remove docs from .dockerignore (#2994)
  • Add workflow to sync changelog with Seqera docs (#2993)
  • Docker build: add workflowdispatch, do not push on pullrequest (#2962)
  • Build and test Docker image in PRs (#2958)
  • Fix Docker CI for forks: skip login step (#2953)
  • Document bar plot's sort_samples (#3006)
  • Update ruff to 0.8.3 (#3029)
  • Add more tests for custom-content (#3028)
  • Add pytest ini options to pyproject.toml - to set default test targets (#3030)

- JavaScript
Published by vladsavelyev over 1 year ago

multiqc - v1.25.2

MultiQC v1.25.2 - 2024-11-20

Multiple bug fixes and minor updates.

Updates

  • Add natural sort for sample sorting (#2959)
  • Custom content: for plot_type: image, support custom_data config with section name and description. Fix misleading logging (#2939)
  • Config validation improvements (group messages, cast types, validate column headers) (#2899)

Fixes

  • Workaround for displaying sample grouping in Safari because of missing visibility: collapse (#2941)
  • Fix table CSV export where a title contains a comma (#2911)
  • Showing table in notebooks: respect col1_header (#2914)
  • Customizing custom_table_header_config: fix docs, support both the old and the new ways (#2955)
  • Table scatter mini-plots: fix rounding and range (#2956)
  • File line block iterator: fix reading long lines that do not fit one block (#2935)
  • Fix cond_formatting_rules type hint to avoid validation error (#2922)
  • Fix config.prepend_dirs or -d -dd 1 (#2913)
  • Sample grouping fixes (#2920):
    • Keep sample name column fix width to avoid jumping
    • Fix hiding columns through the modal
  • Custom content fixes:
    • Avoid showing section_comment both for module and section when they have the same ID (#2954)
    • Address issue of sections without search patterns and headers in files (#2921)
    • Fix duplicated custom content sections in the report (#2921)
    • Fix support for plot_type: violin (#2957)

Module updates

  • ngsbits: add submodule samplegender (#2854)
  • nanoq: change lineplots for barplots (#2934)
  • Qualimap: clarify the direction of the transcript in coverage plot (#2946)
  • picard: add table with all metrics to VariantCallingMetrics section (#2885)
  • Nanostat: add general stats columns (#2961)
  • Samtools: add insert size to general stats table (#2905)

Module fixes

  • bcl2fastq: fix missing R1_*/R2_* metrics (#2965)
  • Cutadapt: fix for null values from r2 data (#2936)
  • Qualimap: fix parsing ∞ value (#2937)
  • bclconvert: fix undetermined barcodes plot (#2976)
  • featurecounts: fix missing section name and anchor (#2967)

Infrastructure

  • Pin kaleido to 0.2.1 (new 0.4.1 does not embed a browser and thus not portable) (#2963)

- JavaScript
Published by vladsavelyev over 1 year ago

multiqc - v1.25.1

MultiQC v1.25.1 - 2024-09-30

Python 3.13, bugs fixed, improved sample grouping UI, and handling freezes in containers with incompatible architectures.

Updates

  • Support Python 3.13 (officially to be released on Oct 7). Python 3.8 is supported for now, but might drop support in future releases, so make sure you update! (#2871)
  • Table sample groups UI: allow clicking the entire row to expand, add cursor pointer (#2871)
  • Disable plot export in incompatible architecture containers (when running through rosetta) (#2888)

Fixes

  • Fix export general stats to multiqc_data.json: flatten row groups for back-compatibility (#2879)
  • Custom content:
    • Fix the generalstats headers key name in documentation (#2901)
    • Validate table header config instead of failing (#2875)

Modules

  • Kraken:
    • Re-add the "Unclassified" tab to the "Top taxa" barplot (#2881)
    • Fix the minimizers heatmap (#2895)
    • Remove the separate Bracken module as it's just Kraken internally. Fixes the issues when Kraken reports without unclassified are misidentified for Bracken reports (#2894)
  • bbmap: support qhist outputs with only R1 and extra header (#2882)
  • Picard HsMetrics: fix collecting data sources (#2880)

Refactoring & infrastructure

  • Test Docker image builds on every PR commit (#2886)
  • Suppress "SyntaxWarning: invalid escape sequence" warnings from colormath (#2889)
  • Check the add_data_source args (either path or f should be specified), use strict_helpers.lint_error (#2865)

- JavaScript
Published by vladsavelyev over 1 year ago

multiqc - v1.25

MultiQC v1.25 - 2024-09-16

Highlights - sample grouping

New feature: grouping samples in the General Statistics table.

Some modules - prominently FastQC - may produce multiple results per sample, e.g. for the forward and the reverse reads. To group such results in the table together, a new configuration option is introduced.

This feature is currently opt-in, you'll need to set table_sample_merge in a MultiQC config file to use (see docs above). We'd love to hear your feedback! We hope to enable it by default for common file suffix patterns in a future release.

Because MultiQC needs to know how to merge each column (sum, average, etc), each module must implement it independently. Currently it's supported by the FastQC and Cutadapt modules. If you'd like support added to another module, please let us know in a GitHub issue. Details of how to add it into module code can be found in the moduel development documentation.

Highlights - box plots in custom content

The new box plot plot type, added in v1.21, are now available to use with custom content! See 2847 for configuration examples.

New modules

Updates

  • Group read pairs in general stats (#2794, #2848)
  • Support boxplot in custom content (#2847)
  • Allow x_band, x_lines, x_minrange for any plot type (specifically, scatter plots) (#2851)
  • When both contents and contents_re are specified in a search patterns, treat it as logical AND (#2828)

Module updates

  • NanoStat: support multi-sample logs (#2852)
  • Samtools coverage: support exclude_contigs and include_contigs (#2840)
  • Ganon: support non-verbose output, fix missing removed with --min-count (#2838)
  • fastp: support the --merged flag (#2834)

Fixes

  • Workaround for the hanging Kaleido duing plot export: run plot export in separate threads with a timeout and fallback on freezes, add try-catch for crashes (#2836, #2819)
  • Fix copying multiqc.log into the multiqc_data output folder (#2829)
  • Fix applying exclude_files in search patterns (#2804)
  • Fix bar plot export from toolbox (#2845)
  • Fix autoselection in plot export toolbox (#2844)
  • Fix config.replace_samples for custom content genstats table (#2841)

Module fixes

  • Picard: fix parsing Sentieon insert size metrics (#2823)
  • Picard: fix a search pattern of CollectRnaSeqMetrics (#2811)
  • Kraken: fix for empty top ranks. Also handle it in bargraph (#2822)
  • Cellranger: fix data source tag (#2821)
  • FastQC: fix the Per Base Sequence Content detail plot click when a module was run multiple times (#2856)
  • FastQC: fix calculating average read length (#2817)

Refactoring and typing

  • Separate anchors and IDs for sections: use IDs for Python configation, and anchors in HTML (#2797, #2833)
  • Prefix table column IDs with namespace, but allow configuration to use both short and long anchors (#2818)
  • More type hinting in plots (#2816, #2850)
  • Use typing.NewType for all Python versions (#2820)
  • Remove pyaml_env dependency and apply the fix for the SyntaxWarning (#2837)

Infrastructure and packaging

  • Add py.typed to mark package as providing type information (#2846)

- JavaScript
Published by vladsavelyev over 1 year ago

multiqc - v1.24.1

MultiQC v1.24.1 - 2024-08-21

A bug fix release mainly to restore compatibility with Python 3.8. Aside from that, few other minor bug fixes:

  • FastQC: fix long-standing issue misplacing status labels when anchor is specified in the custom config (#2790)
  • Freyja: handle empty inputs, and ensure deterministic sample order (#2788)
  • Allow numeric xcats and ycats for the heatmap plot (#2787)
  • Make sure that config's extra_fn_clean_exts and fn_clean_exts don't conflict when both specified (#2783)

- JavaScript
Published by vladsavelyev almost 2 years ago

multiqc - v1.24

MultiQC v1.24

Mostly a maintenance release, containing several bug fixes, performance improvements, plus 6 new modules, along with improvements of the existing modules.

The most significant performance boost got the Kraken and Mosdepth modules, that now don't take way more memory and CPU than any other typical module:

| Tool | Data Set | Memory - Before | CPU - Before | Memory - After | CPU - After | | -------- | -------------- | --------------- | ------------ | -------------- | ----------- | | mosdepth | 1 set of files | 196 Mb | 3.04s | 129 Mb | 2.27s | | | 10 | 464 Mb | 8.48s | 131 Mb | 6.11s | | | 100 | 3,719 Mb | 63.19s | 172 Mb | 43.12s | | kraken | 1 set of files | 155 Mb | 2.07s | 132 Mb | 2.20s | | | 10 | 606 Mb | 8.39s | 180 Mb | 3.47s | | | 100 | 4,970 Mb | 71.89s | 809 Mb | 14.53s |

Large plots that may hang browser are now not loaded by default, and the user can click a button to load, so the heavy plots don't slow down the initial report rendering. This is controlled by the config.plots_defer_loading_numseries: 100 option.

Updates

  • Search patterns: allow multiple values for contents (#2696)
  • Custom content:
    • Allow dict input to heatmap (#2761)
    • Allow multiple custom content to general stats table (#2727)
  • Plots:
    • Defer render of plots if number of samples > config.plots_defer_loading_numseries (#2759, #2777, #2773, #2774)
    • Line plot: show markers when num of data points < config.lineplot_number_of_points_to_hide_markers (=50) (#2760). As a nice consequence, trivial lines of a single data point become visible.
    • Line plot: smooth by default to 500 points on the X axis to avoid inflating the report file size (#2776)
    • Allow to configure the scale of the exported plot fonts through the config option config.plots_export_font_scale (#2758)
    • Improve the performance of loading large tables in browser (#2737)
    • Fix the toolbox highlight of the line plots (#2724)
  • The function that returns built plots in an interactive session now uses the module anchor (or lowercase module name) to key the results (#2741)
  • More helpful config validation error: print the parent model name, if applicable (#2709)

New modules

  • VG (#2690), a toolkit to manipulate graphical genomes. The module parses vg-stats reports that summarize alignment stats from a GAM file.
  • ngs-bits (#2231). A tool that calculating statistics from FASTQ, BAM, and VCF files. The module parses XML output generated for two tools in the ngs-bits collection:
  • Pairtools (#1148). A toolkit for Chromatin Conformation Capture experiments. Handles short-reads paired reference alignments, extracts 3C-specific information, and perform common tasks such as sorting, filtering, and deduplication. The module parses summary statistics generated by pairtools's dedup and stats tools.
  • nanoq (#2723). A tool that reports read quality and length from nanopore sequencing data.
  • Ganon (#1935). A tool for metagenomics classification: quickly assigns sequence fragments to their closest reference among thousands of references via Interleaved Bloom Filters of k-mer/minimizers

Fixes

  • Fix --pdf option to generate multiqc_report.pdf (#2733)
  • Fix saving table plots to file (#2735)
  • Fix adding software versions when config.run_modules is set (#2755)
  • Fix toolbox highlight in line plots (#2724)
  • Refactor write_results to avoid dynamically overriding config, fixes module.write_data_file (#2722)
  • Search stats: do not double-count ignored files (#2708)
  • Escape values passed to HTML properties (e.g. val in tables) (#2706)
  • Fix re-loading explicit user configs in interactive sessions (#2704)
  • Fix file search performance regression (#2762)
  • Fix handling module href string (#2739)
  • Custom content:
    • Cast names to strings instead of asserting (allows numerical sample names) (#2769)
    • Fix parsing custom content tables with numerical samples (#2705)
    • Section order and custom content order: allow skip the *-module suffix from anchors (#2770)

Module updates

  • Kraken: optimize memory and runtime (#2756)
  • Mosdepth: optimize memory and runtime (#2748, #2749)
  • Abstract config.get_cov_thresholds function for mosdepth and qualimap (#2707)
  • Anglerfish: adjust for version 0.6.1 (#2757)
  • Umitools: prefer output for sample name, handle the <stdin>/<stdout> placeholders (#2698)
  • Kraken: fix top % calculation, more efficient total read count calculation (#2744)
  • Bracken: when printing number of found samples, indicate that running Bracken not Kraken (#2743)
  • Nonpareil: Update docs about new version that generates the JSON file (#2734)
  • STAR: add all alignment summary metrics into a new separate table (#1828)
  • Pairtools: fix typos and grammar, remove redundancies (#2711)
  • Peddy sex plot: color predicted sex (#2778)

Module fixes

  • Cellranger: fix for missing analysis_tab data (#2771)
  • Fix setting coverage thresholds for mosdepth and qualimap (#2754)
  • Nonpareil: fix running with >12 samples (#2752)
  • Bracken: fix bug when direct reads not classified (#2738)
  • ngsderive: fix ValueError in the encoding submodule (#2740)
  • RSeQC: fix duplicated namespace (#2732)
  • Glimpse: fix parsing data, add proper type hints (#2721)
  • Fix ignoring samples in spaceranger, ngsbits, isoseq, dragen coverage (#2717)
  • Glimpse: clean and fix filtering samples (#2716)
  • Spaceranger: fix ignoring samples (#2714)

Refactoring

  • Refactor write_results to avoid dynamically overriding config, fixes module.write_data_file (#2722)
  • Modules:
    • isoseq and odgi: fix module warnings and error handling (#2718)
    • Qualimap: refactor and add type hints (#2707)
    • FastQC: refactor and add type hints (#2763)
    • Kraken: refactor and add type hints (#2744)
    • Cell Ranger: refactor, add type hints, get rid of module mixins and fields (#2775)
    • Spaceranger: refactor, add type hints, get rid of module mixins and fields (#2714)
    • RSeQC: refactor, add type hints, and remove multiqc_rseqc.js (#2710)

Infrastructure

  • Split up the tests for sample versions discovery (#2751)
  • Move integration test variations into unit tests, actually test them (#2713)
  • Move module docs to the docstrings & generate docs/modules/*.md from the docstrings using a separate script (#2703)
  • Embed the search patterns into the module docs (#2765)
  • Dockerfile: use COPY instead of ADD to copy only relevant files, update base image to Python 3.12 (#2700)

- JavaScript
Published by vladsavelyev almost 2 years ago

multiqc - v1.23

MultiQC v1.23

Bug fixes, integration of pytest and mypy, and one new module.

From the user perspective, this is mostly a maintenance release, containing several important bugfixes, plus minor improvements and a new module - Glimpse.

For developers, there are two significant additions to the CI workflow:

  • pytest, along with unit tests covering the core library,
  • and mypy, along with ensuring that the core codebase is fully type-annotated.

The core unit tests are located in the multiqc/tests folder, and the module tests are located in the corresponding multiqc/modules/*/tests subfolders. The CI workflows are refactored to separate the integration tests and the unit tests, to improve the granularity and parallelization. The tests are discovered and executed with pytest, and the coverage is reported by codecov.

The multiqc/tests subfolder has several test files that cover most of the core library. It also has a testmodulesrun.py tests that checks that every module didn't crash when being run on the corresponding data in test-data, and added something into the report. That is somewhat of a blanket test for modules, that doesn't check if the modules logic worked correctly. For that reason, the users are encouraged to write more comprehensive tests that take the specific module logic into account, and place them in multiqc/modules/*/tests`. For some initial examples, consider checking:

  • The samtools flagstat test that verifies some logic in the flagstat submodule of the samtools module;
  • The picard tools test that checks that every submodule for each Picard tool worked correctly.

Other changes & fixes

Fixes

  • Custom content"
    • Multiple fixes of the custom content parsing logic (#2674)
    • Fix parsing custom content submodules with a custom config (#2654)
    • Line plot: fix spreading extra_series between multiple datasets (#2684)
    • Line plot series: allow pass lists instead of x/y tuples to work properly with YAML (#2683)
  • Re-enabling the software_version module section (#2670)
  • When --no-ansi is set, disable colors in rich_click too (#2678)
  • Support CWD path filters ( ./path/...) in config (#2676)
  • Fix writing report to stdout with --filename stdout, log to stderr (#2672)
  • Interactive use:
    • Reset all config values in config.reset(), even those that are not in config_default.yaml (#2660)
    • Reset config between multiqc.run() calls (#2655)
    • Better handling of calling write_report twice (#2688)

Updates

  • Run mypy on core library (#2665)
  • Add tests for plot export (#2682)
  • Add tests for command line use, including for passing TMPDIR (#2677)
  • Custom content: allow hash-fenced table columns (#2649)
  • Software versions: parse for sorting, but preserve the original strings (#2671)
  • Allow both table-level and column-level custom plot config for table (#2662)

New modules

Module fixes

  • Fix parsing kraken vs. bracken: respect num_lines in search patterns (#2657)
  • Fix the bbmap/qchist search pattern (#2661)

Module updates

  • Picard HsMetrics: support any custom X coverage metrics (#2663)
  • Samtools coverage: avoid hard crash for invalid file contents (#2664)

Refactoring

  • Abstract code related to temporary directory creation into a separate module (#2675)

Infrastructure

  • Use pull-request labels and milestones for changelog generation (#2691)

- JavaScript
Published by vladsavelyev almost 2 years ago

multiqc - v1.22.3

Contains fixes of multiple bugs collected after the last release along with a few minor improvements.

MultiQC fixes

  • Fix the re_contents search patterns when pattern is found in the middle of the file. Fixes finding logs from several Picard submodules, like CollectRnaSeqMetrics and CollectWgsMetrics in some cases (#2610)
  • Fixes the run_modules option use when the module anchor doesn't match the module entry point ID (e.g. DRAGEN and dragen) (#2633)
  • Fix use of custom search patterns for custom content (#2647)
  • Fix plot export with export_plots: true or --export (#2637)
  • Correctly handle old-style label sections in x_lines or y_lines in line plot configs (#2648)
  • Fix disabling sort_rows in custom content by subclassing TableConfig from ValidatedConfig and use deprecated (#2604)
  • When user provides a search pattern dictionary in config, recursively update instead of replacing (#2620)
  • Fix config update when dict replaced with list, e.g. a search_patterns item is a list that's replaced with a dict (https://github.com/MultiQC/MultiQC/commit/c388178bb6d9f143c6f8e8b0146647a067021ea4)

MultiQC updates

  • Add unit tests for core and some modules (see picard or samtools), as well as codecov report (#2624)
    • Now MultiQC checks if every module does something productive with the provided test data in test-data.
    • For modules with many submodules (picard, dragen), additionally check if every submodule parses the expected number of samples from test-data files.
    • Users can put module tests in test subfolders, e.g. https://github.com/MultiQC/MultiQC/tree/main/multiqc/modules/picard/tests
    • Use pytest for all core unit tests (#2623)
    • Move unit tests from the test-data repo into tests folder (#2622)
  • Plot config validation:
    • Validate line plot x_lines, x_bands, etc. with a Pydantic model, including label subsections (#2648)
    • Validate line plot series and extra_series with a Pydantic model (#2573)
    • Validate table config (#2604)
    • Make the "unrecognised field" error a warning
    • Rename deprecated plot config fields in internal modules (#2636)
  • Show progress bar for exporting flat plot images (#2639)
  • Better error message for incorrect run_modules (#2635)
  • Increase flat plots sample number threshold to 1000 (#2615)
  • Small speed-up of the line block iterator (#2588)
  • Update README logos for better compatibility (#2603)
  • Docs: don't use raw markdown links (#2642)
  • Allow to override showlegend for line config plots. Default to not-show for large datasets to avoid bloated legends (#2615)
  • Show error message if failed to parse custom content header (https://github.com/MultiQC/MultiQC/commit/d736846a0c23410243f80b2bdca984363211ffc3)
  • Load every found config file once https://github.com/MultiQC/MultiQC/commit/422b39bc787720cefea81a85dabbf6411b3421ac

Module fixes and updates

  • Picard
    • Fix finding CollectRnaSeqMetrics and CollectWgsMetrics logs by fixing the re_contents search patterns (#2610)
  • biobambam2
    • Fix parsing markdups logs
  • DRAGEN
    • Coverage histograms: fix duplicated label suffix (#2619)
    • Fix the gc_metrics submodule (#2629)
    • vc_metrics: pre-filter numbers can be zero (#2618)
  • FastQC
    • Default to showlegend: false, as we don't distinguish the sample colors, unless fastqc_config: status_checks: false' is set (#2615)
  • BBTools
    • Fix incorrect calculation of % Q30 Bases (#2628)
  • Samtools
    • markdup: resolve inconsistent non-optical pair duplicate variable name in samtools markdup module (#2626)
  • NanoStat
    • Support different Q cutoffs (#2645)
  • Salmon
    • Fix ignored parsed library_types when its type is list (#2617)
  • UMI-tools
    • Improve extract plots (#2614)
  • BCL Convert
    • Fix 'pecent' typo (#2612)

- JavaScript
Published by vladsavelyev almost 2 years ago

multiqc - v1.22.2

Bug fix release. Two main issues are fixed:

  • Fixed running the same module twice with path_filters (e.g. trimmed vs. raw FastQC),
  • The raw data report_saved_raw_data is re-added in multiqc_data.json by default.

MultiQC fixes

  • Fix running the same module multiple times in the report (e.g. trimmed vs. raw FastQC) (#2592)
  • Preserve report_saved_raw_data in multiqcdata.json by keeping `preservemodulerawdata: false` by default (#2591)
  • Table headers: do not set namespace to None when there is a single namespace (#2590)
  • Re-enable falling back to flat plots for large datasets (#2580)
  • Reset in multiqc.run(*) to allow running it twice interactively (#2598)
  • Fix scatter plot in --flat mode when there are categorical axes (#2600)
  • Fix hiding table column with all empty values in custom content (#2599)
  • Table "Copy" button: include headers (#2594)

Module fixes and updates

  • QUAST
    • Underscore attributes captured by lambdas to avoid wiping them after the module is finished (#2581)
  • Cell Ranger
    • Handle missing vdj_annotation and vdj_enrichment sections (#2579)
  • fgbio
    • Fix links in fgbio.md (#2586)
  • Custom content
    • Support DOI for custom content (#2582)

- JavaScript
Published by vladsavelyev about 2 years ago

multiqc - v1.22.1

Fix running as a Nextflow job

This bug fix release addresses the file search problem when MultiQC is executed as a typical Nextflow job. See #2575 for detail.

- JavaScript
Published by vladsavelyev about 2 years ago

multiqc - v1.22

Highlights - notebooks and performance

Version 1.22 brings some major behind-the-scenes refactoring to MultiQC. This unlocks a number of new features, such as the ability to use MultiQC as a Python library in scripts / notebooks, and run-time validation of plot config attributes.

This release also introduces some huge performance improvements thanks to @rhpvorderman. Compared to v1.21, a typical v1.22 run is 53% faster and has a 6x smaller peak-memory footprint - well worth updating! 🏃🏻‍♂️ 💨

Finally, support for the depreciated HighCharts plotting library is fully removed in v1.22, bringing to a close a long standing project to migrate to Plotly.

For more information, please see the upcoming MultiQC release blog article on the Seqera website: https://seqera.io/blog/

MultiQC updates

  • Remove the highcharts template and Highcharts and Matplotlib dependencies (#2409)
  • Remove CSP.txt and the linting check, move the script that prints missing hashes under scripts. Admins of servers with Content Security Policy can use it to print missing hashes when they install a new MultiQC version with: python scripts/print_missing_csp.py --report full_report.html (#2421)
  • Do not maintain change log between releases (#2427)
  • Use native clipboard API (#2419)
  • Profile runtime: visualize per-module memory and run time (#2548, #2547)
  • Refactoring for performance:
    • Search file blocks rather than individual lines for faster results (#2513)
    • Refactor file content search for a 40% speed increase (#2505)
    • Sort filepatterns for faster searching (#2506)
    • Use array.array for in-memory plot data, stream to render Jinja and dump JSON to reduce memory requirement (#2515)
    • Speed up all modules by caching spectra.scale and using sets instead of lists (#2509)
    • Stream json data to a file to save 30% of the memory (#2510)
    • Do replace_nan in place rather than creating a new object (#2529)
    • Use gzip rather than lzstring for compression and decompression of the plot data (#2504)
    • Use gzip level 6 for faster json compression (#2553)
    • Clean up module raw data after running each module, significantly reduces the memory footprint (#2551)
  • Refactoring for interactivity and validation:
    • Top-level functions for MultiQC use as a library (#2442)
    • Pydantic models for plots and datasets (#2442)
    • Validating plot configs with Pydantic (#2534)
    • Use dataclasses for table and violin columns (#2546)
    • Break up the main run function into submodules (#2446)
    • Deprecate multiqc.utils.config and multiqc.utils.report in favour of multiqc.config and multiqc.report (#2542)
    • Static typing of the report and config modules (#2445)
    • Add type hints into core codebase (#2434)
    • Consistent config options: rename decimalPlaces to tt_decimals (#2451)
    • Remove encoding and shebang headers from module files (#2425)
    • Refactor line plot categories: keep boolean throughout the code, and data points as pairs for simplicity (#2418)
  • Fixes:
    • Fix error when using default sort (#2544)
    • Do not attempt to render flat plot when no data (#2490)
    • Fix export plots with --export and always export data (#2489)
    • Fix: make sure modify lambda not present in JSON dump (#2455)
    • Enable --export even when writing interactive plots (#2444)
    • Replace NaN with null in exported JSON (#2432)
    • Fix y_minrange option (#2415)
  • Reduce report size: exclude plot data for sections in remove_sections (#2460)
  • Add ge and le to cond_formatting_rules (#2494)
  • CI: use uv pip (#2352)
  • Lint check for use of f["content_lines"] (#2485)
  • Allow to set style of line graph (lines or lines+markers) per plot (#2413)
  • Add CMD to Dockerfile so a default run without any parameters displays the --help (#2279)

New modules

  • Hostile (#2501)
    • New module: Hostile is a short and long host reads removal tool
  • Sequali (#2441)
    • New module: Sequali Universal sequencing QC

Module updates

  • Adapter Removal
    • Standardize module names: use the came case (#2433)
  • Bamdst
    • Fix chromosome reports when contig data labels are missing (#2479)
    • Fix for the case when chromosomes.report is not provided (#2477)
    • Stress file name requirements for chromosomes report (#2478)
  • BBTools
    • Set missing values to None for bbmap qahist (#2411)
  • Bcftools
    • Stats: add multialleic sites column (#2414)
  • BCL Convert
    • Show message when no undetermined reads instead of error (#2526)
    • Fix for absent index reads (#2511)
    • Add all file types to sources (#2456)
  • Busco
    • Fix barplot colors (#2453)
  • Cell Ranger
    • Fix parsing antibody tab without antibody_treemap_plot (#2525)
  • Cutadapt
    • Speed up module by caching parsing versions (#2528)
  • DRAGEN
    • Add ploidy estimation table (#2496)
  • fastp
    • When could not parse sample name from command (i.e. stdin), use filename and proceed (#2536)
  • FastQC
    • Skip per tile sequence quality section in FastQC reports for better performance (#2552)
    • Fix a ZeroDivisionError error (#2462)
    • Fix memory leak to make 7 times faster and use 10 times less memory (#2552)
    • Do not keep intermediate data in memory to reduce memory footprint further (#2516 )
    • Add option to ignore FastQC quality thresholds (#2486)
  • goleft indexcov
    • Work correctly even if no valid contigs in input (#2540)
  • mosdepth
    • Fix absolute coverage plot (#2488)
  • nonpareil
    • Change writedatafile label to be consistent with other modules (#2472)
  • Picard
    • WgsMetrics: coverage plot: show % based ≥x, not >x (#2473)
    • CrosscheckFingerprints: support multiple files, preserve sample order in heatmap (#2454)
  • qc3C
    • Fix detecting sample name for relative path (#2502)
  • QualiMap
    • BamQC: when trimming long tails, keep at least 20x (#2431)
  • Samtools
    • Add support for markdup (#2254)
    • Add violin multiple datasets & samtools flagstat percentage switch (#2430)
  • Space Ranger
    • fix for missing genomic_dna section (#2429)
  • xengsort
    • Fix parsing long files (do no use content_lines) (#2484)

New Contributors

  • @clintval made their first contribution in https://github.com/MultiQC/MultiQC/pull/2254
  • @alanhoyle made their first contribution in https://github.com/MultiQC/MultiQC/pull/2279
  • @rhpvorderman made their first contribution in https://github.com/MultiQC/MultiQC/pull/2441
  • @TBradley27 made their first contribution in https://github.com/MultiQC/MultiQC/pull/2473
  • @SumeetTiwari07 made their first contribution in https://github.com/MultiQC/MultiQC/pull/2501

Full Changelog: https://github.com/MultiQC/MultiQC/compare/v1.21...v1.22

- JavaScript
Published by ewels about 2 years ago

multiqc - MultiQC version 1.21

Highlights

Box plot

Added a new plot type: box plot. It's useful to visualise a distribution when you have a set of values for each sample.

py from multiqc.plots import box self.add_section( ..., plot=box.plot( { "sample 1": [4506, 4326, 3137, 1563, 1730, 3254, 2259, 3670, 2719, ...], "sample 2": [2145, 2011, 3368, 2132, 1673, 1993, 6635, 1635, 4984, ...], "sample 3": [1560, 1845, 3247, 1701, 2829, 2775, 3179, 1724, 1828, ...], }, pconfig={ "title": "Iso-Seq: Insert Length", }, ) )

Note the difference with the violin plot: the box plot visualises the distributions of many values within one sample, whereas the violin plot shows the distribution of one metric across many samples.

pyproject.toml

The setup.py file has been superseded by pyproject.toml for the build configuration. Note that now for new modules, an entry point should be added to pyproject.toml instead of setup.py, e.g.:

toml [project.entry-points."multiqc.modules.v1"] afterqc = "multiqc.modules.afterqc:MultiqcModule"

Heatmap

The heatmap plot now supports passing a dict as input data, and also supports a zlab parameter to set the label for the z-axis:

py from multiqc.plots import heatmap self.add_section( ..., plot=heatmap.plot( { "sample 1": {"sample 2": 0, "sample 3": 1}, "sample 2": {"sample 1": 0, "sample 3": 0}, "sample 3": {"sample 1": 1, "sample 2": 0, "sample 3": 1}, }, pconfig={ "title": "Sample comparison", "zlab": "Match", }, ) )

MultiQC updates

  • New plot type: box plot (#2358)
  • Add "Export to CSV" button for tables (#2394)
  • Replace setup.py with pyproject.toml (#2353)
  • Heatmap: allow a dict dicts of data (#2386)
  • Heatmap: add zlab config parameter. Show xlab, ylab, zlab in tooltip (#2387)
  • Warn if run_modules contains a non-existent module (#2322)
  • Catch non-hashable values (dicts, lists) passed as a table cell value (#2348)
  • Always create JSON even when MegaQC upload is disabled (#2330)
  • Use generic font family for Plotly (#2368)
  • Use a padded span with nowrap instead of &nbsp; before suffixes in table cells (#2395)
  • Refactor: fix unescaped regex strings (#2384)

Fixes:

  • Pin the required Plotly version and add a runtime version check (#2325)
  • Bar plot: preserve the sample order (#2339)
  • Bar plot: fix inner gap in group mode (#2321)
  • Violin: filter Inf values (#2380)
  • Table: Fix use of the no_violin (ex-no_beeswarm) table config flag (#2376)
  • Heatmap: prevent from parsing numerical sample names (#2349)
  • Work around call of full_figure_for_development to avoid Kaleido errors (#2359)
  • Auto-generate plot id when pconfig=None (#2337)
  • Fix: infinite dmax or dmin fail JSON dump load in JavaScript (#2354)
  • Fix: dump pconfig for MegaQC (#2344)

New modules

  • IsoSeq
    • Iso-Seq contains the newest tools to identify transcripts in PacBio single-molecule sequencing data (HiFi reads). cluster and refine commands are supported.
  • Space Ranger
    • Works with data from 10X Genomics Visium. Processes sequencing reads and images created using the 10x Visium platform to generate count matrices with spatial information.
    • New MultiQC module parses Space Ranger quality reports.

Module updates

  • bcl2fastq: fix the top undetermined barcodes plot (#2340)
  • DRAGEN: add few coverage metrics in general stats (#2341)
  • DRAGEN: fix showing the number of found samples (#2347)
  • DRAGEN: support gvcf_metrics (#2327)
  • fastp: fix detection of JSON files (#2334)
  • HTSeq Count: robust file reading loop, ignore .parquet files (#2364)
  • Illumina InterOp Statistics: do not set 'scale': False as a default (#2350)
  • mosdepth: fix regression in showing general stats (#2346)
  • Picard: Crosscheck Fingerprints updates (#2388)
    • add a heatmap for LOD scores besides a table
    • if too many pairs in table, skip those with Expected status
    • use the warn status for Inconclusive
    • add a separate sample-wise table instead of general stats
    • sort tables by status, not by sample name
    • add a column "Best match" and "Best match LOD" in tables
    • hide the LOD Threshold column
  • PURPLE: support v4.0.1 output without version column (#2366)
  • Samtools: support new coverage command (#2356)
  • UMI-tools: support new extract command (#2296)
  • Whatshap: make robust when a stdout is appended to TSV (#2361)

New Contributors

  • @grst made their first contribution in https://github.com/MultiQC/MultiQC/pull/1945

Full Changelog: https://github.com/MultiQC/MultiQC/compare/v1.20...v1.21

- JavaScript
Published by vladsavelyev over 2 years ago

multiqc - MultiQC version 1.20

Highlights

New plotting library

MultiQC v1.20 comes with totally new plotting code for MultiQC reports. This is a huge change to the report output. We've done our best to maintain feature parity with the previous plotting code, but please do let us know if you spot any bugs or changes in behaviour by creating a GitHub issue.

This change comes with many improvements and new features, and paves the way for more in the future. To find out more, read the associated blog post.

For now, you can revert to the previous plotting code by using the highcharts report template (multiqc --template highcharts). This will be removed in v1.21.

Note that there are several plotting configuration options which have been removed:

  • click_func
  • cursor
  • tt_percentages (use tt_suffix: "%")
  • Bar plot:
    • use_legend (automatically hidden if there is only 1 category)
  • Line plot:
    • labelSize
    • xDecimals, yDecimals (automatic if all values can be cast to int)
    • xLabelFormat, yLabelFormat (use tt_label)
    • pointFormat
  • Heatmap:
    • datalabel_colour
    • borderWidth

Moved GitHub and docker repositories

The v1.20 release is also the first release we've had since we moved the MultiQC repositories. Please note that the code is now at MultiQC/MultiQC (formerly ewels/MultiQC) and the same for the Docker repository. The GitHub repo should automatically redirect, but it's still good to update any references you may have.

MultiQC updates

  • Support Plotly as a new backend for plots (#2079)
    • The default template now uses Plotly for all plots
    • Added a new plot type violin (replaces beeswarm)
    • Moved legacy Highcharts/Matplotlib code under an optional template highcharts (#2292)
  • Move GitHub repository to MultiQC organisation (#2243)
  • Update all GitHub actions to their latest versions (#2242)
  • Update docs to work with Astro 4 (#2256)
  • Remove unused dependency on future library (#2258)
  • Fix incorrect scale IDs caught by linting (#2272)
  • Docs: fix missing v prefix in docker image tags (#2273)
  • Unicode file reading errors: attempt to skip non-unicode characters (#2275)
  • Heatmap: check if value is numeric when calculating min and max (#2276)
  • Add filesearch_file_shared config option, remove unnecessary per-module shared flags in search patterns (#2227)
  • Use alternative method to walk directory using pathlib (#2277)
  • Export config.output_dir in MegaQC JSON (#2287)
  • Drop support for module tags (#2278)
  • Pin Pillow package, wrap add_logo in try-except (#2312)
  • Custom content: support multiple datasets (#2291)
  • Configuration: fix reading config.outputfnname and --filename (#2314)

New modules

  • Bamdst (#2161)
    • Bamdst is a lightweight tool to stat the depth coverage of target regions of bam file(s).
  • MetaPhlAn (#2262)
    • MetaPhlAn is a computational tool for profiling the composition of microbial communities from metagenomic shotgun sequencing data.
  • MEGAHIT (#2222)
    • MEGAHIT is an ultra-fast and memory-efficient NGS assembler
  • Nonpareil (#2215)
    • Estimate metagenomic coverage and sequence diversity.

Module updates

  • Bcftools: order variant depths plot categories (#2289)
  • Bcftools: add missing self.ignore_samples in stats (#2288)
  • BCL Convert: add index, project names to sample statistics and calculate mean quality for lane statistics. (#2261)
  • BCL Convert: fix duplicated yield for 3.9.3+ when the yield is provided explicitly in Quality_Metrics (#2253)
  • BCL Convert: handle samples with zero yield (#2297)
  • Bismark: fix old link in Bismark docs (#2252)
  • Bismark: fix old link in docs (#2252)
  • Cutadapt: support JSON format (#2281)
  • HiFiasm: account for lines with no asterisk (#2268)
  • HUMID: add cluster statistics (#2265)
  • mosdepth: add additional summaries to general stats #2257 (#2257)
  • Picard: fix using multiple times in report: do not pass module.anchor to self.find_log_files (#2255)
  • QualiMap: address NBSP as thousands separators (#2282)
  • Seqera Platform CLI: updates for v0.9.2 (#2248)
  • Seqera Platform CLI: handle failed tasks (#2286)

New Contributors

  • @a-detiste made their first contribution in https://github.com/MultiQC/MultiQC/pull/2258
  • @mbeavitt made their first contribution in https://github.com/MultiQC/MultiQC/pull/2268
  • @almahans made their first contribution in https://github.com/MultiQC/MultiQC/pull/2262
  • @kmlhpk made their first contribution in https://github.com/MultiQC/MultiQC/pull/2273
  • @studioph made their first contribution in https://github.com/MultiQC/MultiQC/pull/2277
  • @vsmalladi made their first contribution in https://github.com/MultiQC/MultiQC/pull/1503

Full Changelog: https://github.com/MultiQC/MultiQC/compare/v1.19...v1.20

- JavaScript
Published by ewels over 2 years ago

multiqc - MultiQC version 1.19

Highlights

An early Christmas present 🎁 Happy holidays everyone! 🎄

This release is mostly bugfixes and minor additions, whilst we lay the groundwork for some bigger updates coming in the new year. Still, there are plenty of goodies in here. Enjoy!

See the full changes in this release here: https://github.com/ewels/MultiQC/compare/v1.18...v1.19

MultiQC updates

  • Add missing table id in DRAGEN modules, and require id in plot configs in strict mode (#2228)
  • Config table_columns_visible and table_columns_name: support flat config and table_id as a group (#2191)
  • Add sort_samples: false config option for bar graphs (#2210)
  • Upgrade the jQuery tablesorter plugin to v2 (#1666)
  • Refactor pre-Python-3.6 code, prefer f-strings over .format() calls (#2224)
  • Allow specifying default sort columns for tables with defaultsort (#1667)
  • Create CODEOFCONDUCT.md (#2195)
  • Add .cram to sample name cleaning defaults (#2209)

MultiQC bug fixes

  • Re-add run into the multiqc namespace (#2202)
  • Fix the "square": True flag to scatter plot to actually make the plot square (#2189)
  • Fix running with the --no-report flag (#2212)
  • Fix guessing custom content plot type: do not assume first row of a bar plot data are sample names (#2208)
  • Fix detection of changed specific module in Changelog CI (#2234)

Module updates

  • BCLConvert: fix mean quality, fix count-per-lane bar plot (#2197)
  • deepTools: handle missing data in plotProfile (#2229)
  • Fastp: search content instead of file name (#2213)
  • GATK: square the BaseRecalibrator scatter plot (#2189)
  • HiC-Pro: add missing search patterns and better handling of missing data (#2233)
  • Kraken: fix UnboundLocalError (#2230)
  • Kraken: fixed column keys in genstats (#2205)
  • QualiMap: fix BamQC for global-only stats (#2207)
  • Picard: add more search patterns for MarkDuplicates, including MarkDuplicatesSpark (#2226)
  • Salmon: add library_types, compatible_fragment_ratio, strand_mapping_bias to the general stats table (#1485)

New Contributors

  • @gsmashd made their first contribution in https://github.com/ewels/MultiQC/pull/2205
  • @kclem made their first contribution in https://github.com/ewels/MultiQC/pull/2235

Full Changelog: https://github.com/ewels/MultiQC/compare/v1.18...v1.19

- JavaScript
Published by ewels over 2 years ago

multiqc - MultiQC Version 1.18

Highlights

Better configs

As of this release, you can now set all of your config variables via environment variables! (see docs).

Better still, YAML config files can now use string interpolation to parse environment variables within strings (see docs), eg:

yaml report_header_info: - Contact E-mail: !ENV "${NAME:info}@${DOMAIN:example.com}"

Picard refactoring

In this release, there was a significant refactoring of the Picard module. It has been generalized for better code sharing with other Picard-based software, like Sentieon and Parabricks. As a result of this, the standalone Sentieon module was removed: Sentieon QC files will be interpreted directly as Picard QC files.

If you were using the Sentieon module in your pipelines, make sure to update any places that reference the module name:

  • MultiQC command line (e.g. replace --module sentieon with --module picard).
  • MultiQC configs (e.g. replace sentieon with picard in options like run_modules, exclude_modules, module_order).
  • Downstream code that relies on names of the files in multiqc_data or multiqc_plots saves (e.g., multiqc_data/multiqc_sentieon_AlignmentSummaryMetrics.txt becomes multiqc_data/multiqc_picard_AlignmentSummaryMetrics.txt).
  • Code that parses data files like multiqc_data/multiqc_data.json.
  • Custom plugins and templates that rely on HTML anchors (e.g. #sentieon_aligned_reads becomes #picard_AlignmentSummaryMetrics).
  • Also, note that Picard fetches sample names from the commands it finds inside the QC headers (e.g. # net.sf.picard.analysis.CollectMultipleMetrics INPUT=Szabo_160930_SN583_0215_AC9H20ACXX.bam ... -> Szabo_160930_SN583_0215_AC9H20ACXX), whereas the removed Sentieon module prioritized the QC file names. To revert to the old Sentieon approach, use the use_filename_as_sample_name config flag.

MultiQC updates

  • Config can be set with environment variables, including env var interpolation (#2178)
  • Try find config in ~/.config or $XDG_CONFIG_HOME (#2183)
  • Better sample name cleaning with pairs of input filenames (#2181)
  • Software versions: allow any string as a version tag (#2166)
  • Table columns with non-numeric values and now trigger a linting error if scale is set (#2176)
  • Stricter config variable typing (#2178)
  • Remove position:absolute CSS from table values (#2169)
  • Fix column sorting in exported TSV files from a matplotlib linegraph plot (#2143)
  • Fix custom anchors for kraken (#2170)
  • Fix logging spillover bug (#2174)

New Modules

  • Seqera Platform CLI (#2151)
    • Seqera Platform CLI reports statistics generated by the Seqera Platform CLI.
  • Xenome (#1860)
    • A tool for classifying reads from xenograft sources.
  • xengsort (#2168)
    • xengsort is a fast xenograft read sorter based on space-efficient k-mer hashing

Module updates

  • fastp: add version parsing (#2159)
  • fastp: correctly parse sample name from --in1/--in2 in bash command. Prefer file name if not fastp.json; fallback to file name when error (#2139)
  • Kaiju: fix division by zero error (#2179)
  • Nanostat: account for both tab and spaces in v1.41+ search pattern (#2155)
  • Pangolin: update for v4: add QC Note , update tool versions columns (#2157)
  • Picard: Generalize to directly support Sentieon and Parabricks outputs (#2110)
  • Sentieon: Removed the module in favour of directly supporting parsing by the Picard module (#2110)
    • Note that any code that relies on the module name needs to be updated, e.g. -m sentieon will no longer work
    • The exported plot and data files will be now be prefixed as picard instead of sentieon, etc.
    • Note that the Sentieon module used to fetch the sample names from the file names by default, and now it follows the Picard module's logic, and prioritizes the commands recorded in the logs. To override, use the use_filename_as_sample_name config flag

- JavaScript
Published by ewels over 2 years ago

multiqc - MultiQC Version 1.17

The one with the new logo

Highlights:

  • Introducing the new MultiQC logo!
  • Adding support for Python 3.12 and dropping support for Python 3.7
  • New --require-logs to fail if expected tool outputs are not found
  • Rename --lint to --strict
  • Modules should now use ModuleNotFoundError instead of UserWarning when no logs are found
  • 2 new modules and updates to 9 modules.

MultiQC updates

  • Add CI action changelog.yml to populate the changelog from PR titles, triggered by a comment @multiqc-bot changelog (#2025, #2102, #2115)
  • Add GitHub Actions bot workflow to fix code linting from a PR comment (#2082)
  • Use custom exception type instead of UserWarning when no samples are found. (#2049)
  • Lint modules for missing self.add_software_version (#2081)
  • Strict mode: rename config.lint to config.strict, crash early on module or template error. Add MULTIQC_STRICT=1 (#2101)
  • Matplotlib line plots now respect xLog: True and yLog: True in config (#1632)
  • Fix matplotlib linegraph and bargraph for the case when xmax < xmin in config (#2124)
  • Add --require-logs flag to error out if requested modules not used (#2109)
  • Fixes for python 3.12
    • Replace removed distutils (#2113)
    • Bundle lzstring (#2119)
  • Drop Python 3.6 and 3.7 support, add 3.12 (#2121)
  • Just run CI on the oldest + newest supported Python versions (#2074)
  • /// New logo
  • Set name and anchor for the custom content "module" #2131
  • Fix use of shutil.copytree when overriding existing template files in tmp_dir (#2133)

New Modules

  • Bracken
    • A highly accurate statistical method that computes the abundance of species in DNA sequences from a metagenomics sample.
  • Truvari (#1751)
    • Truvari is a toolkit for benchmarking, merging, and annotating structural variants

Module updates

  • Dragen: make sure all inputs are recorded in multiqc_sources.txt (#2128)
  • Cellranger: Count submodule updated to parse Antibody Capture summary (#2118)
  • fastp: parse unescaped sample names with white spaces (#2108)
  • FastQC: Add top overrepresented sequences table (#2075)
  • HiCPro: Fix parsing scientific notation in hicpro-ashic. Thanks @Just-Roma (#2126)
  • HTSeq Count: allow counts files with more than 2 columns (#2129)
  • mosdepth: fix prioritizing region over global information (#2106)
  • Picard: Adapt WgsMetrics to parabricks bammetrics outputs (#2127)
  • Picard: MarkDuplicates: Fix parsing mixed strings/numbers, account for missing trailing 0 (#2083, #2094)
  • Samtools: Add MQ0 reads to the Percent Mapped barplot in Stats submodule (#2123)
  • WhatsHap: Process truncated input with no ALL chromosome (#2095)

- JavaScript
Published by ewels over 2 years ago

multiqc - MultiQC Version 1.16

Highlight: Reporting software versions

New in v1.16 - software version information can now automatically parsed from log output where available, and added to MultiQC in a standardised manner. It's shown in the MultiQC report next to section headings and in a dedicated report section, as well as being saved to multiqc_data. Where version information is not available in logs, it can be submitted manually by using a new special file type with filename pattern *_mqc_versions.yml. There's the option of representing groups of versions, useful for a tool that uses sub-tools, or pipelines that want to report version numbers per analysis step.

There are a handful of new config scopes to control behaviour: software_versions, skip_versions_section, disable_version_detection, versions_table_group_header. See the documentation for more (writing modules, supplying stand-alone)

Huge thanks to @pontushojer for the contribution (#1927). This idea goes way back to issue #290, made in 2016! 🎉

MultiQC updates

  • Removed simplejson unused dependency (#1973)
  • Give config custom_plot_config priority over column-specific settings set by modules
  • When exporting plots, make a more clear error message for unsupported FastQC dot plot (#1976)
  • Fixed parsing of plot_type: "html" data in json custom content
  • Replace deprecated pkg_resources
  • Fix the module groups configuration for modules where the namespace is passed explicitly to general_stats_addcols. Namespace is now always appended to the module name in the general stats (2037).
  • Do not call sys.exit() in the multiqc.run() function, to avoid breaking interactive environments. #2055
  • Fixed the DOI exports in multiqc_data to include more than just the MultiQC paper (#2058)
  • Fix table column color scaling then there are negative numbers (1869)
  • Export plots as static images and data in a ZIP archive. Fixes the issue when only 10 plots maximum were downloaded due to the browser limitation.

New Modules

  • Bakta
    • Rapid and standardized annotation of bacterial genomes, MAGs & plasmids.
  • mapDamage
    • mapDamage2 is a computational framework written in Python and R, which tracks and quantifies DNA damage patterns among ancient DNA sequencing reads generated by Next-Generation Sequencing platforms.
  • Sourmash
    • Quickly search, compare, and analyze genomic and metagenomic data sets.

Module updates

  • BcfTools
    • Stats: fix parsing multi-sample logs (#2052)
  • Custom content
    • Don't convert sample IDs to floats (#1883)
  • DRAGEN
    • Make DRAGEN module use fn_clean_exts instead of hardcoded file names. Fixes working with arbitrary file names (#1994)
  • FastQC:
    • fix UnicodeDecodeError when parsing fastqc_data.txt: try latin-1 or fail gracefully (#2024)
  • Kaiju:
    • Fix UnboundLocalError on outputs when Kanju was run with the -e flag (#2023)
  • Kraken
    • Parametrize top-N through config (#2060)
    • Fix bug where ranks incorrectly assigned to tabs (#1766).
  • Mosdepth
    • Add X/Y relative coverage plot, analogous to the one in samtools-idxstats (#1978)
    • Added the perchrom_fraction_cutoff option into the config to help avoid clutter in contig-level plots
    • Fix a bug happening when both region and global coverage histograms for a sample are available (i.e. when mosdepth was run with --by, see mosdepth docs). In this case, data was effectively merged. Instead, summarise it separately and add a separate report section for the region-based coverage data.
    • Do not fail when all input samples have no coverage (#2005).
  • NanoStat
    • Support new format (#1997).
  • RSeQC
    • Fix max() arg is an empty sequence error (#1985)
    • Fix division by zero on all-zero input (#2040)
  • Samtools
    • Stats: fix "Percent Mapped" plot when samtools was run with read filtering (#1972)
  • Qualimap
    • BamQC: Include % On Target in General Stats table (#2019)
  • WhatsHap
    • Bugfix: ensure that TSV is only split on tab character. Allows sample names with spaces (#1981)

New Contributors

  • @gdrosos made their first contribution in https://github.com/ewels/MultiQC/pull/1973
  • @antass made their first contribution in https://github.com/ewels/MultiQC/pull/1991
  • @lindenb made their first contribution in https://github.com/ewels/MultiQC/pull/1972
  • @odcambc made their first contribution in https://github.com/ewels/MultiQC/pull/1968
  • @ehenrion made their first contribution in https://github.com/ewels/MultiQC/pull/1855
  • @louperelo made their first contribution in https://github.com/ewels/MultiQC/pull/1959
  • @Joon-Klaps made their first contribution in https://github.com/ewels/MultiQC/pull/1903
  • @sstrong99 made their first contribution in https://github.com/ewels/MultiQC/pull/1891
  • @taylorreiter made their first contribution in https://github.com/ewels/MultiQC/pull/1812
  • @FerriolCalvet made their first contribution in https://github.com/ewels/MultiQC/pull/2020

Full Changelog: https://github.com/ewels/MultiQC/compare/v1.15...v1.16

- JavaScript
Published by ewels over 2 years ago

multiqc - MultiQC Version 1.15

Big speedup in file search step

This release of MultiQC introduces speed improvements to the file search (~5-7x fold improvement) 🎉

One way it does this is by limiting the number of lines loaded by each search pattern. For the vast majority of users, this should have no effect except faster searches. However, in some edge cases it may introduce some breaking changes. Hypothetically, for example:

  • If you concatenate log files from multiple tools
  • If you have a custom plugin module that we haven't tested

See the troubleshooting docs for more information.

MultiQC updates

  • Refactor file search for performance improvements (#1904)
  • Bump log_filesize_limit default (to skip large files in the search) from 10MB to 50MB.
  • Table code now tolerates lambda function calls with bad data (#1739)
  • Beeswarm plot now saves data to multiqc_data, same as tables (#1861)
  • Don't print DOI in module if it's set to an empty string.
  • Don't sort table headers alphabetically if we don't have an OrderedDict - regular dicts are fine in Py3 (#1866)
  • New back-end to preview + deploy the new website when the docs are edited.
  • Fixed a lot of broken links in the documentation from the new website change in structure.

New Modules

  • Librarian
    • A tool to predict the sequencing library type from the base composition of a supplied FastQ file.

Module updates

  • Cell Ranger
    • Bugfix: avoid multiple KeyError exceptions when parsing Cell Ranger 7.x web_summary.html (#1853, #1871)
  • DRAGEN
    • Restored functionality to show target BED coverage metrics (#1844)
    • Update filename pattern in RNA quant metrics (#1958)
  • filtlong
    • Handle reports from locales that use . as a thousands separator (#1843)
  • HUMID
    • Fix bug that prevent HUMID stats files from being parsed (#1856)
  • Mosdepth
    • Fix data not written to mosdepth_cumcov_dist.txt and mosdepth_cov_dist.txt (#1868)
    • Update documentation with new file {prefix}.mosdepth.summary.txt (#1868)
    • Fill in missing values for general stats table (#1868)
    • Include mosdepth/summary file paths in multiqc_sources.txt (#1868)
    • Enable log switch for Coverage per contig plot (#1868)
    • Fix y-axis scaling for Coverage distribution plot (#1868)
    • Handle case of intermediate missing coverage x-values in the *_dist.txt file causing a distorted Coverage distribution plot (#1960)
  • Picard
    • WgsMetrics: Fix wrong column label (#1888)
    • HsMetrics: Add missing field descriptions (#1928)
  • Porechop
    • Don't render bar graphs if no samples had any adapters trimmed (#1850)
    • Added report section listing samples that had no adapters trimmed
  • RSeQC
    • Fix ZeroDivisionError error for bam_stat results when there are 0 reads (#1735)
  • UMI-tools
    • Fix bug that broke the module with paired-end data (#1845)

- JavaScript
Published by ewels almost 3 years ago

multiqc - MultiQC Version 1.14

MultiQC new features

  • Rewrote the Dockerfile to build multi-arch images (amd64 + arm), run through a non-privileged user and build tools for non precompiled python binaries (#1541, #1541)
  • Add a new lint test to check that colour scale names are valid (#1835)
  • Update github actions to run tests on a single module if it is the only file affected by the PR (#915)
  • Add CI testing for Python 3.10 and 3.11
  • Optimize line-graph generation to remove an n^2 loop (#1668)
  • Parsing output file column headers is much faster.

MultiQC code cleanup

  • Remove Python 2-3 compatability from __future__ imports
  • Remove unused #!/usr/bin/env python hashbangs from module files
  • Add new code formatting tool isort to standardise the order and formatting of Python module imports
  • Add Pycln pre-commit hook to remove unused imports

MultiQC updates

  • Bugfix: Make config.data_format work again (#1722)
  • Bump minimum version of Jinja2 to >=3.0.0 (#1642)
  • Disable search progress bar if running with --quiet or --no-ansi (#1638)
  • Allow path filters without full paths by trying to prefix analysis dir when filtering (#1308)
  • Fix sorting of table columns with text values
  • Don't crash if a barplot is given an empty list of categories (#1540)
  • New logos! MultiQC is now developed and maintained at Seqera Labs. Updated logos and email addresses accordingly.

New Modules

  • Anglerfish
    • A tool designed to assess pool balancing, contamination and insert sizes of Illumina library dry runs on Oxford Nanopore data.
  • BBDuk
    • Combines most common data-quality-related trimming, filtering, and masking operations via kmers into a single high-performance tool.
  • Cell Ranger
    • Works with data from 10X Genomics Chromium. Processes Chromium single cell data to align reads, generate feature-barcode matrices, perform clustering and other secondary analysis, and more.
    • New MultiQC module parses Cell Ranger quality reports from VDJ and count analysis
  • DIAMOND
    • A high-throughput program for aligning DNA reads or protein sequences against a protein reference database.
  • DRAGEN-FastQC
    • Illumina Bio-IT Platform that uses FPGA for accelerated primary and secondary analysis
    • Finally merged the epic 2.5-year-old pull request, with 3.5k new lines of code.
    • Please report any bugs you find!
  • Filtlong
    • A tool for filtering long reads by quality.
  • GoPeaks
    • GoPeaks is used to call peaks in CUT&TAG/CUT&RUN datasets.
  • HiFiasm
    • A haplotype-resolved assembler for accurate Hifi reads
  • HUMID
    • HUMID is a tool to quickly and easily remove duplicate reads from FastQ files, with or without UMIs.
  • mOTUs
    • Microbial profiling through marker gene (MG)-based operational taxonomic units (mOTUs)
  • Nextclade
    • Tool that assigns clades to SARS-CoV-2 samples
  • Porechop
    • A tool for finding and removing adapters from Oxford Nanopore reads
  • PRINSEQ++
    • PRINSEQ++ is a C++ of prinseq-lite.pl program for filtering, reformating or trimming genomic and metagenomic sequence data.
  • UMI-tools
    • Work with Unique Molecular Identifiers (UMIs) / Random Molecular Tags (RMTs) and single cell RNA-Seq cell barcodes.

Module updates

  • Bcftools stats
    • Bugfix: Do not show empty bcftools stats variant depth plots (#1777)
    • Bugfix: Avoid exception when PSC nMissing column is not present (#1832)
  • BclConvert
    • Handle single-end read data correctly when setting cluster length instead of always assuming paired-end reads (#1697)
    • Handle different R1 and R2 read-lengths correctly instead of assuming they are the same (#1774)
    • Handle single-index paired-end data correctly
    • Added a config option to enable the creation of barplots with undetermined barcodes (create_unknown_barcode_barplots with False as default) (#1709)
  • BUSCO
    • Update BUSCO pass/warning/fail scheme to be more clear for users
  • Bustools
    • Show median reads per barcode statistic
  • Custom content
    • Create a report even if there's only Custom Content General Stats there
    • Attempt to cooerce line / scatter x-axes into floats so as not to lose labels (#1242)
    • Multi-sample line-graph TSV files that have no sample name in row 1 column 1 now use row 1 as x-axis labels (#1242)
  • fastp
    • Add total read count (after filtering) to general stats table (#1744)
    • Don't crash for invalid JSON files (#1652)
  • FastQC
    • Report median read-length for fastqc in addition to mean (#1745)
  • Kaiju
    • Don't crash if we don't have any data for the top-5 barplot (#1540)
  • Kallisto
    • Fix ZeroDivisionError when a sample has zero reads (#1746)
  • Kraken
    • Fix duplicate heatmap to account for missing taxons (#1779)
    • Make heatmap full width
    • Handle empty kreports gracefully (#1637)
    • Fix regex error with very large numbers of unclassified reads (#1639)
  • malt
    • Fixed division by 0 in malt module (#1683)
  • miRTop
    • Avoid KeyError - don't assume all fields present in logs (#1778)
  • Mosdepth
    • Don't pad the General Stats table with zeros for missing data (#1810)
  • Picard
    • HsMetrics: Allow custom columns in General Stats too, with HsMetrics_genstats_table_cols and HsMetrics_genstats_table_cols_hidden
  • Qualimap
    • Added additional columns in general stats for BamQC results that displays region on-target stats if region bed has been supplied (hidden by default) (#1798)
    • Bugfix: Remove General Stats rows for filtered samples (#1780)
  • RSeQC
    • Update geneBody_coverage to plot normalized coverages using a similar formula to that used by RSeQC itself (#1792)
  • Sambamba Markdup
    • Catch zero division in sambamba markdup (#1654)
  • Samtools
    • Added additional column for flagstat that displays percentage of mapped reads in a bam (hidden by default) (#1733)
  • VEP
    • Don't crash with ValueError if there are zero variants (#1681)

- JavaScript
Published by ewels over 3 years ago

multiqc - MultiQC Version 1.13

MultiQC updates

  • Major spruce of the command line help, using the new rich-click package
  • Drop some of the Python 2k compatability code (eg. custom requirements)
  • Improvements for running MultiQC in a Python environment, such as a Jupyter Notebook or script
    • Fixed bug raised when removing logging file handlers between calls that arose when configuring the root logger with dictConfig (#1643)
  • Added new config option custom_table_header_config to override any config for any table header
  • Fixed edge-case bug in custom content where a description that doesn't terminate in . gave duplicate section descriptions.
  • Tidied the verbose log to remove some very noisy statements and add summaries for skipped files in the search
  • Add timezone to time in reports
  • Add nix flake support
  • Added automatic tweet about new releases

Module updates

  • AdapterRemoval
    • Finally merge a fix for counts of reads that are discarded/collapsed (#1647)
  • VEP
    • Fixed bug when General Statistics have a value of - (#1656)
  • Custom content
    • Only set id for custom content when id not set by metadata (#1629)
    • Fix bug where module wouldn't run if all content was within a MultiQC config file (#1686)
    • Fix crash when info isn't set (#1688)
  • Nanostat
    • Removed HTML escaping of special characters in the log to fix bug in jinja2 v3.10 removing jinja2.escape() (#1659)
    • Fix bug where module would crash if input does not contain quality scores (#1717)
  • Pangolin
    • Updated module to handle outputs from Pangolin v4 (#1660)
  • Somalier
    • Handle zero mean X depth in Sex plot (#1670)
  • Fastp
    • Include low complexity and too long reads in filtering bar chart
  • miRTop
    • Fix module crashing when ref_miRNA_sum is missing in file. (#1712)
    • Fix module crashing due to zero division (#1719)
  • FastQC
    • Fixed error when parsing duplicate ratio when there is nan values in the report. (#1725)

- JavaScript
Published by ewels almost 4 years ago

multiqc - MultiQC Version 1.12

Version 1.12 of MultiQC brings with it a modest number of new modules, a few new core features and a swathe of bugfixes and general improvements. I hope that everyone continues to find it useful! You can see the full changes in this release here: https://github.com/ewels/MultiQC/compare/v1.11...v1.12

Special thanks to the 13 people who had their first MultiQC contributions in this release:

New Contributors * @HofLucien made their first contribution in https://github.com/ewels/MultiQC/pull/1486 * @jchorl made their first contribution in https://github.com/ewels/MultiQC/pull/1578 * @bjohnnyd made their first contribution in https://github.com/ewels/MultiQC/pull/1489 * @schorlton made their first contribution in https://github.com/ewels/MultiQC/pull/1567 * @MatthiasZepper made their first contribution in https://github.com/ewels/MultiQC/pull/1584 * @g-pacheco made their first contribution in https://github.com/ewels/MultiQC/pull/1587 * @paulstretenowich made their first contribution in https://github.com/ewels/MultiQC/pull/1605 * @yanick made their first contribution in https://github.com/ewels/MultiQC/pull/1595 * @MillironX made their first contribution in https://github.com/ewels/MultiQC/pull/1594 * @sguizard made their first contribution in https://github.com/ewels/MultiQC/pull/1593 * @maleasy made their first contribution in https://github.com/ewels/MultiQC/pull/1552 * @TomaszSuchan made their first contribution in https://github.com/ewels/MultiQC/pull/1271 * @massiddamt made their first contribution in https://github.com/ewels/MultiQC/pull/1021

✨ MultiQC - new features

  • Added option to customise default plot height in plot config (#1432)
  • Added --no-report flag to skip report generation (#1462)
  • Added support for priting tool DOI in report sections (#1177)
  • Added support for --custom-css-file / config.custom_css_files option to include custom CSS in the final report (#1573)
  • New plot config option labelSize to customise font size for axis labels in flat MatPlotLib charts (#1576)
  • Added support for customising table column names (#1255)

🔨 MultiQC - updates

  • MultiQC now skips modules for which no files were found - gives a small performance boost (#1463)
  • Improvements for running MultiQC in a Python environment, such as a Jupyter Notebook or script
    • Fixed logger bugs when calling multiqc.run multiple times by removing logging file handlers between calls (#1141)
    • Init/reset global state between runs (#1596)
  • Added commonly missing functions to several modules (#1468)
  • Wrote new script to check for the above function calls that should be in every module (.github/workflows/code_checks.py), runs on GitHub actions CI
  • Make table Conditional Formatting work at table level as well as column level. (#761)
  • CSS Improvements to make printed reports more attractive / readable (#1579)
  • Fixed a problem with numeric filenames (#1606)
  • Fixed nasty bug where line charts with a categorical x-axis would take categories from the last sample only (#1568)
  • Ignore any files called multiqc_data.json (#1598)
  • Check that the config path_filters is a list, convert to list if a string is supplied (#1539)

🎁 New Modules

  • CheckQC
    • A program designed to check a set of quality criteria against an Illumina runfolder
  • pbmarkdup
    • Mark duplicate reads from PacBio sequencing of an amplified library
  • WhatsHap
    • WhatsHap is a software for phasing genomic variants using DNA sequencing reads

🌟 Module feature additions

  • BBMap
    • Added handling for qchist output (#1021)
  • bcftools
    • Added a plot with samplewise number of sites, Ts/Tv, number of singletons and sequencing depth (#1087)
  • Mosdepth
    • Added mean coverage #1566
  • NanoStat
    • Recognize FASTA and FastQ report flavors (#1547)

🐛 Module updates

  • BBMap
    • Correctly handle adapter stats files with additional columns (#1556)
  • bclconvert
    • Handle change in output format in v3.9.3 with new Quality_Metrics.csv file (#1563)
  • bowtie
    • Minor update to handle new log wording in bowtie v1.3.0 (#1615)
  • CCS
    • Tolerate compound IDs generated by pbcromwell ccs in the general statistics (#1486)
    • Fix report parsing. Update test on attributes ids (#1583)
  • Custom content
    • Fixed module failing when writing data to file if there is a / in the section name (#1515)
    • Use filename for section header in files with no headers (#1550)
    • Sort custom content bargraph data by default (#1412)
    • Always save custom content data to file with a name reflecting the section name. (#1194)
  • DRAGEN
    • Fixed bug in sample name regular expression (#1537)
  • Fastp
    • Fixed % pass filter statistics (#1574)
  • FastQC
    • Fixed several bugs occuring when FastQC sections are skipped (#1488, #1533)
    • Clarify general statistics table header for length
  • goleft/indexcov
    • Fix ZeroDivisionError if no bins are found (#1586)
  • HiCPro
    • Better handling of errors when expected data keys are not found (#1366)
  • Lima
    • Move samples that have been renamed using --replace-names into the General Statistics table (#1483)
  • miRTrace
    • Replace hardcoded RGB colours with Hex to avoid errors with newer versions of matplotlib (#1263)
  • Mosdepth
    • Fixed issue #1568
    • Fixed a bug when reporting per contig coverage
  • Picard
    • Update ExtractIlluminaBarcodes to recognise more log patterns in newer versions of Picard (#1611)
  • Qualimap
    • Fix ZeroDivisionError in QM_RNASeq and skip genomic origins plot if no aligned reads are found (#1492)
  • QUAST
    • Clarify general statistics table header for length
  • RSeQC
    • Fixed minor bug in new TIN parsing where the sample name was not being correctly cleaned (#1484)
    • Fixed bug in the junction_saturation submodule (#1582)
    • Fixed bug where empty files caused tin submodule to crash (#1604)
    • Fix bug in read_distribution for samples with zero tags (#1571)
  • Sambamba
    • Fixed issue with a change in the format of output from sambamba markdup 0.8.1 (#1617)
  • Skewer
    • Fix ZeroDivisionError if no available reads are found (#1622)
  • Somalier
    • Plot scaled X depth instead of mean for Sex plot (#1546)
  • VEP
    • Handle table cells containing - instead of numbers (#1597)

- JavaScript
Published by ewels over 4 years ago

multiqc - MultiQC Version 1.11

A summer release for MultiQC 🥳 🏖️ Many thanks to everyone who has contributed!

MultiQC new features

  • New interactive slider controls for controlling heatmap colour scales (#1427)
  • Added new --replace-names / config sample_names_replace option to replace sample names during report generation
  • Added use_filename_as_sample_name config option / --fn_as_s_name command line flag (#949, #890, #864, #998, #1390)
    • Forces modules to use the log filename for the sample identifier, even if the module usually takes this from the file contents
    • Required a change to the clean_s_name() function arguments. All core MultiQC modules updated to reflect this.
    • Should be backwards compatible for custom modules. To adopt new behaviour, supply f instead of f["root"] as the second argument.
    • See the documenation for details: Using log filenames as sample names and Custom sample names.

MultiQC updates

  • Make the module crash tracebacks much prettier using rich
  • Refine the cli log output a little (nicely formatted header line + drop the [INFO])
  • Added docs describing tools for downstream analysis of MultiQC outputs.
  • Added CI tests for Python 3.9, pinned networkx package to >=2.5.1 (#1413)
  • Added patterns to config.fn_ignore_paths to avoid error with parsing installation dir / singularity cache (#1416)
  • Print a log message when flat-image plots are used due to sample size surpassing plots_flat_numseries config (#1254)
  • Fix the mqc_colours util function to lighten colours even when passing categorical or single-length lists.
  • Bugfix for Custom Content, using YAML configuration (eg. section headers) for images should now work

New Modules

  • BclConvert
    • Tool that converts / demultiplexes Illumina Binary Base Call (BCL) files to FASTQ files
  • Bustools
    • Tools for working with BUS files
  • ccs
    • Generate highly accurate single-molecule consensus reads from PacBio data
  • GffCompare
    • GffCompare can annotate and estimate accuracy of one or more GFF files compared with a reference annotation.
  • Lima
    • The PacBio Barcode Demultiplexer
  • NanoStat
    • Calculate various statistics from a long read sequencing datasets
  • ODGI
    • Optimized dynamic genome/graph implementation
  • Pangolin
    • Added MultiQC support for Pangolin, the tool that determines SARS-CoV-2 lineages
  • Sambamba Markdup
    • Added MultiQC module to add duplicate rate calculated by Sambamba Markdup.
  • Snippy
    • Rapid haploid variant calling and core genome alignment.
  • VEP
    • Added MultiQC module to add summary statistics of Ensembl VEP annotations.
    • Handle error from missing variants in VEP stats file. (#1446)

Module feature additions

  • Cutadapt
    • Added support for linked adapters #1329]
    • Parse whether trimming was 5' or 3' for Lengths of Trimmed Sequences plot where possible
  • Mosdepth
    • Include or exclude contigs based on patterns for coverage-per-contig plots
  • Picard
    • Add support for CollectIlluminaBasecallingMetrics, CollectIlluminaLaneMetrics, ExtractIlluminaBarcodes and MarkIlluminaAdapters (#1336)
    • New insertsize_xmax configuration option to limit the plotted maximum insert size for InsertSizeMetrics
  • Qualimap
    • Added new percentage coverage plot in QM_RNASeq (#1258)
  • RSeQC
    • Added a long-requested submodule to support showing the TIN (Transcript Integrity Number) (#737)

Module updates

  • biscuit
    • Duplicate Rate and Cytosine Retention tables are now bargraphs.
    • Refactor code to only calculate alignment statistics once.
    • Fixed bug where cytosine retentions values would not be properly read if in scientific notation.
  • bcl2fastq
    • Added sample name cleaning so that prepending directories with the -d flag works properly.
  • Cutadapt
    • Plot filtered reads even when no filtering category is found (#1328)
    • Don't take the last command line string for the sample name if it looks like a command-line flag (#949)
  • Dragen
    • Handled MultiQC crashing when run on single-end output from Dragen (#1374)
  • fastp
    • Handle a ZeroDivisionError if there are zero reads (#1444)
  • FastQC
    • Added check for if overrepresented_sequences is missing from reports (#1281)
  • Flexbar
    • Fixed bug where reports with 0 reads would crash MultiQC (#1407)
  • Kraken
    • Handle a ZeroDivisionError if there are zero reads (#1440)
    • Updated search patterns to handle edge case (#1428)
  • Mosdepth
    • Show barplot instead of line graph for coverage-per-contig plot if there is only one contig.
  • Picard
    • RnaSeqMetrics - fix assignment barplot labels to say bases instead of reads (#1408)
    • CrosscheckFingerprints - fix bug where LOD threshold was not detected when invoked with "new" picard cli style. fixed formatting bug (#1414)
    • Made checker for comma as decimal separator in HsMetrics more robust (#1296)
  • qc3C
    • Updated module to not fail on older field names.
  • Qualimap
    • Fixed wrong units in tool tip label (#1258)
  • QUAST
    • Fixed typo causing wrong number of contigs being displayed (#1442)
  • Sentieon
    • Handled ZeroDivisionError when input files have zero reads (#1420)
  • RSEM
    • Handled ZeroDivisionError when input files have zero reads (#1040)
  • RSeQC
    • Fixed double counting of some categories in read_distribution bar graph. (#1457)

- JavaScript
Published by ewels almost 5 years ago

multiqc - MultiQC Version 1.10.1

This is a relatively small release focussing just on bug fixes 🐛 - the last release revealed a couple of nasty ones which I felt were bad enough to justify a fast patch.. (99 bugs - fix 1, now you've got 103 bugs)

Many thanks to everyone who reports these problems along with example data 🕵🏻

MultiQC updates

  • Allow scientific notation numbers in colour scheme generation
    • Fixed bug with very small minimum numbers that only revelead itself after a bugfix done in the v1.10 release
  • Require at least rich version 9.4.0 to avoid SpinnerColumn AttributeError (#1393)
  • Dropped the Skipping search pattern log message from a warning to debug
  • Moved directory prepending with -d back to before sample name cleaning (as it was before v1.7) (#1264)
  • If linegraph plot data goes above ymax, only discard the data if the line doesn't come back again (#1257)
  • Allow top_modules to be specified as empty dicts (#1274)
  • Properly ignore .snakemake folders as intended (#1395)

Module updates

  • bcftools
    • Fixed bug where QUAL value . would crash MultiQC (#1400)
  • bowtie2
    • Fix bug where HiSAT2 paired-end bar plots were missing unaligned reads (#1230)
  • Deeptools
    • Handle plotProfile data where no upstream / downstream regions have been calculated around genes (#1317)
    • Fix IndexError caused by mysterious -1 in code.. (#1275)
  • FastQC
    • Replace NaN with 0 in the Per Base Sequence Content plot to avoid crashing the plot (#1246)
  • Picard
    • Fixed bug in ValidateSamFile module where additional whitespace at the end of the file would cause MultiQC to crash (#1397)
  • Somalier
    • Fixed bug where using sample name cleaning in a config would trigger a KeyError (#1234)

- JavaScript
Published by ewels about 5 years ago

multiqc - MultiQC Version 1.10

Many thanks to everyone's patience in waiting for this release, it is much appreciated!

Update for developers: Code linting

This is a big change for MultiQC developers. I have added automated code formatting and code linting (style checks) to MultiQC. This helps to keep the MultiQC code base consistent despite having many contributors and helps me to review pull-requests without having to consider whitespace. Specifically, MultiQC now uses three main tools:

All developers must run these tools when submitting changes via Pull-Requests! Automated CI tests now run with GitHub actions to check that all files pass the above tests. If any files do not, that test will fail giving a red :x: next to the pull request.

For further information, please see the documentation.

MultiQC updates

New MultiQC Features

  • --sample-filters now also accepts show_re and hide_re in addition to show and hide. The _re options use regex, while the "normal" options use globbing.
  • MultiQC config files now work with .yml file extension as well as .yaml
    • .yaml will take preference if both found.
  • Section comments can now also be added for General Statistics
    • section_comments: { general_stats: "My comment" }
  • New table header config option bgcols allows background colours for table cells with categorical data.
  • New table header config options cond_formatting_rules and cond_formatting_colours
    • Comparable functionality to user config options table_cond_formatting_rules and table_cond_formatting_colours, allowes module developers to format table cell values as labels.
  • New CI test looks for git merge markers in files
  • Beautiful new progress bar from the amazing willmcgugan/rich package.
  • Added a bunch of new default sample name trimming suffixes (see 8ac5c7b)
  • Added timeout-minutes: 10 to the CI test workflow to check that changes aren't negatively affecting run time too much.
  • New table header option bars_zero_centrepoint to treat 0 as zero width bars and plot bar length based on absolute values

New Modules

  • EigenStratDatabaseTools
    • Added MultiQC module to report SNP coverages from eigenstrat_snp_coverage.py in the general stats table.
  • HOPS
    • Post-alignment ancient DNA analysis tool for MALT
  • JCVI
    • Computes statistics on genome annotation.
  • ngsderive
    • Forensic analysis tool useful in backwards computing information from next-generation sequencing data.
  • OptiType
    • Precision HLA typing from next-generation sequencing data
  • PURPLE
    • A purity, ploidy and copy number estimator for whole genome tumor data
  • Pychopper
    • Identify, orient and trim full length Nanopore cDNA reads
  • qc3C
    • Reference-free QC of Hi-C sequencing data
  • Sentieon
    • Submodules added to catch Picard-based QC metrics files

Module updates

  • DRAGEN
    • Fix issue where missing out fields could crash the module (#1223)
    • Added support for whole-exome / targetted data (#1290)
  • featureCounts
  • fgbio
    • Fix ErrorRateByReadPosition to calculate ymax not just on the overall error_rate, but also specific base errors (ex. a_to_c_error_rate, a_to_g_error_rate, ...). (#1215)
    • Fix ErrorRateByReadPosition plotted line names to no longer concatenate multiple read identifiers and no longer have off-by-one read numbering (ex. Sample1_R2_R3 -> Sample1_R2) ([#1304)
  • Fastp
    • Fixed description for duplication rate (pre-filtering, not post) ([#1313)
  • GATK
    • Add support for the creation of a "Reported vs Empirical Quality" graph to the Base Recalibration module.
  • hap.py
    • Updated module to plot both SNP and INDEL stats (#1241)
  • indexcov
    • Fixed bug when making the PED file plots (#1265)
  • interop
    • Added the % Occupied metric to Read Metrics per Lane table which is reported for NovaSeq and iSeq platforms.
  • Kaiju
    • Fixed bug affecting inputs with taxa levels other than Phylum (#1217)
    • Rework barplot, add top 5 taxons (#1219)
  • Kraken
    • Fix ZeroDivisionError (#1276)
    • Add distinct minimizer heatmap for KrakenUniq style duplication information (#1333)
  • MALT
    • Fix y-axis labelling in bargraphs
  • MACS2
    • Add number of peaks to the General Statistics table.
  • mosdepth
    • Enable prepending of directory to sample names
    • Display contig names in Coverage per contig plot tooltip
  • Picard
    • Fix HsMetrics bait percentage columns (#1212)
    • Fix ConvertSequencingArtifactToOxoG files not being found (#1310)
    • Make WgsMetrics histogram smoothed if more than 1000 data points (avoids huge plots that crash the browser)
    • Multiple new config options for WgsMetrics to customise coverage histogram and speed up MultiQC with very high coverage files.
    • Add additional datasets to Picard Alignment Summary (#1293)
    • Add support for CrosscheckFingerprints (#1327)
  • PycoQC
    • Log10 x-axis for Read Length plot (#1214)
  • Rockhopper
    • Fix issue with parsing genome names in Rockhopper summary files (#1333)
    • Fix issue properly parsing multiple samples within a single Rockhopper summary file
  • Salmon
    • Only try to generate a plot for fragment length if the data was found.
  • verifyBamID
    • Fix CHIP value detection (#1316).

New Custom Content features

  • General Stats custom content now gives a log message
  • If id is not set in JSON or YAML files, it defaults to the sample name instead of just custom_content
  • Data from JSON or YAML now has data keys (sample names) run through the clean_s_name() function to apply sample name cleanup
  • Fixed minor bug which caused custom content YAML files with a string data type to not be parsed

Bug Fixes

  • Disable preservation of timestamps / modes when copying temp report files, to help issues with network shares (#1333)
  • Fixed MatPlotLib warning: FixedFormatter should only be used together with FixedLocator
  • Fixed long-standing min/max bug with shared minimum values for table columns using shared_key
  • Made table colour schemes work with negative numbers (don't strip - from values when making scheme)

- JavaScript
Published by ewels over 5 years ago

multiqc - MultiQC Version 1.9

Another massive release - many thanks to all of the contributors! Keep those pull-requests and issues coming!

Dropped official support for Python 2

Python 2 had its official sunset date on January 1st 2020, meaning that it will no longer be developed by the Python community. Part of the python.org statement reads:

That means that we will not improve it anymore after that day, even if someone finds a security problem in it. You should upgrade to Python 3 as soon as you can.

Very many Python packages no longer support Python 2 and it whilst the MultiQC code is currently compatible with both Python 2 and Python 3, it is increasingly difficult to maintain compatibility with the dependency packages it uses, such as MatPlotLib, numpy and more.

As of MultiQC version 1.9, Python 2 is no longer officially supported. Automatic CI tests will no longer run with Python 2 and Python 2 specific workarounds are no longer guaranteed.

Whilst it may be possible to continue using MultiQC with Python 2 for a short time by pinning dependencies, MultiQC compatibility for Python 2 will now slowly drift and start to break. If you haven't already, you need to switch to Python 3 now.

New MultiQC Features

  • Now using GitHub Actions for all CI testing
    • Dropped Travis and AppVeyor, everything is now just on GitHub
    • Still testing on both Linux and Windows, with multiple versions of Python
    • CI tests should now run automatically for anyone who forks the MultiQC repository
  • Linting with --lint now checks line graphs as well as bar graphs
  • New gathered template with no tool name sections (#1119)
  • Added --sample-filters option to add show/hide buttons at the top of the report (#1125)
    • Buttons control the report toolbox Show/Hide tool, filtering your samples
    • Allows reports to be pre-configured based on a supplied list of sample names at report-generation time.
  • Line graphs can now have Log10 buttons (same functionality as bar graphs)
  • Importing and running multiqc in a script is now a little Better
    • multiqc.run now returns the report and config as well as the exit code. This means that you can explore the MultiQC run time a little in the Python environment.
    • Much more refactoring is needed to make MultiQC as useful in Python scripts as it could be. Watch this space.
  • If a custom module anchor is set using module_order, it's now used a bit more:
    • Prefixed to module section IDs
    • Appended to files saved in multiqc_data
    • Should help to prevent duplicates requiring -1 suffixes when running a module multiple times
  • New heatmap plot config options xcats_samples and ycats_samples
    • If set to False, the report toolbox options (highlight, rename, show/hide) do not affect that axis.
    • Means that the Show only matching samples report toolbox option works on FastQC Status Checks, for example (#1172)
  • Report header time and analysis paths can now be hidden
    • New config options show_analysis_paths and show_analysis_time (#1113)
  • New search pattern key skip: true to skip specific searches when modules look for a lot of different files (eg. Picard).
  • New --profile-runtime command line option (config.profile_runtime) to give analysis of how long the report takes to be generated
    • Plots of the file search results and durations are added to the end of the MultiQC report as a special module called Run Time
    • A summary of the time taken for the major stages of MultiQC execution are printed to the command line log.
  • New table config option only_defined_headers
    • Defaults to true, set to false to also show any data columns that are not defined as headers
    • Useful as allows table-wide defaults to be set with column-specific overrides
  • New module key allowed for config.extra_fn_clean_exts and config.fn_clean_exts
    • Means you can limit the action of a sample name cleaning pattern to specific MultiQC modules (#905)

New Custom Content features

  • Improve support for HTML files - now just end your HTML filename with _mqc.html
    • Native handling of HTML snippets as files, no MultiQC config or YAML file required.
    • Also with embedded custom content configuration at the start of the file as a HTML comment.
  • Add ability to group custom-content files into report sections
    • Use the new parent_id, parent_name and parent_description config keys to group content together like a regular module (#1008)
  • Custom Content files can now be configured using custom_data, without giving search patterns or data
    • Allows you to set descriptions and nicer titles for images and other 'blunt' data types in reports (#1026)
    • Allows configuration of custom content separately from files themselves (tsv, csv, txt formats) (#1205)

New Modules:

  • DRAGEN
    • Illumina Bio-IT Platform that uses FPGA for secondary NGS analysis
  • iVar
    • Added support for iVar: a computational package that contains functions broadly useful for viral amplicon-based sequencing.
  • Kaiju
    • Fast and sensitive taxonomic classification for metagenomics
  • Kraken
    • K-mer matching tool for taxonomic classification. Module plots bargraph of counts for top-5 hits across each taxa rank. General stats summary.
  • MALT
    • Megan Alignment Tool: Metagenomics alignment tool.
  • miRTop
    • Command line tool to annotate miRNAs with a standard mirna/isomir naming (mirGFF3)
    • Module started by @oneillkza and completed by @FlorianThibord
  • MultiVCFAnalyzer
    • Combining multiple VCF files into one coherent report and format for downstream analysis.
  • Picard - new submodules for QualityByCycleMetrics, QualityScoreDistributionMetrics & QualityYieldMetrics
  • Rockhopper
    • RNA-seq tool for bacteria, includes bar plot showing where features map.
  • Sickle
    • A windowed adaptive trimming tool for FASTQ files using quality
  • Somalier
    • Relatedness checking and QC for BAM/CRAM/VCF for cancer, DNA, BS-Seq, exome, etc.
  • VarScan2
    • Variant calling and somatic mutation/CNV detection for next-generation sequencing data

Module updates:

  • BISCUIT
    • Major rewrite to work with new BISCUIT QC script (BISCUIT v0.3.16+)
      • This change breaks backwards-compatability with previous BISCUIT versions. If you are unable to upgrade BISCUIT, please use MultiQC v1.8.
    • Fixed error when missing data in log files (#1101)
  • bcl2fastq
    • Samples with multiple library preps (i.e barcodes) will now be handled correctly (#1094)
  • BUSCO
    • Updated log search pattern to match new format in v4 with auto-lineage detection option (#1163)
  • Cutadapt
    • New bar plot showing the proportion of reads filtered out for different criteria (eg. too short, too many Ns) (#1198)
  • DamageProfiler
    • Removes redundant typo in init name. This makes referring to the module's column consistent with other modules when customising general stats table.
  • DeDup
    • Updates plots to make compatible with 0.12.6
    • Fixes reporting errors - barplot total represents mapped reads, not total reads in BAM file
    • New: Adds 'Post-DeDup Mapped Reads' column to general stats table.
  • FastQC
    • Fixed tooltip text in Sequence Duplication Levels plot (#1092)
    • Handle edge-case where a FastQC report was for an empty file with 0 reads (#1129)
  • FastQ Screen
    • Don't skip plotting % No Hits even if it's 0% (#1126)
    • Refactor parsing code. Avoids error with -0.00 %Unmapped (#1126)
    • New plot for Bisulfite Reads, if data is present
    • Categories in main plot are now sorted by the total read count and hidden if 0 across all samples
  • fgbio
    • New: Plot error rate by read position from ErrorRateByReadPosition
    • GroupReadsByUmi plot can now be toggled to show relative percents (#1147)
  • FLASh
    • Logs not reporting innie and outine uncombined pairs now plot combined pairs instead (#1173)
  • GATK
    • Made parsing for VariantEval more tolerant, so that it will work with output from the tool when run in different modes (#1158)
  • MTNucRatioCalculator
    • Fixed misleading value suffix in general stats table
  • Picard MarkDuplicates
    • Major change - previously, if multiple libraries (read-groups) were found then only the first would be used and all others ignored. Now, values from all libraries are merged and PERCENT_DUPLICATION and ESTIMATED_LIBRARY_SIZE are recalculated. Libraries can be kept as separate samples with a new MultiQC configuration option - picard_config: markdups_merge_multiple_libraries: False
    • Major change - Updated MarkDuplicates bar plot to double the read-pair counts, so that the numbers stack correctly. (#1142)
  • Picard HsMetrics
    • Updated large table to use columns specified in the MultiQC config. See docs. (#831)
  • Picard WgsMetrics
    • Updated parsing code to recognise new java class string (#1114)
  • QualiMap
    • Fixed QualiMap mean coverage calculation #1082, #1077
  • RSeqC
    • Support added for output from geneBodyCoverage2.py script (#844)
    • Single sample view in the "Junction saturation" plot now works with the toolbox properly (rename, hide, highlight) (#1133)
  • RNASeQC2
  • Samblaster
    • Improved parsing to handle variable whitespace (#1176)
  • Samtools
    • Removes hardcoding of general stats column names. This allows column names to indicate when a module has been run twice (https://github.com/ewels/MultiQC/issues/1076).
    • Added an observed over expected read count plot for idxstats (#1118)
    • Added additional (by default hidden) column for flagstat that displays number total number of reads in a bam
  • sortmerna
    • Fix the bug for the latest sortmerna version 4.2.0 (#1121)
  • sexdeterrmine
    • Added a scatter plot of relative X- vs Y-coverage to the generated report.
  • VerifyBAMID
    • Allow files with column header FREEMIX(alpha) (#1112)

Bug Fixes:

  • Added a new test to check that modules work correctly with --ignore-samples. A lot of them didn't:
    • Mosdepth, conpair, Qualimap BamQC, RNA-SeQC, GATK BaseRecalibrator, SNPsplit, SeqyClean, Jellyfish, hap.py, HOMER, BBMap, DeepTools, HiCExplorer, pycoQC, interop
    • These modules have now all been fixed and --ignore-samples should work as you expect for whatever data you have.
  • Removed use of shutil.copy to avoid problems with working on multiple filesystems (#1130)
  • Made folder naming behaviour of multiqc_plots consistent with multiqc_data
    • Incremental numeric suffixes now added if folder already exists
    • Plots folder properly renamed if using -n/--filename
  • Heatmap plotting function is now compatible with MultiQC toolbox hide and highlight (#1136)
  • Plot config logswitch_active now works as advertised
  • When running MultiQC modules several times, multiple data files are now created instead of overwriting one another (#1175)
  • Fixed minor bug where tables could report negative numbers of columns in their header text
  • Fixed bug where numeric custom content sample names could trigger a TypeError (#1091)
  • Fixed custom content bug HTML data in a config file would trigger a ValueError (#1071)
  • Replaced deprecated 'warn()' with 'warning()' of the logging module
  • Custom content now supports section_extra config key to add custom HTML after description.
  • Barplots with ymax set now ignore this when you click the Percentages tab.

- JavaScript
Published by ewels about 6 years ago

multiqc - MultiQC Version 1.8

A huge release, this one has been a long time coming. Due to @ewels being away on paternity leave for over six months it was very delayed and has been nearly a year in the making! During that time there has been 344 commits with 3,370 lines of code added and 1,194 deletions by 19 contributors. That's a lot of changes.

Highlights include:

  • Finally removing the annoying YAML warning
  • Six new modules, and many large updates to existing modules
  • Code restructuring allowing MultiQC to be imported into Python environments and easier running on Windows
  • Lots of tiny bug fixes all over the place.

Enjoy the update! And I promise I'll try not to make everyone wait so long for the next release...

Full changelog

New Modules:

  • fgbio
    • Process family size count hist data from GroupReadsByUmi
  • biobambam2
    • Added submodule for bamsormadup tool
    • Totally cheating - it uses Picard MarkDuplicates but with a custom search pattern and naming
  • SeqyClean
    • Adds analysis for seqyclean files
  • mtnucratio
    • Added little helper tool to compute mt to nuclear ratios for NGS data.
  • mosdepth
    • fast BAM/CRAM depth calculation for WGS, exome, or targeted sequencing
  • SexDetErrmine
    • Relative coverage and error rate of X and Y chromosomes

Module updates:

  • bcl2fastq
    • Added handling of demultiplexing of more than 2 reads
    • Allow bcl2fastq to parse undetermined barcode information in situations when lane indexes do not start at 1
  • BBMap
    • Support for scafstats output marked as not yet implemented in docs
  • DeDup
    • Added handling clusterfactor and JSON logfiles
  • damageprofiler
    • Added writing metrics to data output file.
  • DeepTools
    • Fixed Python3 bug with int() conversion (#1057)
    • Handle varied TES boundary labels in plotProfile (#1011)
    • Fixed bug that prevented running on only plotProfile files when no other deepTools files found.
  • fastp
    • Fix faulty column handling for the after filtering Q30 rate (#936)
  • FastQC
    • When including a FastQC section multiple times in one report, the Per Base Sequence Content heatmaps now behave as you would expect.
    • Added heatmap showing FastQC status checks for every section report across all samples
    • Made sequence content individual plots work after samples have been renamed (#777)
    • Highlighting samples from status - respect chosen highlight colour in the toolbox (#742)
  • FastQ Screen
    • When including a FastQ Screen section multiple times in one report, the plots now behave as you would expect.
  • GATK
    • Refactored BaseRecalibrator code to be more consistent with MultiQC Python style
    • Handle zero count errors in BaseRecalibrator
  • HiC Explorer
    • Fixed bug where module tries to parse QC_table.txt, a new log file in hicexplorer v2.2.
  • HTSeq
    • Fixed bug where module would crash if a sample had zero reads (#1006)
  • LongRanger
    • Added support for the LongRanger Align pipeline.
  • miRTrace
    • Fixed bug where a sample in some plots was missed. (#932)
  • Peddy
    • Fixed bug where sample name cleaning could lead to error. (#1024)
    • All plots (including Het Check and Sex Check) now hidden if no data
  • Picard
    • Modified OxoGMetrics.py so that it will find files created with GATK CollectMultipleMetrics and ConvertSequencingArtifactToOxoG.
  • QoRTs
    • Fixed bug where --dirs broke certain input files. (#821)
  • Qualimap
    • Added in mean coverage computation for general statistics report
    • Creates now tables of collected data in multiqc_data
  • RNA-SeQC
    • Updated broken URL link
  • RSeQC
    • Fixed bug where Junction Saturation plot when clicking a single sample was mislabelling the lines.
    • When including a RSeQC section multiple times in one report, clicking Junction Saturation plot now behaves as you would expect.
    • Fixed bug where exported data in multiqc_rseqc_read_distribution.txt files had incorrect values for _kb fields (#1017)
  • Samtools
    • Utilize in-built read_count_multiplier functionality to plot flagstat results more nicely
  • SnpEff
    • Increased the default summary csv file-size limit from 1MB to 5MB.
  • Stacks
    • Fixed bug where multi-population sum stats are parsed correctly (#906)
  • TopHat
    • Fixed bug where TopHat would try to run with files from Bowtie2 or HiSAT2 and crash
  • VCFTools
    • Fixed a bug where tstv_by_qual.py produced invalid json from infinity-values.
  • snpEff
    • Added plot of effects

New MultiQC Features:

  • Added some installation docs for windows
  • Added some docs about using MultiQC in bioinformatics pipelines
  • Rewrote Docker image
    • New base image czentye/matplotlib-minimal reduces image size from ~200MB to ~80MB
    • Proper installation method ensures latest version of the code
    • New entrypoint allows easier command-line usage
  • Support opening MultiQC on websites with CSP script-src 'self' with some sha256 exceptions
    • Plot data is no longer intertwined with javascript code so hashes stay the same
  • Made config.report_section_order work for module sub-sections as well as just modules.
  • New config options exclude_modules and run_modules to complement -e and -m cli flags.
  • Command line output is now coloured by default :rainbow: (use --no-ansi to turn this off)
  • Better launch comparability due to code refactoring by @KerstenBreuer and @ewels
    • Windows support for base multiqc command
    • Support for running as a python module: python -m multiqc .
    • Support for running within a script: import multiqc and multiqc.run('/path/to/files')
  • Config option custom_plot_config now works for bargraph category configs as well (#1044)
  • Config table_columns_visible can now be given a module namespace and it will hide all columns from that module (#541)

Bug Fixes:

  • MultiQC now ignores all .md5 files
  • Use SafeLoader for PyYaml load calls, avoiding recent warning messages.
  • Hide multiqc_config_example.yaml in the test directory to stop people from using it without modification.
  • Fixed matplotlib background colour issue (@epakarin - #886)
  • Table rows that are empty due to hidden columns are now properly hidden on page load (#835)
  • Sample name cleaning: All sample names are now truncated to their basename, without a path.
    • This includes for regex and replace (before was only the default truncate).
    • Only affects modules that take sample names from file contents, such as cutadapt.
    • See #897 for discussion.

- JavaScript
Published by ewels over 6 years ago

multiqc - MultiQC Version 1.7

An early Christmas present for MultiQC users! 🎅🎁🎄

Many thanks to everyone who has contributed to this release. Happy Christmas and a very happy new year!

New Modules:

  • BISCUIT
    • BISuilfite-seq CUI Toolkit
    • Module written by @zwdzwd
  • DamageProfiler
    • A tool to determine ancient DNA misincorporation rates.
    • Module written by @apeltzer
  • FLASh
    • FLASH (Fast Length Adjustment of SHort reads)
    • Module written by @pooranis
  • MinIONQC
    • QC of reads from ONT long-read sequencing
    • Module written by @ManavalanG
  • phantompeakqualtools
    • A tool for informative enrichment and quality measures for ChIP-seq/DNase-seq/FAIRE-seq/MNase-seq data.
    • Module written by @chuan-wang
  • Stacks
    • A software for analyzing restriction enzyme-based data (e.g. RAD-seq). Support for Stacks >= 2.1 only.
    • Module written by @remiolsen

Module updates:

  • AdapterRemoval
    • Handle error when zero bases are trimmed. See #838.
  • Bcl2fastq
    • New plot showing the top twenty of undetermined barcodes by lane.
    • Informations for R1/R2 are now separated in the General Statistics table.
    • SampleID is concatenate with SampleName because in Chromium experiments several sample have the same SampleName.
  • deepTools
    • New PCA plots from the plotPCA function (written by @chuan-wang)
    • New fragment size distribution plots from bamPEFragmentSize --outRawFragmentLengths (written by @chuan-wang)
    • New correlation heatmaps from the plotCorrelation function (written by @chuan-wang)
    • New sequence distribution profiles around genes, from the plotProfile function (written by @chuan-wang)
    • Reordered sections
  • Fastp
    • Fixed bug in parsing of empty histogram data. See #845.
  • FastQC
    • Refactored Per Base Sequence Content plots to show original underlying data, instead of calculating it from the page contents. Now shows original FastQC base-ranges and fixes 100% GC bug in final few pixels. See #812.
    • When including a FastQC section multiple times in one report, the summary progress bars now behave as you would expect.
  • FastQ Screen
    • Don't hide genomes in the simple plot, even if they have zero unique hits. See #829.
  • InterOp
    • Fixed bug where read counts and base pair yields were not displaying in tables correctly.
    • Number formatting for these fields can now be customised in the same way as with other modules, as described in the docs
  • Picard
    • InsertSizeMetrics: You can now configure to what degree the insert size plot should be smoothed.
    • CollectRnaSeqMetrics: Add warning about missing rRNA annotation.
    • CollectRnaSeqMetrics: Add chart for counts/percentage of reads mapped to the correct strand.
    • Now parses VariantCallingMetrics reports. (Similar to GATK module's VariantEval.)
  • phantompeakqualtools
    • Properly clean sample names
  • Trimmomatic
    • Updated Trimmomatic module documentation to be more helpful
    • New option to use filenames instead of relying on the command line used. See #864.

New MultiQC Features:

  • Embed your custom images with a new Custom Content feature! Just add _mqc to the end of the filename for .png, .jpg or .jpeg files.
  • Documentation for Custom Content reordered to make it a little more sane
  • You can now add or override any config parameter for any MultiQC plot! See the documentation for more info.
  • Allow table_columns_placement config to work with table IDs as well as column namespaces. See #841.
  • Improved visual spacing between grouped bar plots

Bug Fixes:

  • Custom content no longer clobbers col1_header table configs
  • The option --file-list that refers to a text file with file paths to analyse will no longer ignore directory paths
  • Sample name directory prefixes are now added after cleanup.
  • If a module is run multiple times in one report, it's CSS and JS files will only be included once (default template)

- JavaScript
Published by ewels over 7 years ago

multiqc - MultiQC Version 1.6

Some of these updates are thanks to the efforts of people who attended the NASPM 2018 MultiQC hackathon session. Thanks to everyone who attended!

New Modules:

  • fastp
    • An ultra-fast all-in-one FASTQ preprocessor (QC, adapters, trimming, filtering, splitting...)
    • Module started by @florianduclot and completed by @ewels
  • hap.py
    • Hap.py is a set of programs based on htslib to benchmark variant calls against gold standard truth datasets
    • Module written by @tsnowlan
  • Long Ranger
    • Works with data from the 10X Genomics Chromium. Performs sample demultiplexing, barcode processing, alignment, quality control, variant calling, phasing, and structural variant calling.
    • Module written by @remiolsen
  • miRTrace
    • A quality control software for small RNA sequencing data.
    • Module written by @chuan-wang

Module updates:

  • BCFtools
  • BBMap
    • Support added for BBDuk kmer-based adapter/contaminant filtering summary stats (@boulund
  • FastQC
    • New read count plot, split into unique and duplicate reads if possible.
    • Help text added for all sections, mostly copied from the excellent FastQC help.
    • Sequence duplication plot rescaled
  • FastQ Screen
    • Samples in large-sample-number plot are now sorted alphabetically (@hassanfa
  • MACS2
    • Output is now more tolerant of missing data (no plot if no data)
  • Peddy
    • Background samples now shown in ancestry PCA plot (@roryk)
    • New plot showing sex checks versus het ratios, supporting unknowns (@oyvinev)
  • Picard
    • New submodule to handle ValidateSamFile reports (@cpavanrun)
    • WGSMetrics now add the mean and standard-deviation coverage to the general stats table (hidden) (@cpavanrun)
  • Preseq
    • New config option to plot preseq plots with unique old coverage on the y axis instead of read count
    • Code refactoring by @vladsaveliev
  • QUAST
    • Null values (-) in reports now handled properly. Bargraphs always shown despite varying thresholds. (@vladsaveliev)
  • RNA-SeQC
    • Don't create the report section for Gene Body Coverage if no data is given
  • Samtools
    • Fixed edge case bug where MultiQC could crash if a sample had zero count coverage with idxstats.
    • Adds % proper pairs to general stats table
  • Skewer
    • Read length plot rescaled
  • Tophat
    • Fixed bug where some samples could be given a blank sample name (@lparsons)
  • VerifyBamID
    • Change column header help text for contamination to match percentage output (@chapmanb)

New MultiQC Features:

  • New config option remove_sections to skip specific report sections from modules
  • Add path_filters_exclude to exclude certain files when running modules multiple times. You could previously only include certain files.
  • New exclude_* keys for file search patterns
    • Have a subset of patterns to exclude otherwise detected files with, by filename or contents
  • Command line options all now use mid-word hyphens (not a mix of hyphens and underscores)
    • Old underscore terms still maintained for backwards compatibility
  • Flag --view-tags now works without requiring an "analysis directory".
  • Removed Python dependency for enum34 (@boulund)
  • Columns can be added to General Stats table for custom content/module.
  • New --ignore-symlinks flag which will ignore symlinked directories and files.
  • New --no-megaqc-upload flag which disables automatically uploading data to MegaQC

Bug Fixes

  • Fix pathfilters for topmodules/module_order configuration only selecting if all globs match. It now filters searches that match any glob.
  • Empty sample names from cleaning are now no longer allowed
  • Stop prepend_dirs set in the config from getting clobbered by an unpassed CLI option (@tsnowlan)
  • Modules running multiple times now have multiple sets of columns in the General Statistics table again, instead of overwriting one another.
  • Prevent tables from clobbering sorted row orders.
  • Fix linegraph and scatter plots data conversion (sporadically the incorrect ymax was used to drop data points) (@cpavanrun)
  • Adjusted behavior of ceiling and floor axis limits
  • Adjusted multiple file search patterns to make them more specific
    • Prevents the wrong module from accidentally slurping up output from a different tool. By @cpavanrun (see PR #727)
  • Fixed broken report bar plots when -p/--export-plots was specified (see issue #801)

- JavaScript
Published by ewels almost 8 years ago

multiqc - MultiQC Version 1.5

New Modules:

  • DeDup - New module!
    • DeDup: Improved Duplicate Removal for merged/collapsed reads in ancient DNA analysis
    • Module written by @apeltzer,
  • Clip&Merge - New module!
    • Clip&Merge: Adapter clipping and read merging for ancient DNA analysis
    • Module written by @apeltzer,

Module updates:

  • bcl2fastq
    • Catch ZeroDivisionError exceptions when there are 0 reads (@aledj2)
    • Add parsing of TrimmedBases and new General Stats column for % bases trimmed (@matthdsm).
  • BUSCO
    • Fixed configuration bug that made all sample names become 'short'
  • Custom Content
    • Parsed tables now exported to multiqc_data files
  • Cutadapt
    • Refactor parsing code to collect all length trimming plots
  • FastQC
    • Fixed starting y-axis label for GC-content lineplot being incorrect.
  • HiCExplorer
    • Updated to work with v2.0 release.
  • Homer
    • Made parsing of tagInfo.txt file more resilient to variations in file format so that it works with new versions of Homer.
    • Kept order of chromosomes in coverage plot consistent.
  • Peddy
    • Switch Sex error logic to Correct sex for better highlighting (@aledj2)
  • Picard
    • Updated module and search patterns to recognise new output format from Picard version >= 2.16 and GATK output.
  • Qualimap BamQC
    • Fixed bug where start of Genome Fraction could have a step if target is 100% covered.
  • RNA-SeQC
    • Added rRNA alignment stats to summary table @Rolandde
  • RSeqC
    • Fixed read distribution plot by adding category for other_intergenic (thanks to @moxgreen)
    • Fixed a dodgy plot title (Read GC content)
  • Supernova
    • Added support for Supernova 2.0 reports. Fixed a TypeError bug when using txt reports only. Also a bug when parsing empty histogram files.

New MultiQC Features:

  • Invalid choices for --module or --exclude now list the available modules alphabetically.
  • Linting now checks for presence in config.module_order and tags.

Bug Fixes

  • Excluding modules now works in combination with using module tags.
  • Fixed edge-case bug where certain combinations of output_fn_name and data_dir_name could trigger a crash
  • Conditional formatting - values are now longer double-labelled
  • Made config option extra_series work in scatter plots the same way that it works for line plots
  • Locked the matplotlib version to v2.1.0 and below
    • Due to two bugs that appeared in v2.2.0 - will remove this constraint when there's a new release that works again.

- JavaScript
Published by ewels over 8 years ago

multiqc - MultiQC Version 1.4

A slightly earlier-than-expected release due to a new problem with dependency packages that is breaking MultiQC installations since 2018-01-11.

New Modules:

  • Sargasso
    • Parses output from Sargasso - a tool to separate mixed-species RNA-seq reads according to their species of origin
    • Module written by @hxin
  • VerifyBAMID
    • Parses output from VerifyBAMID - a tool to detect contamination in BAM files.
    • Adds the CHIPMIX and FREEMIX columns to the general statistics table.
    • Module written by @aledj2

Module updates:

  • MACS2
    • Updated to work with output from older versions of MACS2 by @avilella
  • Peddy
    • Add het check plot to suggest potential contamination by @aledj2
  • Picard
    • Picard HsMetrics HS_PENALTY plot now has correct axis labels
    • InsertSizeMetrics switches commas for points if it can't convert floats. Should help some european users.
  • QoRTs
    • Added support for new style of output generated in the v1.3.0 release
  • Qualimap
    • New Error rate column in General Statistics table, added by @Cashalow
      • Hidden by default - customise your MultiQC config to always show this column (see docs)
  • QUAST
    • New option to customise the default display of contig count and length (eg. bp instead of Mbp).
    • See documentation. Written by @ewels and @Cashalow
  • RSeQC
    • Removed normalisation in Junction Saturation plot. Now raw counts instead of % of total junctions.

New MultiQC Features:

  • Conditional formatting / highlighting of cell contents in tables
    • If you want to make values that match a criteria stand out more, you can now write custom rules and formatting instructions for tables.
    • For instructions, see the documentation
  • New --lint option which is strict about best-practices for writing new modules
    • Useful when writing new modules and code as it throws warnings
    • Currently only implemented for bar plots and a few other places. More linting coming soon...
  • If MultiQC breaks and shows am error message, it now reports the filename of the last log it found
    • Hopefully this will help with debugging / finding dodgy input data

Bug Fixes

  • Addressed new dependency error with conflicting package requirements
    • There was a conflict between the networkx, colormath and spectra releases.
    • I previously forced certain software versions to get around this, but spectra has now updated with the unfortunate effect of introducing a new dependency clash that halts installation.
  • Fixed newly introduced bug where Custom Content MultiQC config file search patterns had been broken
  • Updated pandoc command used in --pdf to work with new releases of Pandoc
  • Made config table_columns_visible module name key matching case insensitive to make less frustrating

- JavaScript
Published by ewels over 8 years ago

multiqc - MultiQC Version 1.3

There are 34 merged pull-requests in this release - a fantastic example of how an open source community can develop a tool! Many thanks to everyone involved for their hard work.

Breaking changes - custom search patterns

Only for users with custom search patterns for the bowtie or star: you will need to update your config files - the bowtie search key is now bowtie1, star_genecounts is now star/genecounts.

For users with custom modules - search patterns must now conform to the search pattern naming convention: modulename or modulename/anything (the search pattern string beginning with the name of your module, anything you like after the first /).

New Modules:

  • 10X Supernova
    • Parses statistics from the de-novo Supernova software.
    • Module written by @remiolsen
  • BBMap
    • Plot metrics from a number of BBMap tools, a suite of DNA/RNA mapping tools and utilities
    • Module written by @boulund and @epruesse
  • deepTools - new module!
    • Parse text output from bamPEFragmentSize, estimateReadFiltering, plotCoverage, plotEnrichment, and plotFingerprint
    • Module written by @dpryan79
  • Homer Tag Directory - new submodule!
  • illumina InterOp
    • Module to parse metrics from illumina sequencing runs and demultiplexing, generated by the InterOp package
    • Module written by @matthdsm
  • RSEM - new module!
    • Parse .cnt file comming from rsem-calculate-expression and plot read repartitions (Unalignable, Unique, Multi ...)
    • Module written by @noirot
  • HiCExplorer
    • New module to parse the log files of hicBuildMatrix.
    • Module written by @joachimwolff

Module updates:

  • AfterQC
    • Handle new output format where JSON summary key changed names.
  • bcl2fastq
    • Clusters per sample plot now has tab where counts are categoried by lane.
  • GATK
    • New submodule to handle Base Recalibrator stats, written by @winni2k
  • HiSAT2
    • Fixed bug where plot title was incorrect if both SE and PE bargraphs were in one report
  • Picard HsMetrics
    • Parsing code can now handle commas for decimal places
  • Preseq
    • Updated odd file-search pattern that limited input files to 500kb
  • QoRTs
    • Added new plots, new helptext and updated the module to produce a lot more output.
  • Qualimap BamQC
    • Fixed edge-case bug where the refactored coverage plot code could raise an error from the range call.
  • Documentation and link fixes for Slamdunk, GATK, bcl2fastq, Adapter Removal, FastQC and main docs
  • Went through all modules and standardised plot titles
    • All plots should now have a title with the format Module name: Plot name

New MultiQC Features:

  • New MultiQC docker image
    • Ready to use docker image now available at https://hub.docker.com/r/ewels/multiqc/ (200 MB)
    • Uses automated builds - pull :latest to get the development version, future releases will have stable tags.
    • Written by @MaxUlysse
  • New module_order config options allow modules to be run multiple times
    • Filters mean that a module can be run twice with different sets of files (eg. before and after trimming)
    • Custom module config parameters can be passed to module for each run
  • File search refactored to only search for running modules
    • Makes search much faster when running with lots of files and limited modules
    • For example, if using -m star to only use the STAR module, all other file searches now skipped
  • File search now warns if an unrecognised search type is given
  • MultiQC now saves nearly all parsed data to a structured output file by default
    • See multiqc_data/multiqc_data.json
    • This can be turned off by setting config.data_dump_file: false
  • Verbose logging when no log files found standardised. Less duplication in code and logs easier to read!
  • New documentation section describing how to use MultiQC with Galaxy
  • Using shared_key: 'read_counts' in table header configs now applies relevant defaults

Bug Fixes

  • Installation problem caused by changes in upstream dependencies solved by stricter installation requirements
  • Minor default_dev directory creation bug squashed
  • Don't prepend the directory separator (|) to sample names with -d when there are no subdirs
  • yPlotLines now works even if you don't set width

- JavaScript
Published by ewels over 8 years ago

multiqc - MultiQC Version 1.2

CodeFest 2017 Contributions

We had a fantastic group effort on MultiQC at the 2017 BOSC CodeFest. Many thanks to those involved!

New Modules:

  • AfterQC - New module!
    • Added parsing of the AfterQC json file data, with a plot of filtered reads.
    • Work by @raonyguimaraes
  • bcl2fastq
    • bcl2fastq can be used to both demultiplex data and convert BCL files to FASTQ file formats for downstream analysis
    • New module parses JSON output from recent versions and summarises some key statistics from the demultiplexing process.
    • Work by @iimog (with a little help from @tbooth and @ewels)
  • leeHom
    • leeHom is a program for the Bayesian reconstruction of ancient DNA
  • VCFTools
    • Added initial support for VCFTools relatedness2
    • Added support for VCFTools TsTv-by-count TsTv-by-qual TsTv-summary
    • Module written by @mwhamgenomics

Module updates:

  • FastQ Screen
    • Gracefully handle missing data from very old FastQ Screen versions.
  • RNA-SeQC
    • Add new transcript-associated reads plot.
  • Picard
    • New submodule to handle output from TargetedPcrMetrics
  • Prokka
    • Added parsing of the # CRISPR arrays data from Prokka when available (@asetGem)
  • Qualimap
    • Some code refactoring to radically improve performance and run times, especially with high coverage datasets.
    • Fixed bug where Cumulative coverage genome fraction plot could be truncated.

New MultiQC Features:

  • New module help text
    • Lots of additional help text was written to make MultiQC report plots easier to interpret.
    • Updated modules:
      • Bowtie
      • Bowtie 2
      • Prokka
      • Qualimap
      • SnpEff
    • Elite team of help-writers:
  • New config option section_comments allows you to add custom comments above specific sections in the report
  • New --tags and --view_tags command line options
    • Modules can now be given tags (keywords) and filtered by those. So running --tags RNA will only run MultiQC modules related to RNA analysis.
    • Work by @Hammarn
  • Back-end configuration options to specify the order of table columns
    • Modules and user configs can set priorities for columns to customise where they are displayed
    • Work by @tbooth
  • Added framework for proper unit testing
    • Previous start on unit tests tidied up, new blank template and tests for the clean_sample_name functionality.
    • Added to Travis and Appveyor for continuous integration testing.
    • Work by @tbooth
  • Bug fixes and refactoring of report configuration saving / loading
    • Discovered and fixed a bug where a report config could only be loaded once
    • Work by @DennisSchwartz
  • Table column row headers (sample names) can now be numeric-only.
  • Improved sample name cleaning functionality
    • Added option regex_keep to clean filenames by keeping the matching part of a pattern
    • Work by @robinandeer
  • Handle error when invalid regexes are given in reports
    • Now have a nice toast error warning you and the invalid regexes are highlighted
    • Previously this just crashed the whole report without any warning
    • Work by @robinandeer
  • Command line option --dirs-depth now sets -d to True (so now works even if -d isn't also specified).
  • New config option config.data_dump_file to export as much data as possible to multiqc_data/multiqc_data.json
  • New code to send exported JSON data to a a web server
    • This is in preparation for the upcoming MegaQC project. Stay tuned!

Bug Fixes:

  • Specifying multiple config files with -c/--config now works as expected
    • Previously this would only read the last specified
  • Fixed table rendering bug that affected Chrome v60 and IE7-11
    • Table cell background bars weren't showing up. Updated CSS to get around this rendering error.
  • HTML ID cleanup now properly cleans strings so that they work with jQuery as expected.
  • Made bar graph sample highlighting work properly again
  • Config custom_logo paths can now be relative to the config file (or absolute as before)
  • Report doesn't keep annoyingly telling you that toolbox changes haven't been applied
    • Now uses more subtle toasts and only when you close the toolbox (not every click).
  • Switching report toolbox options to regex mode now enables the Apply button as it should.
  • Sorting table columns with certain suffixes (eg. 13X) no works properly (numerically)
  • Fixed minor bug in line plot data smoothing (now works with unsorted keys)

- JavaScript
Published by ewels almost 9 years ago

multiqc - MultiQC Version 1.1

New Modules:

  • BioBloom Tools
    • Create Bloom filters for a given reference and then to categorize sequences
  • Conpair
    • Concordance and contamination estimator for tumor–normal pairs
  • Disambiguate
    • Bargraph displaying the percentage of reads aligning to two different reference genomes.
  • Flexbar
    • Flexbar is a tool for flexible barcode and adapter removal.
  • HISAT2
    • New module for the HISAT2 aligner.
    • Made possible by updates to HISAT2 logging by @infphilo (requires --new-summary HISAT2 flag).
  • HOMER
    • Support for summary statistics from the findPeaks tool.
  • Jellyfish
    • Histograms to estimate library complexity and coverage from k-mer content.
    • Module written by @vezzi
  • MACS2
    • Summary of redundant rate from MACS2 peak calling.
  • QoRTs
    • QoRTs is toolkit for analysis, QC and data management of RNA-Seq datasets.
  • THetA2
    • THeTA2 (Tumor Heterogeneity Analysis) estimates tumour purity and clonal / subclonal copy number.

Module updates:

  • BCFtools
    • Option to collapse complementary changes in substitutions plot, useful for non-strand specific experiments (thanks to @vladsaveliev)
  • Bismark
    • M-Bias plots no longer show read 2 for single-end data.
  • Custom Content
    • New option to print raw HTML content to the report.
  • FastQ Screen
    • Fixed edge-case bug where many-sample plot broke if total number of reads was less than the subsample number.
    • Fixed incorrect logic of config option fastqscreen_simpleplot (thanks to @daler)
    • Organisms now alphabetically sorted in fancy plot so that order is nonrandom (thanks to @daler)
    • Fixed bug where %No Hits was missed in logs from recent versions of FastQ Screen.
  • HTSeq Counts
    • Fixed but so that module still works when --additional-attr is specified in v0.8 HTSeq above (thanks to @nalcala)
  • Picard
    • CollectInsertSize: Fixed bug that could make the General Statistics Median Insert Size value incorrect.
    • Fixed error in sample name regex that left trailing ] characters and was generally broken (thanks to @jyh1 for spotting this)
  • Preseq
    • Improved plots display (thanks to @vladsaveliev)
  • Qualimap
    • Only calculate bases over target coverage for values in General Statistics. Should give a speed increase for very high coverage datasets.
  • QUAST
    • Module is now compatible with runs from MetaQUAST (thanks to @vladsaveliev)
  • RSeQC
    • Changed default order of sections
    • Added config option to reorder and hide module report sections

New MultiQC features:

  • If a report already exists, execution is no longer halted.
    • _1 is appended to the filename, iterating if this also exists.
    • -f/--force still overwrites existing reports as before
    • Feature written by @Hammarn
  • New ability to run modules multiple times in a single report
    • Each run can be given different configuration options, including filters for input files
    • For example, have FastQC after trimming as well as FastQC before trimming.
    • See the relevant documentation for more instructions.
  • New option to customise the order of report sections
    • This is in addition / alternative to changing the order of module execution
    • Allows one module to have sections in multiple places (eg. Custom Content)
  • Tables have new column options floor, ceiling and minRange.
  • Reports show warning if JavaScript is disabled
  • Config option custom_logo now works with file paths relative to config file directory and cwd.

Bug Fixes:

  • Table headers now sort columns again after scrolling the table
  • Fixed buggy table header tooltips
  • Base clean_s_name function now strips excess whitespace.
  • Line graphs don't smooth lines if not needed (number of points < maximum number allowed)
  • PDF output now respects custom output directory.

- JavaScript
Published by ewels almost 9 years ago

multiqc - MultiQC Version 1.0

Version 1.0! This release has been a long time coming and brings with it some fairly major improvements in speed, report filesize and report performance. There's also a bunch of new modules, more options, features and a whole lot of bug fixes.

The version number is being bumped up to 1.0 for a couple of reasons:

  1. MultiQC is now (hopefully) relatively stable. A number of facilities and users are now using it in a production setting and it's published. It feels like it probably deserves v1 status now somehow.
  2. This update brings some fairly major changes which will break backwards compatibility for plugins. As such, semantic versioning suggests a change in major version number.

Breaking Changes

For most people, you shouldn't have any problems upgrading. There are two scenarios where you may need to make changes with this update:

1. You have custom file search patterns

Search patterns have been flattened and may no longer have arbitrary depth. For example, you may need to change the following: yaml fastqc: data: fn: 'fastqc_data.txt' zip: fn: '*_fastqc.zip' to this: yaml fastqc/data: fn: 'fastqc_data.txt' fastqc/zip: fn: '*_fastqc.zip' See the documentation for instructions on how to write the new file search syntax.

See search_patterns.yaml for the new module search keys and more examples.

2. You have custom plugins / modules / external code

To see what changes need to applied to your custom plugin code, please see the MultiQC docs.

Module updates:

  • Adapter Removal - new module!
    • AdapterRemoval v2 - rapid adapter trimming, identification, and read merging
  • BUSCO - new module!
    • New module for the BUSCO v2 tool, used for assessing genome assembly and annotation completeness.
  • Cluster Flow - new module!
    • Cluster Flow is a workflow tool for bioinformatics pipelines. The new module parses executed tool commands.
  • RNA-SeQC - new module!
    • New module to parse output from RNA-SeQC, a java program which computes a series of quality control metrics for RNA-seq data.
  • goleft indexcov - new module! Thanks to @chapmanb and @brentp
    • goleft indexcov uses the PED and ROC data files to create diagnostic plots of coverage per sample, helping to identify sample gender and coverage issues.
  • SortMeRNA - new module! Written by @bschiffthaler
    • New module for SortMeRNA, commonly used for removing rRNA contamination from datasets.
  • Bcftools
    • Fixed bug with display of indels when only one sample
  • Cutadapt
    • Now takes the filename if the sample name is - (stdin). Thanks to @tdido
  • FastQC
    • Data for the Sequence content plot can now be downloaded from reports as a JSON file.
  • FastQ Screen
    • Rewritten plotting method for high sample numbers plot (~ > 20 samples)
    • Now shows counts for single-species hits and bins all multi-species hits
    • Allows plot to show proper percentage view for each sample, much easier to interpret.
  • HTSeq
    • Fix bug where header lines caused module to crash
  • Picard
    • New RrbsSummaryMetrics Submodule!
    • New WgsMetrics Submodule!
    • CollectGcBiasMetrics module now prints summary statistics to multiqc_data if found. Thanks to @ahvigil
  • Preseq
    • Now trims the x axis to the point that meets 90% of min(unique molecules). Hopefully prevents ridiculous x axes without sacrificing too much useful information.
    • Allows to show estimated depth of coverage instead of less informative molecule counts (see details).
    • Plots dots with externally calculated real read counts (see details).
  • Qualimap
    • RNASeq Transcript Profile now has correct axis units. Thanks to @roryk
    • BamQC module now doesn't crash if reports don't have genome gc distributions
  • RSeQC
    • Fixed Python3 error in Junction Saturation code
    • Fixed JS error for Junction Saturation that made the single-sample combined plot only show All Junctions

Core MultiQC updates:

  • Change in module structure and import statements (see details).
  • Module file search has been rewritten (see above changes to configs)
    • Significant improvement in search speed (test dataset runs in approximately half the time)
    • More options for modules to find their logs, eg. filename and contents matching regexes (see the docs)
  • Report plot data is now compressed, significantly reducing report filesizes.
  • New --ignore-samples option to skip samples based on parsed sample name
    • Alternative to filtering by input filename, which doesn't always work
    • Also can use config vars sample_names_ignore (glob patterns) and sample_names_ignore_re (regex patterns).
  • New --sample-names command line option to give file with alternative sample names
    • Allows one-click batch renaming in reports
  • New --cl_config option to supply MultiQC config YAML directly on the command line.
  • New config option to change numeric multiplier in General Stats
    • For example, if reports have few reads, can show Thousands of Reads instead of Millions of Reads
    • Set config options read_count_multiplier, read_count_prefix and read_count_desc
  • Config options decimalPoint_format and thousandsSep_format now apply to tables as well as plots
    • By default, thosands will now be separated with a space and . used for decimal places.
  • Tables now have a maximum-height by default and scroll within this.
    • Speeds up report rendering in the web browser and makes report less stupidly long with lots of samples
    • Button beneath table toggles full length if you want a zoomed-out view
    • Refactored and removed previous code to make the table header "float"
    • Set config.collapse_tables to False to disable table maximum-heights
  • Bar graphs and heatmaps can now be zoomed in on
    • Interactive plots sometimes hide labels due to lack of space. These can now be zoomed in on to see specific samples in more detail.
  • Report plots now load sequentially instead of all at once
    • Prevents the browser from locking up when large reports load
  • Report plot and section HTML IDs are now sanitised and checked for duplicates
  • New template available (called sections) which has faster loading
    • Only shows results from one module at a time
    • Makes big reports load in the browser much more quickly, but requires more clicking
    • Try it out by specifying -t sections
  • Module sections tidied and refactored
    • New helper function self.add_section()
    • Sections hidden in nav if no title (no more need for the hacky self.intro +=)
    • Content broken into description, help and plot, with automatic formatting
    • Empty module sections are now skipped in reports. No need to check if a plot function returns None!
    • Changes should be backwards-compatible
  • Report plot data export code refactored
    • Now doesn't export hidden samples (uses HighCharts export-csv plugin)
  • Handle error when git isn't installed on the system.
  • Refactored colouring of table cells
    • Was previously done in the browser using chroma.js
    • Now done at report generation time using the spectra package
    • Should helpfully speed up report rendering time in the web browser, especially for large reports
  • Docs updates (thanks to @varemo)
  • Previously hidden log file .multiqc.log renamed to multiqc.log in multiqc_data
  • Added option to load MultiQC config file from a path specified in the environment variable MULTIQC_CONFIG_PATH
  • New table configuration options
    • sortRows: False prevents table rows from being sorted alphabetically
    • col1_header allows the default first column header to be changed from "Sample Name"
  • Tables no longer show Configure Columns and Plot buttons if they only have a single column
  • Custom content updates
    • New custom_content/order config option to specify order of Custom Content sections
    • Tables now use the header for the first column instead of always having Sample Name
    • JSON + YAML tables now remember order of table columns
    • Many minor bugfixes
  • Line graphs and scatter graphs axis limits
    • If limits are specified, data exceeding this is no longer saved in report
    • Visually identical, but can make report file sizes considerable smaller in some cases
  • Creating multiple plots without a config dict now works (previously just gave grey boxes in report)
  • All changes are now tested on a Windows system, using AppVeyor
  • Fixed rare error where some reports could get empty General Statistics tables when no data present.
  • Fixed minor bug where config option force: true didn't work. Now you don't have to always specify -f!

- JavaScript
Published by ewels about 9 years ago

multiqc - MultiQC Version 0.9

A major new feature is released in v0.9 - support for custom content. This means that MultiQC can now easily include output from custom scripts within reports without the need for a new module or plugin. For more information, please see the MultiQC documentation.

Module updates: - HTSeq - new module! - New module for the htseq-count tool, often used in RNA-seq analysis. - Prokka - new module! - Prokka is a software tool for the rapid annotation of prokaryotic genomes. - Slamdunk - new module! - Slamdunk is a software tool to analyze SLAMSeq data. - Peddy - new module! - Peddy calculates genotype :: pedigree correspondence checks, ancestry checks and sex checks using VCF files. - Cutadapt - Fixed bug in General Stats table number for old versions of cutadapt (pre v1.7) - Added support for really old cutadapt logs (eg. v.1.2) - FastQC - New plot showing total overrepresented sequence percentages. - New option to parse a file containing a theoretical GC curve to display in the background. - Human & Mouse Genome / Transcriptome curves bundled, or make your own using fastqcTheoreticalGC. See the MultiQC docs for more information. - featureCounts - Added parsing checks and catch failures for when non-featureCounts files are picked up by accident - GATK - Fixed logger error in VariantEval module. - Picard - Fixed missing sample overwriting bug in RnaSeqMetrics - New feature to customise coverage shown from HsMetrics in General Statistics table see the docs for info). - Fixed compatibility problem with output from CollectMultipleMetrics for CollectAlignmentSummaryMetrics - Preseq - Module now recognises output from c_curve mode. - RSeQC - Made the gene body coverage plot show the percentage view by default - Made gene body coverage properly handle sample names - Samtools - New module to show duplicate stats from rmdup logs - Fixed a couple of niggles in the idxstats plot - SnpEff - Fixed swapped axis labels in the Variant Quality plot - STAR - Fixed crash when there are 0 unmapped reads. - Sample name now taken from the directory name if no file prefix found. - Qualimap BamQC - Add a line for pre-calculated reference genome GC content - Plot cumulative coverage for values above 50x, align with the coverage histogram. - New ability to customise coverage thresholds shown in General Statistics table (see the docs for info).

Core Updates: - Support for custom content (see top of release notes). - New ninja report tool: make scatter plots of any two table columns! - Plot data now saved in multiqc_data when 'flat' image plots are created - Allows you easily re-plot the data (eg. in Excel) for further downstream investigation - Added 'Apply' button to Highlight / Rename / Hide. - These tools can become slow with large reports. This means that you can enter several things without having to wait for the report to replot each change. - Report heatmaps can now be sorted by highlight - New config options decimalPoint_format and thousandsSep_format - Allows you to change the default 1 234.56 number formatting for plots. - New config option top_modules allows you to specify modules that should come at the top of the report - Fixed bar plot bug where missing categories could shift data between samples - Report title now printed in the side navigation - Missing plot IDs added for easier plot exporting - Stopped giving warnings about skipping directories (now a debug message) - Added warnings in report about missing functionality for flat plots (exporting and toolbox) - Export button has contextual text for images / data - Fixed a bug where user config files were loaded twice - Fixed bug where module order was random if --module or --exclude was used. - Refactored code so that the order of modules can be changed in the user config - Beefed up code + docs in scatter plots back end and multiple bar plots. - Fixed a few back end nasties for Tables - Shared-key columns are no longer forced to share colour schemes - Fixed bug in lambda modified values when format string breaks - Supplying just data with no header information now works as advertised - Improvements to back end code for bar plots - New tt_decimals and tt_suffix options for bar plots - Bar plots now support yCeiling, yFloor and yMinRange, as with line plots. - New option hide_zero_cats:False to force legends to be shown even when all data is 0 - General Stats Showing x of y columns count is fixed on page load. - Big code whitespace cleanup

- JavaScript
Published by ewels over 9 years ago

multiqc - MultiQC Version 0.8

Module updates: - GATK - new module! - Added support for VariantEval reports, only parsing a little of the information in there so far, but it's a start. - Module originally written by @robinandeer at the OBF Codefest, finished off by @ewels - Bcftools - new module! - QUAST - new module! - QUAST is a tool for assessing de novo assemblies against reference genomes. - Bismark now supports reports from bam2nuc, giving Cytosine coverage in General Stats. - Bowtie1 - Updated to try to find bowtie command before log, handle multiple logs in one file. Same as bowtie2. - FastQC - Sample pass/warn/fail lists now display properly even with large numbers of samples - Sequence content heatmap display is better with many samples - Kallisto - Now supports logs from SE data. - Picard - BaseDistributionByCycle - new submodule! Written by @mlusignan - RnaSeqMetrics - new submodule! This one by @ewels ;) - AlignmentSummaryMetrics - another new submodule! - Fixed truncated files crash bug for Python 3 (#306) - Qualimap RNASeqQC - Fixed parsing bug affecting counts in Genomic Origin plot. - Module now works with European style thousand separators (1.234,56 instead of 1,234.56) - RSeQC - infer_experiment - new submodule! Written by @Hammarn - Samtools - stats submodule now has separate bar graph showing alignment scores - flagstat - new submodule! Written by @HLWiencko - idxstats - new submodule! This one by @ewels again

Core updates: - New --export/-p option to generate static images plot in multiqc_plots (.png, .svg and .pdf) - Configurable with export_plots, plots_dir_name and export_plot_formats config options - --flat option no longer saves plots in multiqc_data/multiqc_plots - New --comment/-b flag to add a comment to the top of reports. - New --dirs-depth/-dd flag to specify how many directories to prepend with --dirs/-d - Specifying a postive number will take that many directories from the end of the path - A negative number will take directories from the start of the path. - Directory paths now appended before cleaning, so fn_clean_exts will now affect these names. - New custom_logo attributes to add your own logo to reports. - New report_header_info config option to add arbitrary information to the top of reports. - New --pdf option to create a PDF report - Depends on Pandoc being installed and is in a beta-stage currently. - Note that specifying this will make MultiQC use the simple template, giving a HTML report with much reduced functionality. - New fn_clean_sample_names config option to turn off sample name cleaning - This will print the full filename for samples. Less pretty reports and rows on the General Statistics table won't line up, but can prevent overwriting. - Table header defaults can now be set easily - General Statistics table now hidden if empty. - Some new defaults in the sample name cleaning - Updated the simple template. - Now has no toolbox or nav, no JavaScript and is better suited for printing / PDFs. - New config.simple_output config flag so code knows when we're trying to avoid JS. - Fixed some bugs with config settings (eg. template) being overwritten. - NFS log file deletion bug fixed by @brainstorm (#265) - Fixed bug in --ignore behaviour with directory names. - Fixed nasty bug in beeswarm dot plots where sample names were mixed up (#278) - Beeswarm header text is now more informative (sample count with more info on a tooltip) - Beeswarm plots now work when reports have > 1000 samples

Many thanks to those at the OpenBio Codefest 2016 who worked on MultiQC projects.

- JavaScript
Published by ewels over 9 years ago

multiqc - MultiQC Version 0.7

Version 0.7 of MultiQC!

This is quite a big release: 141 commits since v0.6, with 4,587 new lines of code and 2,116 lines deleted (or updated), affecting 75 files. Many thanks to everyone for your patience.

MultiQC is published!

If you didn't hear about it - the reason that I wanted to keep the main release stable was because the MultiQC paper was under review. It went well, and MultiQC is now published! You can see it here:

MultiQC: Summarize analysis results for multiple tools and samples in a single report Philip Ewels, Måns Magnusson, Sverker Lundin and Max Käller Bioinformatics (2016) doi: 10.1093/bioinformatics/btw354 PMID: 27312411

Many thanks to everyone involved.

Module updates:

Not a huge number of new modules this time, but lots of fixed bugs and quite a few big extensions / refactoring of big modules. - Kallisto - new module! - Picard - Code refactored to make maintenance and additions easier. - Big update to HsMetrics parsing - more results shown in report, new plots (by @lpantano) - Updated InsertSizeMetrics to understand logs generated by CollectMultipleMetrics (#215) - Newlines in picard output. Fixed by @dakl - Samtools - Code refactored - Rewrote the samtools stats code to display more stats in report with a beeswarm plot. - Qualimap - Rewritten to use latest methods and fix bugs. - Added Percentage Aligned column to general stats for BamQC module. - Extra table thresholds added by @avilella (hidden by default) - General Statistics - Some tweaks to the display defaults (FastQC, Bismark, Qualimap, SnpEff) - Now possible to skip the General Statistics section of the report with --exclude general_stats - Cutadapt module updated to recognise logs from old versions of cutadapt (<= v1.6) - Trimmomatic - Now handles , decimal places in percentage values. - Can cope with line breaks in log files (see issue #212) - FastQC refactored - Now skips zip files if the sample name has already been found. Speeds up MultiQC execution. - Code cleaned up. Parsing and data-structures standardised. - New popovers on Pass / Warn / Fail status bars showing sample names. Fast highlighting and hiding. - New column in General Stats (hidden by default) showing percentage of FastQC modules that failed. - SnpEff - Search pattern now more generic, should match reports from others. - Counts by Effect plot removed (had hundreds of categories, was fairly unusable). - KeyError bug fixed. - Samblaster now gets sample name from ID instead of SM (@dakl) - Bowtie 2 - Now parses overall alignment rate as intended. - Now depends on even less log contents to work with more inputs. - MethylQA now handles variable spacing in logs - featureCounts now splits columns on tabs instead of whitespace, can handle filenames with spaces

Core updates:

  • Galaxy: MultiQC now available in Galaxy! Work by @devengineson / @yvanlebras / @cmonjeau
  • Heatmap: New plot type!
  • Scatter Plot: New plot type!
  • Download raw data behind plots in reports! Available in the Export toolbox.
    • Choose from tab-separated, comma-separated and the complete JSON.
  • Table columns can be hidden on page load (shown through Configure Columns)
    • Defaults are configurable using the table_columns_visible config option.
  • Beeswarm plot: Added missing rename / highlight / hiding functionality.
  • New -l / --file-list option: specify a file containing a list of files to search.
  • Updated HighCharts to v4.2.5. Added option to export to JPEG.
  • Can now cancel execution with a single ctrl+c rather than having to button mash
  • More granular control of skipping files during scan (filename, dirname, path matching)
    • Fixed --exclude so that it works with directories as well as files
  • New Clear button in toolbox to bulk remove highlighting / renaming / hiding filters.
  • Improved documentation about behaviour for large sample numbers.
  • Handle YAML parsing errors for the config file more gracefully
  • Removed empty columns from tables again
  • Fixed bug in changing module search patterns, reported by @lweasel
  • Added timeout parameter to version check to prevent hang on systems with long defaults
  • Fixed table display bug in Firefox
  • Fixed bug related to order in which config files are loaded
  • Fixed bug that broke the "Show only" toolbox feature with multiple names.
  • Numerous other small bugs.

- JavaScript
Published by ewels almost 10 years ago

multiqc - MultiQC Version 0.6

Date released: 2016-04-29

Key highlights in this report are new plot types! Beeswarm and generalised tables. Three new modules, a major internal code refactoring around plotting and tables (now easier to understand, maintain and add to) and a whole swath of other minor module updates, new functions and features..

Changelog: - New Salmon module. - New Trimmomatic module. - New Bamtools stats module. - New beeswarm plot type. General Stats table replaced with this when many samples in report. - New RSeQC module: Actually a suite of 8 new modules supporting various outputs from RSeQC - Rewrote bowtie2 module: Now better at parsing logs and tries to scrape input from wrapper logs. - Made cutadapt show counts by default instead of obs/exp - Dynamic plots now update their labels properly when changing datasets and to percentages - Config files now loaded from working directory if present - Started new docs describing how each module works - Refactored featureCounts module. Now handles summaries describing multiple samples. - Stopped using so many hidden files. .multiqc.log now called multiqc.log - New -c/--config command line option to specify a MultiQC configuration file - Can now load run-specific config files called multiqc_config.yaml in working directory - Large code refactoring - moved plotting code out of BaseModule and into new multiqc.plots submodules - Generalised code used to generate the General Stats table so that it can be used by modules - Removed interactive report tour, replaced with a link to a youtube tutorial - Made it possible to permanently hide the blue welcome message for all future reports - New option to smooth data for line plots. Avoids mega-huge plots. Applied to SnpEff, RSeQC, Picard. - Added percentage view to Picard insert size plot - Bugfix: Qualimap handles infinity symbol (thanks @chapmanb ) - Bugfix: Made SnpEff less fussy about required fields for making plots - Bufgix: UTF-8 file paths handled properly in Py2.7+ - Bugfix: Extending two config variables wasn't working. Now fixed. - Bugfix: Dragging the height bar of plots now works again. - Bugfix: Plots now properly change y axis limits and labels when changing datasets - Bugfix: Flat plots now have correct path in default_dev template

- JavaScript
Published by ewels about 10 years ago

multiqc - MultiQC v0.5

Latest version of MultiQC. Key highlights are big reports (hundreds / thousands of samples now possible), numerous new modules, better plugin framework and a bunch of tweaks for smoother running.

Changelog: - New Skewer module, written by @dakl - New Samblaster module, written by @dakl - New Samtools stats module, written by @lpantano - New HiCUP module - New SnpEff module - New methylQA module - New "Flat" image plots, rendered at run time with MatPlotLib - By default, will use image plots if > 50 samples (set in config as plots_flat_numseries) - Means that very large numbers of samples can be viewed in reports. eg. single cell data. - Templates can now specify their own plotting functions - Use --flat and --interactive to override this behaviour - MultiQC added to bioconda (with help from @dakl) - New plugin hook: config_loaded - Plugins can now add new command line options (thanks to @robinandeer) - Changed default data directory name from multiqc_report_data to multiqc_data - Removed support for depreciated MultiQCOSXApp - Updated logging so that a verbose `multiqcdata/.multiqc.logfile is always written - Now logs more stuff in verbose mode - command used, user configs and so on. - Added a call to multiqc.info to check for new versions. Disable with confignoversioncheck` - Removed general stats manual row sorting. - Made filename matching use glob unix style filename match patterns - Everything (including the data directory) is now created in a temporary directory and moved when MultiQC is complete. - A handful of performance updates for large analysis directories

- JavaScript
Published by ewels about 10 years ago

multiqc - MultiQC v0.4

Latest release of MultiQC with some tasty new functions.. - New multiqc_sources.txt which identifies the paths used to collect all report data for each sample - Export parsed data as tab-delimited text, JSON or YAML using the new -k/--data-format command line option - Updated HighCharts from v4.2.2 to v4.2.3, fixes tooltip hover bug. - Nicer export button. Now tied to the export toolbox, hopefully more intuitive. - FastQC: Per base sequence content heatmap can now be clicked to show line graph for single sample - FastQC: No longer show adapter contamination datasets with <= 0.1% contamination. - Picard: Added support for CollectOxoGMetrics reports. - Changed command line option --name to --filename - --name also used for filename if --filename not specified. - Hide samples toolbox now has switch to show only matching samples - New regex help box with examples added to report - New button to copy general stats table to the clipboard - General Stats table 'floating' header now sorts properly when scrolling - Bugfix: MultiQC default_dev template now copies module assets properly - Bufgix: General Stats table floating header now resizes properly when page width changes

- JavaScript
Published by ewels over 10 years ago

multiqc - MultiQC v0.3.2

New version - a number of new modules, a few new command line options and some new toys in the report user interface.

I've just noticed that this was a bugfix version number bump and it should have been a minor version bump (v0.4). But I've already pushed this to PyPI so I'm not going to change it now. Sorry.

Anyway, here's what's new:

Changelog

  • All modules now load their log file search parameters from a config file, allowing you to overwrite them using your user config file
    • This is useful if your analysis pipeline renames program outputs
  • New Picard (sub)modules - Insert Size, GC Bias & HsMetrics
  • New Qualimap (sub)module - RNA-Seq QC
  • Made Picard MarkDups show percent by default instead of counts
  • Added M-Bias plot to Bismark
  • New option to stream report HTML to stdout
  • Files can now be specified as well as directories
  • New options to specify whether the parsed data directory should be created
    • command line flags: --data / --no-data
    • config option name: make_data_dir
  • Fixed bug with incorrect path to installation dir config YAML file
  • New toolbox drawer for bulk-exporting graph images
  • Report side navigation can now be hidden to maximise horizontal space
  • Mobile styling improved for narrow screen
  • More vibrant colours in the general stats table
  • General stats table numbers now left aligned
  • Settings now saved and loaded to named localstorage locations
    • Simplified interface - no longer global / single report saving
    • Removed static file config. Solves JS error, no-one was doing this since we have standalone reports anyway.
  • Added support for Python 3.5
  • Fixed bug with module specific CSS / JS includes in some templates
  • Made the 'ignore files' config use unix style file pattern matching
  • Fixed some bugs in the FastQ Screen module
  • Fixed some bugs in the FastQC module
  • Fixed occasional general stats table bug
  • Table sorting on sample names now works after renaming
  • Bismark module restructure
    • Each report type now handled independently (alignment / dedup / meth extraction)
    • M-Bias plot now shows R1 and R2
  • FastQC GC content plot now has option for counts or percentages
    • Allows comparison between samples with very different read counts
  • Bugfix for reports javascript
    • Caused by updated to remotely loaded HighCharts export script
    • Export script now bundled with multiqc, so does not depend on internet connection
    • Other JS errors fixed in this work
  • Bugfix for older FastQC reports - handle old style sequence dup data
  • Bugfix for varying Tophat alignment report formats
  • Bugfix for Qualimap RNA Seq reports with paired end data

- JavaScript
Published by ewels over 10 years ago

multiqc - MultiQC v0.3.1

Hotfix to patch a bug that I found immediately after submitting v0.3.0 - sorry! Took the opportunity to add a few more new things to sweeten the deal.

Changelog

  • Hotfix patch to fix broken FastQC module (wasn't finding .zip files properly)
  • General Stats table colours now flat. Should improve browser speed.
  • Empty rows now hidden if appear due to column removal in general stats
  • FastQC Kmer plot removed until we have something better to show.

- JavaScript
Published by ewels over 10 years ago

multiqc - MultiQC v0.3

Lots of spit and polish added to MultiQC in this release - everything should now be properly documented and reports are quite a bit more stable.

Changelog

  • Lots of lovely new documentation!
  • Child templates - easily customise specific parts of the default report template
  • Plugin hooks - allow other tools to execute custom code during MultiQC execution
  • New Preseq module
  • New design for general statistics table (snazzy new background bars)
  • Further development of toolbox
    • New button to clear all filters
    • Warnings when samples are hidden, plus empty plots and table cols are hidden
    • Active toolbar tab buttons are highlighted
  • Lots of refactoring by @moonso to please the Pythonic gods
    • Switched to click instead of argparse to handle command line arguments
    • Code generally conforms to best practices better now.
  • Now able to supply multiple directories to search for reports
  • Logging output improved (now controlled by -q and -v for quiet and verbose)
  • More HTML output dealt with by the base module, less left to the modules
    • Module introduction text
    • General statistics table now much easier to add to (new helper functions)
  • Images, CSS and Javascript now included in HTML, meaning that there is a single report file to make sharing easier
  • More accessible scrolling in the report - styled scrollbars and 'to top' button.
  • Modules and templates now use setuptools entry points, facilitating plugins by other packages. Allows niche extensions whilst keeping the core codebase clean.
  • The general stats table now has a sticky header row when scrolling, thanks to some new javascript wizardry...
  • General stats columns can have a shared key which allows common colour schemes and data ranges. For instance, all columns describing a read count will now share their scale across modules.
  • General stats columns can be hidden and reordered with a new modal window.
  • Plotting code refactored, reports with many samples (>50 by default) don't automatically render to avoid freezing the browser.
  • Plots with highlighted and renamed samples now honour this when exporting to different file types.

- JavaScript
Published by ewels over 10 years ago

multiqc - MultiQC v0.2

A major rewrite of MultiQC, with a lot of stabilisation and sanitisation. Many new functions created in Python and JavaScript to handle common functions, making writing new modules much easier. New toolbox of features created for report, making it much more interactive.

Download MultiQC_v0.2.app.zip for the Mac OS X graphical tool. To run from the command line, download the zip of the code. Can also be installed directly with pip from PyPI:

pip install multiqc

- JavaScript
Published by ewels over 10 years ago

multiqc - MultiQC v0.1

The first public release of MultiQC, after a month of development. Basic structure in place and modules for FastQC, FastQ Screen, Cutadapt, Bismark, STAR, Bowtie, Subread featureCounts and Picard MarkDuplicates. Approaching stability, though still under fairly heavy development.

Download MultiQC.app.zip for the Mac OS X graphical tool. To run from the command line, download the zip of the code. Can also be installed directly with pip from PyPI:

pip install multiqc

- JavaScript
Published by ewels almost 11 years ago