This document is under governance review. When the review completes as appropriate per local and agency processes, the project team will be allowed to remove this notice. This material is draft.

US State and Territory subsample datasets and example builds

Index

1. Overview and purpose
2. Example usage and customization
   • Available files
   • Example build files
   • Data set selection
   • Saving previous data files
   • Reference sequences
   • Pangolin calls
3. Subsampling details
   • Unweighted samples
   • Weighted samples
   • Weighted sampling algorithm
4. Other data sets
5. Implementation
6. FAQ (Frequently Asked Questions)
7. Notices and disclaimers

1. OVERVIEW

This repository contains tools for building custom Nextstrain phylogenetic analyses utilizing publicly-available sequence data and metadata from databases of The National Center for Biotechnology Information (NCBI).

These data provide users with pre-computed, up-to-date sequence and metadata files organized in a way that prioritizes:

• Simplified usage. Users are not required to download or manipulate the data in order to incorporate them into custom Nextstrain builds.
• Powerful customization. Users can further customize analyses by modifying the example Nextstrain builds.yaml files.
• Open data. All data is derived from publicly-available resources.
• Right-sized. Datasets subsampled to varied sizes are available to reduce computational demands.
• Correct formating. Sequence and metadata files are ready to be used in Nextstrain.
• Automatic updates. These datasets are routinely updated by the Centers for Disease Control and Prevention (CDC) and the latest version is automatically incorporated into builds using the remote URL.
• Pre-aligned sequences. Sequences are provided pre-aligned, allowing users to forgo the computationally demanding alignment step.
• Optional weighting. Additional datasets with subsampling weighted to include more recently circulating strains are also provided.

These files differ from data used in the Nextstrain SPHERES group builds, which are derived from GISAID's EpiCov database. See github.com/CDCgov/spheres-augur-build for more information about those builds.

2. EXAMPLE USAGE AND CUSTOMIZATION

Available files

The complete list of available datasets is provided here as remote paths to matched sequence alignment and metadata files with format similar to:
https://s3.amazonaws.com/location/State/State-full-aligned.fasta.xz https://s3.amazonaws.com/location/State/State-full-metadata.tsv.xz

Example build files

• name: statedata aligned: https://s3.amazonaws.com/location/State/State-2000-unweighted-aligned.fasta.xz metadata: https://s3.amazonaws.com/location/State/State-2000-unweighted-metadata.tsv.xz ``` Example build files are also provided to guide users in developing their own custom analyses under three common build scenarios:

1. basic: The "basic" example build combines subsampled state data with a United States contextual background.

2. tristate: The "tristate" example build demonstrates how data from one state can be easily combined with focused contextual data from neighboring states and sparsely sampled data from the rest of the United States.

3. hybrid: The "hybrid" example build combines subsampled state data with the user's local sequence data not submitted to public repositories. Additional contextual background from United States and global sequence data is also added.

To use these example builds: cp -r spheres-state-subsample/modular_nextstrain_profiles <PATH TO NEXTSTRAIN INSTALLATION> cd <PATH TO ncov installation> conda activate nextstrain snakemake --profile spheres-state-subsample/modular_nextstrain_profiles/<BUILD> --jobs 1 Results will be output to the ./auspice directory.

Each of these example builds can be customized by editing builds.yaml to include any of the provided subsampled datasets or adjust any Nextstrain parameters.

If the user is aware their sampled region has been sparsely sequenced (see Subsampling details below), the use of soft subsampling parameters in builds.yaml are recommended, such as max_sequences: <DESIRED SAMPLES>. Hard limits, such as seq_per_group: <DESIRED SAMPLES>, may exceed the number of available sequences in the input dataset and cause a failure in the augur pipeline.

Data set selection

Whether you are using weighted or unweighted data sets, you should always include some sequence data in your final subsample from earlier periods in the pandemic. Not including a diverse enough background may negatively impact tree reconstruction, especially for deep splits in the tree. Consider including a global background input, such as the global data set from data.nextstrain.org, in every build.

Saving previous data files

There is no archive of previous data files provided. Re-running your Nextstrain build will download the most recent data and overwrite previously downloaded data sets. If you wish to save a specific data set, you must manually copy that data to another location prior to re-running the build. You can find the previously downloaded data files in your Nextstrain directory here: ncov/s3.us-east-1.amazonaws.com/io-pe1-prod-ncezid-oamd-nextstrain/<STATE>/ Remember to save both the fasta sequence file and metadata tsv file. The pangolin calls in these data files are current at the time of data set preparation, but when using archived data, we recommend that the user generate up to date lineage calls by calling pangolin in their build.

Reference sequences

Reference data required by Nextstrain is not included in these data sets, however the required data is included in the default ncov installation. Users must include these data in the "inputs" of their builds.yaml file - name: references sequences: data/references_sequences.fasta metadata: data/references_metadata.tsv See the example builds for examples of how to include this in your builds.yaml.

Pangolin calls

While we provide the pangolin lineage call available on NCBI at the time of data download, pangolin lineage classifications are always being updated. Users may want to run pangolin as part of their Nexstrain build to ensure that the latest calls are used or that the same pangolin version is used on both remote and local files. We always recommend re-running pangolin if you are using archived data to ensure that the lineage calls are as up to date as possible.

Users can add pangolin to their workflow by including the following line in their builds.yaml file: run_pangolin: true Enabling pangolin calling by nextstrain requires pangolin to be installed or the "--use-conda" flag provided to snakemake.

3. SUBSAMPLING DETAILS

Unweighted Subsamples

After de-duplication, unweighted subsamples are drawn randomly from all available sequences in the requested state or territory. These random samples will reflect any sampling biases present in the NCBI database. Among other factors, these biases may include case prevalence, sequencing capacity, or public health sequencing initiatives.

Unweighted samples are nested such that the 1000 sequence set is a subset of the 2000 sequence set, which is a subset of the 4000 sample set.

Example: https://s3.amazonaws.com/location/State/State-1000-unweighted-aligned.fasta.xz https://s3.amazonaws.com/location/State/State-1000-unweighted-metadata.tsv.xz https://s3.amazonaws.com/location/State/State-2000-unweighted-aligned.fasta.xz https://s3.amazonaws.com/location/State/State-2000-unweighted-metadata.tsv.xz https://s3.amazonaws.com/location/State/State-4000-unweighted-aligned.fasta.xz https://s3.amazonaws.com/location/State/State-4000-unweighted-metadata.tsv.xz

Weighted Subsamples

After de-duplication, weighted subsamples are intentionally biased towards more recent collection dates to prioritize circulating strains. Sequences are weighted as defined below and the weighted value is used to select the final weighted set. These weighted samples will reflect any sampling biases present in the NCBI database. Among other factors, these may include case prevalence, sequencing capacity, or public health sequencing initiatives. The weighted sampling attempts to populate the data set as described here but is limited by the availability of sequence data.

Weighted samples are not nested, meaning sequences in the 1000 sequence set will not necessarily be present in the 2000 sequence set, nor will the 2000 sequence set necessarily be a subset of the 4000 sequence set.

https://s3.amazonaws.com/location/State/State-1000-weighted-aligned.fasta.xz https://s3.amazonaws.com/location/State/State-1000-weighted-metadata.tsv.xz https://s3.amazonaws.com/location/State/State-2000-weighted-aligned.fasta.xz https://s3.amazonaws.com/location/State/State-2000-weighted-metadata.tsv.xz https://s3.amazonaws.com/location/State/State-4000-weighted-aligned.fasta.xz https://s3.amazonaws.com/location/State/State-4000-weighted-metadata.tsv.xz It is not recommended to also use the subsampling instruction: group_by: "year month" to further subsample a weighted dataset. The group_by: "year month" instruction attempts to unbias the selection by sampling evenly over time. This is contrary to the design of the weighted subsample dataset and may cause unintended effects on the final analysis. If even subsampling over time is desired in the final build, the "unweighted" data set should be used. See the Nextstrain documentation for more information on subsampling parameters.

Data set size take precedence over fitting the subsample collection date distribution to the desired weighting. For example, if a 4000 sequence data set is requested but only 3500 sequences exist across all time points, all 3500 sequences will be included in the weighted data set no matter when they were collected.

Similarly, an insufficient number of sequences in a date bin will cause the weighted algorithm to select data from other date bins to fill out the requested data set size.

Weighted sampling algorithm

4. OTHER DATA SETS

The Nextstrain team has prepared several remote open data files corresponding to global and regional data sets. You can find information on other available files and instruction on adding them to your builds here. These files can be combined with state subsampling data sets and query parameters to perform more complex analyses.

5. PROGRAM USAGE

6. FAQ (Frequently Asked Questions)

1. How frequently are these data sets updated?

   These files are updated weekly on Monday evening.

2. How do I get access to an earlier release of a data set?

   Previous data sets are not archived. To keep a previous data set, see the instructions here.
   If you re-use data sets you have manually archived, please be aware that you should update it with the most current pangolin lineage calls (see here).

3. What changes from week to week?

   New sequences will be added based upon the most recent contents of NCBI. Additionally, any revisions or revocations to the NCBI data will be reflected in the release. Any changes to the pangolin lineage provided by NCBI will also be present in the new data set.

   Downsampled unweighted sets will have a different randomized set of sequences in them.

   Downsampled weighted sets will have a different weighting applied to the sequences to continue to provide data representative of the most recently circulating strains.

4. Can I share this data?

   All data is derived from publicly available data on NCBI.

5. How is this data intended to be used?

   This data is provided to allow users to include the most recently available data in a Nextstrain build without having to modify their nextstrain build file or manually interact with the files or file contents. File paths do not change from week to week, so your build file will always download the most recent version of the file.

   Basic use cases include a weekly build using only remote data that can simply be re-run to provide an updated Nexstrain auspice file or using these files to simplify adding background context to a custom analysis. Advanced users may wish to use these files as a basis for their own filtering or subsampling schemes.

   See the data set selection section for more advice on selecting which input files are appropriate for your use.

7. Notices and Disclaimers

This repository constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. This repository is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication. All contributions to this repository will be released under the CC0 dedication. By submitting a pull request you are agreeing to comply with this waiver of copyright interest.

License Standard Notice

The repository utilizes code licensed under the terms of the Apache Software License and therefore is licensed under ASL v2 or later.

This source code in this repository is free: you can redistribute it and/or modify it under the terms of the Apache Software License version 2, or (at your option) any later version.

This source code in this repository is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Apache Software License for more details.

You should have received a copy of the Apache Software License along with this program. If not, see http://www.apache.org/licenses/LICENSE-2.0.html

The source code forked from other open source projects will inherit its license.

Privacy Standard Notice

This repository contains only non-sensitive, publicly available data and information. All material and community participation is covered by the Disclaimer and Code of Conduct. For more information about CDC's privacy policy, please visit http://www.cdc.gov/other/privacy.html.

Contributing Standard Notice

Anyone is encouraged to contribute to the repository by forking and submitting a pull request. (If you are new to GitHub, you might start with a basic tutorial.) By contributing to this project, you grant a world-wide, royalty-free, perpetual, irrevocable, non-exclusive, transferable license to all users under the terms of the Apache Software License v2 or later.

All comments, messages, pull requests, and other submissions received through CDC including this GitHub page may be subject to applicable federal law, including but not limited to the Federal Records Act, and may be archived. Learn more at http://www.cdc.gov/other/privacy.html.

Records Management Standard Notice

This repository is not a source of government records, but is a copy to increase collaboration and collaborative potential. All government records will be published through the CDC web site.

Additional Standard Notices

Please refer to CDC's Template Repository for more information about contributing to this repository, public domain notices and disclaimers, and code of conduct.