Python Tools for AFSC GAP

| Group | Badges | |-------|--------| | Status | | | Usage | | | Publication | | | Archive | |

Python-based tool chain ("Pyafscgap.org") for working with the public bottom trawl data from the NOAA AFSC GAP. This provides information from multiple survey programs about where certain species were seen and when under what conditions, information useful for research in ocean health.

See webpage, project Github, and example notebook.

Quickstart

Taking your first step is easy!

Explore the data in a UI: To learn about the datasets, try out an in-browser visual analytics app at https://app.pyafscgap.org without writing any code.

Try out a tutorial in your browser: Learn from and modify an in-depth tutorial notebook in a free hosted academic environment (all without installing any local software).

Jump into code: Ready to build your own scripts? Here's an example querying for Pacific cod in the Gulf of Alaska for 2021:

python import afscgap # install with pip install afscgap query = afscgap.Query() query.filter_year(eq=2021) query.filter_srvy(eq='GOA') query.filter_scientific_name(eq='Gadus macrocephalus') results = query.execute()

Continue your exploration in the developer docs.

Installation

Ready to take it to your own machine? Install the open source tools for accessing the AFSC GAP via Pypi / Pip:

bash $ pip install afscgap

The library's only dependency is requests and Pandas / numpy are not expected but supported. The above will install the release version of the library. However, you can also install the development version via:

bash $ pip install git+https://github.com/SchmidtDSE/afscgap.git@main

Installing directly from the repo provides the "edge" version of the library which should be treated as pre-release.

Purpose

Unofficial Python-based tool set for interacting with bottom trawl surveys from the Ground Fish Assessment Program (GAP). It offers:

• Pythonic access to the NOAA AFSC GAP datasets.
• Tools for inference of the "negative" observations not provided by the API service.
• Visualization tools for quickly exploring and creating comparisons within the datasets, including for audiences with limited programming experience.

Note that GAP are an excellent collection of datasets produced by the Resource Assessment and Conservation Engineering (RACE) Division of the Alaska Fisheries Science Center (AFSC) as part of the National Oceanic and Atmospheric Administration's Fisheries organization (NOAA Fisheries).

Please see our objectives documentation for additional information about the purpose, developer needs addressed, and goals of the project.

Usage

This library provides access to the AFSC GAP data with optional zero catch ("absence") record inference.

Examples / tutorial

One of the best ways to learn is through our examples / tutorials series. For more details see our usage guide.

API Docs

Full formalized API documentation is available as generated by pdoc in CI / CD.

Data structure

Detailed information about our data structures and their relationship to the data structures found in NOAA's upstream database is available in our data model documentation.

Absence vs presence data

By default, the NOAA service will only return information on hauls matching a query. So, for example, requesting data on Pacific cod will only return information on hauls in which Pacific cod is found. This can complicate the calculation of important metrics like catch per unit effort (CPUE). That in mind, one of the most important features in afscgap is the ability to infer "zero catch" records as enabled by set_presence_only(False). See more information in our inference docs.

Data quality and completeness

There are a few caveats for working with these data that are important for researchers to understand. These are detailed in our limitations docs.

Community flat files

The upstream datasets have shifted starting in 2024 with one important change including decomposing the dataset into hauls, catches, and species. Without the ability to join through the API endpoint, the entire catch dataset has to be queried or catches named individually in requests in order to retrieve complete records. Therefore, starting with the 2.x releases, this library uses pre-joined community Avro files to speed up requests, offering precomputed indicies such that, where available, hauls can be pre-filtered to reduce download payload size and running time. See flat file documentation for more details about this service.

License

We are happy to make this library available under the BSD 3-Clause license. See LICENSE for more details. (c) 2023 Regents of University of California. See the Eric and Wendy Schmidt Center for Data Science and the Environment at UC Berkeley.

Developing

Intersted in contributing to the project or want to bulid manually? Please see our build docs for details.

People

Sam Pottinger is the primary contact with additional development from Giulia Zarpellon. Additionally some acknowledgements:

• Thank you to Carl Boettiger and Fernando Perez for advice in the Python library.
• Thanks also to Maya Weltman-Fahs, Brookie Guzder-Williams, Angela Hayes, David Joy, and Magali de Bruyn for advice on the visual analytics tool.
• Lewis Barnett, Emily Markowitz, and Ciera Martinez for general guidance.

This is a project of the The Eric and Wendy Schmidt Center for Data Science and the Environment at UC Berkeley where Kevin Koy is Executive Director. Please contact us via dse@berkeley.edu.

Open Source

We are happy to be part of the open source community. We use the following:

• Requests which is available under the Apache v2 License from Kenneth Reitz and other contributors.
• fastavro by Miki Tebeka and Contributors under the MIT License.

In addition to Github-provided Github Actions, our build and documentation systems also use the following but are not distributed with or linked to the project itself:

• mkdocs under the BSD License.
• mkdocs-windmill under the MIT License.
• mypy under the MIT License from Jukka Lehtosalo, Dropbox, and other contributors.
• nose2 under a BSD license from Jason Pellerin and other contributors.
• pdoc under the Unlicense license from Andrew Gallant and Maximilian Hils.
• pycodestyle under the Expat License from Johann C. Rocholl, Florent Xicluna, and Ian Lee.
• pyflakes under the MIT License from Divmod, Florent Xicluna, and other contributors.
• sftp-action under the MIT License from Niklas Creepios.
• ssh-action under the MIT License from Bo-Yi Wu.

Next, the visualization tool has additional dependencies as documented in the visualization readme. Similarly, the community flat files snapshot updater has additional dependencies as documented in the snapshot readme.

Finally, note that the website uses assets from The Noun Project under the NounPro plan. If used outside of https://pyafscgap.org, they may be subject to a different license.

Thank you to all of these projects for their contribution.

Version history

Annotated version history:

• 2.0.1: Some minor changes to better support weaker internet connections.
• 2.0.0: Switch to support new NOAA endpoints.
• 1.0.4: Minor documentation fypo fix.
• 1.0.3: Documentation edits for journal article.
• 1.0.2: Minor documentation touch ups for pyopensci.
• 1.0.1: Minor documentation fix.
• 1.0.0: Release with pyopensci.
• 0.0.9: Fix with issue for certain import modalities and the http module.
• 0.0.8: New query syntax (builder / chaining) and units conversions.
• 0.0.7: Visual analytics tools.
• 0.0.6: Performance and size improvements.
• 0.0.5: Changes to documentation.
• 0.0.4: Negative / zero catch inference.
• 0.0.3: Minor updates in documentation.
• 0.0.2: License under BSD.
• 0.0.1: Initial release.

The community files were last updated on Jan 7, 2025.