Data

Tools

Indexes

Applications

Experiments

Open Source Science

esgpull

ESGF data transfer and replication tool

https://github.com/esgf/esgf-download

Science Score: 52.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ✓
    Institutional organization owner
    Organization esgf has institutional domain (esgf.llnl.gov)
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.3%) to scientific vocabulary

Keywords from Contributors

earth-system-model
Last synced: 7 months ago · JSON representation ·

Repository

ESGF data transfer and replication tool

Basic Info
Statistics
  • Stars: 20
  • Watchers: 7
  • Forks: 6
  • Open Issues: 23
  • Releases: 10
Created about 3 years ago · Last pushed 7 months ago
Metadata Files
Readme License Citation

README.md

esgpull - ESGF data management utility

Rye

esgpull is a tool that simplifies usage of the ESGF Search API for data discovery, and manages procedures related to downloading and storing files from ESGF.

```py from esgpull import Esgpull, Query

query = Query() query.selection.project = "CMIP6" query.options.distrib = True # default=False esg = Esgpull() nbdatasets = esg.context.hits(query, file=False)[0] nbfiles = esg.context.hits(query, file=True)[0] datasets = esg.context.datasets(query, maxhits=5) print(f"Number of CMIP6 datasets: {nbdatasets}") print(f"Number of CMIP6 files: {nb_files}") for dataset in datasets: print(dataset) ```

Features

  • Command-line interface
  • HTTP download (async multi-file)

Installation

esgpull is distributed via PyPI:

shell pip install esgpull esgpull --help

For isolated installation, uv or pipx are recommended:

```shell

with uv

uv tool install esgpull esgpull --help

alternatively, uvx enables running without explicit installation (comes with uv)

uvx esgpull --help ```

```shell

with pipx

pipx install esgpull esgpull --help ```

Usage

```console Usage: esgpull [OPTIONS] COMMAND [ARGS]...

esgpull is a management utility for files and datasets from ESGF.

Options: -V, --version Show the version and exit. -h, --help Show this message and exit.

Commands: add Add queries to the database config View/modify config convert Convert synda selection files to esgpull queries download Asynchronously download files linked to queries login OpenID authentication and certificates renewal remove Remove queries from the database retry Re-queue failed and cancelled downloads search Search datasets and files on ESGF self Manage esgpull installations / import synda database show View query tree status View file queue status track Track queries untrack Untrack queries update Fetch files, link files <-> queries, send files to download... ```

Useful links

Contributions

You can use the common github workflow (through pull requests and issues) to contribute.

Owner

  • Name: Earth System Grid Federation
  • Login: ESGF
  • Kind: organization
  • Location: Earth

Citation (CITATION.cff) 

# This CITATION.cff file was generated with cffinit.

cff-version: 1.2.0
title: esgpull
message: ESGF command line download tool esgpull
type: software
authors:
  - given-names: Sven
    family-names: Rodriguez
    email: sven.rodriguez@ipsl.fr
    affiliation: Sorbonne University
  - given-names: Atef
    family-names: Ben Nasser
    email: abennasser@ipsl.fr
    affiliation: Centre National de Recherche Scientifique
    orcid: 'https://orcid.org/0000-0001-6948-8735'
  - given-names: Guillaume
    family-names: Levavasseur
    email: glipsl@ipsl.fr
    affiliation: Sorbonne University
    orcid: 'https://orcid.org/0000-0002-0801-0890'
repository-code: 'https://github.com/ESGF/esgf-download.git'
url: 'https://esgf.github.io/esgf-download/'
abstract: >-
  esgpull is a modern ESGF data management tool, bundled
  with a custom asynchronous interface with the ESGF Search
  API. It handles scanning, downloading and updating
  datasets, files and queries from ESGF.
license: BSD-3-Clause
commit: 762494e
version: 0.7.4
doi: 10.5281/zenodo.14228984
date-released: '2024-11-27'

GitHub Events

Total
  • Create event: 13
  • Release event: 4
  • Issues event: 40
  • Watch event: 6
  • Delete event: 28
  • Issue comment event: 50
  • Push event: 20
  • Pull request review comment event: 8
  • Pull request review event: 15
  • Pull request event: 21
  • Fork event: 6
Last Year
  • Create event: 13
  • Release event: 4
  • Issues event: 40
  • Watch event: 6
  • Delete event: 28
  • Issue comment event: 50
  • Push event: 20
  • Pull request review comment event: 8
  • Pull request review event: 15
  • Pull request event: 21
  • Fork event: 6

Committers

Last synced: about 2 years ago

All Time
  • Total Commits: 384
  • Total Committers: 4
  • Avg Commits per committer: 96.0
  • Development Distribution Score (DDS): 0.034
Past Year
  • Commits: 335
  • Committers: 4
  • Avg Commits per committer: 83.75
  • Development Distribution Score (DDS): 0.039
Top Committers
Name Email Commits
svenrdz s****r@g****m 371
Zeitsperre 1****e 10
Atef a****r@i****r 2
svenrdz s****z 1
Committer Domains (Top 20 + Academic)
ipsl.fr: 1

Issues and Pull Requests

Last synced: 7 months ago

All Time
  • Total issues: 47
  • Total pull requests: 66
  • Average time to close issues: 4 months
  • Average time to close pull requests: about 1 month
  • Total issue authors: 18
  • Total pull request authors: 4
  • Average comments per issue: 1.26
  • Average comments per pull request: 0.21
  • Merged pull requests: 50
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 24
  • Pull requests: 36
  • Average time to close issues: about 1 month
  • Average time to close pull requests: 11 days
  • Issue authors: 14
  • Pull request authors: 2
  • Average comments per issue: 1.21
  • Average comments per pull request: 0.19
  • Merged pull requests: 22
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • AtefBN (11)
  • meteorologist15 (9)
  • svenrdz (4)
  • Zeitsperre (4)
  • alaniwi (3)
  • orlando-code (2)
  • hanjunkim0617 (2)
  • benkirk (2)
  • sbathiany (1)
  • durack1 (1)
  • sandrocalmanti (1)
  • sashakames (1)
  • dchandan (1)
  • magnuspn (1)
  • DominikStiller (1)
Pull Request Authors
  • svenrdz (58)
  • Zeitsperre (3)
  • alaniwi (3)
  • AtefBN (2)
Top Labels
Issue Labels
enhancement (6) bug (6)
Pull Request Labels
enhancement (7)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 255 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 10
  • Total maintainers: 1
pypi.org: esgpull

ESGF data discovery, download, replication tool

  • Versions: 10
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 255 Last month
Rankings
Dependent packages count: 10.8%
Average: 35.8%
Dependent repos count: 60.8%
Maintainers (1)
svenrodriguez
Last synced: 7 months ago

Dependencies

.github/workflows/ci.yml actions
  • actions/checkout v3 composite
  • pdm-project/setup-pdm v3 composite
.github/workflows/doc.yml actions
  • actions/checkout v3 composite
  • pdm-project/setup-pdm main composite
  • peaceiris/actions-gh-pages v3 composite
pyproject.toml pypi
  • MyProxyClient >=2.1.0
  • aiofiles >=22.1.0
  • aiostream >=0.4.5
  • alembic >=1.8.1
  • attrs >=22.1.0
  • cattrs >=22.2.0
  • click >=8.1.3
  • click-params >=0.4.0
  • httpx >=0.23.0
  • nest-asyncio >=1.5.6
  • platformdirs >=2.6.2
  • pyOpenSSL >=22.1.0
  • pyparsing >=3.0.9
  • pyyaml >=6.0
  • rich >=12.6.0
  • setuptools >=65.4.1
  • sqlalchemy >=2.0.0b2
  • tomlkit >=0.11.5