maridatadownloader

Downloading tools for environmental/marine/weather data.

Currently supported platforms/provider: - Global Forecast System (GFS) - Copernicus Marine Environment Monitoring Service (CMEMS) - Copernicus Climate Data Store (CDS) ECMWF ERA5 reanalysis - ETOPO Global Relief Model (NOAA/NCEI)

Currently supported access services/APIs: - OPeNDAP (via xarray) - Copernicus CDS API - Copernicus Marine Toolbox API

Installation

pip install git+https://github.com/52North/maridatadownloader.git

Usage

Basics

```python from maridatadownloader import DownloaderFactory

Create downloader object

Depending on platform/dataset these might be necessary

kwargs = { 'product': , 'producttype': <producttype>, 'username': , 'password': }

downloader = DownloaderFactory.getdownloader('<downloadertype>', '', **kwargs)

Download data

Define coordinates for sub-setting

sel_dict = { 'time': slice('2023-11-24T010:30:00', '2023-11-25T10:30:00'), 'latitude': slice(51.5, 52.5), 'longitude': slice(7, 8) } parameters = ['']

xarraydataset = downloader.download(parameters=parameters, seldict=sel_dict) ```

Sub-setting

Conventions - Coordinates are named "latitude" and "longitude", independently of the name in the original file/dataset - Latitude is defined from -90° to 90° - Longitude is defined from -180° to 180°

The sub-setting logic is implemented using xarray. Depending on the arguments provided to the download-method a different xarray method is used (sel, isel or interp). Note also that kwargs are passed on to the corresponding xarray method. For a detailed documentation of sub-setting with xarray check the dedicated section on their website: https://docs.xarray.dev/en/latest/user-guide/indexing.html.

Note that it is also possible to apply vectorized indexing (in contrast to the common orthogonal indexing).
E.g. sub-setting along a trajectory is possible by defining a common dimension (here: 'trajectory') for the sub-setting coordinates:

```python from datetime import datetime import xarray

lons = [2.81, 3.19, 4.56, 6.67, 6.68] lats = [51.9, 53.0, 54.0, 54.3, 55.5]

times = [datetime.strptime("2023-09-20 09:00:00", '%Y-%m-%d %H:%M:%S'), datetime.strptime("2023-09-20 11:00:00", '%Y-%m-%d %H:%M:%S'), datetime.strptime("2023-09-20 13:00:00", '%Y-%m-%d %H:%M:%S'), datetime.strptime("2023-09-20 15:00:00", '%Y-%m-%d %H:%M:%S'), datetime.strptime("2023-09-20 17:00:00", '%Y-%m-%d %H:%M:%S')]

lonsxr = xarray.DataArray(lons, dims=['trajectory']) latsxr = xarray.DataArray(lats, dims=['trajectory']) timesxr = xarray.DataArray(times, dims=['trajectory']) seldict = { 'time': timesxr, 'longitude': lonsxr, 'latitude': lats_xr } ```

Further reading: - https://docs.xarray.dev/en/stable/user-guide/indexing.html#vectorized-indexing - https://docs.xarray.dev/en/stable/user-guide/interpolation.html#advanced-interpolation

Important note on the 'cdsapi' downloader:
The downloader method is not yet harmonized with the other downloaders, so the API has to be used differently. To get the settings for the ERA5 CDS API go to their website and select the parameters, time and extent you like to use and click on show API request at the bottom of the page. Then you can copy the dictionary within the request and use it as your settings function parameter.

Chunking

Downloader types based on xarray can use chunking via dask. The desired chunk sizes are stored as an object attribute of the downloader. They can be provided to the DownloaderFactory.get_downloader method and are internally passed to the xarray.open_dataset method.

python chunks = {'latitude': 100, 'longitude': 100} gfs = DownloaderFactory.get_downloader('xarray', 'gfs', chunks=chunks)

Further reading: - https://docs.xarray.dev/en/stable/user-guide/dask.html - https://examples.dask.org/xarray.html

Available datasets/downloader

| Platform/Provider | Downloader type | Type of data | Product | Product type | References | |-------------------|-----------------|----------------------|------------------------------------------|--------------|------------| | cmems¹ | xarray⁵, cmtapi | Ocean waves | cmemsmodglowavanfc0.083degPT3H-i² | nrt³ | [1] | | cmems¹ | xarray⁵, cmtapi | Ocean currents | cmemsmodglophyanfcmerged-uvPT1H-i² | nrt³ | [2] | | cmems¹ | xarray⁵, cmtapi | Ocean physics | cmemsmodglophyanfc0.083degPT1H-m² | nrt³ | [2] | | gfs | xarray | Weather/Atmosphere | - | - | [3] | | etoponcei | xarray | Topology/Bathymetric | - | - | [4] | | era5¹⁴ | cdsapi | Atmosphere/Ocean | - | - | [5] |

¹Registration needed
²Check the CMEMS product catalog for additional products: https://data.marine.copernicus.eu/products
³nrt = near real-time
⁴The download interface differs from the interface of the 'xarray' downloader type (ToDo: harmonize)
⁵Deprecated (see https://marine.copernicus.eu/news/introducing-new-copernicus-marine-data-store)

Dataset references: - [1] https://data.marine.copernicus.eu/product/GLOBALANALYSISFORECASTWAV001027/description - [2] https://data.marine.copernicus.eu/product/GLOBALANALYSISFORECASTPHY001024/description - [3] https://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_0p25deg/catalog.html - [4] https://www.ncei.noaa.gov/products/etopo-global-relief-model - [5] https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels

Funding

| Project/Logo | Description | | :-------------: | :------------- | | | MariGeoRoute is funded by the German Federal Ministry of Economic Affairs and Energy (BMWi) |