oads-download

oads_download.py is a Python script designed to download EarthCARE data products from ESA's Online Access and Distribution System (OADS). The execution of the script is divided into two parts: 1. Based on user inputs via the command-line search requests are send to the OpenSearch API data catalogue EO-CAT. 2. The resulting list of products is then downloaded from the OADS servers: - https://ec-pdgs-dissemination1.eo.esa.int/ - https://ec-pdgs-dissemination2.eo.esa.int/

User credentials are not required for the search part but are needed for the download part and must be provided in a separate configuration file (called config.toml) where OADS data collections must also be selected depending on your data access authorization. The detailed content of each collection is described here.

The code of this script is developed by Leonard König (TROPOS) based on a Jupyter notebook provided by ESA. If you have questions please create an issue or contact koenig@tropos.de.

Table of Contents

• Setup
• Usage
  • Logging
  • Examples
    • How can I download specific frames?
    • How can I select products within the radius of a ground site?
    • How do I obtain data for an entire day?
    • How can I first search for product candidates and then select a single product?
    • How to download orbit ranges?
• Tables of product name aliases
  • Level 1 products
  • Level 2a products
  • Level 2b products
  • Auxiliary data
  • Orbit data

Setup

• Make sure that you are using a Python environment with the following dependencies:
  • python 3.11+ (or 3.10 with tomli)
  • requests
  • numpy
  • pandas
  • beautifulsoup4
  • lxml
• Create a copy of the example_config.toml file and rename it to config.toml. It should be located in the same directory as the script oads_download.py.
• Enter your OADS credentials as well as the path to your desired data folder.
• Comment out or remove all OADS data collections for which you do not have access authorizations. Otherwise you may not be able to download any data.

Usage

For detailed explanations on how to use the script and available search criteria run the help command: $ python oads_download.py -h

By default, products downloaded with the script are unpacked and stored in the local data folder specified in the data_directory of your config.toml file. Also, products are organized in a subfolder structure depending on the product level and the acquisition date: data_directory/ ├── L1/ │ ├── 2024/ │ │ ├── 11/ │ │ │ ├── 01/ │ │ │ ├── 02/ │ │ │ └── ... ├── L2a/ ├── L2b/ ├── Meteo_Supporting_Files/ └── Orbit_Data_Files/ To prevent this, the --no_unzip and --no_subdirs options can be used.

Logging

On execution, log files are created which can be found in the logs folder. These can be used to trace the execution of the script in more detail than from the console. Logging can be disabled by using the --no_log option. By default, a maximum of 10 log files are created (older files are automatically deleted).

Examples

Here are selected examples that illustrate some possible use cases.

Example 1: How can I download specific frames?

To download the ATLNOM1B product for the orbit and frame 02163E you can run the command: $ python oads_download.py ATL_NOM_1B -oaf 2163E If you want to download a product from a specific processor baseline, you can specify its two-letter identifier after a colon or use the --product_version/-pv option: $ python oads_download.py ATL_NOM_1B:AC -oaf 2163E You can also download different product types with the same command and also use alternative shorthand aliases (see this table below). For example, the following command downloads the products ATLNOM1B (baseline AC), CPRNOM1B and MSIRGR1C for frame 02163E. $ python oads_download.py ANOM:AC MRGR CNOM -oaf 2163E You can also specify only a timestamp within the frame, e.g. if you do not know the orbit and frame identifier in advance (the --time/-t option allows flexible timestamp string formats, like 202410142355, 2024-10-14T23:55, ...): $ python oads_download.py ANOM:AC MRGR CNOM -t 2024-10-14T23:55

Example 2: How can I select products within the radius of a ground site?

$ python oads_download.py ATL_EBD_2A --radius 100000 16.878 -24.995 --start_time 2025-01-20T00:00:00 --end_time 2025-01-28T00:00:00 With this command the script downloads all ATLEBD2A products that are found within a 100 km radius around Mindelo CPV between the 20th and 28th of January 2025.

Example 3: How do I obtain data for an entire day?

$ python oads_download.py AALD -st 20250101 -et 20250102 This command downloads all ATLALD2A products for the day of January 1 2025 (125 files) by using the --start_time/-st and --end_time/-et options.

Example 4: How can I first search for product candidates and then select a single product?

$ python oads_download.py XORBP -t 20250130 --no_download This lists all AUX_ORBPRE files predicting the orbit on January 30 2025 without downloading them.

The output shows a list of found products with indices: ... List of files found (total number 11): [ 1] ECA_EXAA_AUX_ORBPRE_20250120T000000Z_20250130T000000Z_0001 [ 2] ECA_EXAA_AUX_ORBPRE_20250121T000000Z_20250131T000000Z_0001 [ 3] ECA_EXAA_AUX_ORBPRE_20250122T000000Z_20250201T000000Z_0001 [ 4] ECA_EXAA_AUX_ORBPRE_20250123T000000Z_20250202T000000Z_0001 [ 5] ECA_EXAA_AUX_ORBPRE_20250124T000000Z_20250203T000000Z_0001 [ 6] ECA_EXAA_AUX_ORBPRE_20250125T000000Z_20250204T000000Z_0001 [ 7] ECA_EXAA_AUX_ORBPRE_20250126T000000Z_20250205T000000Z_0001 [ 8] ECA_EXAA_AUX_ORBPRE_20250127T000000Z_20250206T000000Z_0001 [ 9] ECA_EXAA_AUX_ORBPRE_20250128T000000Z_20250207T000000Z_0001 [10] ECA_EXAA_AUX_ORBPRE_20250129T000000Z_20250208T000000Z_0001 [11] ECA_EXAA_AUX_ORBPRE_20250130T000000Z_20250209T000000Z_0001 Note: To export this list use the option --export_results Note: To select only one specific file use the option -i/--select_file_at_index ... To download a single file from this list you can specify its index. To select the last file set the index to -1: $ python oads_download.py XORBP -t 20250130 -i -1 ... List of files found (total number 11): [ 1] ECA_EXAA_AUX_ORBPRE_20250120T000000Z_20250130T000000Z_0001 [ 2] ECA_EXAA_AUX_ORBPRE_20250121T000000Z_20250131T000000Z_0001 [ 3] ECA_EXAA_AUX_ORBPRE_20250122T000000Z_20250201T000000Z_0001 [ 4] ECA_EXAA_AUX_ORBPRE_20250123T000000Z_20250202T000000Z_0001 [ 5] ECA_EXAA_AUX_ORBPRE_20250124T000000Z_20250203T000000Z_0001 [ 6] ECA_EXAA_AUX_ORBPRE_20250125T000000Z_20250204T000000Z_0001 [ 7] ECA_EXAA_AUX_ORBPRE_20250126T000000Z_20250205T000000Z_0001 [ 8] ECA_EXAA_AUX_ORBPRE_20250127T000000Z_20250206T000000Z_0001 [ 9] ECA_EXAA_AUX_ORBPRE_20250128T000000Z_20250207T000000Z_0001 [10] ECA_EXAA_AUX_ORBPRE_20250129T000000Z_20250208T000000Z_0001 <[11]> ECA_EXAA_AUX_ORBPRE_20250130T000000Z_20250209T000000Z_0001 <-- Select file (user input: -1) Note: To export this list use the option --export_results ...

Further examples: How to download orbit ranges?

Download all D and B frames from orbit 3000 to 3009 (20 files): $ python oads_download.py AALD -f D B -so 3000 -eo 3009

Download all frames between 01300D and 01302B (15 files): $ python oads_download.py AALD -soaf 01300D -eoaf 01302B

Tables of product name aliases

Level 1 products

| Product name | File type | Shorthand | Notes | | ------------ | ---------- | --------- | ------------ | | A-NOM | ATLNOM1B | ANOM | | | M-NOM | MSINOM1B | MNOM | | | B-NOM | BBRNOM1B | BNOM | | | C-NOM | CPRNOM1B | CNOM | JAXA product | | M-RGR | MSIRGR1C | MRGR | |

Level 2a products

| Product name | File type | Shorthand | Notes | | ------------ | ---------- | --------- | ------------ | | A-FM | ATLFM2A | AFM | | | A-AER | ATLAER2A | AAER | | | A-ICE | ATLICE2A | AICE | | | A-TC | ATLTC2A | ATC | | | A-EBD | ATLEBD2A | AEBD | | | A-CTH | ATLCTH2A | ACTH | | | A-ALD | ATLALD2A | AALD | | | M-CM | MSI_CM2A | MCM | | | M-COP | MSICOP2A | MCOP | | | M-AOT | MSIAOT2A | MAOT | | | C-FMR | CPRFMR2A | CFMR | | | C-CD | CPRCD2A | CCD | | | C-TC | CPRTC_2A | CTC | | | C-CLD | CPRCLD2A | CCLD | | | C-APC | CPRAPC2A | CAPC | | | A-CLA | ATLCLA2A | ACLA | JAXA product | | M-CLP | MSICLP2A | MCLP | JAXA product | | C-ECO | CPRECO2A | CECO | JAXA product | | C-CLP | CPRCLP_2A | CCLP | JAXA product |

Level 2b products

| Product name | File type | Shorthand | Notes | | ------------ | ---------- | --------- | ------------ | | AM-MO | AMMO2B | AMMO | | | AM-CTH | AMCTH_2B | AMCTH | | | AM-ACD | AMACD2B | AMACD | | | AC-TC | ACTC2B | ACTC | | | BM-RAD | BMRAD2B | BMRAD | | | BMA-FLX | BMAFLX2B | BMAFLX | | | ACM-CAP | ACMCAP2B | ACMCAP | | | ACM-COM | ACMCOM2B | ACMCOM | | | ACM-RT | ACMRT2B | ACMRT | | | ACMB-3D | ALL3D2B | ALL3D | | | ACMB-DF | ALL_DF2B | ALLDF | | | AC-CLP | AC_CLP2B | ACCLP | JAXA product | | ACM-CLP | ACMCLP2B | ACMCLP | JAXA product | | ACMB-RAD | ALLRAD2B | ALLRAD | JAXA product |

Auxiliary data

| Product name | File type | Shorthand | Notes | | ------------ | ---------- | --------- | ----- | | X-MET | AUXMET1D | XMET | | | X-JSG | AUXJSG1D | XJSG | |

Orbit data

| Product name | File type | Shorthand | Notes | | ------------ | ---------- | --------- | ------------------- | | ORBSCT | MPLORBSCT | MPLORBS | Orbit scenario | | ORBPRE | AUXORBPRE | XORBP | Predicted orbit | | ORBRES | AUX_ORBRES | XORBR | Reconstructed orbit |