Data

Tools

Indexes

Applications

Experiments

Open Source Science

Satip

Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data

https://github.com/openclimatefix/Satip

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    1 of 24 committers (4.2%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (18.5%) to scientific vocabulary

Keywords

eumetsat gis nowcasting python satellite-imagery

Keywords from Contributors

gan mesh climate-model pytorch-lightning particle unitful python-template energy-system-model pypy distribution
Last synced: 6 months ago · JSON representation

Repository

Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data

Basic Info
  • Host: GitHub
  • Owner: openclimatefix
  • License: mit
  • Language: Jupyter Notebook
  • Default Branch: main
  • Homepage: https://satip.readthedocs.io/
  • Size: 467 MB
Statistics
  • Stars: 44
  • Watchers: 1
  • Forks: 35
  • Open Issues: 1
  • Releases: 346
Topics
eumetsat gis nowcasting python satellite-imagery
Created over 5 years ago · Last pushed 6 months ago
Metadata Files
Readme License

README.md

Satip

⚠️ Note this repo will soon be deprecated in favour of satellite-consumer

All Contributors <!-- ALL-CONTRIBUTORS-BADGE:END -->

PyPI version

codecov

Satip is a library for satellite image processing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT data


Installation

To install the satip library please run:

bash pip install satip

Or if you're working in the development environment you can run the following from the directory root:

bash pip install -e .

Conda

Or, if you want to use conda from a cloned Satip repository:

bash conda env create -f environment.yml conda activate satip pip install -e .

If you plan to work on the development of Satip then also consider installing these development tools:

bash conda install pytest flake8 jedi mypy black pre-commit pre-commit install

Development Environment

In order to contribute: - it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development. - At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.

Operation

Getting your own API key

In order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:

  1. Go to https://eoportal.eumetsat.int and register an account.
  2. You can log in and go to https://data.eumetsat.int/ to check available data services. From there go to your profile and choose the option "API key" or go to https://api.eumetsat.int/api-key/ directly.
  3. Please make sure that you added the key and secret to your user's environment variables.

Downloading EUMETSAT Data

We have moved this to here

Converting Native files to Zarr

scripts/convert_native_to_zarr.py converts EUMETSAT .nat files to Zarr datasets, using very mild lossy JPEG-XL compression. (JPEG-XL is the "new kid on the block" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalent bz2 compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range [0, 1]. NaNs are encoded as the value 0.025. All "real" values are in the range [0.075, 1]. We leave a gap between "NaNs" and "real values" because there is very slight "ringing" around areas of constant value (see this comment for more details). Use satip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNs to decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range [0, 1].

Running in Production

The live service uses app.py as the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.

--api-key or API_KEY is the EUMETSAT API key

--api-secret or API_SECRET is the EUMETSAT API secret

--save-dir or SAVE_DIR is the top level directory to save the output files, a latest subfolder will be added to that directory to contain the latest data

--history or HISTORY is the amount of history timesteps to use in the latest.zarr files

--db-url or DB_URL is the URL to the database to save to when a run has finished

--use-rescaler or USE_RESCALER tells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.

--use-iodc or USE_IODC is an option to get the IODC satellite data

Testing

To run tests, simply run pytest . from the root of the repository. To generate the test plots, run python scripts/generate_test_plots.py.

Environmental Variables

Some tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows: - EUMETSAT_USER_KEY: the EUMETSAT API key - EUMETSAT_USER_SECRET: the EUMETSAT API secret

These can be added using the export command in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. "~/.bashrc" if using bash).

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Jacob Bieker
Jacob Bieker
💻		 Jack Kelly
Jack Kelly
💻		 Ayrton Bourn
Ayrton Bourn
💻		 Laurence Watson
Laurence Watson
💻		 Notger Heinz
Notger Heinz
📖		 Peter Dudfield
Peter Dudfield
📖		 Azah Norbline
Azah Norbline
💻
Tom Pughe
Tom Pughe
💻		 Zhenbang Feng
Zhenbang Feng
💻		 jsbaasi
jsbaasi
💻		 Suleman Karigar
Suleman Karigar
💻		 Richa
Richa
💻		 Nathan Simpson
Nathan Simpson
🐛		 peach280
peach280
🚧

This project follows the all-contributors specification. Contributions of any kind welcome!

Owner

  • Name: Open Climate Fix
  • Login: openclimatefix
  • Kind: organization
  • Email: info@openclimatefix.org
  • Location: London

Using open science to mitigate climate change

GitHub Events

Total
  • Create event: 49
  • Release event: 23
  • Issues event: 31
  • Watch event: 2
  • Delete event: 19
  • Issue comment event: 67
  • Push event: 139
  • Pull request review comment event: 4
  • Pull request review event: 9
  • Pull request event: 58
  • Fork event: 6
Last Year
  • Create event: 49
  • Release event: 23
  • Issues event: 31
  • Watch event: 2
  • Delete event: 19
  • Issue comment event: 67
  • Push event: 139
  • Pull request review comment event: 4
  • Pull request review event: 9
  • Pull request event: 58
  • Fork event: 6

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 1,298
  • Total Committers: 24
  • Avg Commits per committer: 54.083
  • Development Distribution Score (DDS): 0.734
Past Year
  • Commits: 271
  • Committers: 14
  • Avg Commits per committer: 19.357
  • Development Distribution Score (DDS): 0.731
Top Committers
Name Email Commits
BumpVersion Action b****n@g****s 345
Jacob Bieker j****b@b****h 264
AyrtonB A****n@O****m 165
peterdudfield p****d@h****m 160
Rabscuttler l****n@g****m 80
Jacqueline James 1****J 73
pre-commit-ci[bot] 6****] 57
Notger Heinz n****z@g****m 34
Jack Kelly j****k@o****g 33
allcontributors[bot] 4****] 27
devsjc 4****c 22
dependabot[bot] 4****] 11
Adimanv a****7@g****m 6
peach280 v****8@g****m 5
Zhenbang Feng j****n@u****u 4
TomPughe 1****e 3
Richa 4****a 2
ludobegins l****n@g****m 1
Aryan Bhosale 3****e 1
Azah Norbline a****e@g****m 1
Matthew Duffin m****5 1
Simon Lemieux 1****x 1
Suleman Karigar 3****2 1
jsbaasi 7****i 1
Committer Domains (Top 20 + Academic)
usc.edu: 1 openclimatefix.org: 1 bieker.tech: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 135
  • Total pull requests: 264
  • Average time to close issues: 8 months
  • Average time to close pull requests: about 1 month
  • Total issue authors: 17
  • Total pull request authors: 28
  • Average comments per issue: 2.51
  • Average comments per pull request: 1.03
  • Merged pull requests: 183
  • Bot issues: 0
  • Bot pull requests: 82
Past Year
  • Issues: 11
  • Pull requests: 54
  • Average time to close issues: 11 days
  • Average time to close pull requests: 3 days
  • Issue authors: 4
  • Pull request authors: 10
  • Average comments per issue: 1.27
  • Average comments per pull request: 0.39
  • Merged pull requests: 38
  • Bot issues: 0
  • Bot pull requests: 5
View more stats
Top Authors
Issue Authors
  • peterdudfield (44)
  • jacobbieker (43)
  • JackKelly (28)
  • Rabscuttler (4)
  • PaulaPerezL (2)
  • devsjc (2)
  • TomPughe (2)
  • phinate (1)
  • AyrtonB (1)
  • AUdaltsova (1)
  • gabrielelibardi (1)
  • tomasvanoyen (1)
  • LuisTellezSirocco (1)
  • BenediktSchulzAlitiq (1)
  • jsbaasi (1)
Pull Request Authors
  • peterdudfield (73)
  • dependabot[bot] (50)
  • jacobbieker (44)
  • devsjc (21)
  • allcontributors[bot] (21)
  • pre-commit-ci[bot] (15)
  • peach280 (7)
  • JasonFengGit (7)
  • JackKelly (7)
  • vikasgrewal16 (4)
  • aryanbhosale (4)
  • TomPughe (4)
  • notger (4)
  • 14Richa (4)
  • jsbaasi (2)
Top Labels
Issue Labels
enhancement (55) bug (36) good first issue (25) timeline (4) help wanted (3) documentation (2) tech-ops (1) Meta (1)
Pull Request Labels
dependencies (50) enhancement (26) bug (9) documentation (3)

Packages

  • Total packages: 2
  • Total downloads: unknown
  • Total dependent packages: 0
    (may contain duplicates)
  • Total dependent repositories: 0
    (may contain duplicates)
  • Total versions: 710
proxy.golang.org: github.com/openclimatefix/satip
  • Versions: 355
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 5.4%
Average: 5.6%
Dependent repos count: 5.8%
Last synced: 6 months ago
proxy.golang.org: github.com/openclimatefix/Satip
  • Versions: 355
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 5.4%
Average: 5.6%
Dependent repos count: 5.8%
Last synced: 6 months ago

Dependencies

.github/workflows/workflows.yaml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
  • codecov/codecov-action v2 composite
  • iterative/setup-cml v1 composite
Dockerfile docker
  • ubuntu latest build
docs/requirements.txt pypi
  • jinja2 <3.1.0
  • pydoc-markdown *
  • readthedocs-custom-steps ==0.5.1
requirements.txt pypi
  • bottleneck *
  • click *
  • eumdac *
  • fsspec *
  • h5netcdf *
  • imagecodecs *
  • netCDF4 *
  • nowcasting_datamodel >=0.0.36
  • numcodecs *
  • numpy ==1.23.0
  • pandas >=1.1.4
  • psutil *
  • pyorbital *
  • pyresample ==1.25.0
  • requests *
  • s3fs *
  • satpy ==0.36.0
  • tqdm *
  • xarray ==2022.3.0
  • zarr *
.github/workflows/ghcr-release.yaml actions
  • actions/checkout v3 composite
  • docker/build-push-action ad44023a93711e3deb337508980b4b5e9bcdc5dc composite
  • docker/login-action f054a8b539a109f9f41c372932f1ae047eff08c9 composite
  • docker/metadata-action 98669ae865ea3cffbcbaa878cf57c20bbf1c6c38 composite
.github/workflows/release.yaml actions
setup.py pypi
environment.yml pypi
  • alembic ==1.13.0
  • blosc2 ==2.3.2
  • eumdac ==2.1.0
  • freezegun ==1.3.1
  • greenlet ==3.0.1
  • imagecodecs ==2023.9.18
  • iniconfig ==2.0.0
  • mako ==1.3.0
  • ndindex ==1.7
  • nowcasting-datamodel ==1.5.26
  • numpy ==1.23.2
  • ocf-blosc2 ==0.0.4
  • pluggy ==1.3.0
  • psycopg2-binary ==2.9.9
  • py-cpuinfo ==9.0.0
  • pydantic ==1.10.10
  • pytest ==7.4.3
  • sqlalchemy ==2.0.23
  • structlog ==23.2.0