Data

Tools

Indexes

Applications

Experiments

Open Source Science

brightsky

JSON API for DWD's open weather data.

https://github.com/jdemaeyer/brightsky

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.4%) to scientific vocabulary

Keywords

api dwd open-data weather
Last synced: 6 months ago · JSON representation

Repository

JSON API for DWD's open weather data.

Basic Info
  • Host: GitHub
  • Owner: jdemaeyer
  • License: mit
  • Language: Python
  • Default Branch: master
  • Homepage: https://brightsky.dev/
  • Size: 2.77 MB
Statistics
  • Stars: 352
  • Watchers: 11
  • Forks: 21
  • Open Issues: 26
  • Releases: 0
Topics
api dwd open-data weather
Created almost 6 years ago · Last pushed 10 months ago
Metadata Files
Readme Funding License

README.md

Bright Sky

API Status Docs Status Build Status PyPI Release Docker Hub Release

JSON API for DWD's open weather data.

The DWD (Deutscher Wetterdienst), as Germany's meteorological service, publishes a myriad of meteorological observations and calculations as part of their Open Data program.

Bright Sky is an open-source project aiming to make some of the more popular data — in particular weather observations from the DWD station network and weather forecasts from the MOSMIX model — available in a free, simple JSON API.

Looking for something specific?

I just want to retrieve some weather data

You can use the free public Bright Sky instance!

I want to run my own instance of Bright Sky

Check out the infrastructure repo!

I want to parse DWD weather files from the command line or in Python

The parsing core for Bright Sky is maintained in a separate package named dwdparse, which has no dependencies outside the standard library. If you find that's not quite serving your needs, check out wetterdienst.

I want to contribute to Bright Sky's source code

Read on. :)

On Bright Sky's versioning

Starting from version 2.0, where we extracted the parsing core into a separate package, Bright Sky is no longer intended to be used as a Python library, but only as the service available at brightsky.dev.

Consequentially, we adjust our version numbers from the perspective of that service and its users – i.e., we will increase the major version number only when we introduce backwards-incompatible (or otherwise very major) changes to the actual JSON API interface, e.g. by changing URLs or parameters. This means that increases of the minor version number may introduce backwards-incompatible changes to the internals of the brightsky package, including the database structure. If you use brightsky as a Python library, please version-pin to a minor version, e.g. by putting brightsky==2.0.* in your requirements.txt.

Quickstart

Running a full-fledged API instance

Note: These instructions are aimed at running a Bright Sky instance for development and testing. Check out our infrastructure repository if you want to set up a production-level API instance.

Just run docker-compose up and you should be good to go. This will set up a PostgreSQL database (with persistent storage in .data), run a Redis server, and start the Bright Sky worker and webserver. The worker periodically polls the DWD Open Data Server for updates, parses them, and stores them in the database. The webserver will be listening to API requests on port 5000.

Architecture

Bright Sky's Architecture

Bright Sky is a rather simple project consisting of four components:

  • The brightsky worker, which leverages the logic contained in the brightsky Python package to retrieve weather records from the DWD server, parse them, and store them in a database. It will periodically poll the DWD servers for new data.

  • The brightsky webserver (API), which serves as gate to our database and processes all queries for weather records coming from the outside world.

  • A PostgreSQL database consisting of two relevant tables:

    • sources contains information on the locations for which we hold weather records, and
    • weather contains the history of actual meteorological measurements (or forecasts) for these locations.

The database structure can be set up by running the migrate command, which will simply apply all .sql files found in the migrations folder.

  • A Redis server, which is used as the backend of the worker's task queue.

Most of the tasks performed by the worker and webserver can also be performed independently. Run docker-compose run --rm brightsky to get a list of available commands.

Hacking

Constantly rebuilding the brightsky container while working on the code can become cumbersome, and the default setting of parsing records dating all the way back to 2010 will make your development database unnecessarily large. You can set up a more lightweight development environment as follows:

  1. Create a virtual environment and install our dependencies: python -m virtualenv .venv && source .venv/bin/activate && pip install -r requirements.txt && pip install -e .

  2. Start a PostgreSQL container: docker-compose run --rm -p 5432:5432 postgres

  3. Start a Redis container: docker-compose run --rm -p 6379:6379 redis

  4. Point brightsky to your containers, and configure a tighter date threshold for parsing DWD data, by adding the following .env file: BRIGHTSKY_DATABASE_URL=postgres://postgres:pgpass@localhost BRIGHTSKY_BENCHMARK_DATABASE_URL=postgres://postgres:pgpass@localhost/benchmark BRIGHTSKY_REDIS_URL=redis://localhost BRIGHTSKY_MIN_DATE=2020-01-01

You should now be able to directly run brightsky commands via python -m brightsky, and changes to the source code should be effective immediately.

Tests

Large parts of our test suite run against a real Postgres database. By default, these tests will be skipped. To enable them, make sure the BRIGHTSKY_DATABASE_URL environment variable is set when calling pytest, e.g. via: BRIGHTSKY_DATABASE_URL=postgres://postgres:pgpass@localhost/brightsky_test pytest

Beware that adding this environment variable to your .env file will not work as that file is not read by pytest. The database will be dropped and recreated on every test run, so don't use your normal Bright Sky database. ;)

Acknowledgements

Bright Sky's development is boosted by the priceless guidance and support of the Open Knowledge Foundation's Prototype Fund program, and is generously funded by Germany's Federal Ministry of Education and Research. Obvious as it may be, it should be mentioned that none of this would be possible without the painstaking, never-ending effort of the Deutscher Wetterdienst.

Prototype Fund     Open Knowledge Foundation Germany     Bundesministerium für Bildung und Forschung     Deutscher Wetterdienst

Owner

  • Name: Jakob de Maeyer
  • Login: jdemaeyer
  • Kind: user
  • Location: Münster, Germany

Freelance Data and Product Engineer

GitHub Events

Total
  • Issues event: 10
  • Watch event: 42
  • Delete event: 1
  • Issue comment event: 31
  • Push event: 13
  • Pull request review comment event: 1
  • Pull request review event: 3
  • Pull request event: 2
  • Fork event: 3
  • Create event: 7
Last Year
  • Issues event: 10
  • Watch event: 42
  • Delete event: 1
  • Issue comment event: 31
  • Push event: 13
  • Pull request review comment event: 1
  • Pull request review event: 3
  • Pull request event: 2
  • Fork event: 3
  • Create event: 7

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 372
  • Total Committers: 3
  • Avg Commits per committer: 124.0
  • Development Distribution Score (DDS): 0.008
Past Year
  • Commits: 31
  • Committers: 1
  • Avg Commits per committer: 31.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Jakob de Maeyer j****1@g****m 369
Gerald Pape u****t 2
Tobias t****h@p****e 1
Committer Domains (Top 20 + Academic)
posteo.de: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 117
  • Total pull requests: 15
  • Average time to close issues: 3 months
  • Average time to close pull requests: 9 days
  • Total issue authors: 43
  • Total pull request authors: 5
  • Average comments per issue: 2.71
  • Average comments per pull request: 0.8
  • Merged pull requests: 6
  • Bot issues: 0
  • Bot pull requests: 7
Past Year
  • Issues: 10
  • Pull requests: 3
  • Average time to close issues: 3 days
  • Average time to close pull requests: 34 minutes
  • Issue authors: 9
  • Pull request authors: 2
  • Average comments per issue: 2.2
  • Average comments per pull request: 0.33
  • Merged pull requests: 2
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • jdemaeyer (56)
  • poetaster (4)
  • tawissus (3)
  • smilingchemist (3)
  • ubergesundheit (3)
  • MHz000 (3)
  • mweinelt (3)
  • BlackScreen (3)
  • martinfricke (2)
  • ptoews (2)
  • jwillmer (1)
  • MatthiasDod (1)
  • rfpaiva (1)
  • papjul (1)
  • gabowi (1)
Pull Request Authors
  • dependabot[bot] (7)
  • jdemaeyer (5)
  • AndiLeni (1)
  • tobi-laa (1)
  • ubergesundheit (1)
Top Labels
Issue Labels
documentation (4) experimental (4) enhancement (3) bug (2) data quality (2) housekeeping (1)
Pull Request Labels
dependencies (7)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 1,995 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 58
  • Total maintainers: 1
pypi.org: brightsky

JSON API for DWD's open weather data.

  • Versions: 58
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 1,995 Last month
Rankings
Stargazers count: 4.2%
Forks count: 8.9%
Dependent packages count: 10.1%
Average: 12.3%
Downloads: 16.5%
Dependent repos count: 21.5%
Maintainers (1)
jdemaeyer
Last synced: 10 months ago

Dependencies

requirements.txt pypi
  • astral ==2.2
  • certifi ==2022.6.15
  • charset-normalizer ==2.1.0
  • click ==8.1.3
  • coloredlogs ==15.0.1
  • cssselect ==1.1.0
  • falcon ==2.0.0
  • falcon-cors ==1.1.7
  • gunicorn ==20.1.0
  • huey ==2.4.3
  • humanfriendly ==10.0
  • idna ==3.3
  • lxml ==4.9.1
  • parsel ==1.6.0
  • psycopg2-binary ==2.9.3
  • python-dateutil ==2.8.2
  • pytz ==2022.1
  • redis ==3.5.3
  • requests ==2.28.1
  • sentry-sdk ==1.7.2
  • six ==1.16.0
  • urllib3 ==1.26.10
  • w3lib ==1.22.0
setup.py pypi
  • astral *
  • click *
  • coloredlogs *
  • falcon ==2.
  • falcon-cors *
  • gunicorn *
  • huey *
  • parsel *
  • psycopg2-binary *
  • python-dateutil *
  • redis <4
  • requests *
  • sentry-sdk *
.github/workflows/main.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v1 composite
  • docker/build-push-action v2 composite
  • docker/login-action v2 composite
  • pypa/gh-action-pypi-publish release/v1 composite
  • postgres 12 docker
Dockerfile docker
  • python 3.10-slim build
docker-compose.yml docker
pyproject.toml pypi