Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/bje-/nemweb

python package to directly download and process AEMO files from http://www.nemweb.com.au/

https://github.com/bje-/nemweb

Science Score: 10.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    1 of 7 committers (14.3%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (15.0%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

python package to directly download and process AEMO files from http://www.nemweb.com.au/

Basic Info
  • Host: GitHub
  • Owner: bje-
  • License: mit
  • Language: Jupyter Notebook
  • Default Branch: master
  • Size: 5.28 MB
Statistics
  • Stars: 0
  • Watchers: 1
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Fork of opennem/nemweb
Created over 6 years ago · Last pushed over 6 years ago
Metadata Files
Readme License

README.md

nemweb

This is a python3 package to directly download and process AEMO files from www.nemweb.com.au. Main module within the package downloads the nemweb files and inserts the tables into a local SQLite database.

This forms part of the backend of the OpenNEM platform. The OpenNEM backend utilises a normalised MySQL database (with foreign key and unique constraints). At present, this package only includes a simple sqlite interface without this support of capability. A mysql interface and module will be added (...eventually - see enhancements).

Getting started

Dependencies

Installation

nemweb can be installed through pip:

bash pip install opennem

Or by running:

bash python3 setup.py install

From the package directory. If installing by setup.py, a post-install script will prompt you input a directory for the sqlite database to live in. For example:

bash 'Enter directory (abs path) to store for sqlite db:'/home/dylan/Data

This value will live in a configuration file in your root directory ($HOME/.nemweb_config.ini).

IMPORTANT If you install via pip you must manually enter the sqlite directory in a file named .nemweb_config.ini in your home directly post- install (...couldn't figure out how to make post-install scripts to work with pip).

Quick example

```python from nemweb import nemweb_current

nemwebcurrent.updatedatasets(['dispatchscada']) DISPATCHUNIT_SCADA doesn't exists. Enter start date [YYYYMMDD]: 20180624 ```

The first time you add a new dataset to you sqlite db, it will prompt you for a date to start donwloading from. From then on, it will only download data beyond what you already have locally.

You can chose to print progress to screen, if desired. For example (and for a table that has already been initialised):

```python from nemweb import nemweb_current

nemwebcurrent.updatedatasets(['dispatchscada'], printprogress = True) 'DispatchSCADA 2018-06-24 13:40:00' 'DispatchSCADA 2018-06-24 13:45:00' 'DispatchSCADA 2018-06-24 13:50:00' 'DispatchSCADA 2018-06-24 13:55:00' '...' ```

Currently, the following dataset are built in to the package, and can be added and downloaded automatically from the Current index of nemweb

  • next_day_actual_gen
  • rooftopPV_actual
  • trading_is
  • dispatch_scada
  • next_day_dispatch
  • dispatch_is

Other datasets can be add by using using the class factory function for containing data for 'Current' datasets (CurrentDataset) found in the nemweb_current module. The namedtuple from this can then be used with CurrentFileHandler class to download and process that dataset.

Documentation

Build Status

Find more information in the documentation at readthedocs.

TODO

In no particular order:

  • Add more datasets from Current index
  • Add module to process archived data (from Archive index)
  • Add more sophisticate support for sqlite database (e.g. selectively inserting fields, tables from dataset)
  • Add module for interfacing with mysql database

Owner

  • Name: Ben Elliston
  • Login: bje-
  • Kind: user
  • Location: Canberra, Australia

GitHub Events

Total
Last Year

Committers

Last synced: over 2 years ago

All Time
  • Total Commits: 77
  • Total Committers: 7
  • Avg Commits per committer: 11.0
  • Development Distribution Score (DDS): 0.403
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Dylan McConnell d****n@l****t 46
Dylan McConnell d****l@g****m 15
Ben Elliston b****e@a****u 6
Dylan McConnell d****e 6
Joel Courtney e****e@g****m 2
Tim Bell t****l@g****m 1
Adam Green a****n@a****m 1
Committer Domains (Top 20 + Academic)
adgefficiency.com: 1 air.net.au: 1

Issues and Pull Requests

Last synced: over 2 years ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels