Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/hcdp/mesonet_station_parser

Code for parsing mesonet loggernet files into Tapis stream

https://github.com/hcdp/mesonet_station_parser

Science Score: 8.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ✓
    Institutional organization owner
    Organization hcdp has institutional domain (hawaii.edu)
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (6.8%) to scientific vocabulary
Last synced: 6 months ago · JSON representation

Repository

Code for parsing mesonet loggernet files into Tapis stream

Basic Info
  • Host: GitHub
  • Owner: HCDP
  • Language: Python
  • Default Branch: main
  • Size: 325 KB
Statistics
  • Stars: 0
  • Watchers: 0
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Fork of jeff-yc-wong/mesonet_station_parser
Created almost 3 years ago · Last pushed about 1 year ago
Metadata Files
Readme

README.md

mesonetstationparser

This repository contains codes that parse mesonet station data (.dat) files into Tapis streams_api.

The repository contains 2 primary script files that achieve this. - streamsprocessor.py - This file parses incoming station data that is stored in a directory(datadir). - There are variations of this file that speed up the parsing process by using Python's multiprocessing module - legacy.py - This file downloads past data by sending API requests to the ikewai gateway, thus allowing us to parse past data into Tapis streams-api. - You are able to provide a startdate and an enddate range for what files you want to parse.

The standard_var folder contains csv files that maps the raw shortnames of the stations' data to the standardized shortnames - Note: the csv files are not extremely accurate as the raw shortnames for the stations' data changes occasionally.

TODO List

  • [ ] Ensure that standardize_variable function is working properly (still waiting for HCDP folks to decide on a more permanent variable naming convention)
  • [ ] Move username, password, datadir, (startdate and end_date for legacy.py) into a config file for easier editability
  • [ ] Potentially add a field in the config file to enable parallelism and define number of workers
  • [ ] For production use, remove iteration counter from projectid, siteid and inst_id

Owner

  • Name: Hawaii Climate Data Portal
  • Login: HCDP
  • Kind: organization

GitHub Events

Total
  • Delete event: 1
  • Push event: 8
  • Pull request event: 2
  • Create event: 1
Last Year
  • Delete event: 1
  • Push event: 8
  • Pull request event: 2
  • Create event: 1