Data

Tools

Indexes

Applications

Experiments

Open Source Science

statxplore

This is a data harvester for the Department for Work and Pensions Stat-Explore

https://github.com/urban-flows/stat-xplore

Science Score: 62.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    1 of 1 committers (100.0%) from academic institutions
  • ✓
    Institutional organization owner
    Organization urban-flows has institutional domain (urbanflows.ac.uk)
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.3%) to scientific vocabulary

Keywords

data-pipeline datagovuk govuk
Last synced: 7 months ago · JSON representation ·

Repository

This is a data harvester for the Department for Work and Pensions Stat-Explore

Basic Info
  • Host: GitHub
  • Owner: urban-flows
  • License: mit
  • Language: Python
  • Default Branch: master
  • Homepage: http://urbanflows.ac.uk/
  • Size: 20.5 KB
Statistics
  • Stars: 1
  • Watchers: 0
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Topics
data-pipeline datagovuk govuk
Created about 4 years ago · Last pushed about 4 years ago
Metadata Files
Readme License Citation

README.md

DWP Harvester

This is a data harvester for the Department for Work and Pensions (DWP) Stat-Explore system. See Stat-Xplore : Open Data API documentation. This API is based on the SuperSTAR 9.5 Open Data API by WingArc1st.

Installation

To install from the Python Package Index (PyPi):

bash pip install statxplore

Authentication

The harvester authenticates against the remote server using an access token.

To use the harvester you must register an account on Stat-Xplore. When you're logged in, click the menu button (the three dots in the top-right corner) and click "Account". The string under "Open Data API Access" which is your API access key/token.

This token must be input to the harvester either in the command line or from a file (the default path is ~/configs/stat_explore.txt) as shown below:

```bash statxplore --apikey <myaccess_token>

or

statxplore --apikeypath ~/configs/stat_explore.txt ```

Usage

To view the available commands and options, run the following command:

bash statxplore --help

Querying

To get a query specification in JSON format, visit Stat-Xplore, log in and select a data set. Choose a table and click "Open Table." You may customise the rows and columns as needed. Next, click the "Download Table" field and select "Open Data API Query (.json)" This JSON file can be used to define a query as shown below.

Run the following command to execute a query and output the result to a CSV file:

bash statxplore -o test.csv -q queries\relative-low-income-by-year-sheffield.json

Where -o (--output) is the output CSV file path, -q (--query) is the query JSON file.

To generate the CSV headers that will result from a particular query, use the --csv (-c) flag:

bash statxplore -o test.csv -q queries\relative-low-income-by-year-sheffield.json -c

Code documentation

An authenticated HTTP session is required to communicate with the API.

```python from statxplore import http_session

session = httpsession.StatSession(apikey='') ```

API objects

The subclasses of objects.StatObject are thin wrappers around the API endpoints. Please refer to the API documentation.

Schema

The /schema endpoint returns information about the Stat-Xplore datasets that are available to you, and their fields and measures.

The root endpoint, /schema, returns details of all datasets and folders at the root level of Stat-Xplore.

```python

from statxplore import objects

List all data schemas

objects.Schema.list(session)

Get info about a schema

objects.Schema('str:folder:fuc').get(session)

Get the schema of a specific table

objects.Schema('str:database:UC_Monthly').get(session) ```

Table examples

The /table endpoint allows you to submit table queries and receive the results. The body of the request contains your query.

```python

Retrieve the number of people on Universal Credit broken down by month

objects.Table('str:database:UCMonthly').runquery(session, measures=['str:count:UCMonthly:VFUCCASELOADFULL'], dimensions=[['str:field:UCMonthly:FUCDATE:DATE_NAME']], ) ```

It's also possible to use JSON to define a query. This is useful for replicating queries generated by the Stat-Xplore graphical user interface. (In Table View, go to Download Table and select "Open Data API Query (.json)" then click Go.)

python query = """{ "database" : "str:database:DLA_In_Payment_New", "measures" : [ "str:count:DLA_In_Payment_New:V_F_DLA_In_Payment_New", "str:statfn:DLA_In_Payment_New:V_F_DLA_In_Payment_New:CAWKLYAMT:MEAN" ], "dimensions" : [ [ "str:field:DLA_In_Payment_New:V_F_DLA_In_Payment_New:COA_CODE" ], [ "str:field:DLA_In_Payment_New:F_DLA_QTR_New:DATE_NAME" ] ] }""" data = objects.Table.query_json(session, query)

Resources

Owner

  • Name: Urban Flows Observatory
  • Login: urban-flows
  • Kind: organization
  • Email: urbanflows@sheffield.ac.uk
  • Location: United Kingdom

Through the gathering of data relating to the physical processes within cities, Urban Observatories enable characterisation of how cities ‘work’.

Citation (CITATION.cff) 

# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: Stat-Explore Harvest
message: 'If you use this software, please cite it.'
type: software
authors:
  - given-names: Joe Salvador Thornhill
    family-names: Heffer
    name-particle: Dr
    email: j.heffer@sheffield.ac.uk
    affiliation: The University of Sheffield
    orcid: 'https://orcid.org/0000-0001-8733-1117'

GitHub Events

Total
Last Year

Committers

Last synced: about 3 years ago

All Time
  • Total Commits: 9
  • Total Committers: 1
  • Avg Commits per committer: 9.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Joe Heffer j****r@s****k 9
Committer Domains (Top 20 + Academic)
sheffield.ac.uk: 1

Issues and Pull Requests

Last synced: 9 months ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 33 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 1
  • Total maintainers: 1
pypi.org: statxplore

This is a data harvester for the Department for Work and Pensions Stat-Explore

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 33 Last month
Rankings
Dependent packages count: 10.1%
Dependent repos count: 21.5%
Forks count: 29.8%
Average: 31.5%
Stargazers count: 31.9%
Downloads: 64.4%
Maintainers (1)
jheffer
Last synced: 8 months ago