Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/uwmadison-chm/masterfile

Tools to organize, access, and catalog the variables of interest in a scientific study

https://github.com/uwmadison-chm/masterfile

Science Score: 20.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ✓
    Committers with academic emails
    1 of 5 committers (20.0%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.0%) to scientific vocabulary

Keywords from Contributors

interactive projection sequences archival optim embedded genomics observability autograding hacking
Last synced: 11 months ago · JSON representation

Repository

Tools to organize, access, and catalog the variables of interest in a scientific study

Basic Info
  • Host: GitHub
  • Owner: uwmadison-chm
  • License: mit
  • Language: Python
  • Default Branch: main
  • Size: 226 KB
Statistics
  • Stars: 2
  • Watchers: 1
  • Forks: 1
  • Open Issues: 17
  • Releases: 0
Created almost 9 years ago · Last pushed about 2 years ago
Metadata Files
Readme License

README.md

masterfile

DOI

Tools to organize, document, and validate the variables of interest in scientific studies

Command line usage

masterfile --help will list all the subcommands.

Create

masterfile create masterfile_path out_file

Join

masterfile join masterfile_path out_file

Extract

masterfile extract [-s|--skip ROWS] [--index_column COL]
                      masterfile_path csv_file out_file

Validate

asterfile validate masterfile_path [file [file ...]]

Draft API usage example

```python import masterfile

Load all of the .csv files from /path, and the dictionary files in

/path/dictionaries. Takes settings info from a 'settings.json' file in

/path.

joins the .csv files on 'participant_id', which will be used as the index

There will be warnings if the data look bad in some way

mf = masterfile.load('/path')

Get the pandas dataframe associated

df = mf.dataframe # aliased as mf.df

All the variable stuff is less important, people can go look in data dicts

So we'll write that stuff later.

v = mf.lookup('srt1panaspa') v.contacts # listof_names v.measure.contact # Someone v.modality # Component("self-report") ```

CSV file format

CSV files should be comma-separated (no surprise there) and have DOS line endings (CRLF). They should not have the stupid UTF-8 signature at the start. UTF-8 characters are fine. Missing data is indicated by an empty cell. Quoting should be like Excel does.

Basically, you want Excel-for-Windows-style CSV files with no UTF-8 signature.

Dictionaries

  • CSV format
  • Has AT LEAST two columns: component, short_name
  • Those are the indexes
  • There shouldn't be any repeats in the index
  • The settings.json file should contain a "components" thing that says what should exist in the component column
  • Things with blank component are ignored (TODO: Maybe?)

Exclusion files

  • CSV format
  • Live in exclusions/
  • One row per ppt, one column per value
  • Has index column, same as data file
  • Blanks mean "Use this value," nonblanks mean "exclude this value"
  • Things in the cells may be codes; these codes may be defined in settings.json
  • If data is excluded for more than one reason, separate codes with ","
  • Not all rows / columns in masterfiles need to be included in exclusion files. Missing rows / columns are treated like blank values.

Data checks

Here are some (all?) of the things to do to verify you have semantically reasonable data:

  • Variable parts not in dictionaries
  • Missing participant_id column
  • Repeated paticipant_id column
  • Blanks in participant_id column
  • Duplicate columns
  • Column names not matching format

Getting started for development

Create a virtualenv:

virtualenv ~/env/masterfile
source ~/env/masterfile/bin/activate

Install the requirements and this module for development:

pip install -r requirements_dev.txt
pip install -e .

Run tests:

pytest

Run tests across all supported Python versions:

tox

To run in a specific python version:

tox -e py37

Credits

Written by Nate Vack njvack@wisc.edu with help from Dan Fitch dfitch@wisc.edu

masterfile packages some wonderful tools: schema and attrs.

schema is copyright (c) 2012 Vladimir Keleshev, vladimir@keleshev.com

attrs is copyright (c) 2015 Hynek Schlawack

Owner

  • Name: Center for Healthy Minds
  • Login: uwmadison-chm
  • Kind: organization
  • Location: Madison, WI

The Center for Healthy Minds at UW-Madison

GitHub Events

Total
Last Year

Committers

Last synced: over 2 years ago

All Time
  • Total Commits: 164
  • Total Committers: 5
  • Avg Commits per committer: 32.8
  • Development Distribution Score (DDS): 0.11
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Nate Vack n****k@w****u 146
Nathan Vack n****e@g****s 9
Dan Fitch d****h@g****m 7
dependabot[bot] 4****] 1
Nate Vack n****k@f****t 1
Committer Domains (Top 20 + Academic)
freshforever.net: 1 gimlet.us: 1 wisc.edu: 1

Issues and Pull Requests

Last synced: 12 months ago

All Time
  • Total issues: 24
  • Total pull requests: 2
  • Average time to close issues: 7 months
  • Average time to close pull requests: 9 minutes
  • Total issue authors: 2
  • Total pull request authors: 2
  • Average comments per issue: 0.5
  • Average comments per pull request: 0.0
  • Merged pull requests: 2
  • Bot issues: 0
  • Bot pull requests: 1
Past Year
  • Issues: 2
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 1
  • Pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • njvack (22)
  • dgfitch (2)
Pull Request Authors
  • njvack (1)
  • dependabot[bot] (1)
Top Labels
Issue Labels
Pull Request Labels
dependencies (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 25 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 5
  • Total maintainers: 3
pypi.org: masterfile

Tools to organize, document, and validate the variables of interest in scientific studies

  • Versions: 5
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 25 Last month
Rankings
Dependent packages count: 7.3%
Dependent repos count: 22.1%
Forks count: 22.8%
Average: 27.0%
Stargazers count: 32.0%
Downloads: 50.6%
Maintainers (3)
dgfitch njvack sshrivastava224
Last synced: 12 months ago

Dependencies

requirements_dev.txt pypi
  • PyYAML * development
  • Sphinx * development
  • bumpversion * development
  • coverage * development
  • flake8 * development
  • pandas * development
  • pytest * development
  • tox * development
  • watchdog * development
  • wheel * development