Data

Tools

Indexes

Applications

Experiments

Open Source Science

RSMTool

RSMTool: collection of tools building and evaluating automated scoring models - Published in JOSS (2016)

https://github.com/educationaltestingservice/rsmtool

Science Score: 93.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 6 DOI reference(s) in README and JOSS metadata
  • ✓
    Academic publication links
    Links to: joss.theoj.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ✓
    JOSS paper metadata
    Published in Journal of Open Source Software

Keywords

automated-scoring-engines fairness hacktoberfest python scikit-learn scoring-models visualization

Scientific Fields

Mathematics Computer Science - 84% confidence
Last synced: 6 months ago · JSON representation

Repository

A Python package to facilitate research on building and evaluating automated scoring models.

Basic Info
  • Host: GitHub
  • Owner: EducationalTestingService
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Homepage: https://rsmtool.readthedocs.io
  • Size: 57.2 MB
Statistics
  • Stars: 70
  • Watchers: 13
  • Forks: 19
  • Open Issues: 47
  • Releases: 33
Topics
automated-scoring-engines fairness hacktoberfest python scikit-learn scoring-models visualization
Created almost 10 years ago · Last pushed about 1 year ago
Metadata Files
Readme License

README.rst 

Rater Scoring Modeling Tool
---------------------------

.. image:: https://gitlab.com/EducationalTestingService/rsmtool/badges/main/pipeline.svg
   :target: https://gitlab.com/EducationalTestingService/rsmtool/-/pipelines
   :alt: Gitlab CI status

.. image:: https://dev.azure.com/EducationalTestingService/RSMTool/_apis/build/status/EducationalTestingService.rsmtool
   :target: https://dev.azure.com/EducationalTestingService/RSMTool/_build?view=runs
   :alt: Azure Pipeline status

.. image:: https://codecov.io/gh/EducationalTestingService/rsmtool/branch/main/graph/badge.svg?token=wMCnQrB49J
   :target: https://codecov.io/gh/EducationalTestingService/rsmtool
   :alt: Coverage status

.. image:: https://img.shields.io/conda/v/ets/rsmtool.svg
   :target: https://anaconda.org/ets/rsmtool
   :alt: Conda package for SKLL

.. image:: https://img.shields.io/readthedocs/rsmtool/v11.3.0.svg
   :target: https://rsmtool.readthedocs.io
   :alt: Docs

.. image:: https://img.shields.io/badge/DOI-10.21105%2Fjoss.00033-blue.svg
   :target: http://joss.theoj.org/papers/10.21105/joss.00033
   :alt: DOI for citing RSMTool

.. image:: https://img.shields.io/pypi/pyversions/rsmtool.svg
   :target: https://pypi.org/project/rsmtool/
   :alt: Supported python versions for RSMTool

.. image:: https://img.shields.io/pypi/v/rsmtool.svg
   :target: https://pypi.org/project/rsmtool/
   :alt: Latest version on PyPI

.. image:: https://img.shields.io/pypi/l/rsmtool.svg
   :alt: License

Introduction
------------

Automated scoring of written and spoken test responses is a growing field in educational natural language processing. Automated scoring engines employ machine learning models to predict scores for such responses based on features extracted from the text/audio of these responses. Examples of automated scoring engines include `Project Essay Grade `_ for written responses and `SpeechRater `_ for spoken responses.

Rater Scoring Modeling Tool (RSMTool) is a python package which automates and combines in a single pipeline multiple analyses that are commonly conducted when building and evaluating such scoring models.  The output of RSMTool is a comprehensive, customizable HTML statistical report that contains the output of these multiple analyses. While RSMTool does make it really simple to run a set of standard analyses using a single command, it is also fully customizable and allows users to easily exclude unneeded analyses, modify the default analyses, and even include custom analyses in the report.

We expect the primary users of RSMTool to be researchers working on developing new automated scoring engines or on improving existing ones. Note that RSMTool is not a scoring engine by itself but rather a tool for building and evaluating machine learning models that may be used in such engines.

RSMTool is driven by a configuration file that users have to supply. Given the large number of available options, this can get complicated especially for new users. That's why RSMTool can help users generate configuration files interactively via guided prompts! The video below demonstrates this feature.

.. image:: https://raw.githubusercontent.com/EducationalTestingService/rsmtool/main/doc/assets/demo.gif
   :alt: Interactive Generation Demo Video

Getting Started
---------------

To get started with RSMTool, please see the extensive `official documentation `_.  If you use the `Dash `_ app on macOS, you can also download the complete RSMTool documentation for offline use. Go to the Dash preferences, click on "Downloads", then "User Contributed", and search for "RSMTool".


Requirements
------------

- ``ipython``
- ``jupyter``
- ``notebook``
- ``numpy``
- ``openpyxl``
- ``pandas``
- ``python>=3.8``
- ``scikit-learn``
- ``scipy``
- ``seaborn``
- ``shap``
- ``skll==5.0.1``
- ``statsmodels``
- ``wandb``
- ``xlrd``
- ``xlwt``

Contributing
------------
Contributions to RSMTool are very welcome. Please refer to the `documentation `_ for how to get started on developing new features or functionality for RSMTool.

Citing
------
If you are using RSMTool in your work, you can cite it as follows:

MLA
===
Madnani, Nitin and Loukina, Anastassia. "RSMTool: A Collection of Tools for Building and Evaluating Automated Scoring Models". Journal of Open Source Software 1(3), 2016.

BibTex
======

.. code:: bib

  @article{MadnaniLoukina2016,
    doi = {10.21105/joss.00033},
    url = {http://dx.doi.org/10.21105/joss.00033},
    year  = {2016},
    month = {jul},
    publisher = {The Open Journal},
    volume = {1},
    number = {3},
    author = {Nitin Madnani and Anastassia Loukina},
    title = {{RSMTool}: A Collection of Tools for Building and Evaluating Automated Scoring Models},
    journal = {{Journal of Open Source Software}}
  }


Changelog
---------
See `GitHub Releases `_.

Owner

  • Name: ETS
  • Login: EducationalTestingService
  • Kind: organization
  • Email: opensource@ets.org
  • Location: Princeton, NJ

Educational Testing Service

JOSS Publication

RSMTool: collection of tools building and evaluating automated scoring models
Published
July 28, 2016
DOI
Volume 1, Issue 3, Page 33
Authors
Nitin Madnani ORCID
Educational Testing Service
Anastassia Loukina ORCID
Educational Testing Service
Editor
Arfon Smith ORCID
Tags
educational applications scoring models statistics visualization
View PDF Review Thread Software Archive

GitHub Events

Total
  • Issues event: 1
  • Watch event: 4
  • Delete event: 1
  • Issue comment event: 9
  • Push event: 3
  • Pull request review event: 1
  • Pull request event: 2
  • Fork event: 2
Last Year
  • Issues event: 1
  • Watch event: 4
  • Delete event: 1
  • Issue comment event: 9
  • Push event: 3
  • Pull request review event: 1
  • Pull request event: 2
  • Fork event: 2

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 2,355
  • Total Committers: 18
  • Avg Commits per committer: 130.833
  • Development Distribution Score (DDS): 0.476
Past Year
  • Commits: 5
  • Committers: 2
  • Avg Commits per committer: 2.5
  • Development Distribution Score (DDS): 0.4
Top Committers
Name Email Commits
Nitin Madnani n****i@e****g 1,233
Anastassia Loukina a****a@e****g 717
jbiggsets j****s@e****g 182
Tamar Lavee t****e@e****g 88
Matthew Mulholland m****d@e****g 26
Frost45 s****p@e****g 24
Zhaoyang Xie d****2@g****m 23
hlepp 4****p 17
Binod Gyawali b****i@e****g 13
Zhaoyang Xie z****e@e****a 10
Robert Pugh r****h@e****g 8
Biggs j****s@M****g 5
srhrshr s****h@g****m 2
jbiggsets 3****s 2
Jeremy G. Kahn j****y@t****t 2
Biggs j****s@w****g 1
Arfon Smith a****n 1
tamarl08 1****8 1
Committer Domains (Top 20 + Academic)
ets.org: 8 wsp014865wss.etsclient.org: 1 trochee.net: 1 m-ah552373-010044116072.etsclient.org: 1 etscanada.ca: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 92
  • Total pull requests: 118
  • Average time to close issues: about 1 year
  • Average time to close pull requests: 26 days
  • Total issue authors: 10
  • Total pull request authors: 9
  • Average comments per issue: 0.84
  • Average comments per pull request: 2.27
  • Merged pull requests: 103
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 1
  • Pull requests: 4
  • Average time to close issues: N/A
  • Average time to close pull requests: 1 day
  • Issue authors: 1
  • Pull request authors: 3
  • Average comments per issue: 5.0
  • Average comments per pull request: 2.5
  • Merged pull requests: 3
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • desilinguist (48)
  • aloukina (20)
  • tamarl08 (8)
  • mulhod (4)
  • damien2012eng (4)
  • jbiggsets (2)
  • twoertwein (2)
  • ghost (1)
  • Frost45 (1)
  • bwriordan (1)
Pull Request Authors
  • desilinguist (71)
  • tamarl08 (16)
  • Frost45 (15)
  • damien2012eng (12)
  • mulhod (6)
  • aloukina (4)
  • srhrshr (2)
  • twoertwein (2)
  • ghost (1)
Top Labels
Issue Labels
enhancement (26) dependency (14) bug (8) good first issue (8) help wanted (6) documentation (6) testing (3) task (1)
Pull Request Labels
documentation (1) dependency (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 92 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 20
  • Total maintainers: 4
pypi.org: rsmtool

Rater scoring modeling tool

  • Versions: 20
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 92 Last month
Rankings
Stargazers count: 8.5%
Forks count: 8.9%
Dependent packages count: 10.1%
Average: 13.2%
Downloads: 17.2%
Dependent repos count: 21.5%
Maintainers (4)
desilinguist tamarl08 Frost45 damien2012eng
Last synced: 6 months ago

Dependencies

doc/requirements.txt pypi
  • ipython *
  • nbconvert *
  • skll *
  • statsmodels *
requirements.txt pypi
  • coverage *
  • coveralls *
  • doc2dash *
  • ipython *
  • jupyter *
  • nose *
  • notebook *
  • numpy *
  • openpyxl *
  • pandas *
  • parameterized *
  • seaborn *
  • skll ==3.0.0
  • sphinx *
  • sphinx_rtd_theme *
  • statsmodels *
  • tqdm *
  • xlrd *
  • xlwt *
pyproject.toml pypi