Data

Tools

Indexes

Applications

Experiments

Open Source Science

rag-for-math-qa

Analysis code for a research paper

https://github.com/digitalharborfoundation/rag-for-math-qa

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 2 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (12.7%) to scientific vocabulary

Keywords

research-paper
Last synced: 10 months ago · JSON representation ·

Repository

Analysis code for a research paper

Basic Info
  • Host: GitHub
  • Owner: DigitalHarborFoundation
  • License: mit
  • Language: Jupyter Notebook
  • Default Branch: main
  • Homepage:
  • Size: 2.79 MB
Statistics
  • Stars: 18
  • Watchers: 3
  • Forks: 1
  • Open Issues: 0
  • Releases: 0
Topics
research-paper
Created almost 3 years ago · Last pushed over 1 year ago
Metadata Files
Readme License Citation

README.md

Retrieval-augmented generation to improve math question-answering: trade-offs between groundedness and human preference

arXiv License

This repository contains analysis code, prompts, surveys, figures, and data for the paper "Retrieval-augmented generation to improve math question-answering: trade-offs between groundedness and human preference".

This repository forks the llm-math-education package.

Cite the paper using the CITATION.cff file and dropdown:

Zachary Levonian, Chenglu Li, Wangda Zhu, Anoushka Gade, Owen Henkel, Millie-Ellen Postle, and Wanli Xing. 2023. Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference. In NeurIPS’23 Workshop on Generative AI for Education (GAIED), New Orleans, USA. DOI:https://doi.org/10.48550/arXiv.2310.03184

Development

Primary code contributor:

Local development setup

This project uses make and Poetry to manage and install dependencies.

On Windows, you'll need to use WSL and maybe make some other changes.

Python development

Use make install to install all needed dependencies (including the pre-commit hooks and Poetry).

You'll probably need to manually add Poetry to your PATH, e.g. by updating your .bashrc (or relevant equivalent):

bash export PATH="$HOME/.local/bin:$PATH"

Run tests

bash make test

Run Jupyter Lab

bash make jupyter

Which really just runs poetry run jupyter lab, so feel free to customize your Jupyter experience.

Other useful commands

  • poetry run <command> - Run the given command, e.g. poetry run pytest invokes the tests.
  • poetry add <package> - Add the given package as a dependency. Use flag -G dev to add it as a development dependency.

## Other notes

### Poster figures

Some logos are present in the posters directory.

The Digital Harbor Foundation logo was created using rsvg-convert, installed via brew.

I manually adjusted the source svg (dhf-logo-vector-blue.svg) to use DHF blue (#0091c9) rather than black (#010101).

bash brew install librsvg rsvg-convert -d 150 -p 150 -h 2in figures/dhf-logo-vector-blue.svg > figures/dhf-poster-logo.png

Converting the system diagram (converted from draw.io as an SVG, with embedded fonts):

bash rsvg-convert -d 150 -p 150 -h 4in figures/system-diagram.svg > figures/system-diagram-poster.png

Owner

  • Name: Digital Harbor Foundation
  • Login: DigitalHarborFoundation
  • Kind: organization
  • Location: Baltimore, MD

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite the paper as below."
authors:
  - family-names: Levonian
    given-names: Zachary
    orcid: https://orcid.org/0000-0002-8932-1489
  - family-names: Li
    given-names: Chenglu
  - family-names: Zhu
    given-names: Wangda
  - family-names: Gade
    given-names: Anoushka
  - family-names: Henkel
    given-names: Owen
  - family-names: Postle
    given-names: Millie-Ellen
  - family-names: Xing
    given-names: Wanli
date-released: 2023-10-04
repository-code: "https://github.com/DigitalHarborFoundation/rag-for-math-qa"
preferred-citation:
  type: conference-paper
  title: "Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference"
  abstract: "For middle-school math students, interactive question-answering (QA) with tutors is an effective way to learn. The flexibility and emergent capabilities of generative large language models (LLMs) has led to a surge of interest in automating portions of the tutoring process - including interactive QA to support conceptual discussion of mathematical concepts. However, LLM responses to math questions can be incorrect or mismatched to the educational context - such as being misaligned with a school's curriculum. One potential solution is retrieval-augmented generation (RAG), which involves incorporating a vetted external knowledge source in the LLM prompt to increase response quality. In this paper, we designed prompts that retrieve and use content from a high-quality open-source math textbook to generate responses to real student questions. We evaluate the efficacy of this RAG system for middle-school algebra and geometry QA by administering a multi-condition survey, finding that humans prefer responses generated using RAG, but not when responses are too grounded in the textbook content. We argue that while RAG is able to improve response quality, designers of math QA systems must consider trade-offs between generating responses preferred by students and responses closely matched to specific educational resources."
  doi: 10.48550/arXiv.2310.03184
  year: 2023
  conference:
      name: "NeurIPS'23 Workshop on Generative AI for Education (GAIED)"
      city: "New Orleans"
      country: "US"
      date-start: "2023-12-15"
      date-end: "2023-12-15"
  authors:
  - family-names: Levonian
    given-names: Zachary
    orcid: https://orcid.org/0000-0002-8932-1489
  - family-names: Li
    given-names: Chenglu
  - family-names: Zhu
    given-names: Wangda
  - family-names: Gade
    given-names: Anoushka
  - family-names: Henkel
    given-names: Owen
  - family-names: Postle
    given-names: Millie-Ellen
  - family-names: Xing
    given-names: Wanli

GitHub Events

Total
  • Watch event: 7
Last Year
  • Watch event: 7