grolts-llm

A repository for testing and improving the GRoLTS checklist on open-source llms

https://github.com/timovdk/grolts-llm

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
✓
DOI references
Found 8 DOI reference(s) in README
✓
Academic publication links
Links to: zenodo.org
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (11.4%) to scientific vocabulary

Last synced: 6 months ago · JSON representation ·

Repository

A repository for testing and improving the GRoLTS checklist on open-source llms

Basic Info

Host: GitHub
Owner: timovdk
License: mit
Language: Jupyter Notebook
Default Branch: main
Size: 157 MB

Statistics

Stars: 0
Watchers: 1
Forks: 1
Open Issues: 0
Releases: 1

Created over 1 year ago · Last pushed 7 months ago

Metadata Files

Readme License Citation

README.md

GRoLTS-llm

This repository supports the testing and improvement of the GRoLTS checklist using open-source large language models (LLMs).

Purpose

The scripts provided here allow you to calculate GRoLTS scores — based on the checklist described in this publication — for the PTSS datasets:

This work is part of the FORAS project, which is pre-registered in PROSPERO under ID CRD42023494027.

Installation

Tested with Python 3.13 1. Install all packages in requirements.txt: pip install -r ./requirements.txt

Place your paper pdfs in the folder ./src/data
Copy the OpenAI API key in the corresponding fields in the files: ./src/embed.py and ./src/generate.py
Update the questions in p4 in the file ./src/grolts_questions.py
Move into the ./src folder: cd ./src
Run ./embed.py to create chunks of the papers and embed these chunks and questions
After ./embed.py completes, run ./generate.py. This will take a while!
After this completes, ensure to update ./human_labels.csv with the ground truth labels (Pay extra attention to the question ids, they should match with the ids in p4 in ./grolts_questions.py. Also, make sure that the paper_ids correspond with the names of the pdfs in the ./data folder)
Run the first 4 cells in ./eval.ipynb and see the agreement proportion between the LLM and yourself!

Funding

This project is funded by the Dutch Research Council (NWO), grant number 406.22.GO.048.

Owner

Login: timovdk
Kind: user

Repositories: 7
Profile: https://github.com/timovdk

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "van der Kuil"
  given-names: "Timo"
  orcid: "https://orcid.org/0009-0004-6930-9169"
title: "GRoLTS LLM"
version: v1.0.0
doi: 10.5281/zenodo.15582826
date-released: 2025-06-03
url: "https://github.com/timovdk/grolts-llm"

GitHub Events

Total

Delete event: 6
Push event: 65
Pull request review event: 1
Pull request event: 11
Fork event: 1
Create event: 6

Last Year

Delete event: 6
Push event: 65
Pull request review event: 1
Pull request event: 11
Fork event: 1
Create event: 6

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 0
Total pull requests: 7
Average time to close issues: N/A
Average time to close pull requests: about 10 hours
Total issue authors: 0
Total pull request authors: 1
Average comments per issue: 0
Average comments per pull request: 0.0
Merged pull requests: 5
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 7
Average time to close issues: N/A
Average time to close pull requests: about 10 hours
Issue authors: 0
Pull request authors: 1
Average comments per issue: 0
Average comments per pull request: 0.0
Merged pull requests: 5
Bot issues: 0
Bot pull requests: 0

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science