https://github.com/cosmaadrian/rocode
Official repository for "RoCode: A Dataset for Measuring Code Intelligence from Romanian Problem Definitions"
Science Score: 10.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org, scholar.google -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.4%) to scientific vocabulary
Keywords
Repository
Official repository for "RoCode: A Dataset for Measuring Code Intelligence from Romanian Problem Definitions"
Basic Info
- Host: GitHub
- Owner: cosmaadrian
- License: other
- Default Branch: master
- Homepage: https://arxiv.org/abs/2402.13222
- Size: 25.4 KB
Statistics
- Stars: 6
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in 🇷🇴 Romanian 🇷🇴
Accepted at the 2024 International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).
In this work, we present RoCode, a competitive programming dataset, consisting of 2,642 problems written in Romanian, 11k solutions in C, C++ and Python and comprehensive testing suites for each problem.
TL;DR
RoCode is a dataset of programming puzzles with problem statements in Romanian for evaluation of LLM code intelligence. Leetcode, but for Romanian.
📘 Abstract
Recently, large language models (LLMs) have become increasingly powerful and have become capable of solving a plethora of tasks through proper instructions in natural language. However, the vast majority of testing suites assume that the instructions are written in English, the de facto prompting language. Code intelligence and problem solving still remain a difficult task, even for the most advanced LLMs. Currently, there are no datasets to measure the generalization power for code-generation models in a language other than English. In this work, we present RoCode, a competitive programming dataset, consisting of 2,642 problems written in Romanian, 11k solutions in C, C++ and Python and comprehensive testing suites for each problem. The purpose of RoCode is to provide a benchmark for evaluating the code intelligence of language models trained on Romanian / multilingual text as well as a fine-tuning set for pretrained Romanian models. Through our results and review of related works, we argue for the need to develop code models for languages other than English.
📖 Citation
If you found our work useful, please cite our paper:
RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in 🇷🇴 Romanian 🇷🇴
@InProceedings{cosma24rocode,
author="Cosma, Adrian and Iordache, Ioan-Bogdan and Rosso, Paolo",
title="RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian",
booktitle="The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
year="2024",
}
📝 License
This work is protected by Attribution-NonCommercial 4.0 International
Owner
- Name: Adrian Cosma
- Login: cosmaadrian
- Kind: user
- Location: Bucharest, Romania
- Company: University Politehnica of Bucharest
- Repositories: 21
- Profile: https://github.com/cosmaadrian
Mercenary Researcher
GitHub Events
Total
- Watch event: 4
Last Year
- Watch event: 4