Data

Tools

Indexes

Applications

Experiments

Open Source Science

pareval

A Parallel Code Evaluation Benchmark

https://github.com/parallelcodefoundry/pareval

Science Score: 62.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ✓
    Institutional organization owner
    Organization parallelcodefoundry has institutional domain (pssg.cs.umd.edu)
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (16.8%) to scientific vocabulary
Last synced: 6 months ago · JSON representation ·

Repository

A Parallel Code Evaluation Benchmark

Basic Info
  • Host: GitHub
  • Owner: parallelcodefoundry
  • License: mit
  • Language: C++
  • Default Branch: develop
  • Homepage:
  • Size: 8.2 MB
Statistics
  • Stars: 34
  • Watchers: 3
  • Forks: 10
  • Open Issues: 2
  • Releases: 2
Created over 2 years ago · Last pushed 9 months ago
Metadata Files
Readme License Citation

README.md

ParEval

HPDC 2024 arXiv GitHub license

This repo contains the Parallel Code Evaluation (ParEval) Benchmark for evaluating the ability of Large Language Models to write parallel code. See the ParEval Leaderboard for up-to-date results on different LLMs.

Overview

The organization of the repo is as follows.

  • prompts/ -- the prompts in ParEval alongside some utility scripts
  • generate/ -- scripts for generating LLM outputs
  • drivers/ -- scripts to evaluate LLM outputs
  • analysis/ -- scripts to analyze driver results and compute metrics
  • tpl/ -- git submodule dependencies

Each subdirectory has further documentation on its contents. The general workflow is to use generate/generate.py to generate LLM outputs, run drivers/run-all.py to evaluate outputs, and analysis/metrics.py to post-process the results.

Setup and Installation

A couple core systems software are assumed to be installed: Python >=3.7, a C++ compiler that supports C++17 and OpenMP, Make, CMake, and an MPI implementation. If you are testing the CUDA and HIP prompts, then you will need access to NVIDIA and AMD GPUs alongside their respective software stacks.

First, clone the repo.

sh git clone --recurse-submodules https://github.com/parallelcodefoundry/ParEval.git

Next, you need to build Kokkos (if you want to include it in testing).

```sh cd tpl/kokkos

mkdir build cd build

depending on your system you may need to pass your c++ compiler to CMAKECXXCOMPILER

cmake .. -DCMAKEINSTALLPREFIX=. -DKokkosENABLETHREADS=ON make install -j4 ```

You will need to build the main C++ drivers before running ParEval. The included makefile will skip CUDA, HIP, and/or Kokkos if their respective libraries cannot be found.

```sh

from the repository root, step into the cpp drivers directory and run make

cd drivers/cpp make ```

Finally, you need to install the Python dependencies. requirements.txt has the set of dependencies pinned at the version they were tested with. Other versions may also work. Note that some of these are only required for parts of the pipeline i.e. PyTorch and Transformers are only needed for generating LLM outputs.

sh pip install -r requirements.txt

Citing ParEval

@misc{nichols2024large, title={Can Large Language Models Write Parallel Code?}, author={Daniel Nichols and Joshua H. Davis and Zhaojun Xie and Arjun Rajaram and Abhinav Bhatele}, year={2024}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, booktitle = {Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing}, series = {HPDC '24} }

License

ParEval is distributed under the terms of the MIT license.

Owner

  • Name: Parallel Code Foundry
  • Login: parallelcodefoundry
  • Kind: organization
  • Location: United States of America

Citation (CITATION.cff) 

cff-version: 1.2.0
title: "Can Large Language Models Write Parallel Code?"
message: "If you use this library and love it, cite the software and the paper \U0001F917"
authors:
  - given-names: Daniel
    family-names: Nichols
    email: dnicho@umd.edu
    affiliation: University of Maryland, College Park
  - given-names: Josh
    family-names: Davis
    email: jhdavis@umd.edu
    affiliation: University of Maryland, College Park
  - given-names: Zhaojun
    family-names: Xie
    email: zxie12@umd.edu
    affiliation: University of Maryland, College Park
  - given-names: Arjun
    family-names: Rajaram
    email: arajara1@umd.edu
    affiliation: University of Maryland, College Park
  - given-names: Abhinav
    family-names: Bhatele
    email: bhatele@cs.umd.edu
    affiliation: University of Maryland, College Park
version: 1.0.0
doi: https://doi.org/10.48550/arXiv.2401.12554
date-released: 2024-01-23
references:
  - type: article
    authors:
      - given-names: Daniel
        family-names: Nichols
        email: dnicho@umd.edu
        affiliation: University of Maryland, College Park
      - given-names: Josh
        family-names: Davis
        email: jhdavis@umd.edu
        affiliation: University of Maryland, College Park
      - given-names: Zhaojun
        family-names: Xie
        email: zxie12@umd.edu
        affiliation: University of Maryland, College Park
      - given-names: Arjun
        family-names: Rajaram
        email: arajara1@umd.edu
        affiliation: University of Maryland, College Park
      - given-names: Abhinav
        family-names: Bhatele
        email: bhatele@cs.umd.edu
        affiliation: University of Maryland, College Park
    title: "Can Large Language Models Write Parallel Code?"
    year: 2024
    journal: ArXiv
    doi: https://doi.org/10.48550/arXiv.2401.12554
    url: https://arxiv.org/abs/2401.12554

abstract: >-
  Large Language Models are becoming an increasingly popular tool for software
  development. Their ability to model and generate source code has been
  demonstrated in a variety of contexts, including code completion,
  summarization, translation, and lookup. However, they often struggle to
  generate code for more complex tasks. In this paper, we explore the ability of
  state-of-the-art language models to generate parallel code. We propose a
  benchmark, PCGBench, consisting of a set of 420 tasks for evaluating the
  ability of language models to generate parallel code, and we evaluate the
  performance of several state-of-the-art open- and closed-source language
  models on these tasks. We introduce novel metrics for comparing parallel code
  generation performance and use them to explore how well each LLM performs on
  various parallel programming models and computational problem types.
keywords:
  - Large Language Models
  - High Performance Computing
  - Parallel Computing
license: Apache-2.0

GitHub Events

Total
  • Create event: 15
  • Release event: 3
  • Issues event: 16
  • Watch event: 22
  • Delete event: 12
  • Issue comment event: 6
  • Push event: 24
  • Pull request review comment event: 6
  • Pull request review event: 17
  • Pull request event: 29
  • Fork event: 6
Last Year
  • Create event: 15
  • Release event: 3
  • Issues event: 16
  • Watch event: 22
  • Delete event: 12
  • Issue comment event: 6
  • Push event: 24
  • Pull request review comment event: 6
  • Pull request review event: 17
  • Pull request event: 29
  • Fork event: 6

Dependencies

requirements.txt pypi
  • datasets ==2.16.1
  • google-generativeai ==0.1.0rc1
  • huggingface-hub ==0.20.2
  • multiprocess ==0.70.15
  • numpy ==1.24.4
  • pandas ==2.0.3
  • torch ==2.1.2
  • torchaudio ==2.1.2
  • torchvision ==0.16.2
  • tqdm ==4.66.1
  • transformers ==4.36.2