SDFeedback

• 1. 📖 Overview
  • 1.1 MigrationBench: Datasets and Evaluation Framework
  • 1.2 SDFeedback: Migration with LLMs
• 2. 🤗 MigrationBench Datasets
• 3. Code Migration with LLMs
  • 3.1 Single Job
  • 3.1.1 Basic Setup
  • 3.1.2 Install MigrationBench and SDFeedback
  • 3.1.3 Local Run
  • 3.2 Batch Job
  • 3.2.1 ~~Local Run~~
  • 3.2.2 EMRS Run
• 4. 📚 Citation

1. 📖 Overview

SDFeedback is a library to conduct code migration with LLMs, and improves efficacy by providing feedback to LLMs as specific as possible, motivated by Teaching Large Language Models to Self-Debug.

• Reference paper: MigrationBench: Repository-Level Code Migration Benchmark from Java 8

1.1 MigrationBench: Datasets and Evaluation Framework

1. 🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages.
   • Current and initial release includes java 8 repositories with the maven build system, as of May 2025.
   • See more details in 2. 🤗 MigrationBench Datasets
2. MigrationBench is the evaluation framework to assess code migration success, from java 8 to 17 or any other long-term support versions.

1.2 SDFeedback: Migration with LLMs

SDFeedback (current package) is to conduct code migration with LLMs as a baseline solution, and it relies on the MigrationBench package for the final evaluation. - It builds an ECR image and then - It runs both code migration and final evaluation with AWS Elastic Map Reduce Serverless (EMRS) in a scalable way.

2. 🤗 MigrationBench Datasets

There are three datasets in 🤗 MigrationBench: - All repositories included in the datasets are available on GitHub, under the MIT or Apache-2.0 license.

| Index | Dataset | Size | Notes | |-------|-----------------------------------------------|-------|-----------------------------------------------------------------------------------------------------| | 1 | 🤗 AmazonScience/migration-bench-java-full | 5,102 | Each repo has a test directory or at least one test case | | 2 | 🤗 AmazonScience/migration-bench-java-selected | 300 | A subset of 🤗 migration-bench-java-full | | 3 | 🤗 AmazonScience/migration-bench-java-utg | 4,814 | The unit test generation (utg) dataset, disjoint with 🤗 migration-bench-java-full|

3. Code Migration with LLMs

We support running code migration for MigrationBench in two modes: 1. Single job mode: For a single repository and 2. Batch job mode: For multiple repositories with EMRS - TL;DR: To run batch mode, one can skip to 3.2.2 EMRS Run directly.

3.1 Single Job

To get started with code migration with LLMs from java 8 to 17, under either minimal migration or maximal migration (See the arXiv paper for the definition):

3.1.1 Basic Setup

Verify you have java 17, maven 3.9.6 and conda (optional) locally:

```

java

~ $ java --version openjdk 17.0.15 2025-04-15 LTS OpenJDK Runtime Environment Corretto-17.0.15.6.1 (build 17.0.15+6-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.15.6.1 (build 17.0.15+6-LTS, mixed mode, sharing) ```

```

maven

~ $ mvn --version Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae) Maven home: /usr/local/bin/apache-maven-3.9.6 Java version: 17.0.15, vendor: Amazon.com Inc., runtime: /usr/lib/jvm/java-17-amazon-corretto.x8664 Default locale: enUS, platform encoding: UTF-8 OS name: "linux", version: "5.10.236-208.928.amzn2int.x86_64", arch: "amd64", family: "unix" ```

```

conda (Optional)

$ conda --version conda 25.1.1 ```

3.1.2 Install MigrationBench and SDFeedback

``` cd ~ git clone https://github.com/amazon-science/MigrationBench.git git clone https://github.com/amazon-science/SDFeedback.git

They're optional if one doesn't need a conda env

export CONDA_ENV=sd-feedback

conda create -n $CONDA_ENV python=3.9

conda activate $CONDA_ENV

cd ~/MigrationBench pip install -r requirements.txt -e .

cd ~/SDFeedback pip install -r requirements.txt -e . ```

3.1.3 Local Run

To run code migration for a single repository:

``` cd ~/SDFeedback/src/self_debug

Explicit max_iteration will override it in the config_file

python runselfdebugging.py --configfile configs/javaconfig.pbtxt # --max_iterations 3 ```

3.2 Batch Job

To run code migration in batch mode for multiple repositories, one can run it ~~either locally or~~ through EMRs.

3.2.1 ~~Local Run~~

TL;DR: Local run for batch job is typically for debugging and integration test purposes, and it's NOT recommended.

See relevant spark scripts for reference: - src/self_debug/batch/spark_build.py - src/self_debug/batch/spark_debug.py

3.2.2 EMRS Run

Before submitting a job to EMRS, make sure you have the following ready: - Set up IAM roles, network, security groups, etc correctly - Set up ECR repository - Set up SES (optional)

1. Build an ECR image

``` cd ~/SDFeedback/src/self_debug/container

To build ECR image: 552793110740.dkr.ecr.us-east-1.amazonaws.com/$USER:java

./image.sh java $USER 1 docker/java.Dockerfile # 999999999999.dkr.ecr.us-west-2.amazonaws.com ```

1. Submit a spark job to EMRS

Note that security keys might be subject to 12h timeout.

``` cd ~/SDFeedback/src/self_debug/batch

Update config file as needed for emrs.py, e.g. use the right ECR image in step #1

CONFIG=... export APPLICATION=emrs-dbg-{user}--{date}--run00 export SCRIPT=debugger

python emrs.py --configfile=$CONFIG --application=$APPLICATION --script=$SCRIPT --user=$USER # --dryrun=1 ```

4. 📚 Citation

bibtex @misc{liu2025migrationbenchrepositorylevelcodemigration, title={MIGRATION-BENCH: Repository-Level Code Migration Benchmark from Java 8}, author={Linbo Liu and Xinle Liu and Qiang Zhou and Lin Chen and Yihan Liu and Hoan Nguyen and Behrooz Omidvar-Tehrani and Xi Shen and Jun Huan and Omer Tripp and Anoop Deoras}, year={2025}, eprint={2505.09569}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2505.09569}, }