https://github.com/assert-kth/drr

Tool & data on the correctness of Defects4 patches generated by program repair tools http://arxiv.org/pdf/1909.13694

https://github.com/assert-kth/drr

Repository

Tool & data on the correctness of Defects4 patches generated by program repair tools http://arxiv.org/pdf/1909.13694

https://github.com/ASSERT-KTH/drr/blob/master/

# Automated Patch Assessment for Program Repair

A tool for automatic correctness assessment for patches generated by program repair systems. We consider the human patch as ground truth oracle and use Random tests based on the Ground Truth (RGT). See [Automated Patch Assessment for Program Repair at Scale](http://arxiv.org/pdf/1909.13694)

If you use this repo, please cite:

```
@Article{Ye2021EMSE,
    author = {Ye, He and Martinez, Matias and Monperrus, Martin},
    title = "Automated Patch Assessment for Program Repair at Scale",
    journal="Empirical Software Engineering",
    volume = "26",
    issn = "1573-7616",
    doi = "https://doi.org/10.1007/s10664-020-09920-w",
    year = "2021"
}

```



## Folder Structure
 ```bash
 Patches257 patches from Dcorrect and 381 patches from Doverfitting

 RGT: incl. tests from Evosuite2019, Randoop2019, EvosuitASE15, RandoopASE15 and EvosuiteEMSE18
 
 DiffTGen
    Results: the running result overfitting patches found by DiffTGen. 
  runDrr.py: a command to reproduce DiffTGen experiment(details see below)

 statistics: our exerimental statistics for all RQs

  run.py: a command to reproduce all experiments

```

## Prerequisites

* JDK 1.7
* OS: Linux and Mac
* Configure the DEFECTS4J_HOME="home_of_defects4j"
* Add submodule defects4j and checkout the commit 486e2b4(Please note our experiment depends on several Defects4J commands)

```
git submodule add https://github.com/rjust/defects4j
git reset --hard 486e2b49d806cdd3288a64ee3c10b3a25632e991
```

## Run

To assess an indiviual patch for Defects4J:
```
./run.py patch_assessment     
example:  ./run.py patch_assessment patch1-Lang-35-ACS.patch Dcorrect 2019_Evosuite
```

To perform different sanity checks:

```
./run.py applicable_check
./run.py plausible_check
```

To identify flaky tests:
```
./run.py flaky_check     
example:  ./run.py flaky_check patch1-Lang-35-ACS.patch Dcorrect 2019_Evosuite
```

To reproduce our Expriments with RGT patch assessment

```
RQ1: ./run.py RQ1
RQ3: ./run.py RQ3
RQ4: ./run.py RQ4
RQ5: cd ./statistics   ./RQ5-randomness-script.py  
```


## Results

* [patches_overview.csv](https://github.com/kth-tcs/defects4-repair-reloaded/blob/master/statistics/patches_overview.csv), an overview of patches. 
* The RGT tests generation log for [Evosuite2019](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RGT_Evosuite2019_Generation_Log.csv) and [Randoop2019](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RGT_Randoop2019_Generation_Log.csv). 
* Individually, Evosuite2019 and Randoop2019 fail to generate cases for 31/3510  and 1080/3510 executions. There are respective 2.2% and 2.4% falky tests are found, please see the log [Flaky_Check_For_Evosuite2019](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RGT_Evosuite2019_Flaky_Check.csv) and [Flaky_Check_For_Randoop2019](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RGT_Randoop2019_Flaky_Check.csv).
* We run Evosuite2019 and Randoop2019 over 257 patches from Dcorrect. The statistics for each test execution is available at [RQ1 and RQ2_Result](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RG1-2_Result.csv).
* The detailed execution logs of RGT compared to DiffTGen are available at [Evosuite2019_Execution_on_Doverfitting](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ3-Evosuite2019-Result.txt) and [Randoop2019_Execution_on_Doverfitting](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ3-Randoop2019-Result.txt)
* Overfitting patches found by Evosite2019 and Randoop2019 are individally summarized in the [statistics](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ3_Overfitting_Patches.csv).Together, they found 274 overfitting patches. 
* Overfitting patches found by DiffTGen is summarized in the [DiffTGen Result](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/DiffTGen/Result.csv).
* Overfitting patches found by RGT from previous research: EvosuiteASE15, RandoopASE15, EvosuiteEMSE18 are summarized [here](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ4_Patches_Found_by_Previous_RGT.csv).
* Experiment statistics of [RQ4_RGT_From_Previous_on_Dcorrect.csv](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ4_RGT_From_Previous_on_Dcorrect.csv): 9 misclassified patches found. 
* Experiment statistics of [RQ4_RGT_From_Previous_on_Doverfitting.csv](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ4_RGT_From_Previous_on_Doverfitting.csv): 219 misclassified patches found. 
* Individually, the failure number for 30 RGT tests of identifying overfitting patches are available [Evosuite2019](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ5-Evosuite_Raw_Statistics.txt) and [Randoop2019](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ5-Randoop_Raw_Statistics.csv).
* We develop a [script](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ5-randomness-script.py) to simulate 1000 random groups that each contain 30 runs test generation. 
* Statistics that record the average test generations that achieve effectiveness of 80%, 85%, 90%, 95% and 100%: [1000 groups of Evosuite](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ5-1000-groups-evosuite.csv) and [1000 groups of Randoop](https://github.com/kth-tcs/defects4j-repair-reloaded/blob/master/statistics/RQ5-1000-groups-randoop.csv).


## Credits

* For more details about Defects4J, see the original repository of the [Defects4J benchmark](https://github.com/rjust/defects4j).
* For more details about DiffTGen, see the original repository of the [DiffTGen](https://github.com/qixin5/DiffTGen).

Owner

GitHub Events

Committers

Last synced: over 1 year ago