modified-lam-experiments

Copyright © 2020 Vincent A. Cicirello

This repository contains code to reproduce the experiments, and analysis of experimental data, from the following paper:

Vincent A. Cicirello. 2020. Optimizing the Modified Lam Annealing Schedule. Industrial Networks and Intelligent Systems, 7(25), Article e1 (December 2020). https://doi.org/10.4108/eai.16-12-2020.167653.

| Related Publication: | | | :--- | :--- | | Source Info: | | | Packages and Releases: | |

Requirements to Build and Run the Experiments

To build and run the experiments on your own machine, you will need the following: * JDK 11: I used OpenJDK 11, but you should be fine with Oracle's JDK as well. Technically, there is nothing in the code that strictly requires Java 11, so you should be able to build and run with JDK 8 or later. However, the Maven pom.xml provided in the repository assumes Java 11. Also, if you want to recreate the experiments in as similar an environment as used in the reported results, then you should use Java 11. * Apache Maven: In the root of the repository, there is a pom.xml for building the Java programs for the experiments. Using this pom.xml, Maven will take care of downloading the exact version of the Chips-n-Salsa library that was used in the experiments (release 2.2.0), as well as Chips-n-Salsa's dependencies. Additionally, there is a file pom-get-from-central.xml, which is an alternate Maven pom file that I provided if you would prefer to download and use a prebuilt jar file of the experiments from Maven Central, rather than building locally on your own machine. * Python 3: The repository contains a Python program that was used to compute summary statistics, as well as statistical significance tests. If you want to run this Python program, you will need Python 3. I specifically used Python 3.8.2. You also need scipy installed to run this Python program. * Make: The repository also contains a Makefile to simplify running the build, running the experiment's Java programs, and running the Python program to analyze the data. If you are familiar with using the Maven build tool, and running Python programs, then you can just run these directly, although the Makefile may be useful to see the specific commands needed.

Building the Experiment's Java Programs

The source code of the Java programs, implementing the experiments is in the src/main directory. You can build the experiment programs in one of the following ways: * Execute mvn compile at the root of the repository. * Or, execute make or make build at the root of the repository (which simply executes a mvn compile).

This produces 5 Java programs for running different parts of the experiments. If you are unfamiliar with the usual structure of the directories of a Java project built with Maven, the .class files, etc will be found in a target directory that is created by the build process.

Downloading a Prebuilt Jar of the Experiment's Java Programs

For convenience, a prebuilt jar file of the experiment's Java programs is available in the Maven Central Repository. I have provided an alternate to the Maven pom.xml file, pom-get-from-central.xml, which can be used to download the prebuilt jar. Use it in one of the following ways: * Execute mvn -f pom-get-from-central.xml clean dependency:copy-dependencies at the root of the repository. * Or, execute make download at the root of the repository (which does the above).

This will download a jar file containing the compiled experiment programs, as well as the jar files of the appropriate versions of the Chips-n-Salsa library, and its dependencies, all of which will be in the target directory.

Running the Experiments

Once you have either successfully executed the build or downloaded the jars from Maven Central, you can then run the experiments. If you built the experiment programs locally on your machine, then run the experiments by executing make experiments at the root of the repository. If you instead downloaded the prebuilt jar, then run the experiments by executing make experimentsCentral. In either case, this will run each of the experiment programs in sequence, with the results piped to text files in the root of the project. The output from my runs are found in the /data directory.

The experiments produce 5 text files with data: * anneal.txt: This has data comparing the two versions of the annealing schedule independent from Simulated Annealing. * onemax.txt: This has results from a comparison on the OneMax problem. * boundmax.txt: This has results from a comparison on the BoundMax problem. * haystack.txt: This has results from a comparison on the Permutation in a Haystack problem. * roots.txt: This has results from a comparison on a Polynomial Root Finding problem.

You can find these 5 files from my runs of the experiments in the /data directory.

Analyzing the Experimental Data

To run the Python program that I used to generate summary statistics and run significance tests, you need Python 3 installed. The source code of this Python program is found in the src/analysis directory, and consists of a single Python file experimentstats.py. To run that Python program, execute make analysis in the root of the project. This assumes that you have run the experiments above, and that the output of the experiments is in the root of the project. If you want to simply run it against the result of my runs, then change the variable pathToDataFiles = "data/" in the Makefile. Executing the Makefile in this way will also take care of installing scipy if you don't already have it installed, and will pass the appropriate command line parameters to the Python program for each of the data files.

The output of this analysis program is piped to a file stats.txt in the root of the project. This file from my run of the experiments is found in the /data directory.

Other Files in the Repository

There are a few other files, potentially of interest, in the repository, which include: * system-stats.txt: This file contains details of the system I used to run the experiments, such as operating system, processor specs, Java JDK and VM. It is in the /data directory. * graphs.xlsx: An Excel spreadsheet used to generate the graphs of the paper. It is also in the /data directory. * /data/figures directory: This is probably of less interest to someone other than me. It contains the figures of the paper in eps format, along with some intermediate formats.