Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods.

This is a work by Luisa Werner, Nabil Layada, Pierre Genevs, Jrme Euzenat, Damien Graux.

This repository contains our re-implementation of the experiments conducted with Knowledge Enhanced Neural Networks (KENN) on the Citeseer Dataset, including the re-implementation of the Experiments in PyTorch and PyTorch Geometric. We also extended the experiments to the datasets Cora and PubMed.

| Name | Description | #nodes | #edges | #features | #Classes | Task | |----------------|----------------------------------|-----------|-------------|-----------|----------|----------------------| | CiteSeer | from Planetoid, Citation Network | 3,327 | 9,104 | 3,703 | 6 | Node classification | | Cora | from Planetoid, Citation Network | 2,708 | 10,556 | 1,433 | 7 | Node Classification | | PubMed | from Planetoid, Citation Network | 19,717 | 88,648 | 500 | 3 | Node Classification |

Overview of this repository

This repository contains to sub-directories that refer to the experiments conducted with the initial implementation and the re-implementation.
1. The initialimplementation contains code from initial experiments here, extended to Cora and PubMed 2. The reimplementation of the experiments.

The results of both approaches are stored in the respective /results subdirectory.

The implementation of KENN layers in PyTorch is used from here.

Before running the experiments in both implementations

1. In order to make sure that the right environment is used, the necessary Python packages and their versions are specified in requirements.txt. We use Python 3.9. To install them go in the project directory and create a conda environment with the following packages. pip install -r requirements.txt The full list of packages in our environment including the dependencies is specified in system_packages.txt.

2. While the Citeseer dataset used by the initial implementation is already stored in the directory CiteSeer_reproduce, the datasets Cora and PubMed have to be loaded from PyTorch Geometric [Source] and preprocessed. Run the following command from the project directory to get PubMed and Cora. python create_datasets.py

To run the initial experiments.

1. To run the initial experiments with the specified parameters in the paper on all three datasets, run the following command from the project directory: cd initial_implementation python test.py

2. To get an overview of the results of the initial implementation. It is important that all experiments are finished before. cd initial_implementation python inspect_results.py

To run the replicated experiment.

We use Weights and Biases (WandB) as experiment tracking tool. The experiments can be run WITHOUT or WITH the use of WandB. 1. To run the experiments without WandB, run the following command.

cd re-implementation python train_and_evaluate.py conf.json

(By default, "wandb_use" : false is set in re-implementation/conf.json)

1. If you want to use weights and biases specify the following parameters in re-implementation/conf.json. "wandb_use" : true, "wandb_label": "<your label>", "wandb_project" : "<your project>", "wandb_entity": "<your entity>"

Then use the following command to run the experiments: cd re-implementation python run_experiments.py conf.json

Interprete the results To get an overview of the results of the re-implementation, run the following command cd re_implementation python inspect_results.py

Comparison between both implementations

To compare the results of both approaches (comparison type 2 reproducibility), go to the project folder and run python compare_results.py

Hyperparameters

In the file re-implementation/conf.json, the hyperparameters and settings of the runs are configured and saved. By default, the conf.json file contains parameters mentioned in the paper. The last column indicates whether the parameter can be modified in this implementation and to which values it should be set.

| Parameter | description | default | state | |----------------------|---------------------------------------------------------------------|----------------------------|-----------------------------------------------------------------------------------------------------| | adambeta1 | Adam optimizer parameter | 0.9 | modifiable | | adambeta2 | Adam optimizer parameter | 0.999 | modifiable | | adameps | Adam optimizer parameter | 1e-07 | modifiable | | bias init | Initialization of bias in NN | "zeroes" | | | binarypreactivation | constant value for activation of binary predicate | 500.0 | modifiable | | clauseweight | initialization of clause weight | 0.5 | modifiable | | dataset | dataset | CiteSeerreproduce | modifiable: [Corareproduce, PubMedreproduce, CiteSeerreproduce] | | device | gpu device for gpu computing | 1 | modifiable | | dropout | dropout rate | 0.0 | modifiable (0.0, 1.0) | | epochs | number of epochs in training | 300 | modifiable (1, ...) | | esenabled | early stopping activated flag | false | modifiable [true, false] | | esmindelta | early stopping delta threshold | 0.001 | modifiable (0, ...) | | espatience | early stopping patience | 10 | modifiable (0, ...) | | evalsteps | prints each n steps during training | 10 | modifiable (1, ...) | | hiddenchannels | hidden layer dimension | 50 | modifiable (1, ...) | | loss function | loss function | categorical cross-entropy | | | lr | learning rate | 0.001 | modifiable (0.0, 1.0) | | minweight | clause weight clipping minimum value | 0.0 | modifiable (..., maxweight) | | maxweight | clause weight clipping maximum value | 500.0 | modifiable (minweight, ...) | | mode | training mode | "transductive" | | | model | standard or KENNStandard (for base NN Standard) | Standard | modifiable: [Standard, KENNStandard] | | numkennlayers | number of KENN layers | 3 | modifiable (0, ...) | | numlayers | number of layers of base NN | 3 | modifiable (1, ...) | | optimizer | optimizer | adam | | | runs | number of runs | 30 | modifiable (1, ...) | | seed | random seed | 0 | modifiable (0,...) | | trainingdimension | training dimension | 0.1 | modifiable: [0.1, 0.25, 0.5, 0.75, 0.9] | | validdim | validation set dimension | 0.2 | modifiable in: (0.0, 1.0) | | wandbuse | if weights and biases should be used | false | modifiable: [true, false] | | wandblabel | label for weights and biases | "None" | modifiable depending on custom WandB settings | | wandbproject | project name for weights and biases | "None" | modifiable depending on custom WandB settings | | wandbentity | entity name for weights and biases | "None" | modifiable depending on custom WandB settings | | weight init | initialization of weights in NN | xavier uniform | |

How to Cite

This work can be cited as follows: @article{Werner_Layada_Genevs_Euzenat_Graux_2024, title={Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/29515}, DOI={10.1609/aaai.v38i14.29515}, abstractNote={Reproducibility is a desirable property of scientific research. On the one hand, it increases confidence in results. On the other hand, reproducible results can be extended on a solid basis. In rapidly developing fields such as machine learning, the latter is particularly important to ensure the reliability of research. In this paper, we present a systematic approach to reproducing (using the available implementation), replicating (using an alternative implementation) and reevaluating (using different datasets) state-of-the-art experiments. This approach enables the early detection and correction of deficiencies and thus the development of more robust and transparent machine learning methods. We detail the independent reproduction, replication, and reevaluation of the initially published experiments with a method that we want to extend. For each step, we identify issues and draw lessons learned. We further discuss solutions that have proven effective in overcoming the encountered problems. This work can serve as a guide for further reproducibility studies and generally improve reproducibility in machine learning.}, number={14}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Werner, Luisa and Layada, Nabil and Genevs, Pierre and Euzenat, Jrme and Graux, Damien}, year={2024}, month={Mar.}, pages={15850-15858} }

The works of KENN can be cited as follows: ```

Knowledge Enhanced Neural Networks for Relational Domains

@InProceedings{10.1007/978-3-031-27181-6_7, author="Daniele, Alessandro and Serafini, Luciano", editor="Dovier, Agostino and Montanari, Angelo and Orlandini, Andrea", title="Knowledge Enhanced Neural Networks for Relational Domains", booktitle="AIxIA 2022 -- Advances in Artificial Intelligence", year="2023", publisher="Springer International Publishing", address="Cham", pages="91--109", isbn="978-3-031-27181-6" }

Knowledge Enhanced Neural Networks

@InProceedings{10.1007/978-3-030-29908-8_43, author="Daniele, Alessandro and Serafini, Luciano", editor="Nayak, Abhaya C. and Sharma, Alok", title="Knowledge Enhanced Neural Networks", booktitle="PRICAI 2019: Trends in Artificial Intelligence", year="2019", publisher="Springer International Publishing", address="Cham", pages="542--554", isbn="978-3-030-29908-8" }

```