athena

by Ned Thaddeus Taylor

ATHENA (Adaptive Training for High Efficiency Neural network Applications) is a Fortran library for developing and handling neural networks (with a focus on convolutional neural networks).

New Repository Location

This repository has been migrated from the University of Exeter GitLab to GitHub to facilitate community interaction and support. The latest version, updates, and collaboration now take place on this GitHub repository.

GitLab Repository (Archived): https://git.exeter.ac.uk/hepplestone/athena

Why the Migration?

It was decided that this project should be migrated to allow for better community support (i.e. allowing community users to raise issues). All information has been ported over where possible. Issues have not been migrated over, these can be found in the old repository. Releases prior to 1.2.0 have not been migrated over, but they can still be found as tags in this repository.

Statement of need

The ATHENA library leverages Fortran's strong support of array arithmatics, and its compatibility with parallel and high-performance computing resources. Additionally, there exist many improvements made available since Fortran 95, specifically in Fortran 2018 (Reid 2018) (and upcoming ones in Fortran 2023), as well as continued development by the Fortran Standards committee. All of this provides a clear incentive to develop further libraries and frameworks focused on providing machine learning capabilities to the Fortran community.

While existing Fortran-based libraries, such as neural-fortran (Curcic 2019), address many aspects of neural networks, ATHENA provides implementation of some well-known features not currently available within other libraries; these features include batchnormalisation, regularisation layers (such as dropout and dropblock), and average pooling layers. Additionally, the library provides support for 1, 2, and 3D input data for most features currently implemented; this includes 1, 2, and 3D data for convolutional layers. Finally, the ATHENA library supports many convolutional techniques, including various data padding types, and stride.

One of the primary intended applications of ATHENA is in materials science, which heavily utilises convolutional and graph neural networks for learning based on charge densities and atomic structures. Given the unique data structure of atomic configurations, specifically their graph-based nature, a specialised API must be developed to accommodate these needs.

References

• Reid, J. (2018). The new features of fortran 2018. SIGPLAN Fortran Forum, 37(1), 5–43. https://doi.org/10.1145/3206214.3206215
• Curcic, M. (2019). A parallel fortran framework for neural networks and deep learning. SIGPLAN Fortran Forum, 38(1), 4–21. https://doi.org/10.1145/3323057.3323059

Documentation

ATHENA is distributed with the following directories:

| Directory | Description | |---|---| | example/ | A set of example programs utilising the ATHENA library | | src/ | Source code | | tools/ | Additional shell script tools for automating learning | | test/ | A set of test programs to check functionality of the library works after compilation |

For extended details on the functionality of this library, please check out the wiki

NOTE: There currently exists no manual document. This will be included at a later date

Setup

The ATHENA library can be obtained from the git repository. Use the following commands to get started: git clone https://github.com/nedtaylor/athena.git cd athena

Dependencies

The library has the following dependencies: - A Fortran compiler (compatible with Fortran 2018 or later) - fpm, CMake, or Spack for building the library

The library has been developed and tested using the following compilers: - gfortran -- gcc 14.1.0 - ifort -- Intel 2021.10.0.20230609 - ifx -- IntelLLVM 2023.2.0

Tested compilers

The library has also been tested with and found to support the following libraries: - gfortran -- gcc 12.3, gcc 13.2

Building with fpm

The library is set up to work with the Fortran Package Manager (fpm).

Run the following command in the repository main directory: fpm build --profile release

Testing with fpm

To check whether ATHENA has installed correctly and that the compilation works as expected, the following command can be run: fpm test --profile release

This runs a set of test programs (found within the test/ directory) to ensure the expected output occurs when layers and networks are set up.

Building with cmake

Run the following commands in the directory containing CMakeLists.txt: mkdir build cd build cmake [-DCMAKE_BUILD_TYPE="optim;mp"] .. make install This will build the library in the build/ directory. All library files will then be found in: ${HOME}/.local/athena Inside this directory, the following files will be generated: include/athena.mod lib/libathena.a

Testing with cmake

To check whether ATHENA has installed correctly and that the compilation works as expected, the following command can be run: ctest

This runs a set of test programs (found within the test/ directory) to ensure the expected output occurs when layers and networks are set up.

Building with Spack

The library can also be installed using the Spack package manager. This can be achieved by running the following commands in the main directory: spack repo add .spack spack install athena

Currently, Spack compilation requires manual download of ATHENA. NOTE: There already exists an athena package directly on Spack, be aware that these are not related.

Examples

After the library has been installed, a set of example programs can be compiled and run to test the capabilities of ATHENA on the MNIST dataset. Some of the examples can be run as-is, and do not require external databases. For those that require the MNIST (a set of 60,000 hand-written numbers for training and 10,000 for testing, 0-9) dataset (i.e. 'example/mnist_' directories ), the dataset must first be downloaded. The example program has been developed to accept a text-based format of the MNIST dataset. The .txt database that these examples have been developed for can be found here: https://github.com/halimb/MNIST-txt/tree/master

The link to the original MNIST database is: http://yann.lecun.com/exdb/mnist/

NOTE: For the mnist examples, the MNIST dataset must be downloaded. By default, the database is expected to be found in the directory path ../../DMNIST. However, this can be chaned by editing the following line in the example/mnist[_VAR]/test_job.in file to point to the desired path:

dataset_dir = "../../DMNIST"

Running examples using fpm

Using fpm, the examples are built alongside the library. To list all available examples, use: fpm run --example --list

To run a particular example, execute the following command:

fpm run --example [NAME] --profile release

where [NAME] is the name of the example found in the list.

Running examples manually

To compile and run the examples, run the following commands in the directory containing CMakeLists.txt: cd example/mnist make build optim [FC=FORTRAN-COMPILER] ./bin/athena_test -f test_job.in After the example program is compiled, the following directories will also exist:

| Directory | Description | |---|---| | example/mnist/bin/ | Contains binary executable | | example/mnist/obj/ | Contains module/object files (non-linked binary files)|

The example will perform a train over the MNIST dataset. Once complete, it will print its weights and biases to file, and test the trained network on the training set. The output from this can then be compared to the file expectedoutputCOMPILER.txt.

In the tools/ directory, there exist scripts that take utilise the wandb python package (Weights and Biases, a machine learning data tracker). Wandb is a Python module and, as such, a Python interface has been provided to call and run the Fortran example. The Python interface then reads the Fortran output files and logs the results to the wandb project.

Example wandb project link: https://wandb.ai/ntaylor/cnnmnisttest/overview?workspace=user-ntaylor

How-to

To call/reference the ATHENA library in a program, include the following use statement at the beginning of the necessary Fortran file: use athena

During compilation, include the following flags in the compilation (gfortran) command: -I${HOME}/.local/athena/include -L${HOME}/.local/athena/lib -lathena

Developers

• Ned Thaddeus Taylor

Contributing

Please note that this project adheres to the Contributing Guide. If you want to contribute to this project, please first read through the guide. If you have any questions, please either discuss then in issues, or contact Ned Taylor.

License

This work is licensed under an MIT license.

Code Coverage

Automated reporting on unit test code coverage in the README is achieved through utilising the cmake-modules and dynamic-badges-action projects.

Files

|Source file | Description| |-----------|------------| |src/athena.f90 | the module file that imports all necessary user-accessible procedures | |src/lib/modaccuracy.f90_ | accuracy calculation procedures | |src/lib/modactivation.f90_ | generic node activation (transfer) setup | |src/lib/modactivation[NAME].f90 | [NAME] activation method | |src/lib/modbaselayer.f90 | abstract layer construct type | |src/lib/modbaselayersub.f90_ | base layer submodule | |src/lib/modclipper.f90_ | gradient clipping procedures | |src/lib/modconstants.f90_ | a set of global constants used in this code | |src/lib/modcontainer.f90_ | layer container construct for handling multiple layers in a network | |src/lib/modcontainersub.f90 | layer container submodule | |_src/lib/mod[NAME]layer.f90_ | [NAME] layer-type | |src/lib/modinitialiser.f90_ | generic kernel (and bias) initialiser setup | |src/lib/modinitialiser[NAME].f90 | [NAME] kernel initialisation method | |src/lib/modloss.f90_ | loss and corresponding derivatives calculation procedures | |src/lib/modlrdecay.f90 | learning rate decay procedures | |src/lib/modmetrics.f90_ | training convergence metric derived type and procedures | |src/lib/modmisc.f90_ | miscellaneous procedures | |src/lib/modmiscml.f90 | miscellaneous machine learning procedures | |srcs/lib/modnetwork.f90_ | neural network derived type and procedures | |srcs/lib/modnetworksub.f90 | network submodule | |src/lib/modnormalisation.f90_ | data normalisation procedures | |src/lib/modoptimiser.f90_ | learning optimisation derived type and procedures | |src/lib/modrandom.f90_ | random number procedures | |src/lib/modtoolsinfile.f90 | tools to read input files | |src/lib/modtypes.f90_ | neural network-associated derived types |

| Additional file | Description | |-----|------| |CHANGELOG | human-readable athena codebase version history | |CMakeLists.txt | the makefile used for compiling the library | |CONTRIBUTING.md | Guidelines for organisation of athena codebase | |fpm.toml | Fortran Package Manager (fpm) compilation file | |LICENSE | licence of ATHENA code | |README.md | a readme file with a brief description of the code and files | |TODO | todo-list in addition to useful machine learning and fortran references | |cmake/CodeCoverage.cmake | cmake-modules file to automate unit test coverage reporting| |example/examplelibrary_ | Utility library shared between the examples | |example/[NAME]/expectedoutput.txt_ | expected output from executing [NAME] example program | |example/[NAME]/testjob.in_ | input file for [NAME] example program | |example/[NAME]/src | source directory for [NAME] example program | |test/test[NAME].f90 | [NAME] test program to check library expected functionality | |tools/coveragebadge.py_ | script to extract code coverage percentage from GitHub Action | |tools/sweepinit.py_ | script to initialise wandb sweep | |tools/sweeptrain.py_ | script to perform training and log learning to wandb | |tools/template.in | input file for program in test/bin/ (once compiled) | |tools/wandb-metadata.json | metadata defining default plots on wandb website |