Magic Castle

The Digital Research Alliance of Canada provides HPC infrastructure and support to every academic research institution in Canada. The Alliance uses CVMFS, a software distribution system developed at CERN, to make its research software stack available on its HPC clusters, and anywhere else with internet access. This enables replication of the user experience outside of The Alliance physical infrastructure.

From these new possibilities emerged an open-source software project named Magic Castle, which aims to recreate the HPC user experience in public clouds. Magic Castle uses the open-source software Terraform and HashiCorp Language (HCL) to define the virtual machines, volumes, and networks that are required to replicate a virtual HPC infrastructure. The infrastructure definition is packaged as a Terraform module that users can customize as they require. After deployment, the user is provided with a complete HPC cluster software environment including a Slurm scheduler, a Globus Endpoint, JupyterHub, LDAP, DNS, and over 3000 research software applications compiled by experts with EasyBuild. Magic Castle is compatible with AWS, Microsoft Azure, Google Cloud, OpenStack, and OVH.

Setup

• Install Terraform (>= 1.5.7)
• Download the latest release of Magic Castle for the cloud provider you wish to use.
• Uncompress the release
• Follow the instructions
  • OpenStack Cloud
  • Amazon Web Services (AWS)
  • Microsoft Azure
  • Google Cloud Platform (GCP)
  • OVH Public Cloud (OVH)
  • Incus
• For more details, refer to Magic Castle Documentation

How Magic Castle Works

This software project integrates multiple parts that come into play at different steps of spawning the cluster. The following list enumerates the steps involved in order for users to better grasp what is happening when they create clusters.

We will refer to the user of Magic Castle as the operator.

1. After downloading the latest release of the cloud provider of choice and adapting the main configuration file, the operator launches Terraform and accepts the proposed plan.
2. Terraform communicates with the cloud provider REST API and requests the creation of the virtual machines.
3. For each virtual machine creation request, Magic Castle provides a cloud-init file. This file is used to initialize the virtual machine base configuration and installs puppet agent. The cloud-init file of the puppet tagged virtual machine installs and configures a Puppet primary server.
4. Terraform uploads on the Puppet primary server instance a YAML file containing information about the roles of each instances specified as tags.
5. The Puppet agents communicate with the Puppet primary server to retrieve and apply their configuration based on the tags defined in the preceding YAML file.

Talks, slides and videos

• FOSDEM 2020 - Magic Castle: Terraforming the Cloud for HPC (2020-02-02)
• EasyBuild User Meeting 2021 - Magic Castle: Terraforming the Cloud for HPC (2021-01-27)
• Campus Research Computing Consortium - Clusters in the Sky: How Canada is Building Beyond Iaas Cloud with Magic Castle (2021-05-20)
• EESSI User Meeting 2022 - Magic Castle: Preview of MC 12 (2022-09-14)
• HashiTalks Québec 2022 - Magic Castle: Le CIP à l'échelle grâce à Terraform Cloud (2022-09-29)
• SIGHPC Education - Magic Castle: Terraforming the cloud to teach HPC (2024-02-08)

List of other cloud HPC cluster open-source projects

• AWS ParallelCluster [AWS]
• Cluster in the cloud [AWS, GCP, Oracle]
• Elasticluster [AWS, GCP, OpenStack]
• Google Cluster Toolkit [GCP]
• illume-v2 [OpenStack]
• NVIDIA DeepOps [Ansible playbooks only]
• StackHPC Ansible Role OpenHPC [Ansible Role for OpenStack]

When I think about the DevOps landscape, we have so many people just like chefs in a restaurant that are experimenting with different ways of doing things. Once they get it, then they create those recipes. Those recipes in our world is source code. [...] That's why we will always have duplicates and similar projects, because there's going to be one ingredient that's going to be slightly different to make you preferred over something else

Kelsey Hightower, Sourcegraph Podcast, Episode 16, 2020

Contributing / Customizing

Refer to the reference design and the developer documentation.