README: Slurm HPC Cluster with LXD Instances Deployment Guide

This document provides step-by-step instructions for setting up a Slurm High-Performance Computing (HPC) cluster where compute nodes are virtualized using LXD containers running on existing "metal" physical nodes. The create_lxd_nodes.sh script automates much of the deployment, leveraging a shared configuration directory (/share/lxd-hpc) hosted on your Slurm controller node, pluto.

1. Overview

This setup transforms your physical node[X] machines into hosts for multiple LXD containers, each acting as a Slurm compute node.

Metal Nodes: Your existing node[0-11,13,15-18,20-25,28-30,33] machines.
LXD Instances: Each metal node will host 3 LXD instances, named metalNodeName-N (e.g., node0-0, node0-1, node0-2).
Resource Allocation: Each LXD instance will be configured with 4 CPUs and 12GB of RAM.
Networking: LXD instances will be directly visible on your main HPC network via macvlan bridging to a br0 interface on the metal host. They will receive static IP addresses.
Centralized Configuration: Slurm's slurm.conf and munge.key will be distributed from /share/lxd-hpc on pluto.

2. Prerequisites (Perform these steps on `pluto` and all metal nodes BEFORE deployment)

On your Slurm Controller (`pluto`):

Shared Configuration Directory Setup:
- Create the shared directory:
```
sudo mkdir -p /share/lxd-hpc
sudo chown -R <your_admin_user>:<your_admin_group> /share/lxd-hpc # Or appropriate permissions
```
- NFS Export (example /etc/exports entry on pluto):
```
/share/lxd-hpc *(rw,sync,no_subtree_check,no_root_squash)
```
  Then, export the filesystem: sudo exportfs -a
- Ensure nfs-kernel-server is installed and running on pluto.

Munge Key and Slurm Configuration in Shared Folder:

Copy munge.key from pluto to the shared directory:

sudo cp /etc/munge/munge.key /share/lxd-hpc/munge.key

Place the slurm.conf file (the latest version provided, adjusted for LXD nodes) into the shared directory:

sudo cp /etc/slurm-llnl/slurm.conf /share/lxd-hpc/slurm.conf
# Verify the content of /share/lxd-hpc/slurm.conf is the LATEST one for LXD nodes!

Copy the create_lxd_nodes.sh script itself to the shared folder on pluto:

sudo cp /path/to/your/create_lxd_nodes.sh /share/lxd-hpc/create_lxd_nodes.sh
sudo chmod +x /share/lxd-hpc/create_lxd_nodes.sh

slurm User Consistency:

Ensure the slurm user exists on pluto with UID/GID 64030. If not, create or modify it:

sudo groupadd -g 64030 slurm || true
sudo useradd -u 64030 -g 64030 -m -s /bin/bash slurm || true
# If user/group exist but with different IDs, you might need:
sudo usermod -u 64030 slurm || true
sudo groupmod -g 64030 slurm || true

Firewall Rules: Ensure necessary ports are open on pluto:
- 6817/tcp (Slurmctld)
- 6819/tcp (Slurmdbd)
- 2049/tcp (NFS) if using NFS for shared folder.

On each Metal Node (`node[0-11,13,...]`):

Install NFS Client:

sudo apt update
sudo apt install -y nfs-common

Mount Shared Configuration Directory:
- Create the mount point:
```
sudo mkdir -p /share/lxd-hpc
```
- Mount the NFS share (add to /etc/fstab for persistence):
```
# Add this line to /etc/fstab on each metal node
pluto:/share/lxd-hpc /share/lxd-hpc nfs defaults 0 0
```
  Then, mount it: sudo mount -a
- Verify: df -h | grep lxd-hpc

slurm User Consistency:

Ensure the slurm user exists on each metal node with UID/GID 64030. If not, create or modify it:

sudo groupadd -g 64030 slurm || true
sudo useradd -u 64030 -g 64030 -m -s /bin/bash slurm || true
sudo usermod -u 64030 slurm || true
sudo groupmod -g 64030 slurm || true

Time Synchronization: Ensure ntp or chrony is installed and synchronized on all metal nodes and pluto. Time skew can cause Munge authentication failures.
Firewall Rules: Ensure necessary ports are open on metal nodes (especially 6818/tcp for Slurmd, and outbound to pluto).

3. Deployment Steps (Execute on each Metal Node)

The deployment script (create_lxd_nodes.sh) has two phases. You will run it twice on each metal node.

Execute the deployment script (create_lxd_nodes.sh) from the shared folder on each metal node.
- Make sure the script is executable (done in Prerequisites step 2 on pluto).
- Run it:
```
sudo /share/lxd-hpc/create_lxd_nodes.sh
```
First Run: Network Bridge Setup:
- When you execute the script for the first time, it will:
  - Install LXD and initialize it (if not present).
  - Detect the metal node's primary network interface and its IP.
  - Generate a Netplan YAML file (e.g., /share/lxd-hpc/01-bridge.yaml.node0) for creating br0 and enslaving the primary interface.
  - Exit at this point, providing instructions for manual Netplan application.

Manual Network Configuration Application:

For each metal node, after the first run of the script, perform these manual steps:

# Copy the generated Netplan file to the correct location
sudo cp /share/lxd-hpc/01-bridge.yaml.$(hostname) /etc/netplan/01-bridge.yaml

# Apply the Netplan configuration. This might cause a brief network interruption.
sudo netplan apply

Verify br0 is up and has the correct IP: ip a show br0

Second Run: LXD Instance Creation & Slurm/Munge Setup:
- Once br0 is active on the metal node, re-run the script on that same metal node:
```
sudo /share/lxd-hpc/create_lxd_nodes.sh
```
- The script will now detect that br0 exists and proceed to:
  - Create 3 LXD instances (metalNodeName-0, metalNodeName-1, metalNodeName-2).
  - Configure each instance with 4 CPUs and 12GB RAM.
  - Assign static IP addresses to instances.
  - Use cloud-init to:
    - Add pluto's IP to /etc/hosts inside the instance.
    - Ensure slurm user/group (UID/GID 64030) exists.
    - Install munge and slurm-wlm (which includes slurmd).
    - Copy munge.key and slurm.conf from /tmp (where the host copied them) to the correct locations.
    - Start munge and slurmd services.
  - Restart each LXD instance to apply all configurations.

4. Final Steps (Execute on `pluto` - SlurmctldHost)

Update slurm.conf on pluto:
- Ensure the slurm.conf file on your pluto machine (typically /etc/slurm-llnl/slurm.conf) is the latest version that defines the LXD instances as nodes. This file should be identical to the slurm.conf you placed in /share/lxd-hpc.
Restart Slurm Controller:
```
sudo systemctl restart slurmctld
```
Verify Cluster Status:
- Check node status:
```
sinfo -N
```
  You should see all your node[X]-[Y] instances listed, hopefully in IDLE state.
- Check details for a specific LXD node (e.g., node0-0):
```
scontrol show node node0-0
```
  Verify its State, CPUs, RealMemory, and network configuration.
- Test job submission:
```
srun -N1 --wrap="hostname"
```

5. Important Notes & Troubleshooting

IP Address Ranges: Ensure the static IP ranges defined in the script (STARTING_IP_LAST_OCTET, NETWORK_PREFIX) do NOT conflict with any existing devices or DHCP ranges on your network.
UID/GID Consistency: The slurm user having the same UID/GID (64030) across pluto, metal nodes, and all LXD instances is critical for Munge and Slurm to function correctly.
Firewalls: Double-check firewall rules (ufw, firewalld, iptables) on all hosts to ensure Slurm (ports 6817, 6818, 6819) and Munge (port 6811) communication is permitted.
LXD Storage: By default, LXD stores container data in /var/snap/lxd/common/lxd/storage-pools/default/containers. Ensure the underlying disk on your metal nodes has sufficient space for all LXD instances.
Debugging LXD Instances:
- List instances: lxc list
- Get instance details: lxc info <instance-name>
- Access instance console: lxc console <instance-name> (useful for debugging network or service issues inside the container)
- Execute commands inside instance: lxc exec <instance-name> -- <command>
Logs: Monitor Slurm logs on pluto (/var/log/slurm/slurmctld.log) and on each LXD instance (/var/log/slurm/node[X]-[Y].log) for any errors.
Restarting Services: After any changes to slurm.conf or munge.key, remember to restart slurmctld on pluto and slurmd/munge services on the affected compute nodes (LXD instances).

@media print { .ms-editor-squiggler { display:none !important; } } .ms-editor-squiggler { all: initial; display: block !important; height: 0px !important; width: 0px !important; }

aa3025/lxd-hpc

About

Resources

Stars

Watchers

Forks

Releases

Languages