In this example Slurm cluster we have 3 nodes, node1, node2 and node3<\/p>\n\n\n\n
node1 is on IP 10.0.0.1, node2 is on 10.0.0.2 and node3 is on 10.0.0.3<\/p>\n\n\n\n
A critical pre-req is that your \/etc\/hosts or DNS forward and reverse hostname and IP Address lookup commands all work and return the correct and same information about the hostnames and IPs of each node in the cluster.
In the context of this lab cluster, the \/etc\/hosts file is listing all hostnames and IPs used in the cluster.<\/p>\n\n\n\n
All our cluster nodes are running the latest CentOS 9 Linux, updated and rebooted after running “dnf distro-sync” to be all on the same CentOS\/RHEL software versions.<\/p>\n\n\n\n
Firewalld is enabled and the following firewall-cmd commands have been run on each node:<\/p>\n\n\n\n
Whether you need munge or not, its best to install it, so do:<\/p>\n\n\n\n And then set the various files and directory permissions on the RHEL world are needed, so run the following on each node: Some tests\/debugs need each node to be able to ssh into the others, so setup your authorized_keys files on each node as you need.<\/p>\n\n\n\n <\/p>\n\n\n\n Now install Slurm with the following on node1:<\/p>\n\n\n\n And do the following installs on every compute node:<\/p>\n\n\n\n To configure Slurm. all nodes have the same \/etc\/slurm\/slurm.conf<\/strong> file, which critically has the following changes from the default, being:<\/p>\n\n\n\n Check slurm cluster status with the sinfo command from any node, all should return the same info in standard operations. <\/p>\n","protected":false},"excerpt":{"rendered":" In this example Slurm cluster we have 3 nodes, node1, node2 and node3 node1 is on IP 10.0.0.1, node2 is on 10.0.0.2 and node3 is on 10.0.0.3 A critical pre-req is that your \/etc\/hosts or DNS forward and reverse hostname … Continue reading firewall-cmd --permanent --zone=public --add-rich-rule='rule family=\"ipv4\" source address=\"10.0.0.0\/24\" port protocol=\"tcp\" port=\"6817\" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family=\"ipv4\" source address=\"10.0.0.0\/24\" port protocol=\"tcp\" port=\"6818\" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family=\"ipv4\" source address=\"10.0.0.0\/24\" port protocol=\"tcp\" port=\"6819\" accept'
firewall-cmd --permanent --zone=public --add-rich-rule='rule family=\"ipv4\" source address=\"10.0.0.0\/24\" port protocol=\"tcp\" port=\"60001-60100\" accept' && firewall-cmd --reload<\/code><\/p>\n\n\n\ndnf -y install epel-release
dnf install -y munge munge-libs
<\/code>
On node1 (your “main” or “head” node) run: \/usr\/sbin\/create-munge-key<\/p>\n\n\n\nchown -R munge: \/etc\/munge\/ \/var\/log\/munge\/ \/var\/lib\/munge\/ \/run\/munge\/
chmod 0700 \/etc\/munge\/ \/var\/log\/munge\/ \/var\/lib\/munge\/
chmod 0755 \/run\/munge\/
chmod 0700 \/etc\/munge\/munge.key<\/code><\/p>\n\n\n\n
Then copy the \/etc\/munge\/munge.key<\/strong> file to all your nodes.
Get munge going on each node with the above dnf install command and then:systemctl enable munge && systemctl start munge
systemctl status munge<\/code><\/p>\n\n\n\ndnf install -y slurm slurm-slurmctld slurm-slurmdbd mariadb-server slurm-slurmd<\/code><\/p>\n\n\n\ndnf install -y slurm slurm-slurmd<\/code>
<\/p>\n\n\n\n\n
SlurmctldHost=node1(10.0.0.1)<\/code><\/li>\n\n\n\nNodeName=node1 CPUs=2 State=UNKNOWN
NodeName=node2 CPUs=2 State=UNKNOWN
NodeName=node3 CPUs=2 State=UNKNOWN<\/code><\/li>\n\n\n\nSrunPortRange=60001-60100<\/code>
Restart slurmctld on the head\/main node (node1) and then restart slurmd on all nodes.
On node1, run: systemctl restart slurmctld && systemctl restart slurmd<\/code>
On just the compute nodes, not node1, just run systemctl restart slurmd<\/li>\n<\/ul>\n\n\n\n
Debug with logfiles in \/var\/log\/slurm\/<\/p>\n\n\n\n