12 Mar

How to setup HPC Pack 2019 cluster in Azure, AWS, GCP

in Azure

To setup and install a HPC Pack cluster on any of the cloud platforms, the recommended way is to use the Microsoft HPC Pack 2019 image that is available on the cloud marketplaces. Choose to create a head node with either locally installed Microsoft SQL Express or use an existing Microsoft remote SQL server. Deploy as many nodes as you need. Check the following marketplaces for the HCP Pack server image.

Getting Started

Once your HPC server has been deployed, the following links explain how to connect to a Windows VM:

Once connected and logged in, the following section explains how to configure HPC Pack as per your requirements

System Requirements for HPC Pack 2019

Microsoft recommends the following hardware specs for your VMs:

Head node role

Processor (x64-based):

Minimum: 4 cores
Recommended: 8 cores or more

RAM:

Minimum: 8 GB
Recommended: 16 GB or more

Available disk space:

Minimum: 50 GB
Recommended: 100 GB or more

Other node roles

Processor (x64-based):

Minimum: 4 cores
Recommended: 4 cores or more

RAM:

Minimum: 2 GB
Recommended: 4 GB or more

Available disk space:

Minimum: 50 GB
Recommended: 80 GB or more

Refer to Microsoft documentation for more information on HPC Pack system requirements: https://docs.microsoft.com/en-us/powershell/high-performance-computing/system-requirements-for-hpc-pack?view=hpc19-ps

Deploy an HPC Pack 2019 cluster

From the desktop click on “HPC Pack 2019 Setup“

This will launch the setup wizard

You will need to decide the following before setting up your HPC Cluster:

Decide if you want to configure your head node for high availability
Decide if you want to deploy your cluster with remote databases
Decide what type of nodes you want to add to your cluster and how many:
1. Compute nodes
2. Broker nodes
3. Workstation nodes & unmanaged server nodes
Choose the Active Directory domain or use Azure AD depending which cloud platform you’re using for your cluster
Choose a domain account for adding nodes
Choose a network topology for your cluster
Prepare certificates used to secure the communication between HPC nodes

Refer to the following Microsoft article that explains in detail the detailed steps above – https://docs.microsoft.com/en-us/powershell/high-performance-computing/step-1-prepare-for-your-deployment?view=hpc19-ps

HPC Cluster Firewall Ports

For a list of HPC Pack Cluster firewall ports that are used, please refer to the following Microsoft documentation:

https://docs.microsoft.com/en-us/powershell/high-performance-computing/appendix-1-hpc-cluster-networking?view=hpc19-ps#BKMK_ports

To setup AWS firewall rules refer to – AWS Security Groups

To setup Azure firewall rules refer to – Azure Network Security Groups

To setup Google GCP firewall rules refer to – Creating GCP Firewalls

Configure the HPC Cluster

Refer to the following Microsoft documentation on the steps to configure your HPC Cluster

https://docs.microsoft.com/en-us/powershell/high-performance-computing/step-3-configure-the-cluster?view=hpc19-ps

HPC Cluster Manager

Further detailed documentation on HPC Cluster manager:

https://docs.microsoft.com/en-us/powershell/high-performance-computing/hpc-cluster-manager?view=hpc19-ps

Adding Windows / Linux nodes to the HPC Cluster

You can use the HPC Pack 2019 image from the marketplace to create Windows nodes as it has the HPC Pack installation media pre installed. Refer the following links for the steps to add nodes to your cluster:

Windows Nodes

https://docs.microsoft.com/en-us/powershell/high-performance-computing/step-4-add-windows-nodes-to-the-cluster?view=hpc19-ps#42-manually-add-nodes-to-the-cluster

Linux Nodes

https://docs.microsoft.com/en-us/powershell/high-performance-computing/step-5-add-linux-nodes-to-the-cluster?view=hpc19-ps

Submit jobs to HPC Pack Cluster

The following links explain how to submit jobs to your HPC Pack Cluster.

Azure

To submit jobs to HPC Pack cluster in Azure, refer to – https://docs.microsoft.com/en-us/powershell/high-performance-computing/hpcpack-cluster-submit-jobs-in-azure?view=hpc19-ps

Also check –

https://docs.microsoft.com/en-us/powershell/high-performance-computing/creating-and-submitting-jobs?view=hpc19-ps

AWS

To submit jobs to HPC Pack cluster in AWS, refer to – https://docs.microsoft.com/en-us/powershell/high-performance-computing/creating-and-submitting-jobs?view=hpc19-ps

GCP

To submit jobs to HPC Pack cluster in GCP, refer to – https://docs.microsoft.com/en-us/powershell/high-performance-computing/creating-and-submitting-jobs?view=hpc19-ps

Overview of HPC Job Manager

Refer to the following Microsoft documentation on using HPC Job Manager

https://docs.microsoft.com/en-us/powershell/high-performance-computing/overview-of-hpc-job-manager?view=hpc19-ps

Support

Any questions regarding the installation of this solution, please leave a comment below or contact us and we will assist as much as we can to help you get your HPC Cluster up and running in your environment.

Legal Terms By clicking the Create button, I acknowledge that I am getting this software from Microsoft and that the legal terms of Microsoft apply to it. Microsoft does not provide rights for third-party software. Also see the privacy statement from Microsoft.

How to setup HPC Pack 2019 cluster in Azure, AWS, GCP

Setup HPC Pack 2019 on Azure

Setup HPC Pack 2019 on AWS

Setup HPC Pack 2019 on GCP