How to setup HPC Pack 2019 cluster in Azure, AWS, GCP

To setup and install a HPC Pack cluster on any of the cloud platforms, the recommended way is to use the Microsoft HPC Pack 2019 image that is available on the cloud marketplaces. Choose to create a head node with either locally installed Microsoft SQL Express or use an existing Microsoft remote SQL server. Deploy as many nodes as you need. Check the following marketplaces for the HCP Pack server image.

Getting Started

 

Once your HPC server has been deployed, the following links explain how to connect to a Windows VM:

 

 

Once connected and logged in, the following section explains how to configure HPC Pack as per your requirements

System Requirements for HPC Pack 2019

 

Microsoft recommends the following hardware specs for your VMs:

 

Head node role

 

Processor (x64-based):

  • Minimum: 4 cores
  • Recommended: 8 cores or more

 

RAM:

  • Minimum: 8 GB
  • Recommended: 16 GB or more

 

Available disk space:

  • Minimum: 50 GB
  • Recommended: 100 GB or more

 

Other node roles

 

Processor (x64-based):

  • Minimum: 4 cores
  • Recommended: 4 cores or more

RAM:

  • Minimum: 2 GB
  • Recommended: 4 GB or more

Available disk space:

  • Minimum: 50 GB
  • Recommended: 80 GB or more

 

Refer to Microsoft documentation for more information on HPC Pack system requirements: https://docs.microsoft.com/en-us/powershell/high-performance-computing/system-requirements-for-hpc-pack?view=hpc19-ps

Deploy an HPC Pack 2019 cluster 

 

From the desktop click on “HPC Pack 2019 Setup

 

This will launch the setup wizard

Setup HPC Pack Cluster

 

You will need to decide the following before setting up your HPC Cluster:

 

  1. Decide if you want to configure your head node for high availability
  2. Decide if you want to deploy your cluster with remote databases
  3. Decide what type of nodes you want to add to your cluster and how many:
    1. Compute nodes
    2. Broker nodes
    3. Workstation nodes & unmanaged server nodes
  4. Choose the Active Directory domain or use Azure AD depending which cloud platform you’re using for your cluster
  5. Choose a domain account for adding nodes
  6. Choose a network topology for your cluster
  7. Prepare certificates used to secure the communication between HPC nodes

 

Refer to the following Microsoft article that explains in detail the detailed steps above – https://docs.microsoft.com/en-us/powershell/high-performance-computing/step-1-prepare-for-your-deployment?view=hpc19-ps

HPC Cluster Firewall Ports

 

For a list of HPC Pack Cluster firewall ports that are used, please refer to the following Microsoft documentation:

 

https://docs.microsoft.com/en-us/powershell/high-performance-computing/appendix-1-hpc-cluster-networking?view=hpc19-ps#BKMK_ports

 

To setup AWS firewall rules refer to – AWS Security Groups

To setup Azure firewall rules refer to – Azure Network Security Groups

To setup Google GCP firewall rules refer to – Creating GCP Firewalls

Configure the HPC Cluster

 

Refer to the following Microsoft documentation on the steps to configure your HPC Cluster

https://docs.microsoft.com/en-us/powershell/high-performance-computing/step-3-configure-the-cluster?view=hpc19-ps

 

HPC Cluster Manager

Further detailed documentation on HPC Cluster manager:

https://docs.microsoft.com/en-us/powershell/high-performance-computing/hpc-cluster-manager?view=hpc19-ps

Adding Windows / Linux nodes to the HPC Cluster

 

You can use the HPC Pack 2019 image from the marketplace to create Windows nodes as it has the HPC Pack installation media pre installed. Refer the following links for the steps to add nodes to your cluster:

 

Windows Nodes

https://docs.microsoft.com/en-us/powershell/high-performance-computing/step-4-add-windows-nodes-to-the-cluster?view=hpc19-ps#42-manually-add-nodes-to-the-cluster

 

Linux Nodes

https://docs.microsoft.com/en-us/powershell/high-performance-computing/step-5-add-linux-nodes-to-the-cluster?view=hpc19-ps

 

Submit jobs to HPC Pack Cluster

 

The following links explain how to submit jobs to your HPC Pack Cluster.

 

Azure

To submit jobs to HPC Pack cluster in Azure, refer to – https://docs.microsoft.com/en-us/powershell/high-performance-computing/hpcpack-cluster-submit-jobs-in-azure?view=hpc19-ps

 

Also check –

https://docs.microsoft.com/en-us/powershell/high-performance-computing/creating-and-submitting-jobs?view=hpc19-ps

 

AWS

To submit jobs to HPC Pack cluster in AWS, refer to – https://docs.microsoft.com/en-us/powershell/high-performance-computing/creating-and-submitting-jobs?view=hpc19-ps

 

GCP

To submit jobs to HPC Pack cluster in GCP, refer to – https://docs.microsoft.com/en-us/powershell/high-performance-computing/creating-and-submitting-jobs?view=hpc19-ps

 

Overview of HPC Job Manager

Refer to the following Microsoft documentation on using HPC Job Manager

https://docs.microsoft.com/en-us/powershell/high-performance-computing/overview-of-hpc-job-manager?view=hpc19-ps

Support

 

Any questions regarding the installation of this solution, please leave a comment below or contact us and we will assist as much as we can to help you get your HPC Cluster up and running in your environment.

Legal Terms By clicking the Create button, I acknowledge that I am getting this software from Microsoft and that the legal terms of Microsoft apply to it. Microsoft does not provide rights for third-party software. Also see the privacy statement from Microsoft.

Andrew Fitzgerald

Cloud Solution Architect. Helping customers transform their business to the cloud. 20 years experience working in complex infrastructure environments and a Microsoft Certified Solutions Expert on everything Cloud

4 2 votes
Article Rating
Subscribe
Notify of
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x