VM Types for DevOps – Pets vs Cattle vs Immutable
VM Types for DevOps – Pets vs Cattle vs Immutable. Every DevOps team should think about designing a VM pattern if you have a large server infrastructure either on prem or in the cloud. You should design your VM pattern based on your security requirements for your applications, data, access, interoperability between systems and anything else your servers need in order to function. I will try to highlight the pros and cons for each VM type and use cases for each. Then we will look at tooling to automate the management of the different VM types.
What is Immutable Infrastructure
Immutable infrastructures were designed to rely on virtualization technologies that are meant for fast provisioning of architecture components, like cloud computing and virtualization servers that led to simplicity, reliability and consistency. That resulted in eliminating common failure points.
The mutable infrastructure is a much older infrastructure paradigm developed to use the physical servers. The traditional mutable infrastructures was irreplaceable and running at all times. SSH into servers and installing, updating or upgrading packages, changing, and optimizing configuration files. Mutable infrastructure is like pets: unique, inimitable and tended to by hand. Losing one could be a big disturbance.
The servers in immutable infrastructures, on the other hand, are disposable and can be easily replicated or scale with automated tools. Instead of logging in via SSH into the server and updating the software version, every change in the application starts with a software push to the code repository (git push). So they are like cattle: many in a herd, where there is no uniqueness or indispensability. The moving away trend from mutable to immutable infrastructure happened when servers could be considered disposable and it became possible with virtual servers. Immutable infrastructure is cloud computing
Follow this article VM Types for DevOps – Pets vs Cattle vs Immutable to learn more about immutable infrastructure pros.
Immutable infrastructure benefits
Cloud computing provides the automation required to construct and deploy images for immutable operations:
- cost reduction,
- fewer deployment failures (eliminate the risk of snowflake servers and configuration drift),
- full automation of your entire deployment pipeline,
- a service oriented architecture similar to IaaS, PaaS,
- servers are in virtualized environment (containers) ,
- consistent staging environments and horizontal scaling,
- easier maintainability,
- improved availability,
- easy rollback and recovery processes,
- cloud is elastic (you can grow your infrastructure immensely),
- infrastructure is cheaper (capital required to get your product out there is reduced),
- part of cloud appeal are containers, which have enabled true immutability and go in line with CI/CD flows.
Follow this article VM Types for DevOps – Pets vs Cattle vs Immutable to learn about VM Types next.
VM Types - What are Pets, Cattle & Immutable Servers
Pets
Pets and snowflake servers are mutable infrastructure and they are servers or server pairs that are treated as indispensable or unique systems that can never be down. Typically they are manually built, managed and “hand fed”. Examples include mainframes, solitary servers, HA loadbalancers/firewalls (active/active or active/passive), database systems designed as master/slave (active/passive), and so on.
Cattle
Cattle also known as immutable VMs are arrays of more than two servers, that are built using automated tools, and are designed for failure, where no one, two, or even three servers are irreplaceable. Typically, during failure events no human intervention is required as the array exhibits attributes of “routing around failures” by restarting failed servers or replicating data through strategies like triple replication or erasure coding. Examples include phoenix servers ,web server , no-sql clusters, caching reverse proxy cluster, multi-master datastores such as Cassandra clusters, multiple racks of gear put together in clusters, and just about anything that is load-balanced and multi-master.
Immutable Servers
Immutable Servers (Immutable Infrastructure) are most restrictive – No configuration changes are made to any server ever. Servers are provisioned again with any changes that are required. E.g. If a web server is deployed that references a specific virtual IP and that changes, the web server would be demised and a new web server would be created to take it’s place with the new virtual IP in it’s configuration.
VM Types Explained
Type 1 Pets - Manual Application Deployment, Configuration & Maintenance
- Customers require interactive logon to servers to manage and support their application
- The application layer may need customisation and integration and adjustments through it’s lifetime
- The application teams may need interactive access to the server for troubleshooting purposes
- Break Glass – DevOps will need interactive access – privileged and ‘just enough’
- Patching – at runtime (in situ)
Type 2 Cattle – Automated Application Deployment, Configuration & Maintenance
- As above however the application layer is also delivered via a fully automated deployment and management framework and any customisations are made via this framework and/or a configuration management tool.
- No interactive login is required
- Break Glass – DevOps will need interactive access – privileged and ‘just enough’
- Patching – at runtime (in situ)
Type 3 Immutible – Automated Application Deployment, Configuration & Maintenance – Rapid Scaleout (Restricted OS tooling/Operational mode)
- No interactive logons or accounts that could adjust configuration can be allowed on this server. No application layer framework can exist. All server images include the application and operating system desired state.
- The application is part of the operating system image
- Patching – at Image creation time – Not at runtime; running instances need to be replaced with the new patched image
VM Type Use Cases
Examples of Pets, Cattle & Immutable VMs
Immutable (Type 3) | Immutable Cattle (Type 2) | Mutable Pets ( Type 1) |
Stateless Servers | Web array servers | Mainframes |
Management Servers | Solitary Servers | |
Application Server | Multi-master datastores | |
HPC Head Node / Compute Nodes | Firewalls | |
Designing The Immutable VM Type
The following i put together for a banking client who needed a VM pattern designed for deploying virtual servers in the cloud (Azure/AWS/GCP), so you can adjust the Architecture building blocks and solution building blocks to meet your needs, this is just an example of how we did it for our client.
ABB – Defines what functionality will be implemented
SBB – Defines what products and components are available to implement the functionality
In this example VM Types for DevOps – Pets vs Cattle vs Immutable- for our Immutable VMs we put together a list of requirements from the security team (Architecture Building Blocks) and then looked for solutions (Solution Building Blocks) that would meet the requirements we received from the security team. All the VMs were deployed in the cloud.
Architecture Building Block (ABB) | Solution Building Blocks (SBB) | ||
Option 1- (Immutable) | |||
Implement a CI-CD pipeline | Jenkins , GitLab or Azure DevOps | Jenkins, GitLab or GCP native Cloud Composer | Jenkins, GitLab or AWS native CodeStar |
As part of the pipeline scan the images for vulnerabilities If any Vulns found reject them, If not approve them as part of the build pipeline and harden configurations appropriately | Nessus or other vulnerability scanning tools, apply any OS security lockdown configuration, Pester test OS scanning | Nessus or other vulnerability scanning tools, apply any OS security lockdown configuration, Pester test OS scanning | Nessus or other vulnerability scanning tools, apply any OS security lockdown configuration, Pester test OS scanning |
All Infrastructure deployments must be automated thru CI-CD pipeline only using automation and NO Manual deployments | Terraform, Azure DevOps, Ansible, Puppet, Chef, Jenkins, Azure ARM, Azure Bicep | Terraform, Jenkins, Puppet, Chef Ansible, GCP Cloud Deployment Manager | Terraform, Ansible, Puppet, Chef |
No Internet access to the deployed infrastructure or services | Azure Network Security Groups (NSG) rules to block access to internet | GCP VPC Firewall polices block access | AWS Security Groups |
No Interactive access to the deployed infra or services e.g. No SSH, RDP etc. for troubleshooting etc., if problems with service then kill the instance and re- instantiate the service | OS lockdown configuration & NSG Rules controlling access, Azure Squid Proxy | OS lockdown configuration, Server side protocols disabled, GCP Squid Proxy, GCP Firewall Rules | OS lockdown configuration & Security Groups controlling access, AWS Squid Proxy |
This Infrastructure and services deployed should be ephemeral, which means automatically after 28 days | Azure DevOps & Azure Automation Runbooks | Google Cloud Functions with Jenkins or Cloud Composer | AWS Lambda & AWS DevOps Tooling (CodePipelines, Codebuild, CodeDeploy, CodeStar) or Jenkins |
Have an automated function check for any instances running longer than 28 days or predefined period and auto-kill them, or auto reinitiate them thru the pipeline using new image which is pre-scanned thru the pipeline. | Azure Automation Runbooks / Azure Inventory / Azure DevOps | Jenkins, GitLab or GCP Cloud Composer | AWS Lambda, Jenkins or AWS Lambda AWS DevOps Tooling (CodePipelines, Codebuild, CodeDeploy, CodeStar) |
Malware detection solution | McAfee AV VSE (VirusScan Enterprise) or ENS (End Point Security), Crowdstrike | McAfee AV VSE (VirusScan Enterprise) or ENS (End Point Security), Crowdstrike | McAfee AV VSE (VirusScan Enterprise) or ENS (End Point Security), Crowdstrike |
Immutable infrastructure tools
Designing The Cattle (Immutable) VM Type
Architecture Building Block (ABB) | Solution Building Blocks (SBB) | ||
Option 2 (Cattle) | Solution Option | Solution Option – GCP | Solution Option – AWS |
Implement a CI-CD pipeline | Jenkins or Azure DevOps | Jenkins or GCP Tooling (Google Cloud Composer, GC Deployment manager, GC Build, GC Source Repos) | Jenkins or AWS DevOps Tooling (CodePipelines, Codebuild, CodeDeploy, CodeStar) |
As part of the pipeline scan the images for vulnerabilities If any Vulns found reject them, If not approve them as part of the build pipeline and harden configurations appropriately | Nessus, security standards, lockdown configuration, Pester test framework, CCS Scanning | Nessus, security standards, lockdown configuration, Pester test framework, CCS Scanning | Nessus, security standards, lockdown configuration, Pester test framework, CCS Scanning |
All Infrastructure deployments must be automated thru CI-CD pipeline only using automation and NO Manual deployments | Terraform, Azure DevOps, Ansible, or Jenkins | Terraform, Ansible, Jenkins or GCP Tooling (GC Composer, GC Deployment manager, GC Build) | Terraform, AWS DevOps Tooling or Jenkins & Ansible |
Deploy a Runtime vulnerability management solution to detect any unauthorized/unapproved images running in the environment | Azure Policy & Azure Automation Runbooks | Organization Policy & Google Cloud Functions | Service Control Policies & AWS Lambda |
Have a Malware and threat detection solution in place to detect any malicious software injected thru different channels | McAfee AV VSE (VirusScan Enterprise) or ENS (End Point Security), Crowdstrike, Azure Security Center | McAfee AV VSE (VirusScan Enterprise) or ENS (End Point Security), Crowdstrike, Azure Security Center | McAfee AV VSE (VirusScan Enterprise) or ENS (End Point Security), Crowdstrike, Amazon GuardDuty, Amazon Inspector |
Internet access to the deployed infrastructure or services maybe enabled on need basis with other security protections | NSG Rules | GCP Firewall rules | Security Groups |
In extreme circumstances, Interactive access to the deployed infra or services e.g. using SSH, RDP etc. for troubleshooting etc., appropriate SSH key mgmt. and secure private key mgmt.. with appropriate detection of admin actions. | Just-In-Time solution via Azure security center | Question for security / GCP | Question for security / AWS |
Vulnerability scanner must check for runtime instances from stale images and alert and appropriate action be taken automatically to rebuild those compute instances (Self- Healing). | Azure Security center, Azure DSC, AzureDevOps pipeline | Question for security (Stackdriver for log alerting) | Question for security |
VMs will need a configuration drift management solution in place to prevent VMs being misconfigured | Azure DSC | Puppet, Chef | AWS OpsWorks for (Puppet Enterprise or Chef Automate) |
VMs will need to be regularly patched with the latest security updates and monitored | Azure Update Management or SCCM | SCCM | AWS Systems Manager or SCCM |
Designing Your Server Image Lifecycle
Now that we have a better understanding of how each virtual machine type should look, you now need to put together your CI/CD pipeline and start developing the steps to create your server images.
We defined the image creation in 3 stages
Stage -1 Base line image
This stage will be managed by your server team (infrastructure team) who are responsible for deploying servers and maintaining server images for application teams.
Stage 2 – Application Automation / Configuration (Cattle)
This stage will be where application developers deploy a new VM using stage1 baseline image and baking their applications / configuration. Running sysprep to create a stage 2 image.
Stage 3 – Immutable lockdown image
Current ecosystem
VM Types for DevOps – Pets vs Cattle vs Immutable. Cloud computing where you program the infrastructure as code and apply immutable production with containers is on the rise. Hybrid cloud benefits the infrastructure because it allows businesses to leverage powerful technologies while meeting their unique needs. Hybrid cloud with the popular choices: Terraform or Ansible.
In the big data or streaming space Kafka or Hadoop are popular choices.
Today the industry is moving towards cloud native and cloud based applications due to automation possiblities.
VM Types for DevOps – Pets vs Cattle vs Immutablec Conclusion
There will still be a use cases on both types of servers. But the IT world is moving from mutable to immutable infrastructure. The trend is rising immensely for Cloud based ecosystem. Cloud native space requires automation of your IT infrastructure that increases service and response time.
The transition from (mutable-immutable)-pets to a cattle service model is crucial in your operational process to offer better and faster service.
Cloud native applications rely on the cattle service model and go independently to scale without underlying on the underlying server.
The reason why to go away from the old way and have “pets” is to implement immutability where the whole CI/CD process can be traced to checkpoints in code.
