How to Setup RabbitMQ Cluster on Ubuntu 20.04 (Tutorial)
How to Setup RabbitMQ Cluster on Ubuntu 20.04. This article will explain what RabbitMQ is with its benefits and how to install RabbitMQ Cluster.
What is RabbitMQ?
RabbitMQ is a message broker software that stores and passes on the messages to the receiver only when it connects with the queue and there is a demand for transferring the message. It is open source software that works as a broker between the sender and the receiver. The software is highly beneficial as it helps reduce the excessive load and improves the delivery time.
Before the introduction of RabbitMQ, TCP connections were used to connect the application components and deliver messages. However, the technique was not good enough, and dropping the idea was the only solution due to certain limitations. The sender could forward the information only when he received the request. If the receiver went down, the sender kept transferring messages unless a connection was generated. As a result, there were high chances of system failure. Thus, to avoid such mishaps, the Message Queuing system was introduced.
Pros of RabbitMQ
- RabbitMQ software defines and configures the queue to help create a better communication system between the sender and the receiver.
2. Delivery and order message guaranteed. It helps applications to connect with the queues and transfer single or multiple messages without bogging down the system. The messaging queue system runs a consumer/producer model where the sender (producer) publishes a message to the queue to forward and process by the receiver (consumer). The message can be a simple test or binary information that is retained in the queue system unless the consumer passes a transfer request. The main purpose of introducing this system was to create a space that could store or fill in N number of messages from the sender without failing the system. The idea got successful and came into practice. However, there is a minor twist, RabbitMQ uses exchanges to receive messages from the sender to the queue.
3. Type of RabbitMQ exchanges: Direct, Fanout, Topic and Headers. These exchanges receive messages from the sender and use binary keys to link queues and transfer information.
4. Deployment in public and private clouds is much easier
5. Supports multiple languages
6. Uses HTTP API, command line tool, and UI for management and monitoring of the Message Queuing system.
7. Supports Asynchronous Messaging
8. Flexible Routing.
9. The RabbitMQ broker is scalable and flexible.
How Does RabbitMQ Cluster Works?
Next in how to Setup RabbitMQ Cluster on Ubuntu 20.04 is to introduce RabbitMQ cluster. It is a group of nodes that share users, queues, exchanges, virtual hosts, runtime parameters, bindings and additional distributed states. The software must include infra for the availability of the RabbitMQ queues. A minimum of three nodes cluster must be formed to create it. It helps in retaining quorum amid the various servers. Unlike other clusters, RabbitMQ cluster mode does not follow the concept of master and slave.
In the beginning, a single node is used to run all RabbitMQ brokers. As the nodes start connecting, a cluster is generated. However, these clusters can later be converted into a single broker as per the preference. The only exception in the RabbitMQ cluster mode is message queues reside on a single node by default. However, these are reachable to all nodes in the cluster.
Create RabbitMQ clusters
- DNS server based discovery.
- Use a config file to list cluster nodes.
- Use a plugin for AWS (EC2) instance discovery.
- Use a plugin for Kubernetes discovery.
- Use a plugin for Consul based discovery.
- Use a plugin for etcd based discovery.
To begin the process, make sure you check two things:
Hostname Resolution
RabbitMQ nodes have individual names to address each other in the cluster mode. They often use command line tools to discover and direct one another. Make sure that the hostname of each cluster must be resolved. DNS records or Localhost files methods are used in most cases for Hostname resolution. One can also opt for Erlang VM if there is any restriction in the DNS record method.
Port Access
Each RabbitMQ node must be linked to the ports for smooth communication and functioning. The binding between the nodes and the port helps the receiver connect with the message broker and transfer request. For open connections, one can use any of the three sources:
- Command Line Tools.
- Client Libraries.
- RabbitMQ nodes.
Next, we will show you how to install two node RabbitMQ cluster on Ubuntu 20.04.
Setup Hostname Resolution
Before setting up the RabbitMQ cluster, you will need to set up hostname resolution on both servers. So each server can communicate with each other by hostname.
To do so, edit the /etc/hosts file on both servers:
nano /etc/hosts
Add the following lines:
104.219.55.246 server1
104.219.54.225 server2
Save and close the file when you are finished.
Where:
- 104.219.55.246 is the IP address of the first server.
- 104.219.54.225 is the IP address of the second server.
Install RabbitMQ Server
Next, you will need to install the RabbitMQ server on both servers. You can install it by running the following command:
apt-get install rabbitmq-server -y
Once the RabbitMQ server package is installed, start and enable the RabbitMQ service with the following command:
systemctl start rabbitmq-server
systemctl enable rabbitmq-server
Next, verify the status of the RabbitMQ using the following command:
systemctl status rabbitmq-server
You will get the following output:
● rabbitmq-server.service - RabbitMQ Messaging Server
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2022-02-11 07:06:24 UTC; 4s ago
Main PID: 1369 (beam.smp)
Status: "Initialized"
Tasks: 85 (limit: 2348)
Memory: 73.8M
CGroup: /system.slice/rabbitmq-server.service
├─1365 /bin/sh /usr/sbin/rabbitmq-server
├─1369 /usr/lib/erlang/erts-10.6.4/bin/beam.smp -W w -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P >
├─1623 erl_child_setup 65536
├─1646 inet_gethost 4
└─1647 inet_gethost 4
Feb 11 07:06:15 server1 systemd[1]: Starting RabbitMQ Messaging Server...
Configure RabbitMQ Admin Interface
RabbitMQ provides a management plugin to monitor and manage RabbitMQ nodes and clusters through a web browser. But, it is disabled by default. So you will need to enable it on both servers.
Run the following command to enable the RabbitMQ management plugin:
rabbitmq-plugins enable rabbitmq_management
You will get the following output:
Enabling plugins on node rabbit@server1:
rabbitmq_management
The following plugins have been configured:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@server1...
The following plugins have been enabled:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_web_dispatch
started 3 plugins.
Next, restart the RabbitMQ service to apply the changes:
systemctl restart rabbitmq-server
At this point, the RabbitMQ management plugin is enabled and listens on port 15672. You can check it with the following command:
ss -antpl | grep 15672
You will get the following output:
LISTEN 0 1024 0.0.0.0:15672 0.0.0.0:* users:(("beam.smp",pid=2233,fd=91))
Configure RabbitMQ Cluster
RabbitMQ uses a cookie to determine whether they can communicate with each other. So you will need to copy a cookie file from server1 to server2.
Run the following command on server1 to copy a cookie file to server2:
scp /var/lib/rabbitmq/.erlang.cookie root@104.219.54.225:/var/lib/rabbitmq/
Next, go to the server2 and restart the RabbitMQ service with the following command:
systemctl restart rabbitmq-server
Next, stop the application and reset the RabbitMQ with the following command:
rabbitmqctl stop_app
rabbitmqctl reset
Next, join the server2 to the RabbitMQ cluster using the following command:
rabbitmqctl join_cluster rabbit@server1
You will get the following output:
Clustering node rabbit@server2 with rabbit@server1
Next, start the application process with the following command:
rabbitmqctl start_app
Finally, check the RabbitMQ cluster status using the following command:
rabbitmqctl cluster_status
You will get the following output:
Cluster status of node rabbit@server1 ...
Basics
Cluster name: rabbit@server1
Disk Nodes
rabbit@server1
rabbit@server2
Running Nodes
rabbit@server1
rabbit@server2
Versions
rabbit@server1: RabbitMQ 3.8.2 on Erlang 22.2.7
rabbit@server2: RabbitMQ 3.8.2 on Erlang 22.2.7
Add RabbitMQ Administration Account
In order to access the RabbitMQ web interface, you will need to set an admin username and password. To set an admin user, password, and proper permissions, run the following command on server1:
rabbitmqctl add_user admin yourpassword
rabbitmqctl set_user_tags admin administrator
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
You can now list all users using the following command:
rabbitmqctl list_users
You will get the following output:
Listing users ...
user tags
admin [administrator]
guest [administrator]
Next, it is recommended to delete the guest user account. You can delete it using the following command:
rabbitmqctl delete_user guest
Configure RabbitMQ Queue Mirroring
Next, you will need to create a policy that allows for queues mirroring to all servers in the cluster.
rabbitmqctl set_policy ha-all ".*" '{"ha-mode":"all"}'
You will get the following output:
Setting policy "ha-all" for pattern ".*" to "{"ha-mode":"all"}" with priority "0" for vhost "/" ...
Now, run the following command to list all policies:
rabbitmqctl list_policies;
You will get the following output:
Listing policies for vhost "/" ...
vhost name pattern apply-to definition priority
/ ha-all .* all {"ha-mode":"all"} 0
Access RabbitMQ Dashboard
Now, open your web browser and access the RabbitMQ dashboard using the URL http://104.219.55.246:15672/. You will be redirected to the RabbitMQ login page:
Provide your admin username, password, and click on the Login button. You should see the RabbitMQ cluster dashboard on the following page:
You have successfully setup RabbitMQ Cluster on Ubuntu 20.04.
How to Setup RabbitMQ Cluster on Ubuntu 20.04 Conclusion
In the above guide, we explained how to set up a two-node RabbitMQ cluster on Ubuntu 20.04 server. You can now add more nodes to scale the cluster. For more information, visit the RabbitMQ Clustering guide.
