Install Elasticsearch, Logstash, Kibana (Elastic Stack) Ubuntu 22.04/20.04
Install Elasticsearch, Logstash, Kibana (Elastic Stack) Ubuntu 22.04/20.04. In this post, we will introduce ELK stack, then show you how to install Elasticsearch, Kibana, Logstash, and Filebeat on Ubuntu 22.04 or 20.04.
What is Elastic Stack?
Elastic Stack is a collection of open source software components developed by Elastic. Used for centralized logging purposes in a production environment. The software allows users to search, analyze, and visualize logs generated from any source in any format. It helps the system administrator to troubleshoot application related issues from a central place. The ELK stacks have four main components. A brief explanation of each component is shown below.
Elasticsearch: It is a free, open source and powerful search engine built on Apache Lucene that is used to store all of the collected data. Generally, Elasticsearch is used for full-text search, log analytics, business analytics, security intelligence and operational intelligence use cases. The stack works by gathering data from different locations, then stores and indexes it according to user requirements.
Logstash: Server side data processing component that gathers data from a multitude of sources, transforms it, and then sends it to your desired location. Generally, Logstash is used as a data pipeline for Elasticsearch. Logstash offers several plugins to parse and transform the logging data into any user desirable format.
Kibana: It is a free, and open source frontend application for ELK stack. The software provides a web interface where users visualize log and time-series analytics. Kibana comes with a rich set of features, including, pie charts, heat maps, histograms, line graphs, geospatial support and many more. Mainly it is used to explore and build a dashboard from the Elasticsearch log.
Filebeats: The software is a log shipper used to ship log files to your desired location. Filebeat is lightweight and used to monitor log files, collect log events, and forwards them to Elasticsearch or other sources for indexing.
Install Elasticsearch, Logstash, Kibana (Elastic Stack) on Ubuntu 22.04/20.04
In this section, we explain how to install the ELK stack on Ubuntu 22.04 and 20.04.
Prerequisites
- A server running Ubuntu 22.04 or 20.04
- Minimum 4 GB of RAM.
- A root user or a user with sudo privileges.
Install Required Dependencies
Before starting, update all the system packages to the latest version with the following command.
apt update -y
apt upgrade -y
After updating all the system packages, install additional required dependencies using the following command.
apt install curl gnupg2 unzip wget -y
Once all the dependencies are installed, proceed to install Elasticsearch.
Install Elasticsearch
By default, the latest version of Elasticsearch is not included in the Ubuntu default repository. So you will need to add the Elastic official repository to APT.
First, add the GPG key with the following command.
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elastic.gpg
Next, add the Elastic repository using the following command.
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list
Then, update the repository cache and install the latest version of Elasticsearch using the following command.
apt update
apt install elasticsearch -y
Once Elasticsearch is installed, you need to edit the Elasticsearch configuration file and define your host.
nano /etc/elasticsearch/elasticsearch.yml
Change the following line as per your needs.
network.host: localhost
Save and close the file when you are done. Then, start and enable the Elasticsearch service using the following command.
systemctl start elasticsearch
systemctl enable elasticsearch
Next, verify the Elasticsearch using the curl command.
curl -X GET "localhost:9200"
If everything is fine, you will get the following output.
Install Kibana and Nginx
Next, install the Kibana to visualize the data via the web interface. Install it with Nginx using the following command.
apt install kibana nginx
After the installation, start and enable the Kibana service using the following command.
systemctl start kibana
systemctl enable kibana
Next, generate a password to secure the Kibana web interface.
echo "kibanaadmin:`openssl passwd -apr1`" | tee -a /etc/nginx/htpasswd.users
You are asked to set a password as shown below.
Next, create an Nginx virtual host configuration file for Kibana.
nano /etc/nginx/conf.d/kibana.conf
Add the following configuration.
server {
listen 80;
server_name your-domain-name;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/htpasswd.users;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
Save and close the file then verify the Nginx for a syntax errors.
nginx -t
Finally, restart the Nginx service to implement the changes.
systemctl restart nginx
Now, open your web browser and verify the Kibana status using the URL http://your-domain-name/status. You should see the Kibana login page.
Provide your admin username, password and click on the Sign in button. You should see the Kibana status on the following screen.
Install and Configure Logstash
Next, install the Logstach to send incoming data to Elasticsearch. Install it with the following command.
apt install logstash
Once the installation is finished, create a new beats configuration file to set Filebeat input.
nano /etc/logstash/conf.d/02-beats-input.conf
Add the following configuration.
input {
beats {
port => 5044
}
}
Save and close the file then create an Elasticsearch file to define the output.
nano /etc/logstash/conf.d/30-elasticsearch-output.conf
Add the following lines.
output {
if [@metadata][pipeline] {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
pipeline => "%{[@metadata][pipeline]}"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
}
Save and close the file when you are done. Then, test the Logstash configuration using the following command.
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
You should see the following screen.
Next, start and enable the Logstash service with the following command.
systemctl start logstash
systemctl enable logstash
Install Filebeat
Next, install Filebeat to forward logs to the Elastic stack. You can install it with the following command.
apt install filebeat
Once the Filebeat is installed, edit the Filebeat configuration file.
nano /etc/filebeat/filebeat.yml
Comment on the following line:
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
And, uncomment the following lines:
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
Save and close the file. Next, you will need to install the Filebeat system module to collect and parse Syslog. You can install it with the following command.
filebeat modules enable system
After that, set up an ingest pipeline to parse the log data before sending it through logstash to Elasticsearch.
filebeat setup --pipelines --modules system
Then, load the index template into Elasticsearch with the following command.
filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
Finally, disable the Logstash output and enable Elasticsearch output with the following command.
filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601
You should see the following screen.
Next, start and enable the Filebeat service with the following command.
systemctl start filebeat
systemctl enable filebeat
Now, verify whether Elasticsearch receives data from Filebeat using the following command.
curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
If everything is fine, you will get the following screen.
Access Kibana Dashboard
At this point, the ELK stack is installed and configured. Now, let’s access and explore the Kibana dashboard using the URL http://your-domain-name. You should see the Kibana dashboard on the following command.
Now, search for Discover to see the Filebeat data as shown below.
Next, search for Filebeat System dashboards that come with Filebeat’s system module. You should see the following screen.
Install Elasticsearch, Logstash, Kibana (Elastic Stack) Ubuntu 22.04/20.04 Conclusion
In this guide, we showed you how to install Elastic Stack on Ubuntu 22.04/20.04. We used Elasticsearch as a search index, Logstash as a data processing component, Filebeat to ship data, and Kibana for searching and visualizing logs via web UI. Elastic stack is very useful for centralized logging to identify application and server related problems. I would recommend deploying the Elastic stack in a production environment as a centralized monitoring server.
