How to Setup Apache Kafka Server on Azure/AWS/GCP
Apache Kafka is an open source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Follow the steps below to setup Apache Kafka Server in the Cloud on Azure, AWS or GCP. Deploy using the public image available in the marketplaces.
Kafka Cloud
Kafka on Azure
Kafka on AWS
Kafka on Google Cloud
Getting Started with Apache Kafka
Once your Apache Kafka server has been deployed, the following links explain how to connect to a Linux VM:
- How to connect to a Linux VM on Azure
- How to connect to a Linux VM on AWS
- How to connect to a Linux VM on GCP
Once connected and logged in, the following section explains how to start using Kafka.
Kafka Installation Directory
For reference, the installation of Apache Kafka is in the following directory:
/opt/kafka/
Start the Kafka Environment
Run the following commands in order to start all services in the correct order:
First change your current path to the Kafka installation directory:
cd /opt/kafka/
Run the following command to start the ZooKeeper service.
sudo bin/zookeeper-server-start.sh config/zookeeper.properties
You will now need to open another terminal session to complete the next steps.
Start the Kafka Broker Service
Run the following command to start the Kafka broker service:
First change your current path to the Kafka installation directory:
cd /opt/kafka/
sudo bin/kafka-server-start.sh config/server.properties
Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.
You will now need to open another terminal session to complete the next steps.
Create a Kafka topic to store your events
Kafka is a distributed event streaming platform that lets you read, write, store, and process events (also called records or messages in the documentation) across many machines.
Example events are payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements from IoT devices or medical equipment, and much more. These events are organized and stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.
So before you can write your first events, you must create a topic. Open another terminal session and run:
First change your current path to the Kafka installation directory:
cd /opt/kafka/
sudo bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
All of Kafka’s command line tools have additional options: run the ./kafka-topics.sh command in the bin directory without any arguments to display usage information. For example, it can also show you details such as the partition count of the new topic:
sudo bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
The output should look like this:
Topic:quickstart-events PartitionCount:1 ReplicationFactor:1 Configs:
Topic: quickstart-events Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Write Events into the Kafka Topic
Now that you have a Kafka Topic created, we can now setup some events into the Topic. A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events. Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you need, even forever.
Run the console producer client to write a few events into your topic. By default, each line you enter will result in a separate event being written to the topic.
sudo bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
This is my first test event
This is my second test event
You can stop the producer client with Ctrl-C at any time.
Read the Kafka Events
Open another terminal session and run the console consumer client to read the events you just created:
sudo bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
You should get an output of your events like this:
This is my first test event
This is my second test event
Press Ctrl+C to stop the consumer client.
Kafka Documentation
To learn more, we suggest the following next steps:
- Read through the brief Introduction to learn how Kafka works at a high level, its main concepts, and how it compares to other technologies. To understand Kafka in more detail, head over to the Documentation.
- Browse through the Use Cases to learn how other users in our world-wide community are getting value out of Kafka.
Kafka Firewall Ports
By default Kafka uses the following ports:
| Service | Servers | Default Port | Default Ambari Port | Protocol | Description | Need End User Access? | Configuration Parameters |
|---|---|---|---|---|---|---|---|
| Kafka | Kafka Server | 9092 | 6667 | TCP | The port for Kafka server. |
If you are using any of the cloud security groups and need to change / add ports refer to the following guides:
To setup AWS firewall rules refer to – AWS Security Groups
To setup Azure firewall rules refer to – Azure Network Security Groups
To setup Google GCP firewall rules refer to – Creating GCP Firewalls
Disclaimer: KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by Cloud Infrastructure Services. Cloud Infrastructure Services has no affiliation with and is not endorsed by The Apache Software Foundation.
