How to Setup Apache Kafka Server on Azure/AWS/GCP

Apache Kafka is an open source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.  Follow the steps below to setup Apache Kafka Server in the Cloud on Azure, AWS or GCP. Deploy using the public image available in the marketplaces.

Kafka Cloud

Kafka on Azure

Kafka on AWS

Kafka on Google Cloud

Getting Started with Apache Kafka

Once your Apache Kafka server has been deployed, the following links explain how to connect to a Linux VM:

 

 

Once connected and logged in, the following section explains how to start using Kafka.

Kafka Installation Directory

For reference, the installation of Apache Kafka is in the following directory:

 

/opt/kafka/

Start the Kafka Environment

Run the following commands in order to start all services in the correct order:

First change your current path to the Kafka installation directory:

				
					cd /opt/kafka/
				
			

Run the following command to start the ZooKeeper service.

				
					sudo bin/zookeeper-server-start.sh config/zookeeper.properties
				
			

You will now need to open another terminal session to complete the next steps.

Start the Kafka Broker Service

Run the following command to start the Kafka broker service:

First change your current path to the Kafka installation directory:

				
					cd /opt/kafka/
				
			
				
					sudo bin/kafka-server-start.sh config/server.properties
				
			

Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.

You will now need to open another terminal session to complete the next steps.

Create a Kafka topic to store your events

Kafka is a distributed event streaming platform that lets you read, write, store, and process events (also called records or messages in the documentation) across many machines.

 

Example events are payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements from IoT devices or medical equipment, and much more. These events are organized and stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.

 

So before you can write your first events, you must create a topic. Open another terminal session and run:

First change your current path to the Kafka installation directory:

				
					cd /opt/kafka/
				
			
				
					sudo bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
				
			

All of Kafka’s command line tools have additional options: run the ./kafka-topics.sh command in the bin directory without any arguments to display usage information. For example, it can also show you details such as the partition count of the new topic:

				
					sudo bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
				
			

The output should look like this:

				
					Topic:quickstart-events  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: quickstart-events Partition: 0    Leader: 0   Replicas: 0 Isr: 0
				
			

Write Events into the Kafka Topic

Now that you have a Kafka Topic created, we can now setup some events into the Topic.  A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events. Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you need, even forever.

 

Run the console producer client to write a few events into your topic. By default, each line you enter will result in a separate event being written to the topic.

				
					sudo bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
				
			
				
					This is my first test event
This is my second test event
				
			

You can stop the producer client with Ctrl-C at any time.

Read the Kafka Events

Open another terminal session and run the console consumer client to read the events you just created:

				
					sudo bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
				
			

You should get an output of your events like this:

				
					This is my first test event
This is my second test event
				
			

Press Ctrl+C to stop the consumer client.

 

Kafka Documentation

To learn more, we suggest the following next steps:

 

  • Read through the brief Introduction to learn how Kafka works at a high level, its main concepts, and how it compares to other technologies. To understand Kafka in more detail, head over to the Documentation.
  • Browse through the Use Cases to learn how other users in our world-wide community are getting value out of Kafka.

Kafka Firewall Ports

By default Kafka uses the following ports:

Kafka Service Ports
ServiceServersDefault PortDefault Ambari PortProtocolDescriptionNeed End User Access?Configuration Parameters
KafkaKafka Server90926667TCPThe port for Kafka server. 

 

Ref: Configuring Kafka Ports

 

If you are using any of the cloud security groups and need to change / add ports refer to the following guides:

 

To setup AWS firewall rules refer to – AWS Security Groups

To setup Azure firewall rules refer to – Azure Network Security Groups

To setup Google GCP firewall rules refer to – Creating GCP Firewalls

Disclaimer: KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by Cloud Infrastructure Services. Cloud Infrastructure Services has no affiliation with and is not endorsed by The Apache Software Foundation.

Avatar for Andrew Fitzgerald
Andrew Fitzgerald

Cloud Solution Architect. Helping customers transform their business to the cloud. 20 years experience working in complex infrastructure environments and a Microsoft Certified Solutions Expert on everything Cloud.

5 2 votes
Article Rating
Subscribe
Notify of
0 Comments
Most Voted
Newest Oldest
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x