Apache Kafka is an open source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Follow the steps below to setup Apache Kafka Server in the Cloud on Azure, AWS or GCP. Deploy using the public image available in the marketplaces.
Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.
You will now need to open another terminal session to complete the next steps.
Create a Kafka topic to store your events
Kafka is a distributed event streaming platform that lets you read, write, store, and process events (also called records or messages in the documentation) across many machines.
Example events are payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements from IoT devices or medical equipment, and much more. These events are organized and stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.
So before you can write your first events, you must create a topic. Open another terminal session and run:
First change your current path to the Kafka installation directory:
All of Kafka’s command line tools have additional options: run the ./kafka-topics.sh command in the bin directory without any arguments to display usage information. For example, it can also show you details such as the partition count of the new topic:
Now that you have a Kafka Topic created, we can now setup some events into the Topic. A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events. Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you need, even forever.
Run the console producer client to write a few events into your topic. By default, each line you enter will result in a separate event being written to the topic.
You should get an output of your events like this:
This is my first test event
This is my second test event
Press Ctrl+C to stop the consumer client.
Kafka Documentation
To learn more, we suggest the following next steps:
Read through the brief Introduction to learn how Kafka works at a high level, its main concepts, and how it compares to other technologies. To understand Kafka in more detail, head over to the Documentation.
Browse through the Use Cases to learn how other users in our world-wide community are getting value out of Kafka.
Disclaimer: KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by Cloud Infrastructure Services. Cloud Infrastructure Services has no affiliation with and is not endorsed by The Apache Software Foundation.
Cloud Solution Architect. Helping customers transform their business to the cloud. 20 years experience working in complex infrastructure environments and a Microsoft Certified Solutions Expert on everything Cloud.