How to Configure an Apache Kafka Cluster on Ubuntu 16.04

  1. Support for parallel data load into Hadoop.
  2. High throughput, supporting hundreds of thousands of messages per second, even with modest hardware.
  3. Persistent messaging with O(1) disk structures that provide constant time performance, even with terabytes of stored messages.
  4. The distributed system scales easily with no downtime.

Prerequisites

  1. A fresh Alibaba cloud instance with Ubuntu 16.04 server installed.
  2. A static IP address 192.168.0.103 is configured on the instance.
  3. A Root password is setup on the server.

Launch an Alibaba Cloud ECS Instance

First, login to your Alibaba Cloud ECS Console. Create a new ECS instance choosing Ubuntu 16.04 as the operating system with at least 2GB RAM. Connect to your ECS instance and log in as the root user.

apt-get update -y

Install Java

Apache Kafka needs a Java runtime environment, so you will need to install the latest version of Java to your system. By default, the latest version of the java is not available in Ubuntu 16.04 repository. So, you will need to add Java repository to your system. You can do this by running the following command:

add-apt-repository ppa:webupd8team/java
apt-get install oracle-java8-installer -y
java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

Install ZooKeeper

Apache Kafka depends on ZooKeeper for maintaining configuration information, providing distributed synchronization, and naming and providing group services. So, you will need to install ZooKeeper to your system. You can install it by running the following command:

apt-get install zookeeperd -y
netstat -nlpt | grep ':2181'
tcp6       0      0 :::2181                 :::*                    LISTEN

Install Apache Kafka

First, you will need to download the latest version of the Kafka from the Apache website. You can download it by running the following command:

wget  http://redrockdigimark.com/apachemirror/kafka/1.1.0/kafka_2.12-1.1.0.tgz
tar -xvzf kafka_2.12-1.1.0.tgz
cp -r kafka_2.12-1.1.0 /opt/Kafka
/opt/Kafka/bin/kafka-server-start.sh /opt/Kafka/config/server.properties
[2018-05-20 08:13:54,271] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2018-05-20 08:13:54,449] INFO Kafka version : 1.1.0 (org.apache.kafka.common.utils.AppInfoParser)
[2018-05-20 08:13:54,461] INFO Kafka commitId : fdcf75ea326b8e07 (org.apache.kafka.common.utils.AppInfoParser)
[2018-05-20 08:13:54,466] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Test Apache Kafka

Now, create your first topic named Topic1 with a single partition and only one replica by running the following command:

/opt/Kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1  --partitions 1 --topic Topic1
Created topic "Topic1".
/opt/Kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181
Topic1
/opt/Kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Topic1
/opt/Kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic Topic1 --from-beginning
Hello Kafka
How R You
Ok

Related Alibaba Cloud Products

Auto Scaling is a service to automatically adjust computing resources based on your volume of user requests. When demand for computing resources increase, Auto Scaling automatically adds ECS instances to serve additional user requests, or alternatively removes instances in the case of decreased user requests. This service is available free of cost. You will be charged only for the standard cost of adding additional ECS resources.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com