大数据运维(41)kafka 2.3.0集群部署
安装kafka之前先安装zookeeper。
配置DNS解析:
1
2
3
4
5
6
7
8
9
10
|
cat > /etc/hosts <<EOF
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.0.51 zookeeper1 192.168.0.52 zookeeper2 192.168.0.53 zookeeper3 192.168.0.57 kafka1 192.168.0.58 kafka2 192.168.0.59 kafka3 EOF |
配置免秘钥:
1
2
3
4
|
ssh -keygen
ssh -copy- id -i . ssh /id_rsa .pub root@kafka1
ssh -copy- id -i . ssh /id_rsa .pub root@kafka2
ssh -copy- id -i . ssh /id_rsa .pub root@kafka3
|
安装java:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
tar -xf jdk-8u231-linux-x64. tar .gz -C /usr/local/
cd /usr/local/
ln -sv jdk1.8.0_231/ jdk
cat > /etc/profile .d /java .sh <<EOF
export JAVA_HOME= /usr/local/jdk
export JRE_HOME=\$JAVA_HOME /jre
export CLASSPATH=.:\$JAVA_HOME /lib/dt .jar:\$JAVA_HOME /lib/tools .jar:\$JRE_HOME /lib
export PATH=\$PATH:\$JAVA_HOME /bin :\$JRE_HOME /bin
EOF . /etc/profile .d /java .sh
java -version javac -version |
下载安装包:包的前面是对应scala的版本后面的2.3.0才是kafka的版本。
1
|
wget http: //mirrors .tuna.tsinghua.edu.cn /apache/kafka/2 .3.0 /kafka_2 .12-2.3.0.tgz
|
复制安装包:
1
|
scp kafka_2.12-2.3.0.tgz root@kafka1: /root
|
解压安装:
1
2
3
|
tar -xf kafka_2.12-2.3.0.tgz -C /usr/local/
cd /usr/local/
ln -sv kafka_2.12-2.3.0/ kafka
|
设置环境变量:
1
2
3
4
5
|
cat > /etc/profile .d /kafka .sh <<EOF
KAFKA_HOME= /usr/local/kafka
PATH=\$PATH:\$KAFKA_HOME /bin
EOF . /etc/profile .d /kafka .sh
|
普通用户环境变量:
1
2
3
4
5
|
cat >> ~/.bashrc <<EOF
export KAFKA_HOME= /usr/local/kafka
export PATH=\$PATH:\$KAFKA_HOME /bin
EOF . ~/.bashrc |
创建日志目录:
1
|
mkdir /usr/local/kafka/logs
|
修改配置:
1
2
|
cd /usr/local/kafka/config
vim server.properties |
kafka1的server.properties:不是追加是修改如下几项
1
2
3
4
|
broker. id =0
listeners=PLAINTEXT: //kafka1 :9092
log. dirs = /usr/local/kafka/logs
zookeeper.connect=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 |
kafka2的server.properties:
1
2
3
4
|
broker. id =1
listeners=PLAINTEXT: //kafka2 :9092
log. dirs = /usr/local/kafka/logs
zookeeper.connect=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 |
kafka3的server.properties:
1
2
3
4
|
broker. id =2
listeners=PLAINTEXT: //kafka3 :9092
log. dirs = /usr/local/kafka/logs
zookeeper.connect=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 |
修改属组属主:
1
|
chown -R hadoop.hadoop /usr/local/kafka/ /usr/local/kafka
|
启动kafka:到各节点分别启动kafka。
1
|
. /kafka-server-start .sh .. /config/server .properties &
|
查看运行的进程:
1
2
3
|
]$ jps 28493 Kafka ... |
停止kafka:
1
2
|
cd /usr/local/kafka/bin/
. /kafka-server-stop .sh
|