Skip to main content

docker 安装 kafka

kafka docker-compose​

注意此配置仅为开发测试使用, 对外网暴露, 且无密码和加密等安全措施

version: "2"

services:
  kafka:
    image: docker.io/bitnami/kafka:3.5
    ports:
      - "9092:9092"
      - "9094:9094"
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://kafka.gee.cool:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT

如果需要外网访问, 需要配置KAFKA_CFG_ADVERTISED_LISTENERS的EXTERNAL为部署所在机器的ip/hostname.

https://hub.docker.com/r/bitnami/kafka/

bitnami kafka properties: https://github.com/bitnami/charts/tree/main/bitnami/kafka

基础操作​

bitnami的kafka镜像, bin位置在/opt/bitnami/kafka.

docker exec -it kafka_kafka_1 bash
cd /opt/bitnami/kafka/

创建topic: 

bin/kafka-topics.sh   --bootstrap-server localhost:9092 --create  --replication-factor 1 --partitions 1 --topic test

查看topics列表

bin/kafka-topics.sh   --list   --bootstrap-server localhost:9092

远程测试连接检查kafka​

可以利用bitami的kafka镜像快速连接访问kafka server.

docker run -it --rm bitnami/kafka:latest kafka-topics.sh --list  --bootstrap-server kafka.gee.cool:9094

errors​

error: java.net.UnknownHostException: kafka​

 docker run -it --rm bitnami/kafka:latest kafka-topics.sh --list  --bootstrap-server kafka.gee.cool:9092

内部访问的域名配置的是kafka, 导致docker外访问9002端口直接报错java.net.UnknownHostException: kafka: Name or service not known

ADVERTISED_LISTENERS, 其实就是外部能识别的ip和port地址, 因此如果要宿主机能访问, 估计需要填写这个pod的内网ip才行.

- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://kafka.gee.cool:9094
[2023-07-10 06:26:55,091] WARN [AdminClient clientId=adminclient-1] Error connecting to node kafka:9092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka: Name or service not known
    at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:510)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1030)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1141)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1401)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344)
    at java.base/java.lang.Thread.run(Thread.java:833)

在宿主机访问localhost:9092, 也会报错, 因为宿主机的localhost是宿主机, 并不是docker的ip.

 docker run -it --rm bitnami/kafka:latest kafka-topics.sh --list  --bootstrap-server localhost:9092
kafka 06:31:49.48
kafka 06:31:49.48 Welcome to the Bitnami kafka container
kafka 06:31:49.48 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 06:31:49.48 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 06:31:49.48

[2023-07-10 06:31:50,801] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-07-10 06:31:50,910] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-07-10 06:31:51,012] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-07-10 06:31:51,215] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-07-10 06:31:51,617] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

error:错误配置对外开放导致无法启动docker​

先前看文档对如何配置对外访问不清楚, 在KAFKA_CFG_ADVERTISED_LISTENERS中错误配置了EXTERNAL://0.0.0.0:9094, 导致kafka一直无法启动, 浪费了好多时间...

services:
  kafka:
    image: docker.io/bitnami/kafka:3.5
    ports:
      - "9092:9092"
      - "9094:9094"
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://0.0.0.0:9094
      - ADVERTISED_HOST_NAME=10.0.8.16
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://0.0.0.0:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT

error

Recreating kafka_kafka_1 ... done
Attaching to kafka_kafka_1
kafka_1  | kafka 04:29:11.68
kafka_1  | kafka 04:29:11.68 Welcome to the Bitnami kafka container
kafka_1  | kafka 04:29:11.69 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka_1  | kafka 04:29:11.69 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka_1  | kafka 04:29:11.69
kafka_1  | kafka 04:29:11.70 INFO  ==> ** Starting Kafka setup **
kafka_1  | kafka 04:29:11.76 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka_1  | kafka 04:29:11.76 INFO  ==> Initializing Kafka...
kafka_1  | kafka 04:29:11.77 INFO  ==> No injected configuration files found, creating default config files
kafka_1  | kafka 04:29:11.90 INFO  ==> Configuring Kafka for external client communications with PLAINTEXT authentication.
kafka_1  | kafka 04:29:11.90 WARN  ==> External client communications are configured using PLAINTEXT listeners. For safety reasons, do not use this in a production environment.
kafka_1  | kafka 04:29:11.90 INFO  ==> Initializing KRaft...
kafka_1  | kafka 04:29:11.90 WARN  ==> KAFKA_KRAFT_CLUSTER_ID not set - If using multiple nodes then you must use the same Cluster ID for each one
kafka_1  | kafka 04:29:13.51 INFO  ==> Generated Kafka cluster ID 'eC22NLHzTyG926ilAFrDyA'
kafka_1  | kafka 04:29:13.51 INFO  ==> Formatting storage directories to add metadata...
kafka_kafka_1 exited with code 1

https://hub.docker.com/r/bitnami/kafka/

Note: To connect from an external machine, change localhost above to your host's external IP/hostname and include EXTERNAL://0.0.0.0:9093 in KAFKA_CFG_LISTENERS to allow for remote connections.

kafka server关闭报错 Connection to node -1​

如果kafka server关闭, 远程连接报错为 Connection to node -1


docker run -it --rm bitnami/kafka:latest kafka-topics.sh --list  --bootstrap-server kafka.gee.cool:9094


kafka 03:57:39.33
kafka 03:57:39.33 Welcome to the Bitnami kafka container
kafka 03:57:39.33 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 03:57:39.33 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 03:57:39.34

[2023-07-10 03:57:40,811] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka.gee.cool/43.136.75.92:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-07-10 03:57:42,036] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka.gee.cool/43.136.75.92:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-07-10 03:57:42,259] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka.gee.cool/43.136.75.92:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-07-10 03:57:42,671] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka.gee.cool/43.136.75.92:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

9094远程端口被防火墙关闭​

如果端口在server端机器没有开放, 则外部机器连接的报错为Timed out waiting for a node assignment.

docker run -it --rm bitnami/kafka:latest kafka-topics.sh --list  --bootstrap-server kafka.gee.cool:9094

kafka 03:38:11.83
kafka 03:38:11.83 Welcome to the Bitnami kafka container
kafka 03:38:11.83 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 03:38:11.83 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 03:38:11.83

Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2023-07-10 03:39:13,129] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
 (kafka.admin.TopicCommand$)

其他tips​

kafka docker 网络配置​

https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

There are three main requirements for configuring Kafka networking.

  • Each Broker must be able to talk to Zookeeper - for leader election etc.
  • Each Broker must be able to talk to every other Broker - for replication etc.
  • Each Consumer/Producer must be able to talk to every Broker - for reading/writing data etc.
  • The following diagram represents the different communication paths:

picture 0

This means for a complete working Kafka setup, each one of the components must be able to route to the other and have accessible ports.