kafka + zookeeper 配置SASL
目录
Zk开启SASL
配置conf/jaas.conf
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
uorumServer { org.apache.zookeeper.server.auth.DigestLoginModule required user_qihoo="qihoo@360"; }; QuorumLearner { org.apache.zookeeper.server.auth.DigestLoginModule required username="test" password="123456"; }; Server { org.apache.zookeeper.server.auth.DigestLoginModule required user_kafka="123456"; }; |
配置conf/zoo.cfg
|
1 2 3 4 5 6 7 8 9 10 |
quorum.auth.enableSasl=true quorum.auth.learnerRequireSasl=true quorum.auth.serverRequireSasl=true quorum.auth.learner.saslLoginContext=QuorumLearner quorum.auth.server.saslLoginContext=QuorumServer quorum.cnxn.threads.size=6 requireClientAuthScheme=sasl authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider aasLoginRenew=3600000 |
配置conf/java.env
|
1 |
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/home/cloud/apps/zookeeper/conf/jaas.conf" |
重启zk服务
|
1 |
supervisorctl restart zookeeper |
Kafka
- SASL/PLAIN 需要配置在jaas中，使用 org.apache.kafka.common.security.plain.PlainLoginModule
- SASL/SCRAM 验证可以动态新增用户并分配权限，使用 org.apache.kafka.common.security.scram.ScramLoginModule
配置config/jaas.conf
- kafkaServer中的 username和 user_xxx 为 集群间通信的认证信息
- Client中的配置为 zk中的认证信息
|
1 2 3 4 5 6 7 8 9 10 11 |
KafkaServer { org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="kafka" user_admin="kafka"; }; Client { org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" password="123456"; }; |
配置config/server.properties
- advertised.listeners 改为对外提供服务的IP地址
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# 启用ACL authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer # 设置admin为超级用户 super.users=User:admin zookeeper.set.acl=true # 启用SCRAM机制，采用SCRAM-SHA-256算法 sasl.enabled.mechanisms=SCRAM-SHA-256 # 为broker间通讯开启SCRAM机制，采用SCRAM-SHA-256算法 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 # broker间通讯使用PLAINTEXT，为了提高性能在内网中不设置TLS security.inter.broker.protocol=SASL_PLAINTEXT # 禁止自动创建topic auto.create.topics.enable=false listeners=SASL_PLAINTEXT://0.0.0.0:9092 advertised.listeners=SASL_PLAINTEXT://172.16.100.132:9092 |
配置config/test.conf
|
1 2 3 |
security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="test" password="test"; |
配置config/admin.conf
|
1 2 3 |
security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="kafka"; |
配置bin/kafka-run-class.sh
- -Djava.security.auth.login.config 指向新增的jaas.conf
|
1 2 3 |
219 if [ -z "$KAFKA_OPTS" ]; then 220 KAFKA_OPTS="-Djava.security.auth.login.config=/home/cloud/apps/kafka/config/jaas.conf" 221 fi |
Kafka ACL设置
admin用户凭据
|
1 2 3 4 5 |
# 增加 admin用户 ./bin/kafka-configs.sh --zookeeper localhost:2181/kafka --alter --add-config 'SCRAM-SHA-256=[password=kafka],SCRAM-SHA-512=[password=kafka]' --entity-type users --entity-name admin # 查看用户 ./bin/kafka-configs.sh --zookeeper localhost:2181/kafka --describe --entity-type users --entity-name admin |
test用户凭据
|
1 2 3 4 5 |
# 增加 test用户 ./bin/kafka-configs.sh --zookeeper localhost:2181/kafka --alter --add-config 'SCRAM-SHA-256=[password=123456],SCRAM-SHA-512=[password=123456]' --entity-type users --entity-name test # 查看 test用户 ./bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-type users --entity-name consumer |
增加test用户的 写权限
|
1 |
./bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181/kafka --add --allow-principal User:test --operation Write --topic test --group test-group --allow-host '*' |
增加test用户的 读权限
|
1 |
./bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181/kafka --add --allow-principal User:test --operation Read --topic test --group test-group --allow-host '*' |
查看权限列表
|
1 |
./bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181/kafka --list |
测试
|
1 2 3 4 5 6 7 8 9 |
# 创建topic, 需要指定 admin.conf ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test --partitions 1 --replication-factor 1 --command-config config/admin.conf # 生产数据 ./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test --producer.config config/test.conf # 消费数据 ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --consumer.config config/test.conf |
错误排查
Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256
日志信息： ERROR [Controller id=2, targetBrokerId=2] Connection to node 2 (test2.devse.cn/172.16.100.132:9092) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (org.apache.kafka.clients.NetworkClient)
有未经验证的用户请求集群，可能的原因：
- 有客户端未提供认证信息
- jaas.conf中有错误，列如： username，password 和 user_xxx = <password> 不正确