Kafka won't start again - apache-kafka

Question

Dears,
I do not have any idea what is wrong in my configuration. When I want to start Kafka service I've got an error:
[2020-11-30 08:52:33,502] DEBUG Created SSL context with keystore SecurityStore(path=/etc/pki/CA/certs/node1.corp.jks, modificationTime=Mon Nov 30 08:40:08 CET 2020), truststore SecurityStore(path=/etc/pki/ca-trust/extracted/java/cacerts, modificationTime=Wed Nov 25 09:20:22 CET 2020), provider SunJSSE. (org.apache.kafka.common.security.ssl.SslEngineBuilder)
[2020-11-30 08:52:33,702] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: org.apache.kafka.common.config.ConfigException: Invalid value javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings.
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:158)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:85)
at kafka.network.Processor.<init>(SocketServer.scala:753)
at kafka.network.SocketServer.newProcessor(SocketServer.scala:394)
at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:279)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:278)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:241)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:238)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:238)
at kafka.network.SocketServer.startup(SocketServer.scala:121)
at kafka.server.KafkaServer.startup(KafkaServer.scala:263)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.config.ConfigException: Invalid value javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings.
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:100)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:154)
... 18 more
I've got three files from other Team: CAroot certificate (PEM *.crt), Private key (PEM private *.pem) and certificate (PEM *.crt) for each node.
The keystore was create like this:
keytool -noprompt -keystore node1.corp.jks -alias rootca -import -file testcorp.crt -storepass kafka123
Next key was imported to truststore and keystore like this:
openssl pkcs12 -export -in node1.crt -inkey node1.pem -out node1.p12 -password pass:kafka123
keytool -noprompt -importkeystore -srckeystore node1.p12 -srcstoretype PKCS12 -destkeystore node1.corp.jks -dname "CN=node1, OU=ITC, O=ITC, L=CITY, ST=SOME, C=PL" -deststoretype JKS -storepass kafka123 -keypass kafka123 -keyalg RSA -validity 365
Kafka configuration file looks like:
broker.id=1
delete.topic.enable=true
auto.create.topics.enable=true
listeners=SASL_SSL://:9093
advertised_listeners=SASL_SSL://192.168.1.101:9093
ssl.endpoint.identification.algorithm=
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
security.inter.broker.protocol=SASL_SSL
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.secure.random.implementation=SHA1PRNG
super.users=User:admin
ssl.client.auth=none
ssl.keystore.location=/etc/pki/CA/certs/node1.corp.jks
ssl.keystore.password=kafka123
ssl.key.password=kafka123
ssl.truststore.location=/etc/pki/ca-trust/extracted/java/cacerts
ssl.truststore.password=kafka123
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
log.dirs=/kafka_data
num.partitions=1
num.recovery.threads.per.data.dir=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181
I do not have any idea what is wrong. Did you have similar problems? Do you have any idea what is wrong?
Best Regards,
Dan

Answer 1

Something changed in our DNS serwer. Recreation of keystores solved problems.