The Cluster ID XXXXXXXXXXXX doesn't match stored clusterId Some - apache-kafka

Question

On two nodes Kafka cluster after restart one node I have get error:
ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID hJGLXQwERc2rTXI_dWLwcg doesn't match stored clusterId Some(9VcFGDwSfke5JWuzA6f9mw) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:223)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Where hJGLXQwERc2rTXI_dWLwcg - ID of worked cluster from meta.properties
9VcFGDwSfke5JWuzA6f9mw - new generated ID.
Kafka version 3.0.0
On Node1 :
server.properties:
broker.id=1
listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://loc-kafka1:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=loc-kafka1:2181, loc-kafka2:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
compression.type=gzip
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
ssl.endpoint.identification.algorithm=
super.users=User:admin
sasl.jaas.config=/opt/kafka/config/kafka_server_jaas.conf
delete.topic.enable=true
zookeeper.properties
dataDir=/opt/kafka/zookeeper/data
clientPort=2181
maxClientCnxns=10
admin.enableServer=false
server1=loc-kafka1:2888:3888
server2=loc-kafka2:2888:3888
On Node2 the same config except broker.id=2 and myid file in dataDir.
All information what I can find sad : you should place log.dirs and dataDir not in /tmp folder. And all will done. But it's not like that.
One solution is clear log.dirs and dataDir folders allow this node connect to cluster. But this doesn't seem good solution.
If I just delete meta.properties, new cluster.id is plased in it and node don't connect to cluster. At the same time, in server.logs, the following lines show that the node is connecting to another cluster node:
INFO Opening socket connection to server loc-kafka1/10.1.1.3:2181. (org.apache.zookeeper.ClientCnxn)
INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
INFO Socket connection established, initiating session, client: loc-kafka2/10.1.1.4:38672, server: loc-kafka1/10.1.1.3:2181 (org.apache.zookeeper.ClientCnxn)
INFO Session establishment complete on server loc-kafka1/10.0.0.3:3001, session id = 0x100000043bb0006, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
What I can do to fix it?