Trying to run Apache/NIFI on Zookeeper from Confluent - apache-zookeeper

Question

I am trying to run Apache/NIFI on confluent-zookeeper. NIFI ver 1.11.3 installed in /opt/nifi by unpacking tar container, confluent is community edition, ver 5.3. installed using confluent repo https://packages.confluent.io/rpm/5.3.
So NIFI works using integrated zookeper, NIFI works if I download zookeeper separatly from Apache/zookeeper site. Confluent Kafka also works with separate zookeeper and NIFI-integrated. BUT I cannot make it works using zookeeper from confluent.
In logs I see only one warning which is:
WARN Received packet at server of unknown type 15 (org.apache.zookeeper.server.ZooKeeperServer)
My config file for all three zookeepers are the same:
tickTime=2000
dataDir=/var/lib/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=myhost1:2888:3888
server.2=myhost2:2888:3888
server.3=myhost3:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
I do not think that Confluent really changed smth in their zookeeper. What could be the reason of this error?

Answer 1

As #BryanBende said:
NiFi 1.11.x (In our particular case, 1.11.4) requires ZK 3.5, please
confirm the version of ZK used in Confluent platform, IF ITS 3.4 THEN
ITS NOT GOING TO WORK – Bryan Bende Mar 27 at 14:35
The typical error you will see in zookeeper log:
Oct 08 17:22:23 some-pro-zk03 zookeeper-server-start[14136]:
[2020-10-08 17:22:23,275] INFO Accepted socket connection from
/10.10.10.1:53794 (org.apache.zookeeper.server.NIOServerCnxnFactory)
Oct 08 17:22:23 some-pro-zk03 zookeeper-server-start[14136]:
[2020-10-08 17:22:23,275] INFO Refusing session request for client
/10.10.10.1:53794 as it hasseen zxid 0x400000000 our last zxid is
0x300000004 client must try another server
(org.apache.zookeeper.server.ZooKeeperServer)
Oct 08 17:22:23
some-pro-zk03 zookeeper-server-start[14136]: [2020-10-08 17:22:23,275]
INFO Closed socket connection for client /10.159.164.93:53794 (no
session established for client)
(org.apache.zookeeper.server.NIOServerCnxn)
The typical error you will see in client log:
2020-10-07 16:00:09,112 ERROR [Curator-Framework-0]
o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave
uporg.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLossat
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)at
org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862)at
org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990)at
org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)at
org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)at
org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)at
java.util.concurrent.FutureTask.run(FutureTask.java:266)at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at
java.lang.Thread.run(Thread.java:748)
This takes a full day of work (reading logs!!) to find the error.
So, check your zookeeper version:
FOR Zookeeper 3.5+
echo srvr | nc localhost 2181
FOR Zookeeper 3.5<
echo stats | nc localost 2181
Also you can use telnet
FOR Zookeeper 3.5+
telnet localhost 2181
srvr
FOR Zookeeper 3.5<
telnet localhost 2181
stats