I launch kafkadrop to connect to a kafka cluster running in confluent cloud:
java -jar kafdrop-3.30.0.jar
---kafka.brokerConnect=.......confluent.cloud:9092
I see error:
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
, message=Request processing failed; nested exception is kafdrop.service.KafkaAdminClientException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes, path=/}
2022-07-31 11:25:45.662 INFO 44652 [ionShutdownHook] i.u.Undertow : stopping server: Undertow - 2.2.16.Final
2022-07-31 11:25:45.672 INFO 44652 [ionShutdownHook] i.u.s.s.ServletContextImpl : Destroying Spring FrameworkServlet 'dispatcherServlet'
What do I miss here?
Related
I have Spring Boot application running on Kubernetes connected to Mongo database (deployed with Mongo Operator)
Sometimes - I can't see a pattern so far - the connection between Spring application & MongoDB breaks. After some time connection is restored but this may cause an error to the user.
Logs from application:
12:00:32.606 INFO org.mongodb.driver.connection : Closed connection [connectionId{localValue:3, serverValue:17292}] to kompas2mongo-0.kompas2mongo-svc.dev.svc.cluster.local:27017 because there was a socket exception raised by this connection.
12:00:32.607 INFO org.mongodb.driver.cluster : No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=kompas2mongo-0.kompas2mongo-svc.dev.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}]}. Waiting for 30000 ms before timing out
12:00:32.612 INFO org.mongodb.driver.cluster : Exception in monitor thread while connecting to server kompas2mongo-0.kompas2mongo-svc.dev.svc.cluster.local:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70)
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:143)
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:188)
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:144)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:589)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333)
at java.base/java.net.Socket.connect(Socket.java:648)
at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:107)
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79)
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65)
... 4 common frames omitted
12:00:42.808 INFO org.mongodb.driver.cluster : Cluster description not yet available. Waiting for 30000 ms before timing out
12:01:02.612 ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/api] threw exception [Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Prematurely reached end of stream; nested exception is com.mongodb.MongoSocketReadException: Prematurely reached end of stream] with root cause
com.mongodb.MongoSocketReadException: Prematurely reached end of stream
Logs from MongoDB:
2022-08-21T12:00:33.583Z INFO controllers/replica_set_controller.go:132 Reconciling MongoDB {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.583Z DEBUG controllers/replica_set_controller.go:134 Validating MongoDB.Spec {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.583Z DEBUG controllers/replica_set_controller.go:143 Ensuring the service exists {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.600Z INFO controllers/replica_set_controller.go:390 The service already exists... moving forward: services "kompas2mongo-svc" already exists {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.600Z INFO controllers/replica_set_controller.go:308 Creating/Updating AutomationConfig {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.628Z INFO agent/agent_readiness.go:52 All 1 Agents have reached Goal state {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.628Z INFO controllers/replica_set_controller.go:288 Creating/Updating StatefulSet {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.639Z DEBUG controllers/replica_set_controller.go:298 Ensuring StatefulSet is ready, with type: RollingUpdate {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.639Z INFO controllers/mongodb_status_options.go:110 ReplicaSet is not yet ready, retrying in 10 seconds
2022-08-21T12:00:34.612Z INFO controllers/replica_set_controller.go:132 Reconciling MongoDB {"ReplicaSet": "dev/kompas2mongo"}
Mongo is configured as a Replica Set and application.yaml looks like this:
spring.data.mongodb:
host: ${MONGO_HOST}
database: ${MONGO_DATABASE}
username: ${MONGO_USERNAME}
password: ${MONGO_PASSWORD}
In documentation there is different connection string for replica set. Could this be the issue? Or do you have other ideas?
I'm trying to do change data capture with Debezium using Postgres, Kafka, Kafka connect and debezium Postgres connector. Having an issue when trying to start Kafka Connect service with Debezium-Postgres-connector.
This is the plugin.path in my config/connect-standalone.properties:
plugin.path=/opt/kafka/kafka_2.13-3.1.0/connect/debezium-connector-postgres/
The connect-debezium-postgres.properties file:
name=first-connector
connector.class=io.debezium.connector.postgresql.PostgresConnector
database.hostname=postgres
database.port=5432
database.user=postgres
database.password=password
database.server.id=1
database.server.name=bankserver1
database.include.list=bank
table.inlcude.list=bank.holding
database.history.kafka.bootstrap.servers=localhost:9092
database.history.kafka.topic=dbhistory.test
include.schema.changes=true
tombstones.on.delete=false
The command when starting Kafka Connect service with Debezium-Postgres-Connector:
bin/connect-standalone.sh config/connect-standalone.properties config/connect-debezium-postgres.properties
The error Kafka Connect service:
[2022-02-22 10:41:50,571] ERROR Failed to create job for config/connect-debezium-postgres.properties (org.apache.kafka.connect.cli.ConnectStandalone:107)
[2022-02-22 10:41:50,581] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:117)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
A value is required
You can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:115)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:99)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:114)
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
A value is required
You can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`
at org.apache.kafka.connect.runtime.AbstractHerder.maybeAddConfigErrors(AbstractHerder.java:691)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:207)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.lambda$null$0(StandaloneHerder.java:193)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
[2022-02-22 10:41:50,597] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:67)
[2022-02-22 10:41:50,597] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:311)
[2022-02-22 10:41:50,630] INFO Stopped http_8083#26be6ca7{HTTP/1.1, (http/1.1)}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:381)
[2022-02-22 10:41:50,631] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:149)
[2022-02-22 10:41:50,646] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:328)
[2022-02-22 10:41:50,646] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:106)
[2022-02-22 10:41:50,649] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:199)
[2022-02-22 10:41:50,650] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:66)
[2022-02-22 10:41:50,650] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics:659)
[2022-02-22 10:41:50,651] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics:663)
[2022-02-22 10:41:50,651] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics:669)
[2022-02-22 10:41:50,652] INFO App info kafka.connect for 10.0.2.15:8083 unregistered (org.apache.kafka.common.utils.AppInfoParser:83)
[2022-02-22 10:41:50,652] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:220)
[2022-02-22 10:41:50,663] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:124)
[2022-02-22 10:41:50,664] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:7
the content of the folders that put in the plugin path
I have Artemis configuration (shared storage) with following ha-policy (for master and backup):
<ha-policy>
<shared-store>
<master>
<failover-on-shutdown>true</failover-on-shutdown>
</master>
</shared-store>
</ha-policy>
<ha-policy>
<shared-store>
<slave>
<failover-on-shutdown>true</failover-on-shutdown>
</slave>
</shared-store>
</ha-policy>
Client connection string:
(tcp://master:61616,tcp://backup:61616)?ha=true&retryInterval=1000&retryIntervalMultiplier=1.0&reconnectAttempts=10
At ~18:38 the server crashed, then at ~18:48 it recovered.
Some applications were unable to reconnect correctly without restarting with following errors...
APP 1
Master node crash:
2020-08-06 18:38:37,873 [Thread-0 (ActiveMQ-client-global-threads)] WARN org.apache.activemq.artemis.core.client - AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
Master Node now active (the backup went into passive mode):
2020-08-06 18:47:50,949 [Thread-1 (ActiveMQ-client-global-threads)] WARN org.apache.activemq.artemis.core.client - AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
Why? Other applications reconnected correctly
2020-08-06 18:47:59,994 [Thread-1 (ActiveMQ-client-global-threads)] WARN org.apache.activemq.artemis.core.client - AMQ212005: Tried 10 times to connect. Now giving up on reconnecting it.
2020-08-06 18:47:59,998 [Camel (camel-1) thread #4 - JmsConsumer[xxx]] WARN org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'xxx' - trying to recover. Cause: Session is closed
2020-08-06 18:47:59,999 [Camel (camel-1) thread #4 - JmsConsumer[xxx]] INFO org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Successfully refreshed JMS Connection
2020-08-06 18:48:00,006 [Camel (camel-1) thread #3 - JmsConsumer[xxx]] WARN org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'xxx' - trying to recover. Cause: Session is closed
This error this error was not fixed although the cluster was recovered:
2020-08-06 18:49:25,033 [Camel (camel-1) thread #5 - JmsConsumer[xxx]] INFO org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Successfully refreshed JMS Connection
2020-08-06 18:49:25,033 [Camel (camel-1) thread #7 - JmsConsumer[xxx]] WARN org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'xxx' - trying to recover. Cause: AMQ119010: Connection is destroyed
APP 2
Master node crash:
2020-08-06 18:38:37.883 WARN 1 --- [Thread-1 (ActiveMQ-client-global-threads)] org.apache.activemq.artemis.core.client : AMQ212037: Connection failure to master/master:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-08-06 18:38:46.935 WARN 1 --- [Thread-1 (ActiveMQ-client-global-threads)] org.apache.activemq.artemis.core.client : AMQ212005: Tried 10 times to connect. Now giving up on reconnecting it.
2020-08-06 18:38:46.939 WARN 1 --- [DefaultMessageListenerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'yyyy' - trying to recover. Cause: Session is closed
2020-08-06 18:38:46.945 WARN 1 --- [DefaultMessageListenerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'yyyy' - trying to recover. Cause: Session is closed
2020-08-06 18:38:46.963 INFO 1 --- [Thread-7] o.s.j.c.SingleConnectionFactory : Encountered a JMSException - resetting the underlying JMS Connection
javax.jms.JMSException: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:750) ~[artemis-jms-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:771) ~[artemis-jms-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:704) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:640) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:507) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$600(ClientSessionFactoryImpl.java:73) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1229) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:77) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:220) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:220) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1018) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) ~[artemis-commons-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) ~[artemis-commons-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) ~[artemis-commons-2.10.1.jar!/:2.10.1]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.10.1.jar!/:2.10.1]
Caused by: org.apache.activemq.artemis.api.core.ActiveMQDisconnectedException: AMQ219015: The connection was disconnected because of server shutdown
... 7 common frames omitted
2020-08-06 18:38:51.945 WARN 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'xxx' - trying to recover. Cause: AMQ219010: Connection is destroyed
2020-08-06 18:39:21.965 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=0, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
2020-08-06 18:39:52.983 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyy' - retrying using FixedBackOff{interval=5000, currentAttempts=0, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
2020-08-06 18:40:23.992 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=1, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
Master Node now active (the backup went into passive mode):
2020-08-06 18:47:50.949 WARN 1 --- [Thread-5 (ActiveMQ-client-global-threads)] org.apache.activemq.artemis.core.client : AMQ212037: Connection failure to backup/backup:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-08-06 18:47:53.145 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=8, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
2020-08-06 18:47:53.146 WARN 1 --- [Thread-5 (ActiveMQ-client-global-threads)] org.apache.activemq.artemis.core.client : AMQ212004: Failed to connect to server.
This error this error was not fixed although the cluster was recovered:
2020-08-06 18:47:58.147 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=9, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ219024: Could not select a TransportConfiguration to create SessionFactory]
2020-08-06 18:48:53.160 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=20, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ219024: Could not select a TransportConfiguration to create SessionFactory]
I tried to reproduce the error but could not. Maybe there are ideas for the correct setting?
Kafka is throwing an exception while starting,
Timed out waiting for connection while in state: CONNECTING
Followed by null pointer exception.
I have increased the config/server.properties connection time out but still has the issue.
Here am using the windows Os.
Thanks
ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:230)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:226)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:95)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1580)
at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
[2018-09-04 23:31:35,864] INFO shutting down (kafka.server.KafkaServer)
After restarting the system it got started working.it did not work even changing the kafka server.properties, but its working now as my machine got started.
Thanks every one for helping me to debug.
I'm following this guide to configure messos 3 node master and 3 node slave cluster. However when I start master zookeepers I get following error log
2017-07-05 09:46:18,568 - INFO [main:FileSnap#83] - Reading snapshot /var/lib/zookeeper/version-2/snapshot.100000016
2017-07-05 09:46:18,606 - ERROR [main:FileTxnSnapLog#210] - Parent /mesos/log_replicas missing for /mesos/log_replicas/0000000002
2017-07-05 09:46:18,607 - ERROR [main:QuorumPeer#453] - Unable to load database on disk
java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /mesos/log_replicas
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /mesos/log_replicas
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
... 6 more
2017-07-05 09:46:18,610 - ERROR [main:QuorumPeerMain#89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:454)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: Failed to process transaction type: 1 error: KeeperErrorCode = NoNode for /mesos/log_replicas
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
... 4 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /mesos/log_replicas
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
... 6 more
When slaves are started obviously it cannot discover the masters since it cannot connect to zookeeper. Slaves gives this error
I0705 09:33:43.593530 25710 provisioner.cpp:410] Provisioner recovery complete
I0705 09:33:43.593668 25710 slave.cpp:5970] Finished recovery
W0705 09:33:53.529522 25717 group.cpp:494] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
I0705 09:33:53.530243 25717 group.cpp:510] ZooKeeper session expired
W0705 09:34:03.532635 25710 group.cpp:494] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
I0705 09:34:03.533331 25710 group.cpp:510] ZooKeeper session expired
Any ideas how to troubleshoot this.
Reinstalling master nodes solved the first problem.
Still I had the 2nd problem, where slaves could not find zookeeper. Documentation seems to indicate slaves could discover the master nodes. Was not working for me. However when I pointed zookeeper nodes in slaves in (/etc/mesos/zk) it started working