ERROR Failed to create job for config/connect-debezium-postgres.properties (org.apache.kafka.connect.cli.ConnectStandalone:107) - apache-kafka

I'm trying to do change data capture with Debezium using Postgres, Kafka, Kafka connect and debezium Postgres connector. Having an issue when trying to start Kafka Connect service with Debezium-Postgres-connector.
This is the plugin.path in my config/connect-standalone.properties:
plugin.path=/opt/kafka/kafka_2.13-3.1.0/connect/debezium-connector-postgres/
The connect-debezium-postgres.properties file:
name=first-connector
connector.class=io.debezium.connector.postgresql.PostgresConnector
database.hostname=postgres
database.port=5432
database.user=postgres
database.password=password
database.server.id=1
database.server.name=bankserver1
database.include.list=bank
table.inlcude.list=bank.holding
database.history.kafka.bootstrap.servers=localhost:9092
database.history.kafka.topic=dbhistory.test
include.schema.changes=true
tombstones.on.delete=false
The command when starting Kafka Connect service with Debezium-Postgres-Connector:
bin/connect-standalone.sh config/connect-standalone.properties config/connect-debezium-postgres.properties
The error Kafka Connect service:
[2022-02-22 10:41:50,571] ERROR Failed to create job for config/connect-debezium-postgres.properties (org.apache.kafka.connect.cli.ConnectStandalone:107)
[2022-02-22 10:41:50,581] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:117)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
A value is required
You can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:115)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:99)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:114)
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
A value is required
You can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`
at org.apache.kafka.connect.runtime.AbstractHerder.maybeAddConfigErrors(AbstractHerder.java:691)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:207)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.lambda$null$0(StandaloneHerder.java:193)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
[2022-02-22 10:41:50,597] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:67)
[2022-02-22 10:41:50,597] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:311)
[2022-02-22 10:41:50,630] INFO Stopped http_8083#26be6ca7{HTTP/1.1, (http/1.1)}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:381)
[2022-02-22 10:41:50,631] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:149)
[2022-02-22 10:41:50,646] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:328)
[2022-02-22 10:41:50,646] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:106)
[2022-02-22 10:41:50,649] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:199)
[2022-02-22 10:41:50,650] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:66)
[2022-02-22 10:41:50,650] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics:659)
[2022-02-22 10:41:50,651] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics:663)
[2022-02-22 10:41:50,651] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics:669)
[2022-02-22 10:41:50,652] INFO App info kafka.connect for 10.0.2.15:8083 unregistered (org.apache.kafka.common.utils.AppInfoParser:83)
[2022-02-22 10:41:50,652] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:220)
[2022-02-22 10:41:50,663] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:124)
[2022-02-22 10:41:50,664] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:7
the content of the folders that put in the plugin path

Related

Snowflake kafka connector failing after initializing TEST connector

Not sure what is wrong. Trying to setup Snowflake kafka connect and it seems to be failing without throwing any useful logs
[2021-04-07 21:09:25,024] INFO Creating connector TEST_CONNECTOR of type com.snowflake.kafka.connector.SnowflakeSinkConnector (org.apache.kafka.connect.runtime.Worker:202)
[2021-04-07 21:09:25,028] INFO Instantiated connector TEST_CONNECTOR with version 1.5.0 of type class com.snowflake.kafka.connector.SnowflakeSinkConnector (org.apache.kafka.connect.runtime.Worker:205)
[2021-04-07 21:09:25,029] INFO
[SF_KAFKA_CONNECTOR] Snowflake Kafka Connector Version: 1.5.0 (com.snowflake.kafka.connector.Utils:99)
[2021-04-07 21:09:25,092] WARN
[SF_KAFKA_CONNECTOR] Connector update is available, please upgrade Snowflake Kafka Connector (1.5.0 -> 1.5.2) (com.snowflake.kafka.connector.Utils:136)
[2021-04-07 21:09:25,092] INFO
[SF_KAFKA_CONNECTOR] SnowflakeSinkConnector:start (com.snowflake.kafka.connector.SnowflakeSinkConnector:91)
[2021-04-07 21:09:25,330] INFO
[SF_KAFKA_CONNECTOR] initialized the snowflake connection (com.snowflake.kafka.connector.internal.SnowflakeConnectionServiceV1:38)
[2021-04-07 21:09:25,336] INFO Finished creating connector TEST_CONNECTOR (org.apache.kafka.connect.runtime.Worker:224)
[2021-04-07 21:09:25,337] INFO Skipping reconfiguration of connector sflksink since it is not running (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:285)
[2021-04-07 21:09:25,338] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:113)
java.lang.NullPointerException
at org.apache.kafka.connect.cli.ConnectStandalone$1.onCompletion(ConnectStandalone.java:104)
at org.apache.kafka.connect.cli.ConnectStandalone$1.onCompletion(ConnectStandalone.java:98)
at org.apache.kafka.connect.util.ConvertingFutureCallback.onCompletion(ConvertingFutureCallback.java:44)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:185)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:107)
[2021-04-07 21:09:25,340] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:65)
[2021-04-07 21:09:25,341] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:211)
[2021-04-07 21:09:25,345] INFO Stopped http_8083#2cc0fa2a{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:306)
[2021-04-07 21:09:25,354] INFO Stopped o.e.j.s.ServletContextHandler#5c83ae01{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:865)
[2021-04-07 21:09:25,360] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:222)
[2021-04-07 21:09:25,360] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:77)
[2021-04-07 21:09:25,360] INFO Stopping connector TEST_CONNECTOR (org.apache.kafka.connect.runtime.Worker:305)
[2021-04-07 21:09:25,361] INFO
[SF_KAFKA_CONNECTOR] SnowflakeSinkConnector:stop (com.snowflake.kafka.connector.SnowflakeSinkConnector:141)
[2021-04-07 21:09:25,362] INFO Stopped connector TEST_CONNECTOR (org.apache.kafka.connect.runtime.Worker:321)
[2021-04-07 21:09:25,362] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:151)
[2021-04-07 21:09:25,365] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:67)
[2021-04-07 21:09:25,365] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:172)
[2021-04-07 21:09:25,369] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:87)
[2021-04-07 21:09:25,371] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:70)
The config file looks like below
name=sflksink
connector.class=com.snowflake.kafka.connector.SnowflakeSinkConnector
tasks.max=1
topics=snowflake-connect-test
buffer.count.records=10
buffer.flush.time=60
buffer.size.bytes=50
snowflake.url.name=url
snowflake.user.name=<user>
snowflake.database.name=<database>
snowflake.schema.name=<schema>
snowflake.private.key=<private_key>
snowflake.warehouse.name=MY_WAREHOUSE
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=com.snowflake.kafka.connector.records.SnowflakeJsonConverter
Any pointers will be helpful
In the log message it says
Connector update is available, please upgrade Snowflake Kafka Connector (1.5.0 -> 1.5.2)
So I would suggest that you update your connector JAR to the latest version and try again.

To achieve Kafka-mule integration using Mule 4, I installed zookeeper&kafka on my machine. Both the servers connection goes off after sometime

I started by creating zookeeper and kafka servers respectively on my machine. The connection gets established for both the servers but goes off after a few mins, throwing some errors.
ZOOKEEPER ERROR LOG-
'''2020-05-11 14:32:36,908 [myid:] - INFO [SyncThread:0:FileTxnLog#284] - Creating new log file: log.1
2020-05-11 14:36:05,170 [myid:] - WARN [NIOWorkerThread-1:NIOServerCnxn#373] - Close of session 0x10000cc587b0003
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:324)
at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-05-11 14:36:10,589 [myid:] - INFO [SessionTracker:ZooKeeperServer#600] - Expiring session 0x10000cc587b0003, timeout of 6000ms exceeded
'''
KAFKA ERROR LOG - Below are the kafka server error logs## Heading ##
'''[2020-05-11 14:33:41,077] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2020-05-11 14:36:04,762] ERROR Error while writing to checkpoint file C:\Kafka\Kafkakafka-logs\replication-offset-checkpoint (kafka.server.LogDirFailureChannel)
java.nio.file.FileAlreadyExistsException: C:\Kafka\Kafkakafka-logs\replication-offset-checkpoint.tmp -> C:\Kafka\Kafkakafka-logs\replication-offset-checkpoint
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
.....
[2020-05-11 14:36:04,776] INFO [ReplicaManager broker=0] Stopping serving replicas in dir C:\Kafka\Kafkakafka-logs (kafka.server.ReplicaManager)
[2020-05-11 14:36:04,776] ERROR [ReplicaManager broker=0] Error while writing to highwatermark file in directory C:\Kafka\Kafkakafka-logs (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.KafkaStorageException: Error while writing to checkpoint file C:\Kafka\Kafkakafka-logs\replication-offset-checkpoint
Caused by: java.nio.file.FileAlreadyExistsException: C:\Kafka\Kafkakafka-logs\replication-offset-checkpoint.tmp -> C:\Kafka\Kafkakafka-logs\replication-offset-checkpoint
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
'''

org.apache.kafka.connect.runtime.rest.errors.BadRequestException

I am trying to write a kafka connector to move data that is in kafka topic into mongodb(sink). For i have added required configurations in connect-json-standalone.properties file and also in connect-mongo-sink.properties file in kafka folder. In this process while starting the connector I am getting below exception
[2019-07-23 18:07:17,274] INFO Started o.e.j.s.ServletContextHandler#76e3b45b{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:855)
[2019-07-23 18:07:17,274] INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer:231)
[2019-07-23 18:07:17,274] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:56)
[2019-07-23 18:07:17,635] INFO Cluster created with settings {hosts=[localhost:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500} (org.mongodb.driver.cluster:71)
[2019-07-23 18:07:17,636] INFO Adding discovered server localhost:27017 to client view of cluster (org.mongodb.driver.cluster:71)
[2019-07-23 18:07:17,760] INFO Closing all connections to repracli/localhost:27017 (io.debezium.connector.mongodb.ConnectionContext:86)
[2019-07-23 18:07:17,768] ERROR Failed to create job for ./etc/kafka/connect-mongodb-sink.properties (org.apache.kafka.connect.cli.ConnectStandalone:104)
[2019-07-23 18:07:17,769] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:115)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
A value is required
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:112)
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
A value is required
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.runtime.AbstractHerder.maybeAddConfigErrors(AbstractHerder.java:423)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:188)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:109)
[2019-07-23 18:07:17,782] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:66)
[2019-07-23 18:07:17,782] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:239)
[2019-07-23 18:07:17,790] INFO Stopped http_localhost8084#5f96f6a2{HTTP/1.1,[http/1.1]}{localhost:8084} (org.eclipse.jetty.server.AbstractConnector:341)
[2019-07-23 18:07:17,790] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:167)
[2019-07-23 18:07:17,792] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:256)
[2019-07-23 18:07:17,793] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:94)
[2019-07-23 18:07:17,793] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:185)
[2019-07-23 18:07:17,794] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:66)
[2019-07-23 18:07:17,796] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:206)
[2019-07-23 18:07:17,799] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:111)
[2019-07-23 18:07:17,800] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:71)
I have tried to solve it by changing connection.uri in connect-mongo-sink.properties in several ways which didn't worked out well. I have also googled some links that also didn't solved my problem.
referal_link : https://groups.google.com/forum/#!topic/debezium/bC4TUld5NGw
https://github.com/confluentinc/kafka-connect-jdbc/issues/334
connect-json-standalone.properties:
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schema.registry.url=http://localhost:8081
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
connect-mongo-sink.properties:
name=mongodb-sink-connector
connector.class=io.debezium.connector.mongodb.MongoDbConnector
tasks.max=1
topics=sample-consumerr-sink-topic
type.name=kafka-connect
mongodb.hosts=repracli/localhost:27017
mongodb.collection=conn_mongo_sink_collc
mongodb.connection.uri=mongodb://localhost:27017/conn_mongo_sink_db?w=1&journal=true
I want the sink connector to work inorder to consume topic data into mongodb collection name "conn_mongo_sink_collc". Can anyone help me how to resolve that error ?
Note : I am using 3 replicaSet mongodb in which port-27017 is primary , 27018-secondary, 27019-secondary.
io.debezium.connector.mongodb.MongoDbConnector is a Source connector, for getting data from MongoDB into Kafka.
To stream data from MongoDB into Kafka use a Sink connector. MongoDB recently launched their own sink connector and blogged about its use, including sample configuration.
Faced same issue. This can also happen when connector config is not complete or invalid.
check whether the necessary properties are set by looking debezium documentation

unable to start kafka server/broker

when starting the kafka broker i am getting some error:
I am giving last few lines of the error log:
INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2016-05-12 01:07:01,759] INFO Log directory '/var/logs/service-bridge-logs' not found, creating it. (kafka.log.LogManager)
[2016-05-12 01:07:01,778] INFO Loading logs. (kafka.log.LogManager)
[2016-05-12 01:07:01,796] INFO Logs loading complete. (kafka.log.LogManager)
[2016-05-12 01:07:01,797] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,806] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,874] INFO Awaiting socket connections on gns3-d.cloudapp.net:9092. (kafka.network.Acceptor)
[2016-05-12 01:07:01,875] INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer)
[2016-05-12 01:07:02,042] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-05-12 01:07:02,168] INFO 2 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-05-12 01:07:02,386] INFO Registered broker 2 at path /brokers/ids/2 with address 10.1.0.4:9092. (kafka.utils.ZkUtils$)
[2016-05-12 01:07:02,416] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
[2016-05-12 01:07:02,529] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-05-12 01:07:25,798] ERROR Closing socket for /40.122.64.23 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at kafka.utils.Utils$.read(Utils.scala:380)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
while in zookeeper sidealso am getting few error:
INFO Established session 0x154a35b64f40000 with negotiated timeout 6000 for client /10.1.0.4:36673 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:07:02,313] INFO Got user-level KeeperException when processing sessionid:0x154a35b64f40000 type:delete cxid:0x1d zxid:0x52 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,001] INFO Expiring session 0x154a35b64f40000, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:08:33,001] INFO Processed session termination for sessionid: 0x154a35b64f40000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,017] INFO Closed socket connection for client /10.1.0.4:36673 which had sessionid 0x154a35b64f40000 (org.apache.zookeeper.server.NIOServerCnxn)
Any idea guys??
Thanks in advance..
As user avr pointed out, this is a known bug in Kafka 0.8.2.x
These are routine informational messages misclassified as ERROR.
As of Kafka v0.9.0.0, the loglevel has been corrected to WARN.

kafka cant connect to zookeeper- FATAL Fatal error during KafkaServerStable startup

Well..every service in the world can connect to my zookeeper expect kafka. Below is my connection string in server.properties file
zk.connect=1.dzk.syd.druid.neo.com:2181, 2.dzk.syd.druid.neo.com:2181
Have have all ports on the two zookeeper servers ....total promiscuous mode. I can even telnet into the zookeeper server from the kafka server..
telnet 2.dzk.syd.druid.neo.com 2181
Trying 54.252.183.218...
Connected to 2.dzk.syd.druid.neo.com.
Escape character is '^]'.
So....rather confused on why kafka will not connect to zookeeper?
I am using ubuntu 12.04 and kafka 0.7.2
[2013-07-16 04:36:49,915] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2013-07-16 04:36:49,915] INFO Client environment:user.dir=/etc/sv/kafka (org.apache.zookeeper.ZooKeeper)
[2013-07-16 04:36:49,916] INFO Initiating client connection, connectString=1.dzk.syd.druid.neo.com:2181, 2.dzk.syd.druid.neo.com:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#39cc65b1 (org.apache.zookeeper.ZooKeeper)
[2013-07-16 04:36:49,935] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2013-07-16 04:36:49,938] FATAL Fatal error during KafkaServerStable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkException: Unable to connect to 1.dzk.syd.druid.neo.com:2181, 2.dzk.syd.druid.neo.com:2181
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66)
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.server.KafkaZooKeeper.startup(KafkaZooKeeper.scala:44)
at kafka.log.LogManager.<init>(LogManager.scala:93)
at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:47)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.net.UnknownHostException: 2.dzk.syd.druid.neo.com: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:894)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1286)
at java.net.InetAddress.getAllByName0(InetAddress.java:1239)
at java.net.InetAddress.getAllByName(InetAddress.java:1155)
at java.net.InetAddress.getAllByName(InetAddress.java:1091)
at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:387)
at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:332)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383)
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64)
... 9 more
[2013-07-16 04:36:49,942] INFO Shutting down Kafka server (kafka.server.KafkaServer)
[2013-07-16 04:36:49,943] INFO shutdown scheduler kafka-logcleaner- (kafka.utils.KafkaScheduler)
[2013-07-16 04:36:49,944] INFO Kafka server shut down completed (kafka.server.KafkaServer)
In your kafka/config/server.properties, there should be a property
#host.name=localhost
if you have uncommented this, or set this to another name, then that name should be in the /etc/hosts file
It's been a while since this has been answered, but in case it could help someone here is how i fixed it :
Actually i am using an Ansible playbook to install Kafka cluster and the params generated in zookeeper.properties file were not correctly ordered :
server.1=0.0.0.0:2888:3888
server.2=kafka-4:2888:3888
server.3=kafka-5:2888:3888
server.4=kafka-3:2888:3888
server.5=kafka-2:2888:3888
Putting them in the right order,
server.1=0.0.0.0:2888:3888
server.2=kafka-2:2888:3888
server.3=kafka-3:2888:3888
server.4=kafka-4:2888:3888
server.5=kafka-5:2888:3888
Then restart Kafka service, fixed it.
Change this in zookeeper.properties
maxClientCnxns=0 to maxClientCnxns=1