I have installed zookeeper and kafka,zookeeper is ruuning fine but as per my requirement I want to run Kafka-connect standalone to connect kafka and cassandsra. I am using following command to start kafka server kafka/bin/connect-standalone.sh kafka/config/connect-standalone.properties kafka-connect-cassandra-sink-1.4.0/conf/cassandra-sink-standalone.properties . The connect-standalone.properties configuration file is given below:
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/home/shamama/kafka-connect-cassandra-sink-1.4.0/kafka-connect-cassandra-sink-1.4.0.jar
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
listeners=PLAINTEXT://localhost:9092
#advertised.listeners=PLAINTEXT://localhost:9092
The following error is occurring after starting kafka standalone server:
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1628583433705, tries=1, nextAllowedTryMs=1628583433806) timed out at 1628583433706 after 1 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
I cannot figure out my mistake.Please help.
The error indicates Kafka isn't running.
Based on your previous question, you're running kafka-server-start with the wrong property file
Specifically, zookeeper.connect, listeners, and advertised.listeners should not be part of connect-standalone.properties, and you need broker.id, at a minimum, to start a broker (already part of server.properties)
Related
I run Kafka and Kafka-connect on different servers (Let`s say serverA and serverB)
serverA for kafka connect
# vi /home/kafka/config/connect-distributed.properties
bootstrap.servers=serverB:9092
rest.host.name=localhost
rest.port=8083
serverB for kafka
# vi server.properties
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://serverA:9092
delete.topic.enable = true
But, when i run kafka connect on serverA, i got an error.
[2020-04-30 16:59:37,053] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:84)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1588233577048) timed out at 1588233577049 after 1 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1588233577048) timed out at 1588233577049 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
FYI, If i run kafka connect on kafka server(serverB), It worked. but, I want to run these on different server.
How can I connect kafka-connect to kafka?
In your server.properties you have
advertised.listeners=PLAINTEXT://serverA:9092
but Kafka connnects uses
bootstrap.servers=serverB:9092
instead of
bootstrap.servers=serverA:9092
I am trying to start Kafka connect in distributed mode even in standalone also i am not able to proceed
This is my elastic search sink properties
name=elasticsearch-sink
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=5
topics=fsp-audit
key.ignore=true
connection.url=https://****.amazonaws.com
type.name=kafka-connect
errors.tolerance = all
errors.deadletterqueue.topic.name = fsp-dlq-audit-event
this is my connect-distributed.properties
bootstrap.servers=***:9092,***:9092,***:9092
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=1
schema.enabled=false
config.storage.topic=connect-configs
config.storage.replication.factor=1
status.storage.topic=connect-status
status.storage.replication.factor=1
offset.flush.interval.ms=10000
plugin.path=/usr/local/confluent/share/java
I also have created three topi upfront
connect-offsets
connect-configs
connect-status
I am running this on EC2 and using MSK as Kafka .
I checked connectivity from my EC2 to MSK and i am able to telent
The error i get this
[2020-01-30 08:53:12,126] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:237)
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
[2020-01-30 08:53:12,145] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:237)
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
[2020-01-30 08:53:12,149] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:83)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:94)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
Question :If i have to run Kafka connect in distributed mode,do i have to use more than one EC2/vm ?
Ok after looking into more details i found out that issue was in NACL which was blocking few subnets ip addresses .
So, I checked the MSK side Security group/Network ACL/Route table and found them to be fine. This means that the issue could be with the EC2 instance so I checked the Security group/Route table of the instance and found them to be properly configured.
However, on checking the Network ACL (acl-***) attached with the EC2 instance, I saw that there is an Inbound rule allowing 0.0.0.0/0 for the ephemeral ports which should allow the brokers to talk to the EC2 instance. However, looking at the Outbound rules, I saw it allowing only the subnet range where b-2 is present but it didn't have any explicit Outbound rule to allow either b-3 (10.**.**.0/24) or b-4 (10.**.**.0/24) subnet ranges.
When i added new rule then i was able to ping and connect success fully
If i have to run Kafka connect in distributed mode,do i have to use more than one EC2/vm ?
No, you can run one "psuedo-distributed" instance. The main differerence between standalone and distributed is how it handles storage of offsets and configurations.
Following instructions in lesson 28 of Learn Apache Kafka for Beginners Udemy course to start zookeeper and then start a kafka server broker on Windows 10. Zookeeper runs fine on port 2181:
C:\kafka_2.12-2.3.1> zookeeper-server-start.bat config/zookeeper.properties
...
INFO binding to port 0.0.0.0/0.0.0.0:2181
But after adding the bat files to path, running kafka server does not work:
C:\kafka_2.12-2.3.1> kafka-server-start.bat config/server.properties
...
ERROR There was an error in one of the threads during logs loading: org.apache.kafka.common.KafkaException: Found directory C:\kafka_2.12-2.3.1\data\kafka, 'kafka' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).
Kafka's log directories (and children) should only contain Kafka topic data. (kafka.log.LogManager)
Some of the stdout logging in zookeeper looks informative:
Accepted socket connection from /127.0.0.1:49439 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2019-11-03 17:22:42,278] INFO Client attempting to establish new session at /127.0.0.1:49439 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-11-03 17:22:42,286] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
...
INFO Processed session termination for sessionid: 0x1007b0044a40000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-11-03 17:22:42,987] INFO Closed socket connection for client /127.0.0.1:49439 which had sessionid 0x1007b0044a40000 (org.apache.zookeeper.server.NIOServerCnxn)
Under the data folder I created, there are two folders I created, the second of which was filled after I tried running the kafka broker:
kafka/
|-- empty
zookeeper/
|-- version-2/
|--log.1
Why does this error happen and how can I start a Kafka server on Windows 10?
EDIT:
Contents of config/server.properties:
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=C:/kafka_2.12-2.3.1/data/
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
The data/ folder contains empty zookeeper/ and kafka/ directories
Did you create those folders? Kafka's log.dirs can only contain folders in the format of topic-partition or topic-partition.uniqueId-delete (if marked for deletion), as mentioned by the error. That folder should be empty (no subdriectories) if starting fresh installation.
Plus, Kafka data directory should not hold Zookeeper data as Zookeeper should be independent of Kafka
Following is my architecture
2 Servers:
Server 1: running Apache Nifi + Zookeeper (Not embedded)
Server 2: running Apache Nifi + Zookeeper (Not embedded)
To test failovers, I close down the Server that has been selected as Cluster Coordinator
In this case, zookeeper should automatically elect the remaining one server as leader. But it keeps failing and goes into continuous loop of trying to connect to the first server
Zookeeper Logs in Server 2 when leader (Server 1) went down:
2019-10-22 18:44:01,135 [myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn#370] - Exception causing close of session 0x0: ZooKeeperServer not running
2019-10-22 18:44:02,925 [myid:2] - WARN [NIOWorkerThread-3:NIOServerCnxn#370] - Exception causing close of session 0x0: ZooKeeperServer not running
2019-10-22 18:44:03,320 [myid:2] - WARN [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumCnxManager#677] -
Cannot open channel to 1 at election address ec2-server-1.compute-1.amazonaws.com/172.xx.x.x:3888
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
Server 2 Config files:
zoo.cfg
tickTime=2000
initLimit=5
syncLimit=2
dataDir=/home/ec2-user/zookeeper
clientPort=2181
server.1=ec2-server-1.compute-1.amazonaws.com:2888:3888
server.2=0.0.0.0:2888:3888
nifi.properties
nifi.cluster.is.node=true
nifi.cluster.node.address=ec2-server-2.compute-1.amazonaws.com
nifi.cluster.node.protocol.port=8082
nifi.cluster.flow.election.max.wait.time=2 mins
nifi.cluster.flow.election.max.candidates=1
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=localhost:2181
nifi.zookeeper.root.node=/nifi
Server 1 Config files:
zoo.cfg
tickTime=2000
initLimit=5
syncLimit=2
dataDir=/home/ec2-user/zookeeper
clientPort=2181
server.1=0.0.0.0:2888:3888
server.2=ec2-server-2.compute-1.amazonaws.com:2888:3888
nifi.properties
nifi.cluster.is.node=true
nifi.cluster.node.address=ec2-server-1.compute-1.amazonaws.com
nifi.cluster.node.protocol.port=8082
nifi.cluster.flow.election.max.wait.time=2 mins
nifi.cluster.flow.election.max.candidates=1
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=localhost:2181
nifi.zookeeper.root.node=/nifi
What am I doing wrong?
You need at least three nodes to be able to handle the failure of one node.
Check the Admin guide:
Clustered (Multi-Server) Setup For reliable ZooKeeper service, you
should deploy ZooKeeper in a cluster known as an ensemble. As long as
a majority of the ensemble are up, the service will be available.
Because Zookeeper requires a majority, it is best to use an odd number
of machines. For example, with four machines ZooKeeper can only handle
the failure of a single machine; if two machines fail, the remaining
two machines do not constitute a majority. However, with five machines
ZooKeeper can handle the failure of two machines.
A simpler explanation here also
I setup kafka cluster in vm.
but i have a problem.
My kafka version is kafka 2.11-0.0.0
$ bin/kafka-server-start.sh config/zookeeper.properties
[2017-01-31 09:15:55,216] FATAL (kafka.Kafka$)
org.apache.kafka.common.config.ConfigException: Missing required configuration "zookeeper.connect" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:148)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:49)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:56)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:702)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:691)
at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
please help me .
my zookeeper.properties (down)
zookeeper.properties
dataDir=/home/kafka01/zookeeper-data
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
initLimit=5
syncLimit=2
server.1=kafka01:2888:3888
server.2=kafka02:2888:3888
server.3=kafka03:2888:3888
server.properties
zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181
what is the problem??
You are trying to start a kafka server but you are passing the wrong config file.
Usually you want to start a kafka server using the following command:
./kafka-server-start.sh ../config/server.properties
And is in this file where you specify the address of the zookeeper:
zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181
The quickstart guide in the kafka official documentation is quite good, I recommend you to have a look at it. You can find it here.
You are trying to start kafka server with zookeeper config properties file and that's why you are getting the above error.
Use the commands below instead:
Zookeeper:
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Kafka:
.\bin\windows\kafka-server-start.bat .\config\server.properties
You stared the wrong script. It should be:
bin/zookeeper-server-start.sh config/zookeeper.properties
There are two ways you can start Kafka server:
Start the zookeeper server separately(on your local or another machine)
Start the zookeeper server which is embedded with Kafka server (kafka_2.12/libs/zookeeper-3.4.13.jar)
In first case:
I am assuming you have started the zookeeper server already and that is running on localhost:2181
So now you are good to start kafka server using following command
bin\windows\kafka-server-start.bat ..\..\config\server.properties
In server.properties file you have to tell Kafka server where is/are the zookeeper(s) running. Using property:
## zookeeper.connect=localhost:2181 [default]
zookeeper.connect=zk1:2181, zk2:2181, localhost:2181
In second case
Start the zookeeper first (embedded with the Kafka server)
bin\windows\zookeeper-server-start.bat ..\..\config\zookeeper.properties
Then, start the Kafka server:
bin\windows\kafka-server-start.bat ..\..\config\server.properties
This is the error because there is an issue with your command.
To start zookeeper run -
zookeeper-server-start.sh config/zookeeper.properties
To start server run -
kafka-server-start.sh config/zookeeper.properties
you need to start the zookeeper with
zookeeper-server-start.sh ~/kafka/config/zookeeper.properties
then you
need to start the kafka-server using
kafka-server-start.sh ~/kafka/config/server.properties
Now you kafka will run smoothly ....