I am following the steps given in the link http://www.confluent.io/blog/how-to-build-a-scalable-etl-pipeline-with-kafka-connect to install the kafka -connect on EC2 having HDP2.3 platform.
But I am getting the error :
ERROR Failed to flush WorkerSourceTask{id=test-mysql-jdbc-0}, timed out while waiting for producer to flush outstanding messages, 1 left
Complete error can be seen in the following :
image
Is this a kafka issue or HDP issue ?, because i did the same thing on AWS EMR and it worked .
Related
My rabbitmq connector works fine when I run it in a server with no SASL. Actually it was working in a SASL activated server too but after restarting the Kafka Connect service now it won't start working. The error is:
org.apache.kafka.common.errors.TimeoutException: License topic could not be created
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
Is it a licensing issue? I don't think we purchased any license for it and it says there is a 30 days free trial. But also, I think we were using it for more than 30 days.
Edit: Found this in connect.log file:
INFO [AdminClient clientId=RabbitMQSinkConnector2-license-manager] Metadata update failed
Edit2: It has something to do with SASL. After enabling SASL for my broker, rabbitmq connector started giving this error.
Solved it by adding a little extra configuration for a sasl enabled broker, like I did for a debezium connector. You need to add these lines to your connector config:
"confluent.topic.sasl.mechanism": "PLAIN",
"confluent.topic.security.protocol": "SASL_PLAINTEXT",
"confluent.topic.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=admin password=secret;"
If you don't want to expose your sasl credentials, you can save it in a file and change the last one like that:
"confluent.topic.sasl.jaas.config": "${file:/kafka/vty/pass.properties:sasl}"
But of course you need to enable reading from a file first.
I'm beginner using Kafka and druid. Recently I get an error when I upload data to druid at the parse data step. I don't know about this issue and error. I have assumed that this zookeeper of the Kafka server getting errors. I ran the zookeeper by following ./bin/zookeeper-server-start.sh config/zookeeper.properties but the result has timed out like the picture
I've a single node cluster setup of Apache kafka_2.12-2.5.1 (embedded zookeeper) on the same host. Enabled SSL on Zookeeper and it starts just fine. Kafka, however, throws fatal error "NoAuth for /brokers/ids" at start up.
Please help out if you've any pointers.
I'am trying to run kafka from ambari UI. sandbox hdp. it does not start. i checked the log the error is :
java.lang.IllegalArgumentException: Error creating broker listeners from 'sandbox-hdp.hortonworks.com:9092': Unable to parse sandbox-hdp.hortonworks.com:9092 to a broker $.
It seems this was resolved on the Cloudera Community. The key conclusion:
It's worked for me after changing my listener ip from localhost to
exact ip of virtualbox.
I'm using a local KSQL Server (latest version of Confluent Platform), trying to connect to a Tunneled Kafka cluster on my server (AWS). However, I'm facing an issue of cannot start KSQL server because of the below error:
ERROR Failed to initialize TopicClient: Unexpected broker id, expected 5 or empty string, but received ConfigResource(type=BROKER, name='1').name (io.confluent.ksql.services.KafkaTopicClient:279)
ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:53)
io.confluent.ksql.util.KsqlException: Could not fetch broker information. KSQL cannot initialize
at
io.confluent.ksql.services.KafkaTopicClientImpl.isTopicDeleteEnabled (KafkaTopicClientImpl.java:280)
at io.confluent.ksql.services.KafkaTopicClientImpl.<init>(KafkaTopicClientImpl.java:66)
at io.confluent.ksql.services.DefaultServiceContext.create(DefaultServiceContext.java:43)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:303)
at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:85)
at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:50)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id, expected 5 or empty string, but received ConfigResource(type=BROKER, name='1').name
My tunneled kafka cluster is running on the version 2.0.0. Even I tried to use the previous version of KSQL (5.0.x) that's compatible with the Kafka version 2.0.x. However the issue still happen.