I'm using a local KSQL Server (latest version of Confluent Platform), trying to connect to a Tunneled Kafka cluster on my server (AWS). However, I'm facing an issue of cannot start KSQL server because of the below error:
ERROR Failed to initialize TopicClient: Unexpected broker id, expected 5 or empty string, but received ConfigResource(type=BROKER, name='1').name (io.confluent.ksql.services.KafkaTopicClient:279)
ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:53)
io.confluent.ksql.util.KsqlException: Could not fetch broker information. KSQL cannot initialize
at
io.confluent.ksql.services.KafkaTopicClientImpl.isTopicDeleteEnabled (KafkaTopicClientImpl.java:280)
at io.confluent.ksql.services.KafkaTopicClientImpl.<init>(KafkaTopicClientImpl.java:66)
at io.confluent.ksql.services.DefaultServiceContext.create(DefaultServiceContext.java:43)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:303)
at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:85)
at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:50)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id, expected 5 or empty string, but received ConfigResource(type=BROKER, name='1').name
My tunneled kafka cluster is running on the version 2.0.0. Even I tried to use the previous version of KSQL (5.0.x) that's compatible with the Kafka version 2.0.x. However the issue still happen.
Related
I am trying to connect with kafka with my kafka microservice,
connection configuration kafkajs
but throwing following error :
errors even after initial connection
Ny best guess it is something regarding my kafka environment ,
which is like following right now:
kafka setup server
It could also be because of how kafka works when configured to work with kafka connect, does it behaviour or environment change.
I was trying to connect my RDS MS SQL server with Debezium SQL Server Connector to stream changes to Kafka Cluster on Amazon MSK.
I configured connector and Kafka Connect worker well run the Connect by
bin/connect-standalone.sh ../worker.properties connect/dbzmmssql.properties
Got WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 10 : {TestKafkaDB=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient:1031)
I've solved this problem and just want to share my possible solution with other fresher with Kafka.
TestKafkaDB=UNKNOWN_TOPIC_OR_PARTITION basically means the connector didn't find the a usable topic in Kafka broker. The reason I am facing this is the Kafka broker didn't automatically create a new topic for the stream.
To solving this, I changed Cluster Configuration in AWS MSK console, change auto.create.topics.enable from default false to true and update this configuration to the Cluster, then my problem solved.
We are running Kafka Connect using the Confluent JDBC Source Connector to read from a DB2 database. Periodically, we see issues like this in our Kafka Connect logs:
kafkaconnect-deploy-prod-967ddfffb-5l4cm 2021-04-23 10:39:43.770 ERROR Failed to run query for table TimestampIncrementingTableQuerier{table="PRODSCHEMA"."VW_PRODVIEW", query='null', topicPrefix='some-topic-prefix-', incrementingColumn='', timestampColumns=[UPDATEDATETIME]}: {} (io.confluent.connect.jdbc.source.JdbcSourceTask:404)
com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-668, SQLSTATE=57007, SQLERRMC=1;PRODSCHEMA.SOURCE_TABLE, DRIVER=4.28.11
at com.ibm.db2.jcc.am.b7.a(b7.java:815)
...
at com.ibm.db2.jcc.am.k7.bd(k7.java:785)
at com.ibm.db2.jcc.am.k7.executeQuery(k7.java:750)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.executeQuery(TimestampIncrementingTableQuerier.java:200)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.maybeStartQuery(TimestampIncrementingTableQuerier.java:159)
at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:371)
This appears to be an intermittent issue connecting to DB2, and is semi-expected; for reasons outside the scope of this question, we know that the network between the two is unreliable.
However, what we are trying to establish is whether in this circumstance data loss is likely to have occurred. I've found this article which talks about error handling in Kafka Connect, but it only refers to errors due to broken messages, not the actual connectivity between Kafka Connect and the data source.
In this case, how would we know if the failure to connect had caused data loss? (i.e. records in our data source that were not processed for target topic). Would there be errors in the Kafka Connect log? Will Kafka Connect always retry indefinitely when it has a connectivity issue? Are there any controls over its retry?
(If it matters, Kafka Connect is version 2.5; it is deployed in a Kubernetes cluster, in distributed mode, but with only one actual running worker/container.)
While i'm connecting memsql to kafka using Docker... I'm unable to fetch the data that i'm pushed into kafka topic....
I'm getting error like
ERROR 1933 ER_EXTRACTOR_EXTRACTOR_GET_LATEST_OFFSETS: Cannot get source metadata for pipeline. Could not fetch Kafka metadata (are the Kafka brokers reachable from the Master Aggregator?)
I'm trying to use Kafka connect in Confluent platform 3.2.1 and everything works fine in my local env. Then I encountered this problem when I try to use Kafka source connector to send data to another machine.
I deploy Kafka JDBC source connector in machine A and trying to capture database A. Then I deploy a Kafka borker B(along with zk, schema registry) in machine B. The source connector cannot send data to broker B and throws the following exception:
[2017-05-19 16:37:22,709] ERROR Failed to commit offsets for WorkerSourceTask{id=test-multi-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:112)
[2017-05-19 16:38:27,711] ERROR Failed to flush WorkerSourceTask{id=test-multi-0}, timed out while waiting for producer to flush outstanding 3 messages (org.apache.kafka.connect.runtime.WorkerSourceTask:304)
I tried config the server.properties in broker B like this:
listeners=PLAINTEXT://:9092
and leave the advertised.listeners setting commented.
Then I use
bootstrap.servers=192.168.19.234:9092
in my source connector where 192.168.19.234 is the IP of machine B. Machine A and B are in the same subnet.
I suspect this has something to do with my server.properties.
How should I config to get the things done? Thanks in advance.