We use spring cloud with stream Kafka, it running well, but can't visit the spring mvc Urls, it works fine if I remove the #EnableBinding(Sink.class).
It seems that #EnableBinding(Sink.class) affect the spring mvc functionality.
2018-01-11 15:54:19.218 [restartedMain] WARN org.apache.kafka.clients.consumer.ConsumerConfig - The configuration 'value.serializer' was supplied but isn't a known config.
2018-01-11 15:54:19.218 [restartedMain] WARN org.apache.kafka.clients.consumer.ConsumerConfig - The configuration 'key.serializer' was supplied but isn't a known config.
2018-01-11 15:54:19.219 [restartedMain] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.10.1.1
2018-01-11 15:54:19.219 [restartedMain] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : f10ef2720b03b247
I had the same problem
In order for your application to be able to communicate with Kafka, need to define an outbound stream to write messages to a Kafka topic, and an inbound stream to read messages from a Kafka topic.
Spring Cloud provides a convenient way to do this by simply creating an interface that defines a separate method for each stream.
It seems that your interface is not in Spring context
You should use the #Component annotation,
Related
How do we configure value.subject.name.strategy based on https://docs.confluent.io/platform/current/schema-registry/connect.html#json-schema
I put various configuration names in worker.properties but it seems that nothing is recognized by kafka sink connector. As you can see in the logs, it's always defaulted to topicNameStrategy.
[2022-11-21 16:40:23,663] WARN The configuration 'value.converter.subject.name.strategy' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy
[2022-11-21 16:40:23,690] WARN The configuration 'converter.subject.name.strategy' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2022-11-21 16:40:23,690] WARN The configuration 'value.subject.name.strategy' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2022-11-21 16:40:23,690] WARN The configuration 'value.converter.subject.name.strategy' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2022-11-21 16:40:23,719] WARN The configuration 'converter.subject.name.strategy' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
I put all of these variations in worker.properties and feed it to connector_distributed to start.
grep -i "name.strategy" /plugins/worker.properties
value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
value.converter.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
consumer.value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
consumer.value.converter.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
Those logs can be ignored. Consumer properties don't use those, only the config within the serializer does. That will be printed separately (where you may be seeing the default applied).
There's an open JIRA to silence the logs from passing converter properties all over the consumer.
To configure the serializer, you use converters. To configure converters you need to use
value.converter.[property]=[value]
So, like schema.registry.url,
value.converter.value.subject.name.strategy=OtherStrategy
I have added the below configuration in application.properties file of Spring Boot with Camel implementation but the messages are not getting consumed. Am I missing any configuration? Any pointers to implement consumer from Azure event hub using kafka protocol and Camel ?
bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
The route looks like this:
from("kafka:{{topicName}}?brokers=NAMESPACENAME.servicebus.windows.net:9093" )
.log("Message received from Kafka : ${body}");
I found the solution for this issue. Since I was using the Spring Boot Auto configuration (camel-kafka-starter), the entry on the application.properties file had to be modified as given below:
camel.component.kafka.brokers=NAMESPACENAME.servicebus.windows.net:9093
camel.component.kafka.security-protocol=SASL_SSL
camel.component.kafka.sasl-mechanism=PLAIN
camel.component.kafka.sasl-jaas-config =org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
The consumer route for the Azure event hub with Kafka protocol will look like this:
from("kafka:{{topicName}}")
.log("Message received from Kafka : ${body}");
Hope this solution helps to consume events from Azure event hub in Camel using Kafka protocol
I'm using Confluent's Kafka Connectors to sink data from Kafka to a MinIO bucket. I'm using io.confluent.connect.s3.S3SinkConnector inside a Kubernetes environment. This is my current S3 Sink configuration:
kafkaConnectSpec:
class: io.confluent.connect.s3.S3SinkConnector
config:
tasks.max: 1
topics: topic-name
s3.bucket.name: bucket
flush.size: 1
store.url: http://minio:9000
format.class: io.confluent.connect.s3.format.json.JsonFormat
storage.class: io.confluent.connect.s3.storage.S3Storage
After the cloud environment deploys, the customer wants to be able to control topics dynamically (ie. adding and deleting topics at will). While I understand why this might not be ideal, I yield to the higher authorities.
So in order to perform a topic addition, I'm using the Kafka REST API:
def update_sink(topic, connector):
configuration = requests.get("http://kafka-connect-cluster-connect-api:8083/connectors/" + str(connector)).json()
if "config" not in configuration:
return {
"status": 500,
"message": "Kafka Sink " + str(connector) + " does not have a configuration"
}
# Topics must be comma delimited
if "topics" in configuration["config"]:
configuration["config"]["topics"] += ',' + topic
requests.put("http://kafka-connect-cluster-connect-api:8083/connectors/" + str(connector) + "/config", json=configuration["config"])
print("http://kafka-connect-cluster-connect-api:8083/connectors/" + str(connector) + "/config")
print(configuration["config"])
return {
"status": 200,
"message": "Kafka Sink " + str(connector) + " successfully updated"
}
I know the code is not pretty, but it gets the job done for now. It essentially makes a PUT request to the /connectors/my-sink/config endpoint with my new topics appended.
This works. My sink has the new topic and I can send messages.
However, within 3-5 minutes, my Kafka Sink Pod begins restarting (I think) the Kafka connector:
2021-03-19 23:02:55,086 INFO [Worker clientId=connect-1, groupId=connect-cluster] Connector minio-s3-sink config updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [KafkaBasedLog Work Thread - connect-cluster-configs]
2021-03-19 23:02:55,589 INFO [Worker clientId=connect-1, groupId=connect-cluster] Handling connector-only config update by restarting connector minio-s3-sink (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2021-03-19 23:02:55,589 INFO Stopping connector minio-s3-sink (org.apache.kafka.connect.runtime.Worker) [DistributedHerder-connect-1-1]
2021-03-19 23:02:55,589 INFO Shutting down S3 connector minio-s3-sink (io.confluent.connect.s3.S3SinkConnector) [DistributedHerder-connect-1-1]
2021-03-19 23:02:55,598 INFO Stopped connector minio-s3-sink (org.apache.kafka.connect.runtime.Worker) [DistributedHerder-connect-1-1]
2021-03-19 23:02:55,598 INFO [Worker clientId=connect-1, groupId=connect-cluster] Starting connector minio-s3-sink (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
... # Performs the restart here
At which point the pod loses the topic.
I believe this is caused by the config.action.reload = restart configuration value. I think after receiving a new configuration the connector is slated for a restart after N-minutes. However, I can't seem to find any documentation on how to change that behavior. Perhaps I should be doing it during my PUT request but that feels hacky. It's also just a shot in the dark.
Does anyone know why my connector is restarting after the PUT request with the updated configuration? Is there anyway to prevent this?
Edit #1: I attempted to add config.action.reload = none, however the connector still restarted.
I watched the logs in the Kafka Operator and it did not trigger the reset. Seems to be entirely isolated to the Kafka Connector Operator.
The problem was documented here in Strimzi:
If KafkaConnectors are enabled, manual changes made directly using the Kafka Connect REST API are reverted by the Cluster Operator
https://strimzi.io/docs/operators/latest/deploying.html#availability_of_the_kafka_connect_rest_api
I had no idea this was happening in my deployment but apparently we have to turn it off. Which is unfortunate since the K8 connector deployment was nice for a simple start.
This is the relevant configuration to turn them "off":
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
name: kafka-connect-cluster
annotations:
# # use-connector-resources configures this KafkaConnect
# # to use KafkaConnector resources to avoid
# # needing to call the Connect REST API directly
strimzi.io/use-connector-resources: "false"
The strimzi.io/use-connector-resources: "false" will make it so you cannot add connectors via your YAML file, but you can add the connector via your REST API (and those changes will persist as long as the pod runs)
I'm trying to implement Kafka consumer with topic names as a pattern. E.g. #KafkaListener(topicPattern="${kafka.topics}") where property kafka.topics is 'sss.*'. Now when I send message to topic 'sss.test' or any other topic name like 'sss.xyz', 'sss.pqr', it's throwing error as below:
WARN o.apache.kafka.clients.NetworkClient - Error while fetching metadata with correlation id 12 : {sss.xyz-topic=LEADER_NOT_AVAILABLE}
I tried to enable listeners & advertised.listeners in the server.properties file but when I re-start Kafka it consumes messages from all old topics which were tried. The moment I use new topic name, it throws above error.
Kafka doesn't support pattern matching? Or there's some configuration which I'm missing? Please suggest.
I have an small app in scala that creates kafka producer and that run with Apache Spark.
when I run the command
spark-submit --master local[2] --deploy-mode client <into the jar file> <app Name> <kafka broker> <kafka in queue> <kafka out queue> <interval>
I am getting this WARN:
WARN AppInfoParser: Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=
The code is not relevant because I am getting this exception when scala creates the KafkaProducer: val producer = new KafkaProducerObject,Object
Does anybody have a solution for this?
Thank you!
When a Kafka Producer is created, it attempts to register an MBean using the client.id as its unique identifier.
There are two possibilities of why you are getting the InstanceAlreadyExistsException warning:
You are attempting to initialize more than one Producer at a time with the same client.id property on the same JVM.
You are not calling close() on an existing Producer before initializing another Producer. Calling close() unregisters the MBean.
If you leave the client.id property blank when initializing the producer, a unique one will be created for you. Giving your producers unique client.id values or allowing them to be auto-generated would resolve this problem.
In the case of Kafka, MBeans can be used for tracking statistics.