In a standard/custom kafkaconnect sink how can we specify that it should only consume read_comitted messages from kafka topics. I can see the config here but can't see any options (unless it is the default behaviour). Thanks.
https://docs.confluent.io/current/installation/configuration/connect/sink-connect-configs.html
If you want to override consumer properties (isolation.level in your case) for a specific connector, enable client overrides in the worker configuration and then use consumer.override.* for a sink connector config:
"consumer.override.isolation.level": "read_committed"
Source: Kafka Connect Worker Configuration Properties: Override the Worker Configuration
Note: Prior to Kafka 2.3 you will need to set it at worker level by adding the line below in worker config
consumer.isolation.level=read_committed
Please see
how to set kafka connect auto.offset.reset with rest api
Related
In kafka connect distributed mode, we need to submit kafka connect source and kafka connect sink configuration through REST api. How does Kafka Connect leverage kafka group coordinator to form a consumer group?
Kafka sink connectors form consumer groups using regular consumer API with a group.id named connect-<name>, using the name you've configured in the connector config
I am using java Spring-Boot framework and trying to prevent our consumer from creating topic in kafka by setting the config properties.
where Configurations are:
From broker side:
auto.create.topics.enable=true
From consumer side
auto.create.topics.enable=false
for consumer we made auto creation topic false where on broker it is true.
Above configs are not working for us,
and Also if We have any other ways to archive the same we can discuss.
auto.create.topics.enable is not a consumer config. It needs to be allow.auto.create.topics, but is only a valid option in kafka-clients version 2.3+
There may be other Spring related settings; refer latest comment thread here. Disable auto topic creation from Spring Kafka Consumer
How can I enable Kafka source connector idempotency feature?
I know in confluent we can override producer configs by producer.* properties in the worker configuration, but how about Kafka itself? is it the same?
After setting these configs where can I see applied configs for my connect worker?
Confluent doesn't modify the base Kafka Connect properties.
For configuration of the producers used by Kafka source tasks and the consumers used by Kafka sink tasks, the same parameters can be used but need to be prefixed with producer. and consumer. respectively
Starting with 2.3.0, client configuration overrides can be configured individually per connector by using the prefixes producer.override. and consumer.override. for Kafka sources or Kafka sinks respectively
https://kafka.apache.org/documentation/#connect_running
However, Kafka Connect sources aren't idenpotent - KAFKA-7077 & KIP-308
After setting these configs where can I see applied configs for my connect worker
In the logs, it should show the ProducerConfig or ConsumerConfig when the tasks start
I install confluent and it has own kafka.
I want to change kafka from own to another?
Which .properties or whatelse file I must change to look different kafka.
thanks in advance
In your Kafka Connect worker configuration, you need to set bootstrap.servers to point to the broker(s) on your source Kafka cluster.
You can only connect to one source Kafka cluster per Kafka Connect worker. If you need to stream data from multiple Kafka clusters, you would run multiple Kafka Connect workers.
Edit If you're using Confluent CLI then the Kafka Connect worker config is taken from etc/schema-registry/connect-avro-distributed.properties.
We are currently on HDF (Hortonworks Dataflow) 3.3.1 which bundles Kafka 2.0.0 and are trying to use Kafka Connect in distributed mode to launch a Google Cloud PubSub Sink connector.
We are planning on sending back some metadata into a Kafka Topic and need to integrate a Kafka producer into the flush() function of the Sink task java code.
Would this have a negative impact on the process where Kafka Connect commits back the offsets to Kafka (as we would be adding a overhead of running a Kafka producer before the flush).
Also, how does Kafka Connect get the Bootstrap servers list from the configuration when it is not specified in the Connector Properties for either the sink or the source? I need to use the same Bootstrap server list to start the producer.
Currently I am changing the config for the sink connector, adding bootstrap server list as a property and parsing it in the Java code for the connector. I would like to use bootstrap server list from the Kafka Connect worker properties if that is possible.
Kindly help on this.
Thanks in advance.
need to integrate a Kafka producer into the flush() function of the Sink task java code
There is no producer instance exposed in the SinkTask API...
Would this have a negative impact on the process where Kafka Connect commits back the offsets to Kafka (as we would be adding a overhead of running a Kafka producer before the flush).
I mean, you can add whatever code you want. As far as negative impacts go, that's up to you to benchmark on your own infrastructure. Obviously adding more blocking code makes the other processes slower overall
how does Kafka Connect get the Bootstrap servers list from the configuration when it is not specified in the Connector Properties for either the sink or the source?
Sinks and sources are not workers. Look at connect-distributed.properties
I would like to use bootstrap server list from the Kafka Connect worker properties if that is possible
It's not possible. Adding extra properties to the sink/source configs are the only way. (Feel free to make a Kafka JIRA requesting such a feature of exposing the worker configs, though)