Kafka Sink Connector with custom consumer-group name - apache-kafka

Question

In kafka connect, all the sink connectors will use the different group with the naming conversion of connect-connector_name. But I want to use a custom name as the prefix.(we can do in the sink config - name properties, but looking for set it by default)
I tried to setup this in the consumer.properties file, but no use.
Does anyone know how it set it? also, What happens if I set a single group for all my sink connector?

Answer 1

Sink tasks always have connect- prefix for their ConsumerConfig group.id
https://issues.apache.org/jira/browse/KAFKA-4400
consumer.properties is used (optionally) for kafka-console-consumer, not Connect API
happens if I set a single group for all my sink connector?
You mean a single connector with one name? Then you'd want tasks.max to be equal to the total partitions of all topics its consuming.
If you mean multiple connectors, then you can't; all connectors within the same Connect cluster need a unique name/connector.class pair

Answer 2

You can override any consumer or producer properties. You have to use connector.client.config.override.policy = All in worker configuration (default is None). Then you can override consumer.group.id for your task in property consumer.override.group.id. For example:
{
"consumer.override.group.id": "testgroup",
"name": "Elasticsearch",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"topics": "orders",
"tasks.max": 1,
"connection.url": "http://elasticsearch:9200",
"type.name": "type.name=kafkaconnect",
"key.ignore": "true",
"schema.ignore": "false",
"transforms": "renameTopic",
"transforms.renameTopic.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.renameTopic.regex": "orders",
"transforms.renameTopic.replacement": "orders-latest"
}'
Documentation is here
If you use kafka-connect in docker from image confluentinc/cp-kafka-connnect-base, you can set this configuration from environment variable CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY