Retry Attempt without data loss when sink side solr is down during runtime - apache-kafka

Question

curl -X POST -H "Content-Type: application/json" --data '{
"name": "t1",
"config": {
"tasks.max": "1",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"key.converter.schemas.enable": "false",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"connector.class": "com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector",
"topics": "TRAN",
"solr.queue.size": "100",
"solr.commit.within": "10",
"solr.url": "http://192.168.2.221:27052/solr/TRAN",
"errors.retry.delay.max.ms":"5000",
"errors.retry.timeout":"600000",
"errors.tolerance":"all",
"errors.log.enable":"true",
"errors.log.include.messages":"false",
"errors.deadletterqueue.topic.name":"DEAD_TRAN",
"errors.deadletterqueue.topic.replication.factor":"1",
"retry.backoff.ms":"1000",
"reconnect.backoff.ms":"5000",
"reconnect.backoff.max.ms":"600000"
}
}' http://localhost:8083/connectors
Need to retry ( without any data loss) based on count from connector config if solr server is down during runtime.
In my case, Its working perfectly whenboth connector and solr are in running state [Active].
But while only solr server is down, there is no retry process until my data passed to the solr leads to data loss..
Error Information shown below
Connector Config from the Kafka Connect Log

Answer 1

I've just checked the SinkTask implementation of that specific connector and it does throw a RetriableException in the put() method.
In theory, and according to your connector configuration, it should block for 10 minutes ("errors.retry.timeout" : "600000"). If your SolR instance recover within the 10 minutes there shouldn't be any problem in terms of data loss.
If you want to fully block your connector until solR is up on his feet, have you tried to set "errors.retry.timeout" : "-1"?
As per the documentation of errors.retry.timeout:
The maximum duration in milliseconds that a failed operation will be
reattempted. The default is 0, which means no retries will be
attempted. Use -1 for infinite retries.
PS: IMHO this might lead to a deadlock situation if for some reason a single message is permanently failing its sink operation (i.e: if the sink is rejecting the operation).