Automatically reconnect failed tasks in Kafka-Connect - apache-kafka

Question

I'm using a mongo-source plugin with Kafka-connect.
I checked the source task state, and it was running and listening on a mongo collection.
I manually stopped mongod service and waited about 1 minute, then I start it back again.
I checked the source task to see if anything will fix itself, and after 30 minutes nothing seems to work.
Only after restarting the connector it started working again.
Since, mongo-source doesn't have the options to set retries + backoff when timeout, I searched for a configuration that will fit a simple scenario: restart failed task after X time using Kafka-connect configuration. couldn't find any.. :/
I can do that with a simple script, but there must be something in Kafka-connect that manages failed tasks. or even in mongo-source... I don't want it to fail so fast after just 1 minute... :/

Answer 1

There isn't any way other than using the REST API to find a failed task and submit a restart request - and then running this on a periodic basis. For example
curl -s "http://localhost:8083/connectors?expand=status" | \
jq -c -M 'map({name: .status.name } + {tasks: .status.tasks}) | .[] | {task: ((.tasks[]) + {name: .name})} | select(.task.state=="FAILED") | {name: .task.name, task_id: .task.id|tostring} | ("/connectors/"+ .name + "/tasks/" + .task_id + "/restart")' | \
xargs -I{connector_and_task} curl -v -X POST "http://localhost:8083"\{connector_and_task\}
Source: https://rmoff.net/2019/06/06/automatically-restarting-failed-kafka-connect-tasks/