Kafka connect schema registry timeout - apache-kafka

Question

EDIT: Turns out I turned on a firewall which limited connectivity from containers to host. Adding firewall rules solved the issue.
I am running a kafka JDBC sink connector with the following properties:
{
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"table.name.format": "events",
"connection.password": "******",
"tasks.max": "1",
"topics": "events",
"value.converter.schema.registry.url": "http://IP:PORT",
"db.buffer.size": "8000000",
"connection.user": "postgres",
"name": "cp-sink-events",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"connection.url": "jdbc:postgresql://IP:PORT/postgres?stringtype=unspecified",
"insert.mode": "upsert",
"pk.mode": "record_value",
"pk.fields": "source,timestamp,event,event_type,value"
}
It was working fine before, but since this week I have been getting the following errors while trying to sink my data to Postgres:
Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro value schema for id 4
Caused by: java.net.SocketTimeoutException: connect timed out
It appears my kafka connect cannot acces my schema registry server anymore. I coulnd't manage to figure out why or how. I have tried multiple things but yet to find the solution.
I did install NGINX on this VM over last week, and killed apache2 which was running on port 80. But I haven't found any dependencies that this would cause any problems.
When I curl the schema registry address from the VM to retrieve the schemas of the mentioned IDs it works fine (http://IP:PORT/schemas/ids/4). any clue how to proceed?
EDIT:
If I configure the IP to random value I get
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable).
So my host schema registry seems reachable when right IP is configured, I don't know where the timeout limit comes from.
Tried to set timeout limit but didn't work:
SCHEMA_REGISTRY_KAFKASTORE_TIMEOUT_MS: 10000
My compose connect config is set as such:
CONNECT_BOOTSTRAP_SERVERS: kafka0:29092
CONNECT_GROUP_ID: compose-connect-group
CONNECT_CONFIG_STORAGE_TOPIC: _connect_configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_STORAGE_TOPIC: _connect_offset
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: _connect_status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schemaregistry0:8085
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schemaregistry0:8085
CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect0
CONNECT_PLUGIN_PATH: /usr/share/java,/usr/share/confluent-hub-components,/usr/share/local-connectors
Docker Kafka network:
[
{
"Name": "kafka_default",
"Id": "89cd2fe68f2ea3923a76ada4dcb89e505c18792e1abe50fa7ad047e10ee6b673",
"Created": "2023-01-16T18:42:35.531539648+01:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.25.0.0/16",
"Gateway": "172.25.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"41ac45882364494a357c26e14f8e3b2aede4ace7eaab3dea748c9a5f94430529": {
"Name": "kafka-schemaregistry1-1",
"EndpointID": "56612fbe41396799a8249620dc07b0a5c84c65d311959214b955f538225757ac",
"MacAddress": "02:42:ac:19:00:06",
"IPv4Address": "172.25.0.6/16",
"IPv6Address": ""
},
"42c0847ffb7545d35b2d4116fb5c590a869aec87037601d33267a35e7fe0cb2f": {
"Name": "kafka-kafka-connect0-1",
"EndpointID": "68eef87346aed70bc17ab9960daca4b24073961dcd93bc85c8f7bcdb714feac3",
"MacAddress": "02:42:ac:19:00:08",
"IPv4Address": "172.25.0.8/16",
"IPv6Address": ""
},
"46160f183ba8727fde7b4a7d1770b8d747ed596b8e6b7ca7fea28b39c81dcf7f": {
"Name": "kafka-zookeeper0-1",
"EndpointID": "512970666d1c07a632e0f450bef7ceb6aa3281ca648545ef22de4041fe32a845",
"MacAddress": "02:42:ac:19:00:03",
"IPv4Address": "172.25.0.3/16",
"IPv6Address": ""
},
"6804e9d36647971afe95f5882e7651e39ff8f76a9537c9c6183337fe6379ced9": {
"Name": "kafka-ui",
"EndpointID": "9e9a2a7a04644803703f9c8166d80253258ffba621a5990f3c1efca1112a33a6",
"MacAddress": "02:42:ac:19:00:09",
"IPv4Address": "172.25.0.9/16",
"IPv6Address": ""
},
"8b79e3af68df7d405567c896858a863fecf7f2b32d23138fa065327114b7ce83": {
"Name": "kafka-zookeeper1-1",
"EndpointID": "d5055748e626f1e00066642a7ef60b6606c5a11a4210d0df156ce532fab4e753",
"MacAddress": "02:42:ac:19:00:02",
"IPv4Address": "172.25.0.2/16",
"IPv6Address": ""
},
"92a09c7d3dfb684051660e84793b5328216bf5da4e0ce075d5918c55b9d4034b": {
"Name": "kafka-kafka0-1",
"EndpointID": "cbeba237d1f1c752fd9e4875c8694bdd4d85789bcde4d6d3590f4ef95bb82c6f",
"MacAddress": "02:42:ac:19:00:05",
"IPv4Address": "172.25.0.5/16",
"IPv6Address": ""
},
"e8c5aeef7a1a4a2be6ede2e5436a211d87cbe57ca1d8c506d1905d74171c4f6b": {
"Name": "kafka-kafka1-1",
"EndpointID": "e310477b655cfc60c846035896a62d32c0d07533bceea2c7ab3d17385fe9507b",
"MacAddress": "02:42:ac:19:00:04",
"IPv4Address": "172.25.0.4/16",
"IPv6Address": ""
},
"ecebbd73e861ed4e2ef8e476fa16d95b0983aaa0876a51b0d292b503ef5e9e54": {
"Name": "kafka-schemaregistry0-1",
"EndpointID": "844136d5def798c3837db4256b51c7995011f37576f81d4929087d53a2da7273",
"MacAddress": "02:42:ac:19:00:07",
"IPv4Address": "172.25.0.7/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "kafka",
"com.docker.compose.version": "2.12.2"
}
}
]
curl http://kafka-schemaregistry0-1:8085/schemas/ids/5
Curling the container works from inside the connect container.
EDIT 5:
After changing the URL to the docker container name, I now have access to schema registry.
"value.converter.schema.registry.url": "http://kafka-schemaregistry0-1:8085"
However; now my postgres connection fails.
Caused by: org.postgresql.util.PSQLException: The connection attempt failed
I think the conclusion here would be that my connect container was previously able to access containers via the IP of the host machine, and that it now is not able to anymore. I am curious to know how this can be fixed.