Debezium 1.5: Elasticsearch sink connector not adding anything to cluster - apache-kafka

Question

I recently updated my Debezium images from 1.4 to 1.5.
However now when I insert my connectors it seems like the sink conenctor just kind of 'dies' without giving any output at all..
Everything runs fine on 1.4 but I need to upgrade to 1.5 due to some new features that are necessary for my project.
below is my DockerFile for the debezium connect image:
FROM debezium/connect:1.5
ENV KAFKA_CONNECT_JDBC_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-jdbc \
KAFKA_CONNECT_ES_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-elasticsearch
ARG POSTGRES_VERSION=42.2.20
ARG KAFKA_JDBC_VERSION=10.0.0
ARG KAFKA_ELASTICSEARCH_VERSION=10.0.0
# Deploy PostgreSQL JDBC Driver
RUN cd /kafka/libs && curl -sO https://jdbc.postgresql.org/download/postgresql-$POSTGRES_VERSION.jar
# Deploy Kafka Connect JDBC
RUN mkdir $KAFKA_CONNECT_JDBC_DIR && cd $KAFKA_CONNECT_JDBC_DIR &&\
curl -sO https://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/$KAFKA_JDBC_VERSION/kafka-connect-jdbc-$KAFKA_JDBC_VERSION.jar
# Deploy Confluent Elasticsearch sink connector
RUN mkdir $KAFKA_CONNECT_ES_DIR && cd $KAFKA_CONNECT_ES_DIR &&\
curl -sO https://packages.confluent.io/maven/io/confluent/kafka-connect-elasticsearch/$KAFKA_ELASTICSEARCH_VERSION/kafka-connect-elasticsearch-$KAFKA_ELASTICSEARCH_VERSION.jar && \
curl -sO https://repo1.maven.org/maven2/io/searchbox/jest/6.3.1/jest-6.3.1.jar && \
curl -sO https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore-nio/4.4.4/httpcore-nio-4.4.4.jar && \
curl -sO https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.1/httpclient-4.5.1.jar && \
curl -sO https://repo1.maven.org/maven2/org/apache/httpcomponents/httpasyncclient/4.1.1/httpasyncclient-4.1.1.jar && \
curl -sO https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar && \
curl -sO https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar && \
curl -sO https://repo1.maven.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar && \
curl -sO https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar && \
curl -sO https://repo1.maven.org/maven2/io/searchbox/jest-common/6.3.1/jest-common-6.3.1.jar && \
curl -sO https://repo1.maven.org/maven2/com/google/code/gson/gson/2.8.6/gson-2.8.6.jar && \
curl -sO https://repo1.maven.org/maven2/com/google/guava/guava/20.0/guava-20.0.jar
My Elasticsearch sink connector:
{
"name": "es-sink-connector",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"tasks.max": "1",
"topics": "report",
"connection.url": "http://elasticsearch:9200",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.drop.tombstones": "false",
"transforms.unwrap.drop.deletes": "false",
"key.ignore": "false",
"type.name": "_doc",
"behavior.on.null.values": "delete",
"transforms": "ExtractKey",
"transforms.ExtractKey.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
"transforms.ExtractKey.field": "id"
}
}
And lastly, the docker-compose configurations for my debeium-connect
connect:
build: ./debezium-jdbc-es
ports:
- "8083:8083"
- "5005:5005"
depends_on:
- kafka
- elasticsearch
environment:
- BOOTSTRAP_SERVERS=kafka:9092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_source_connect_statuses
- host.docker.internal= host.docker.internal
Any idea as to how to fix it/ why it might not be working is appreciated!
Thank you!
UPDATE:
So after looking a bit more into it, it seems to be due to my Postgres connector not finding my table and not my Elasticsearch sink connector.
Debezium 1.4:
Snapshot step 3 - Locking captured tables [repo.report] [io.debezium.relational.RelationalSnapshotChangeEventSource]
Debezium 1.5:
Snapshot step 3 - Locking captured tables [] [io.debezium.relational.RelationalSnapshotChangeEventSource]
This is my Postgres connector:
{
"name": "reporting-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "host.docker.internal",
"database.port": "5432",
"database.user": "debezium_user",
"database.password": "postgres",
"database.server.id": "184054",
"database.dbname": "reportdatabase",
"database.server.name": "reporting",
"plugin.name": "pgoutput",
"database.include.list": "reportdatabase",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "reporting",
"schema.include.list":"repo",
"table.include.list":"repo.report",
"transforms": "route",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement": "$3"
}
}