Debezium server error: java.lang.OutOfMemoryError: Java heap space - mongodb

Question

Debezium server: v 1.9.0.Final
MongoDB Atlas: v 4.2.20
Running on AWS ECS with Fargate w/ 1GB CPU & 4GB MEMORY
Overview:
Debezium starts an initial snapshot and it sends some data to kinesis but it runs into an error (below) before it finishes the snapshot. I've tried increasing the memory of the container to 4GB but not sure if that's the issue. The one collection I'm testing this with is 28GB total and 11M documents.
Debezium config (in Terraform):
environment = [
{
"name" : "DEBEZIUM_SINK_TYPE",
"value" : "kinesis"
},
{
"name" : "DEBEZIUM_SINK_KINESIS_REGION",
"value" : "us-east-1"
},
{
"name" : "DEBEZIUM_SINK_KINESIS_CREDENTIALS_PROFILE",
"value" : "default"
},
{
"name" : "DEBEZIUM_SINK_KINESIS_ENDPOINT",
"value" : "https://kinesis.us-east-1.amazonaws.com"
},
{
"name" : "DEBEZIUM_SOURCE_CONNECTOR_CLASS",
"value" : "io.debezium.connector.mongodb.MongoDbConnector"
},
{
"name" : "DEBEZIUM_SOURCE_OFFSET_STORAGE_FILE_FILENAME",
"value" : "data/offsets.dat"
},
{
"name" : "DEBEZIUM_SOURCE_OFFSET_FLUSH_INTERVAL_MS",
"value" : "0"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_NAME",
"value" : "test"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_HOSTS",
"value" : "test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_SSL_ENABLED",
"value" : "true"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_MEMBERS_AUTO_DISCOVER",
"value" : "true"
},
{
"name" : "DEBEZIUM_SOURCE_DATABASE_INCLUDE_LIST",
"value" : "test"
},
{
"name" : "DEBEZIUM_SOURCE_COLLECTION_INCLUDE_LIST",
"value" : "test.testCollection"
},
{
"name" : "DEBEZIUM_SOURCE_CAPTURE_MODE",
"value" : "change_streams_update_full"
},
{
"name" : "DEBEZIUM_SOURCE_DATABASE_HISTORY",
"value" : "io.debezium.relational.history.FileDatabaseHistory"
},
{
"name" : "DEBEZIUM_SOURCE_DATABASE_HISTORY_FILE_FILENAME",
"value" : "history.dat"
},
{
"name" : "QUARKUS_LOG_CONSOLE_JSON",
"value" : "false"
}
]
secrets = [
{
"name" : "DEBEZIUM_SOURCE_MONGODB_USER",
"valueFrom" : "${data.aws_secretsmanager_secret.test-debezium-read.arn}:username::"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_PASSWORD",
"valueFrom" : "${data.aws_secretsmanager_secret.test-debezium-read.arn}:password::"
}
]
Stacktrace:
2022-06-01 18:22:23,976 ERROR [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Error while attempting to sync 'test-mongodb-shard-0.test.testCollection': : java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
at java.base/java.lang.StringBuffer.append(StringBuffer.java:317)
at java.base/java.io.StringWriter.write(StringWriter.java:106)
at org.bson.json.StrictCharacterStreamJsonWriter.write(StrictCharacterStreamJsonWriter.java:368)
at org.bson.json.StrictCharacterStreamJsonWriter.writeStartObject(StrictCharacterStreamJsonWriter.java:204)
at org.bson.json.LegacyExtendedJsonDateTimeConverter.convert(LegacyExtendedJsonDateTimeConverter.java:22)
at org.bson.json.LegacyExtendedJsonDateTimeConverter.convert(LegacyExtendedJsonDateTimeConverter.java:19)
at org.bson.json.JsonWriter.doWriteDateTime(JsonWriter.java:129)
at org.bson.AbstractBsonWriter.writeDateTime(AbstractBsonWriter.java:394)
at org.bson.codecs.DateCodec.encode(DateCodec.java:32)
at org.bson.codecs.DateCodec.encode(DateCodec.java:29)
at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:203)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:200)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:200)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:159)
at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:46)
at org.bson.Document.toJson(Document.java:453)
at io.debezium.connector.mongodb.JsonSerialization.lambda$new$0(JsonSerialization.java:57)
at io.debezium.connector.mongodb.JsonSerialization$$Lambda$521/0x0000000840448840.apply(Unknown Source)
at io.debezium.connector.mongodb.JsonSerialization.getDocumentValue(JsonSerialization.java:89)
at io.debezium.connector.mongodb.MongoDbSchema$$Lambda$580/0x00000008404ce840.apply(Unknown Source)
at io.debezium.connector.mongodb.MongoDbCollectionSchema.valueFromDocumentOplog(MongoDbCollectionSchema.java:90)
at io.debezium.connector.mongodb.MongoDbChangeSnapshotOplogRecordEmitter.emitReadRecord(MongoDbChangeSnapshotOplogRecordEmitter.java:68)
at io.debezium.connector.mongodb.MongoDbChangeSnapshotOplogRecordEmitter.emitReadRecord(MongoDbChangeSnapshotOplogRecordEmitter.java:27)
at io.debezium.pipeline.AbstractChangeRecordEmitter.emitChangeRecords(AbstractChangeRecordEmitter.java:42)
at io.debezium.pipeline.EventDispatcher.dispatchSnapshotEvent(EventDispatcher.java:163)
I noticed that during the snapshot, the number of records sent and the last recorded offset doesn't seem to change while the amount of time elapsed between each of those messages gets longer. This seems like an exponential backoff thing but I'm not entirely sure.
Example:
2022-06-01 16:20:37,789 INFO [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Beginning snapshot of 'test-mongodb-shard-0' at {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:20:37,804 INFO [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Exporting data for collection 'test-mongodb-shard-0.test.testCollection'
2022-06-01 16:20:42,983 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 717 records sent during previous 00:00:06.159, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:20:57,417 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:14.434, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:21:05,107 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
2022-06-01 16:21:16,624 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:19.207, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:21:35,107 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
2022-06-01 16:21:53,130 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:36.505, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:22:05,107 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:23:17,521 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:01:24.391, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:23:35,106 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:26:06,523 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:02:49.003, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:26:35,107 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:31:18,075 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:05:11.552, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:31:35,106 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:42:07,711 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:10:49.636, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:42:35,106 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 17:03:12,872 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:21:05.161, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 17:03:35,117 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 17:45:58,637 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:42:45.765, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 17:46:05,106 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 18:22:23,976 ERROR [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Error while attempting to sync 'test-mongodb-shard-0.test.testCollection': : java.lang.OutOfMemoryError: Java heap space

Answer 1

Beside increasing the container memory to 4GB , you can set also bigger heap size , the maximum and initial heap size can be set for example to 2GB :
-Xms2048m -Xmx2048m
If the issue continue follow this steps:
Start JVM with argument -XX:+HeapDumpOnOutOfMemoryError this will give you a heap dump when program goes in to OOM.
Use a tool like visualVM to analyze the heap dump obtained. That will help in identifying the memory leak.

Answer 2

Not really answering the OP, just wanted to share my experience.
I too occasionally receive java.lang.OutOfMemoryError and would like to find out what's causing it.
My setup:
Debezium 1.9.5
Kafka 2.8
Docker container memory - 6Gi
Java heap - 4Gi both min and max
max.queue.size.in.bytes - 512Mi
max.batch.size - 16384
The errors from stdout:
2022-07-20T16:47:16.348943181Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.config"
2022-07-20T16:47:27.628395682Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.status"
2022-07-20T16:47:28.970536167Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | connector-producer-REDACTED-202207200823-0"
2022-07-20T16:47:33.787361085Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-3"
2022-07-20T16:47:45.067373810Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "server-timer"
2022-07-20T16:47:46.987669188Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | REDACTED-dbhistory"
2022-07-20T16:48:03.396881812Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-2"
2022-07-20T16:48:04.017710798Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-coordinator-heartbeat-thread | production"
2022-07-20T16:48:09.709036280Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "prometheus-http-1-3"
2022-07-20T16:48:14.667691706Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "mysql-cj-abandoned-connection-cleanup"
2022-07-20T16:48:17.182623196Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-62"
2022-07-20T16:48:25.227925660Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-58"
2022-07-20T16:48:43.598026645Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP-Dispatcher"
2022-07-20T16:48:45.543984655Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-1"
2022-07-20T16:48:52.284810255Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SourceTaskOffsetCommitter-1"
2022-07-20T16:48:56.992674380Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DistributedHerder-connect-1-1"
2022-07-20T16:49:18.691603140Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Session-HouseKeeper-47a3d56a-1"
2022-07-20T16:49:19.350459393Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.offset"
2022-07-20T16:49:26.256350455Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-admin-client-thread | adminclient-8"
2022-07-20T16:49:33.154845201Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-59"
2022-07-20T16:49:34.414951745Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-60"
2022-07-20T16:49:40.871967276Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-61"
2022-07-20T16:49:56.007111292Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "debezium-sqlserverconnector-REDACTED-change-event-source-coordinator"
2022-07-20T16:50:00.410800756Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-admin-client-thread | connector-adminclient-REDACTED-0"
Some context:
I transfer a table with rows of ~1000 bytes each. There is a useful max.queue.size.in.bytes setting which I expect to work as a top bound for the connector, but my dashboard shows that the queue size goes only as much as 122Mi fluctuating between 30Mi and 60Mi most of the time. My calculations are 1000*16384 should never reach 512Mi.
In this particular case, the connector had been working for about 3 hours normally and then there was no new data for streaming for a few minutes. Soon after that, the OutOfMemory appeared.
I also notice that every time it happens, the container takes its maximum CPU allowed (which is 4 cores in my case).