I am unable to restore the Db2 backup. Getting error while executing the following line
DB2 Version : 9.7, AIX Version 5.3
rollforward db VATAMC2 stop
SQL0752N Connecting to a database is not permitted within a logical unit of
work when the CONNECT type 1 setting is in use. SQLSTATE=0A001
Attached are the set of commands that I executed to restore the db.
I am getting Error due to the last statement only and the rest all the statements are executed successfully.
Logs attached.
Can someone help me out with the same?
Code which I executed:
restore db PRODVAT user mstadmin using password from /data/db2_backupfs taken at 20160304 into VATAMC2 redirect without prompting;
set tablespace containers for 29 using (path '/data/VAT_AMC2/tbs29/cont1');
set tablespace containers for 46 using (path '/data/VAT_AMC2/tbs46/cont1');
set tablespace containers for 47 using (path '/data/VAT_AMC2/tbs47/cont1');
set tablespace containers for 43 using (path '/data/VAT_AMC2/tbs43/cont1');
set tablespace containers for 48 using (path '/data/VAT_AMC2/tbs48/cont1');
set tablespace containers for 44 using (path '/data/VAT_AMC2/tbs44/cont1');
set tablespace containers for 34 using (file '/data/VAT_AMC2/tbs34/cont1' 589824);
set tablespace containers for 35 using (file '/data/VAT_AMC2/tbs35/cont1' 4680192, file '/data/VAT_AMC2/tbs35/cont1a' 4680192);
set tablespace containers for 36 using (file '/data/VAT_AMC2/tbs36/cont2' 3669984, file '/data/VAT_AMC2/tbs36/cont2a' 3669984);
set tablespace containers for 37 using (file '/data/VAT_AMC2/tbs37/cont1' 3898624, file '/data/VAT_AMC2/tbs37/cont1a' 3898624);
set tablespace containers for 38 using (file '/data/VAT_AMC2/tbs38/cont1' 3295788 , file '/data/VAT_AMC2/tbs38/cont1a' 1647892);
set tablespace containers for 39 using (file '/data/VAT_AMC2/tbs39/cont1' 3669976 , file '/data/VAT_AMC2/tbs39/cont12a' 3669976, file '/data/VAT_AMC2/tbs39/cont12b' 3669976, file '/data/VAT_AMC2/tbs39/cont13a' 3669976);
set tablespace containers for 40 using (file '/data/VAT_AMC2/tbs40/cont1' 5509291, file '/data/VAT_AMC2/tbs40/cont1a' 5509290, file '/data/VAT_AMC2/tbs40/cont1b' 5509290);
set tablespace containers for 41 using (file '/data/VAT_AMC2/tbs41/cont1' 2294656);
set tablespace containers for 42 using (file '/data/VAT_AMC2/tbs42/cont1' 4718592);
set tablespace containers for 45 using (file '/data/VAT_AMC2/tbs45/cont1' 1987872, file '/data/VAT_AMC2/tbs45/cont2' 1987872);
set tablespace containers for 49 using (file '/data/VAT_AMC2/tbs49/cont1' 4476974, file '/data/VAT_AMC2/tbs49/cont2' 4476973, file '/data/VAT_AMC2/tbs49/cont3' 4476973);
set tablespace containers for 50 using (file '/data/VAT_AMC2/tbs50/cont1' 3744512);
set tablespace containers for 51 using (file '/data/VAT_AMC2/tbs51/cont1' 5639424);
set tablespace containers for 52 using (file '/data/VAT_AMC2/tbs52/cont1' 4132096);
set tablespace containers for 53 using (file '/data/VAT_AMC2/tbs53/cont1' 3932160);
set tablespace containers for 54 using (file '/data/VAT_AMC2/tbs54/cont1' 4469376, file '/data/VAT_AMC2/tbs54/cont2' 4469376, file '/data/VAT_AMC2/tbs54/cont3' 4469376);
set tablespace containers for 55 using (file '/data/VAT_AMC2/tbs55/cont1' 4194304, file '/data/VAT_AMC2/tbs55/cont2a' 4194304, file '/data/VAT_AMC2/tbs55/cont2b' 4194304, file '/data/VAT_AMC2/tbs55/cont3a' 4194304);
restore db PRODVAT continue;
rollforward db VATAMC2 stop;
Db2 Log:
2016-03-10-09.06.41.911428-360 E6918736A469 LEVEL: Event
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::FirstConnect, probe:1000
START : DATABASE: VATAMC2 : ACTIVATED: NO
2016-03-10-09.06.41.913584-360 E6919206A481 LEVEL: Info
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqluxGetDegreeParallelism, probe:774
DATA #1 :
Autonomic BAR - using parallelism = 2.
2016-03-10-09.06.41.915158-360 E6919688A492 LEVEL: Info
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqludPrintStartingMsg, probe:1338
DATA #1 :
Starting a full database restore.
Agent EDU ID: 2572
2016-03-10-09.06.41.915354-360 E6920181A554 LEVEL: Warning
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqluCompareDB, probe:1445
DATA #1 : Sqlcode, PD_TYPE_SQLCODE, 4 bytes
2528
DATA #2 : Hexdump, 4 bytes
0x07000000491947D8 : 0000 09E0 ....
2016-03-10-09.06.41.915502-360 E6920736A985 LEVEL: Warning
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqluCompareDB, probe:1445
MESSAGE : SQL2540W Restore is successful, however a warning "" was encountered
during Database Restore while processing in No Interrupt mode.
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: 2540 sqlerrml: 4
sqlerrmc: 2528
sqlerrp : sqluComp
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2016-03-10-09.06.41.919710-360 E6921722A520 LEVEL: Info
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqluxGetAvailableHeapPages, probe:888
DATA #1 :
Autonomic BAR - heap consumption.
Targetting (90%) - 112118 of 124576 pages.
2016-03-10-09.06.41.919864-360 E6922243A508 LEVEL: Info
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqludTuneBuffers, probe:1139
DATA #1 :
Autonomic restore - tuning enabled.
Using buffer size = 4097, number = 4.
2016-03-10-09.06.42.976640-360 I6922752A522 LEVEL: Error
PID : 389280 TID : 13109 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000
EDUID : 13109 EDUNAME: db2bm.2572.0 (VATAMC2) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbDMSDoContainerOp, probe:810
MESSAGE : ZRC=0x8402001E=-2080243682=SQLB_CONTAINER_NOT_ACCESSIBLE
"Container not accessible"
DATA #1 :
Error checking container 0 (/dsexp11/cont/tbs60/cont1) for tbsp 56. Rc = 870F0011
2016-03-10-09.06.43.094056-360 E6923275A555 LEVEL: Warning
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqludBMResponse, probe:310
DATA #1 : Sqlcode, PD_TYPE_SQLCODE, 4 bytes
1277
DATA #2 : Hexdump, 4 bytes
0x07000000491947D8 : 0000 04FD ....
2016-03-10-09.06.43.094275-360 E6923831A1057 LEVEL: Warning
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqludBMResponse, probe:310
MESSAGE : SQL1277W A redirected restore operation is being performed. Table
space configuration can now be viewed and table spaces that do not
use automatic storage can have their containers reconfigured.
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: 1277 sqlerrml: 0
sqlerrmc:
sqlerrp : sqludBMR
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2016-03-10-09.06.43.883679-360 E6924889A567 LEVEL: Warning
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqludCheckRedirectedStatus, probe:1065
DATA #1 : Sqlcode, PD_TYPE_SQLCODE, 4 bytes
1277
DATA #2 : Hexdump, 4 bytes
0x07000000491947D8 : 0000 04FD ....
2016-03-10-09.06.43.883857-360 E6925457A1069 LEVEL: Warning
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqludCheckRedirectedStatus, probe:1065
MESSAGE : SQL1277W A redirected restore operation is being performed. Table
space configuration can now be viewed and table spaces that do not
use automatic storage can have their containers reconfigured.
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: 1277 sqlerrml: 0
sqlerrmc:
sqlerrp : sqludChe
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2016-03-10-09.06.43.884062-360 I6926527A840 LEVEL: Warning
PID : 389280 TID : 2572 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPHDL : 0-7 APPID: *LOCAL.db2inst1.160310150641
AUTHID : MSTADMIN
EDUID : 2572 EDUNAME: db2agent (VATAMC2) 0
FUNCTION: DB2 UDB, database utilities, sqludCheckRedirectedStatus, probe:1068
DATA #1 :
Insufficient space in tablespace DMS_TB6_TS; you must have at least 6402208
usable pages. (The "usable pages" total does not include pages used
internally by DB2, so the value specified on the SET TABLESPACE
CONTAINERS operation should be increased by one extent per container.
Based on the latest SET TABLESPACE CONTAINERS values specifed, the
tablespace should have 6402272 pages in total.)
2016-03-10-09.06.43.884818-360 I6927368A345 LEVEL: Severe
PID : 360608 TID : 1 PROC : db2bp
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPID : *LOCAL.db2inst1.160310150641
EDUID : 1
FUNCTION: DB2 UDB, base sys utilities, sqleStartUsingDatabase, probe:2
RETCODE : ZRC=0xFFFFFD10=-752
2016-03-10-09.06.43.885281-360 I6927714A376 LEVEL: Error
PID : 360608 TID : 1 PROC : db2bp
INSTANCE: db2inst1 NODE : 000 DB : VATAMC2
APPID : *LOCAL.db2inst1.160310150641
EDUID : 1
FUNCTION: DB2 UDB, data protection services, sqlpsrllf, probe:317
MESSAGE : Rollforward failed to connect to database. sqlcode -752.
Related
I have Java + maven project - https://github.com/petersuchy/mongock-test-java based on Mongock reactive example (https://github.com/mongock/mongock-examples/tree/master/mongodb/springboot-reactive)
And everything works well.
I tried to migrate that project into Kotlin + gradle - https://github.com/petersuchy/mongock-test-kotlin
And I am able to run it. But my ChangeUnit is ignored. Mongock is set up properly beacause in the end I have 2 collections created - mongockLock and mongockChangeLog.
I tried to get rid of #Value annotations in MongockConfig and MongoClientConfig, but there was no change in behaviour.
Can you please point me why is this happening? I think it can be something with these Reflections becasue it is only difference in logs.
Kotlin:
2023-02-12T00:49:58.455+01:00 INFO 80854 --- [ main] i.m.r.c.e.system.SystemUpdateExecutor : Mongock has finished the system update execution
2023-02-12T00:49:58.457+01:00 INFO 80854 --- [ main] org.reflections.Reflections : Reflections took 0 ms to scan 0 urls, producing 0 keys and 0 values
2023-02-12T00:49:58.458+01:00 INFO 80854 --- [ main] org.reflections.Reflections : Reflections took 1 ms to scan 0 urls, producing 0 keys and 0 values
2023-02-12T00:49:58.458+01:00 INFO 80854 --- [ main] i.m.r.c.e.o.migrate.MigrateExecutorBase : Mongock skipping the data migration. There is no change set item.
Java:
2023-02-12T00:29:48.064+01:00 INFO 78548 --- [ main] i.m.r.c.e.system.SystemUpdateExecutor : Mongock has finished the system update execution
2023-02-12T00:29:48.072+01:00 INFO 78548 --- [ main] org.reflections.Reflections : Reflections took 6 ms to scan 1 urls, producing 1 keys and 2 values
2023-02-12T00:29:48.075+01:00 INFO 78548 --- [ main] org.reflections.Reflections : Reflections took 3 ms to scan 1 urls, producing 1 keys and 2 values
2023-02-12T00:29:48.081+01:00 INFO 78548 --- [ main] i.m.driver.core.lock.LockManagerDefault : Mongock trying to acquire the lock
Here is the full log from Kotlin project
2023-02-12T00:49:55.863+01:00 INFO 80854 --- [ main] c.e.m.MongockTestKotlinApplicationKt : No active profile set, falling back to 1 default profile: "default"
2023-02-12T00:49:56.764+01:00 INFO 80854 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Reactive MongoDB repositories in DEFAULT mode.
2023-02-12T00:49:57.019+01:00 INFO 80854 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 246 ms. Found 1 Reactive MongoDB repository interfaces.
2023-02-12T00:49:57.919+01:00 INFO 80854 --- [ main] org.mongodb.driver.client : MongoClient with metadata {"driver": {"name": "mongo-java-driver|reactive-streams|spring-boot", "version": "4.8.2"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.15.0-60-generic"}, "platform": "Java/Private Build/17.0.5+8-Ubuntu-2ubuntu122.04"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=NettyStreamFactoryFactory{eventLoopGroup=io.netty.channel.nio.NioEventLoopGroup#631cb129, socketChannelClass=class io.netty.channel.socket.nio.NioSocketChannel, allocator=PooledByteBufAllocator(directByDefault: true), sslContext=null}, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider#3d20e575]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=100, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=JAVA_LEGACY, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-02-12T00:49:57.968+01:00 INFO 80854 --- [ main] i.m.r.core.builder.RunnerBuilderBase : Mongock runner COMMUNITY version[5.2.2]
2023-02-12T00:49:57.970+01:00 INFO 80854 --- [ main] i.m.r.core.builder.RunnerBuilderBase : Running Mongock with NO metadata
2023-02-12T00:49:58.034+01:00 INFO 80854 --- [localhost:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=62515465, setName='myReplicaSet', canonicalAddress=mongo1:27017, hosts=[mongo3:27017, mongo2:27017, mongo1:27017], passives=[], arbiters=[], primary='mongo1:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000002, setVersion=1, topologyVersion=TopologyVersion{processId=63e7c5a7d11b71e048698dab, counter=6}, lastWriteDate=Sun Feb 12 00:49:53 CET 2023, lastUpdateTimeNanos=45870970528894}
2023-02-12T00:49:58.336+01:00 INFO 80854 --- [ main] org.reflections.Reflections : Reflections took 33 ms to scan 1 urls, producing 2 keys and 2 values
2023-02-12T00:49:58.343+01:00 INFO 80854 --- [ main] org.reflections.Reflections : Reflections took 2 ms to scan 1 urls, producing 2 keys and 2 values
2023-02-12T00:49:58.367+01:00 INFO 80854 --- [ main] i.m.driver.core.lock.LockManagerDefault : Mongock trying to acquire the lock
2023-02-12T00:49:58.400+01:00 INFO 80854 --- [ main] i.m.driver.core.lock.LockManagerDefault : Mongock acquired the lock until: Sun Feb 12 00:50:58 CET 2023
2023-02-12T00:49:58.401+01:00 INFO 80854 --- [ Thread-1] i.m.driver.core.lock.LockManagerDefault : Starting mongock lock daemon...
2023-02-12T00:49:58.404+01:00 INFO 80854 --- [ main] i.m.r.c.e.system.SystemUpdateExecutor : Mongock starting the system update execution id[2023-02-12T00:49:57.955733372-712]...
2023-02-12T00:49:58.408+01:00 INFO 80854 --- [ main] i.m.r.c.executor.ChangeLogRuntimeImpl : method[io.mongock.runner.core.executor.system.changes.SystemChangeUnit00001] with arguments: []
2023-02-12T00:49:58.411+01:00 INFO 80854 --- [ main] i.m.r.c.executor.ChangeLogRuntimeImpl : method[beforeExecution] with arguments: [io.mongock.driver.mongodb.reactive.repository.MongoReactiveChangeEntryRepository]
2023-02-12T00:49:58.413+01:00 INFO 80854 --- [ main] i.m.r.core.executor.ChangeExecutorBase : APPLIED - {"id"="system-change-00001_before", "type"="before-execution", "author"="mongock", "class"="SystemChangeUnit00001", "method"="beforeExecution"}
2023-02-12T00:49:58.425+01:00 INFO 80854 --- [ main] i.m.r.c.executor.ChangeLogRuntimeImpl : method[execution] with arguments: [io.mongock.driver.mongodb.reactive.repository.MongoReactiveChangeEntryRepository]
2023-02-12T00:49:58.429+01:00 INFO 80854 --- [ main] i.m.r.core.executor.ChangeExecutorBase : APPLIED - {"id"="system-change-00001", "type"="execution", "author"="mongock", "class"="SystemChangeUnit00001", "method"="execution"}
2023-02-12T00:49:58.447+01:00 INFO 80854 --- [ main] i.m.driver.core.lock.LockManagerDefault : Mongock waiting to release the lock
2023-02-12T00:49:58.447+01:00 INFO 80854 --- [ main] i.m.driver.core.lock.LockManagerDefault : Mongock releasing the lock
2023-02-12T00:49:58.455+01:00 INFO 80854 --- [ main] i.m.driver.core.lock.LockManagerDefault : Mongock released the lock
2023-02-12T00:49:58.455+01:00 INFO 80854 --- [ main] i.m.r.c.e.system.SystemUpdateExecutor : Mongock has finished the system update execution
2023-02-12T00:49:58.457+01:00 INFO 80854 --- [ main] org.reflections.Reflections : Reflections took 0 ms to scan 0 urls, producing 0 keys and 0 values
2023-02-12T00:49:58.458+01:00 INFO 80854 --- [ main] org.reflections.Reflections : Reflections took 1 ms to scan 0 urls, producing 0 keys and 0 values
2023-02-12T00:49:58.458+01:00 INFO 80854 --- [ main] i.m.r.c.e.o.migrate.MigrateExecutorBase : Mongock skipping the data migration. There is no change set item.
2023-02-12T00:49:58.458+01:00 INFO 80854 --- [ main] i.m.r.c.e.o.migrate.MigrateExecutorBase : Mongock has finished
2023-02-12T00:49:59.190+01:00 INFO 80854 --- [ main] o.s.b.web.embedded.netty.NettyWebServer : Netty started on port 8080
2023-02-12T00:49:59.201+01:00 INFO 80854 --- [ main] c.e.m.MongockTestKotlinApplicationKt : Started MongockTestKotlinApplicationKt in 4.086 seconds (process running for 4.773)
The problem is in your application.yam. The migration-scan-package name is wrong. That's the reason Mongock doesn't find any ChangeUnit
Debezium server: v 1.9.0.Final
MongoDB Atlas: v 4.2.20
Running on AWS ECS with Fargate w/ 1GB CPU & 4GB MEMORY
Overview:
Debezium starts an initial snapshot and it sends some data to kinesis but it runs into an error (below) before it finishes the snapshot. I've tried increasing the memory of the container to 4GB but not sure if that's the issue. The one collection I'm testing this with is 28GB total and 11M documents.
Debezium config (in Terraform):
environment = [
{
"name" : "DEBEZIUM_SINK_TYPE",
"value" : "kinesis"
},
{
"name" : "DEBEZIUM_SINK_KINESIS_REGION",
"value" : "us-east-1"
},
{
"name" : "DEBEZIUM_SINK_KINESIS_CREDENTIALS_PROFILE",
"value" : "default"
},
{
"name" : "DEBEZIUM_SINK_KINESIS_ENDPOINT",
"value" : "https://kinesis.us-east-1.amazonaws.com"
},
{
"name" : "DEBEZIUM_SOURCE_CONNECTOR_CLASS",
"value" : "io.debezium.connector.mongodb.MongoDbConnector"
},
{
"name" : "DEBEZIUM_SOURCE_OFFSET_STORAGE_FILE_FILENAME",
"value" : "data/offsets.dat"
},
{
"name" : "DEBEZIUM_SOURCE_OFFSET_FLUSH_INTERVAL_MS",
"value" : "0"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_NAME",
"value" : "test"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_HOSTS",
"value" : "test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_SSL_ENABLED",
"value" : "true"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_MEMBERS_AUTO_DISCOVER",
"value" : "true"
},
{
"name" : "DEBEZIUM_SOURCE_DATABASE_INCLUDE_LIST",
"value" : "test"
},
{
"name" : "DEBEZIUM_SOURCE_COLLECTION_INCLUDE_LIST",
"value" : "test.testCollection"
},
{
"name" : "DEBEZIUM_SOURCE_CAPTURE_MODE",
"value" : "change_streams_update_full"
},
{
"name" : "DEBEZIUM_SOURCE_DATABASE_HISTORY",
"value" : "io.debezium.relational.history.FileDatabaseHistory"
},
{
"name" : "DEBEZIUM_SOURCE_DATABASE_HISTORY_FILE_FILENAME",
"value" : "history.dat"
},
{
"name" : "QUARKUS_LOG_CONSOLE_JSON",
"value" : "false"
}
]
secrets = [
{
"name" : "DEBEZIUM_SOURCE_MONGODB_USER",
"valueFrom" : "${data.aws_secretsmanager_secret.test-debezium-read.arn}:username::"
},
{
"name" : "DEBEZIUM_SOURCE_MONGODB_PASSWORD",
"valueFrom" : "${data.aws_secretsmanager_secret.test-debezium-read.arn}:password::"
}
]
Stacktrace:
2022-06-01 18:22:23,976 ERROR [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Error while attempting to sync 'test-mongodb-shard-0.test.testCollection': : java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
at java.base/java.lang.StringBuffer.append(StringBuffer.java:317)
at java.base/java.io.StringWriter.write(StringWriter.java:106)
at org.bson.json.StrictCharacterStreamJsonWriter.write(StrictCharacterStreamJsonWriter.java:368)
at org.bson.json.StrictCharacterStreamJsonWriter.writeStartObject(StrictCharacterStreamJsonWriter.java:204)
at org.bson.json.LegacyExtendedJsonDateTimeConverter.convert(LegacyExtendedJsonDateTimeConverter.java:22)
at org.bson.json.LegacyExtendedJsonDateTimeConverter.convert(LegacyExtendedJsonDateTimeConverter.java:19)
at org.bson.json.JsonWriter.doWriteDateTime(JsonWriter.java:129)
at org.bson.AbstractBsonWriter.writeDateTime(AbstractBsonWriter.java:394)
at org.bson.codecs.DateCodec.encode(DateCodec.java:32)
at org.bson.codecs.DateCodec.encode(DateCodec.java:29)
at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:203)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:200)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:200)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:159)
at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:46)
at org.bson.Document.toJson(Document.java:453)
at io.debezium.connector.mongodb.JsonSerialization.lambda$new$0(JsonSerialization.java:57)
at io.debezium.connector.mongodb.JsonSerialization$$Lambda$521/0x0000000840448840.apply(Unknown Source)
at io.debezium.connector.mongodb.JsonSerialization.getDocumentValue(JsonSerialization.java:89)
at io.debezium.connector.mongodb.MongoDbSchema$$Lambda$580/0x00000008404ce840.apply(Unknown Source)
at io.debezium.connector.mongodb.MongoDbCollectionSchema.valueFromDocumentOplog(MongoDbCollectionSchema.java:90)
at io.debezium.connector.mongodb.MongoDbChangeSnapshotOplogRecordEmitter.emitReadRecord(MongoDbChangeSnapshotOplogRecordEmitter.java:68)
at io.debezium.connector.mongodb.MongoDbChangeSnapshotOplogRecordEmitter.emitReadRecord(MongoDbChangeSnapshotOplogRecordEmitter.java:27)
at io.debezium.pipeline.AbstractChangeRecordEmitter.emitChangeRecords(AbstractChangeRecordEmitter.java:42)
at io.debezium.pipeline.EventDispatcher.dispatchSnapshotEvent(EventDispatcher.java:163)
I noticed that during the snapshot, the number of records sent and the last recorded offset doesn't seem to change while the amount of time elapsed between each of those messages gets longer. This seems like an exponential backoff thing but I'm not entirely sure.
Example:
2022-06-01 16:20:37,789 INFO [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Beginning snapshot of 'test-mongodb-shard-0' at {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:20:37,804 INFO [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Exporting data for collection 'test-mongodb-shard-0.test.testCollection'
2022-06-01 16:20:42,983 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 717 records sent during previous 00:00:06.159, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:20:57,417 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:14.434, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:21:05,107 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
2022-06-01 16:21:16,624 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:19.207, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:21:35,107 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
2022-06-01 16:21:53,130 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:36.505, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:22:05,107 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:23:17,521 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:01:24.391, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:23:35,106 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:26:06,523 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:02:49.003, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:26:35,107 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:31:18,075 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:05:11.552, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:31:35,106 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:42:07,711 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:10:49.636, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:42:35,106 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 17:03:12,872 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:21:05.161, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 17:03:35,117 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 17:45:58,637 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:42:45.765, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 17:46:05,106 INFO [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 18:22:23,976 ERROR [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Error while attempting to sync 'test-mongodb-shard-0.test.testCollection': : java.lang.OutOfMemoryError: Java heap space
Beside increasing the container memory to 4GB , you can set also bigger heap size , the maximum and initial heap size can be set for example to 2GB :
-Xms2048m -Xmx2048m
If the issue continue follow this steps:
Start JVM with argument -XX:+HeapDumpOnOutOfMemoryError this will give you a heap dump when program goes in to OOM.
Use a tool like visualVM to analyze the heap dump obtained. That will help in identifying the memory leak.
Not really answering the OP, just wanted to share my experience.
I too occasionally receive java.lang.OutOfMemoryError and would like to find out what's causing it.
My setup:
Debezium 1.9.5
Kafka 2.8
Docker container memory - 6Gi
Java heap - 4Gi both min and max
max.queue.size.in.bytes - 512Mi
max.batch.size - 16384
The errors from stdout:
2022-07-20T16:47:16.348943181Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.config"
2022-07-20T16:47:27.628395682Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.status"
2022-07-20T16:47:28.970536167Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | connector-producer-REDACTED-202207200823-0"
2022-07-20T16:47:33.787361085Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-3"
2022-07-20T16:47:45.067373810Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "server-timer"
2022-07-20T16:47:46.987669188Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | REDACTED-dbhistory"
2022-07-20T16:48:03.396881812Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-2"
2022-07-20T16:48:04.017710798Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-coordinator-heartbeat-thread | production"
2022-07-20T16:48:09.709036280Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "prometheus-http-1-3"
2022-07-20T16:48:14.667691706Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "mysql-cj-abandoned-connection-cleanup"
2022-07-20T16:48:17.182623196Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-62"
2022-07-20T16:48:25.227925660Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-58"
2022-07-20T16:48:43.598026645Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP-Dispatcher"
2022-07-20T16:48:45.543984655Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-1"
2022-07-20T16:48:52.284810255Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SourceTaskOffsetCommitter-1"
2022-07-20T16:48:56.992674380Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DistributedHerder-connect-1-1"
2022-07-20T16:49:18.691603140Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Session-HouseKeeper-47a3d56a-1"
2022-07-20T16:49:19.350459393Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.offset"
2022-07-20T16:49:26.256350455Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-admin-client-thread | adminclient-8"
2022-07-20T16:49:33.154845201Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-59"
2022-07-20T16:49:34.414951745Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-60"
2022-07-20T16:49:40.871967276Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-61"
2022-07-20T16:49:56.007111292Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "debezium-sqlserverconnector-REDACTED-change-event-source-coordinator"
2022-07-20T16:50:00.410800756Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-admin-client-thread | connector-adminclient-REDACTED-0"
Some context:
I transfer a table with rows of ~1000 bytes each. There is a useful max.queue.size.in.bytes setting which I expect to work as a top bound for the connector, but my dashboard shows that the queue size goes only as much as 122Mi fluctuating between 30Mi and 60Mi most of the time. My calculations are 1000*16384 should never reach 512Mi.
In this particular case, the connector had been working for about 3 hours normally and then there was no new data for streaming for a few minutes. Soon after that, the OutOfMemory appeared.
I also notice that every time it happens, the container takes its maximum CPU allowed (which is 4 cores in my case).
We have Db2 10.5.0.7 on centos 6.9 and TSAMP 3.2 as our high availability solution, when we create a tablespace in primary database we encounter the following errors in the standby:
2019-08-31-08.47.32.164952+270 I87056E2779 LEVEL: Error (OS)
PID : 4046 TID : 47669095425792 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : SAMDB
APPHDL : 0-8 APPID: *LOCAL.DB2.190725231126
HOSTNAME: samdb-b EDUID : 155 EDUNAME: db2redom
(SAMDB) 0 FUNCTION: DB2 Common, OSSe, ossGetDiskInfo, probe:130
MESSAGE : ECF=0x90000001=-1879048191=ECF_ACCESS_DENIED
Access denied CALLED : OS, -, fopen OSERR: EACCES (13) DATA #1 : String, 12 bytes /proc/mounts DATA #2 :
String, 25 bytes /dbdata1/samdbTsContainer DATA #3 : unsigned integer,
8 bytes
2019-08-31-08.47.32.185625+270 E89836E494 LEVEL: Error PID
: 4046 TID : 47669095425792 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : SAMDB
APPHDL : 0-8 APPID: *LOCAL.DB2.190725231126
HOSTNAME: samdb-b EDUID : 155 EDUNAME: db2redom
(SAMDB) 0 FUNCTION: DB2 UDB, high avail services,
sqlhaGetLocalDiskInfo, probe:9433 MESSAGE :
ECF=0x90000001=-1879048191=ECF_ACCESS_DENIED
Access denied
2019-08-31-08.47.32.186258+270 E90331E484 LEVEL: Error PID
: 4046 TID : 47669095425792 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : SAMDB
APPHDL : 0-8 APPID: *LOCAL.DB2.190725231126
HOSTNAME: samdb-b EDUID : 155 EDUNAME: db2redom
(SAMDB) 0 FUNCTION: DB2 UDB, high avail services, sqlhaCreateMount,
probe:9746 RETCODE : ZRC=0x827300AA=-2106392406=HA_ZRC_FAILED "SQLHA
API call error"
2019-08-31-08.47.32.186910+270 I90816E658 LEVEL: Error PID
: 4046 TID : 47669095425792 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : SAMDB
APPHDL : 0-8 APPID: *LOCAL.DB2.190725231126
HOSTNAME: samdb-b EDUID : 155 EDUNAME: db2redom
(SAMDB) 0 FUNCTION: DB2 UDB, buffer pool services,
sqlbDMSAddContainerRequest, probe:812 MESSAGE :
ZRC=0x827300AA=-2106392406=HA_ZRC_FAILED "SQLHA API call error" DATA
: String, 36 bytes Cluster add mount operation failed: DATA #2 : String, 37 bytes /dbdata1/samdbTsContainer/TSPKGCACH.1 DATA #3 :
String, 8 bytes SAMDB
2019-08-31-08.47.32.190537+270 E113909E951 LEVEL: Error PID
: 4046 TID : 47669095425792 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : SAMDB
APPHDL : 0-8 APPID: *LOCAL.DB2.190725231126
HOSTNAME: samdb-b EDUID : 155 EDUNAME: db2redom
(SAMDB) 0 FUNCTION: DB2 UDB, buffer pool services,
sqlblog_reCreatePool, probe:3134 MESSAGE : ADM6106E Table space
"TSPKGCACH" (ID = "49") could not be created
during the rollforward operation. The most likely cause is that there
is not enough space to create the containers associated with the
table space. Connect to the database after the rollforward operation
completes and use the SET TABLESPACE CONTAINERS command to assign
containers to the table space. Then, issue another ROLLFORWARD
DATABASE command to complete recovery of this table space.
2019-08-31-08.47.32.200949+270 E114861E592 LEVEL: Error PID
: 4046 TID : 47669095425792 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : SAMDB
APPHDL : 0-8 APPID: *LOCAL.DB2.190725231126
HOSTNAME: samdb-b EDUID : 155 EDUNAME: db2redom
(SAMDB) 0 FUNCTION: DB2 UDB, buffer pool services, sqlbIncPoolState,
probe:4628 MESSAGE : ADM12512W Log replay on the HADR standby has
stopped on table space
"TSPKGCACH" (ID "49") because it has been put into "ROLLFORWARD
PENDING" state.
There is free space available for the database and the specified path (/dbdata1/samdbTsContainer) exists on the server and we can create file manually on it .
all settings are equivalent on the primary and standby. db2inst1 is the owner of /dbdata1/samdbTsContainer and permission is drwxr-xr-x, the result of su - db2inst1 “ulimit -Hf” is unlimited and ext3 is file system type and create tablespace statement is as follows:
CREATE LARGE TABLESPACE TSPKGCACH IN DATABASE PARTITION GROUP IBMDEFAULTGROUP PAGESIZE 8 K MANAGED BY DATABASE USING (FILE '/dbdata1/samdbTsContainer/TSPKGCACH.1' 5120) ON DBPARTITIONNUM (0) EXTENTSIZE 64 PREFETCHSIZE 64 BUFFERPOOL BP8KPKGCACH OVERHEAD 10.5 TRANSFERRATE 0.14 DATA TAG NONE NO FILE SYSTEM CACHING;
SELinux is disabled and the sector size is 512 bytes. The mount options are as follws:
/dev/sdf1 /dbdata1 ext3 rw,relatime,errors=continue,barrier=1,data=ordered 0 0
We can not recreate the problem sometimes this problem occur and we don't know the reason of it, but the problem remains until server reboot.
When we restart the standby server problem solves but we need to drop the tablespace and recreate it, is there any idea for this problem?
From the error it looks to me that problem is not with the file access itself but rather /proc/mounts, which Db2 uses to do the mapping between containers and filesystems (to know e.g. the FS type). Hence I suggest to test whether all:
cat /proc/mounts
cat /proc/self/mounts
mount
work OK run as Db2 instance owner ID (db2inst1). If not, this implies some odd OS issue that Db2 is a victim of and we would need more OS diagnostics (e.g strace from the cat /proc/mounts command) to understand it.
Edit:
To confirm this theory I've run a quick test with Db2 11.1. Note this must be TSA-controlled environment for Db2 to follow sqlhaCreateMount code path (because if this will be a separate mount, Db2 will add it to the TSA resource model)
On both primary and standby:
mkdir /db2data
chown db2v111:db2iadm /db2data
then on standby:
chmod o-rx /proc
(couldn't find a "smarter" way to hit EACCES on mount info).
When I will run on primary:
db2 "create tablespace test managed by database using (file '/db2data/testts' 100 M)"
it completes fine on primary but standby hits exactly the error you are seeing:
2019-06-21-03.00.37.087693+120 I1774E2661 LEVEL: Error (OS)
PID : 10379 TID : 46912992438016 PROC : db2sysc 0
INSTANCE: db2v111 NODE : 000 DB : SAMPLE
APPHDL : 0-4492 APPID: *LOCAL.DB2.190621005919
HOSTNAME: rhel-hadrs.kkuduk.com
EDUID : 61 EDUNAME: db2redom (SAMPLE) 0
FUNCTION: DB2 Common, OSSe, ossGetDiskInfo, probe:130
MESSAGE : ECF=0x90000001=-1879048191=ECF_ACCESS_DENIED
Access denied
CALLED : OS, -, fopen OSERR: EACCES (13)
DATA #1 : String, 12 bytes
/proc/mounts
DATA #2 : String, 8 bytes
/db2data
DATA #3 : unsigned integer, 8 bytes
1
CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol)
[0] 0x00002AAAB9CFD84B /home/db2v111/sqllib/lib64/libdb2osse.so.1 + 0x23F84B
[1] 0x00002AAAB9CFED51 ossLogSysRC + 0x101
[2] 0x00002AAAB9D19647 ossGetDiskInfo + 0xF07
[3] 0x00002AAAAC52402C _Z21sqlhaGetLocalDiskInfoPKcjPcjS1_jS1_ + 0x26C
[4] 0x00002AAAAC523C5F _Z16sqlhaGetDiskInfoPKcS0_jPcjS1_jS1_ + 0x29F
[5] 0x00002AAAAC521CA0 _Z16sqlhaCreateMountPKcS0_m + 0x350
[6] 0x00002AAAACDE8D5D _Z26sqlbDMSAddContainerRequestP12SQLB_POOL_CBP16SQLB_POOLCONT_CBP12SQLB_GLOBALSP14SQLB_pfParIoCbbm + 0x90D
[7] 0x00002AAAACE14FF9 _Z29sqlbDoDMSAddContainerRequestsP12SQLB_POOL_CBP16SQLB_POOLCONT_CBjP26SQLB_AS_CONT_AND_PATH_INFOP12SQLB_GLOBALS + 0x2D9
[8] 0x00002AAAACE0C20F _Z17sqlbDMSCreatePoolP12SQLB_POOL_CBiP16SQLB_POOLCONT_CBbP12SQLB_GLOBALS + 0x103F
[9] 0x00002AAAACDB1EAC _Z13sqlbSetupPoolP12SQLB_GLOBALSP12SQLB_POOL_CBPKciiiihiP19SQLB_CONTAINER_SPECllblsib + 0xE4C
-> it is an issue with /proc/mounts access, not the target path itself, where i can write with no issues:
[db2v111#rhel-hadrs ~]$ echo "test" > /db2data/testfile
If that would be path access issue:
chmod o+rx /proc
chmod a-rw /db2data
then an error during the "CREATE TABLESPACE" redo on standby will be different:
2019-06-21-03.07.29.175486+120 I35023E592 LEVEL: Error
PID : 10379 TID : 46912992438016 PROC : db2sysc 0
INSTANCE: db2v111 NODE : 000 DB : SAMPLE
APPHDL : 0-4492 APPID: *LOCAL.DB2.190621005919
HOSTNAME: rhel-hadrs.kkuduk.com
EDUID : 61 EDUNAME: db2redom (SAMPLE) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbCreateAndLockParent, probe:918
MESSAGE : ZRC=0x8402001E=-2080243682=SQLB_CONTAINER_NOT_ACCESSIBLE
"Container not accessible"
DATA #1 : <preformatted>
Failed at directory /db2data.
2019-06-21-03.07.29.175799+120 I35616E619 LEVEL: Severe
PID : 10379 TID : 46912992438016 PROC : db2sysc 0
INSTANCE: db2v111 NODE : 000 DB : SAMPLE
APPHDL : 0-4492 APPID: *LOCAL.DB2.190621005919
HOSTNAME: rhel-hadrs.kkuduk.com
EDUID : 61 EDUNAME: db2redom (SAMPLE) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbCreateAndLockParent, probe:722
MESSAGE : ZRC=0x8402001E=-2080243682=SQLB_CONTAINER_NOT_ACCESSIBLE
"Container not accessible"
DATA #1 : <preformatted>
Failed to create a portion of the path /db2data/testts2
(few more errors follow pointing directly to the permissions on /db2data)
This proves it is the /proc access issue and you need to debug it with your OS team. Perhaps /proc gets completely unmounted?
In any case, the actual issue is db2sysc process hitting EACCES running fopen on /proc/mounts and you need debug it further with OS team.
Edit:
When it comes to the debugging and proving the error is returned by the OS, we would have to trace open() syscalls done by Db2. Strace can do that, but overhead is too high for a production system. If you can get SystemTap installed on the system, I suggest a script like this (this is a basic version):
probe nd_syscall.open.return
{
if ( user_string( #entry( pointer_arg(1) ) ) =~ ".*mounts")
{
printf("exec: %s pid: %d uid: %d (euid: %d) gid: %d (egid: %d) run open(%s) rc: %d\n", execname(), pid(), uid(), euid(), gid(), egid(), user_string(#entry(pointer_arg(1)), "-"), returnval() )
}
}
it uses nd_syscall probe, so it will work even without kernel debuginfo package. You can run it like this:
$ stap open.stap
exec: cat pid: 24159 uid: 0 (euid: 0) gid: 0 (egid: 0) run open(/proc/mounts) rc: 3
exec: cat pid: 24210 uid: 0 (euid: 0) gid: 0 (egid: 0) run open(/proc/mounts) rc: 3
exec: cat pid: 24669 uid: 1111 (euid: 1111) gid: 1001 (egid: 1001) run open(/proc/mounts) rc: 3
exec: cat pid: 24734 uid: 1111 (euid: 1111) gid: 1001 (egid: 1001) run open(/proc/mounts) rc: -13
exec: cat pid: 24891 uid: 1111 (euid: 1111) gid: 1001 (egid: 1001) run open(/proc/self/mounts) rc: -13
exec: ls pid: 24971 uid: 1111 (euid: 1111) gid: 1001 (egid: 1001) run open(/proc/mounts) rc: -13
-> at some point I've revoked access from /proc and open attempt failed with -13 (EACCES). You just need to enable it on the system when you see the error and see if something is logged when Db2 fails.
For example, my postgresql-main.log contains entries such as the following:
process 20234 session 5901502e.4f0a vtransaction transaction 0 : LOG: connection received: host=[local]
process 20234 session 5901502e.4f0a vtransaction 2/1 transaction 0 : LOG: connection authorized: user=postgres database=postgres
process 20234 session 5901502e.4f0a vtransaction 2/2 transaction 0 : LOG: statement: SELECT d.datname as "Name",
pg_catalog.pg_get_userbyid(d.datdba) as "Owner",
pg_catalog.pg_encoding_to_char(d.encoding) as "Encoding",
d.datcollate as "Collate",
d.datctype as "Ctype",
pg_catalog.array_to_string(d.datacl, E'\n') AS "Access privileges"
FROM pg_catalog.pg_database d
ORDER BY 1;
process 20234 session 5901502e.4f0a vtransaction transaction 0 : LOG: disconnection: session time: 0:00:00.004 user=postgres database=postgres host=[local]
process 20237 session 5901502f.4f0d vtransaction transaction 0 : LOG: connection received: host=[local]
process 20237 session 5901502f.4f0d vtransaction 2/3 transaction 0 : LOG: connection authorized: user=postgres database=postgres
2017-04-26 19:58:07 MDT process 20237 remote [local] session 5901502f.4f0d vtransaction 2/4 transaction 0 : LOG: statement: SELECT d.datname as "Name",
pg_catalog.pg_get_userbyid(d.datdba) as "Owner",
pg_catalog.pg_encoding_to_char(d.encoding) as "Encoding",
d.datcollate as "Collate",
d.datctype as "Ctype",
pg_catalog.array_to_string(d.datacl, E'\n') AS "Access privileges"
FROM pg_catalog.pg_database d
ORDER BY 1;
process 20237 session 5901502f.4f0d vtransaction transaction 0 : LOG: disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
Is there any relation between virtual transaction ids 2/1 and 2/2 above since they share the prefix 2/?
I am working on Db2 ATS jobs.
This is what I have tried. Created one procedure p1 on database d1.This procedure will update a table t1 on database d1.From the procedure p1 on a direct call i can update table t1.
And now I have created one table t2 on database d2. From database d1 I am trying to update this table t2 . This is also working fine.
From the database d1 I can update the two tables t1 & t2.
Now I tried the same with a ATS jobs.
I can submit a job to call procedure p1 to update table t1 on database d1 .The job is working fine & able to view the updated records.
tried to submit a job from d1 which will update a table t2 on database d2.
Job is running fine without any error. But the table is not updating.
Below is my procedure & logs Can some one tell me what I am missing here:
CREATE or REPLACE PROCEDURE D_JUMP12(IN text VARCHAR(32672))
p1:BEGIN
DECLARE jobno NUMBER;
execute IMMEDIATE TEXT;
COMMIT;
END p1
This is the way I am calling the procedure:
call DBMS_JOB.SUBMIT (?, 'CALL D_JUMP12(''UPDATE sample.reg.CABS SET FIRSTNAME = reg_f_fixed_char_repos(FIRSTNAME)'')',NULL);
These are my logs:
2015-03-17-18.37.10.360000+330 E110713390F554 LEVEL: Error
PID : 2460 TID : 444 PROC : db2fmp64.exe
INSTANCE: DB2 NODE : 000 DB : test1
APPID : *LOCAL.DB2.150317130717
HOSTNAME: MANGO-04
EDUID : 444
FUNCTION: DB2 UDB, Administrative Task Scheduler, AtsConnection::commit, probe:900
MESSAGE : ECF=0x82BA00E1=-2101739295
DATA #1 : String, 124 bytes
[IBM][CLI Driver][DB2/NT64] SQL0903N COMMIT statement failed, transaction rolled back. Reason code: "2". SQLSTATE=40504
2015-03-17-18.37.10.364000+330 I110713946F425 LEVEL: Error
PID : 2460 TID : 444 PROC : db2fmp64.exe
INSTANCE: DB2 NODE : 000 DB : test1
APPID : *LOCAL.DB2.150317130717
HOSTNAME: MANGO-04
EDUID : 444
FUNCTION: DB2 UDB, Administrative Task Scheduler, AtsTask::run, probe:1300
MESSAGE : ECF=0x82BA00E1=-2101739295
DATA #1 : Codepath, 8 bytes
4
Kindly let me know what I am missing here