Postgres automatically switch to READ-ONLY mode - ERROR: cannot execute UPDATE in a read-only transaction - postgresql

When I increase load on PostgreSQL server, by hitting REST APIs through LOAD Testing Tool. Postgres starts throwing exception ERROR: cannot execute UPDATE in a read-only transaction. Once it starts throwing exception, Even If I try to execute a update query directly to PG SQL command line, same exception is thrown there also. Then around 5-10 minutes later, it automatically comes in normal mode.
In the test suite, I have multiple APIs performing mulitple select, update, insert operations on various tables. Here We have one such table having a column with bytea data type stores a excel file in binary format. Don't know this information is relevant or not.
But my concern is why Postgres automatically switches to READ-ONLY mode, that causes my APIs fails abruptly. It is an intermittent issue, we need to increase lots of load on database server to replicate it.I am not sure it is happening due to LOAD or above mentioned "bytea" column having large binary data causing such issue?
PostgreSql version: 12
PG Driver: 42.2.18
Any suggestion or clue please?
below are logs:
Caused by: org.hibernate.exception.GenericJDBCException: could not execute statement
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:190) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:62) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3281) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3183) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
......
....
Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:473) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:393) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164) ~[postgresql-42.2.18.jar:42.2.18]
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:130) ~[postgresql-42.2.18.jar:42.2.18]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) ~[HikariCP-2.4.5.jar:na]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) ~[HikariCP-2.4.5.jar:na]
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:187) ~[hibernate-core-4.3.5.Final.jar:4.3.5.Final]
... 46 common frames omitted

Related

Telemetry data unable to pass through root rule chain with node save timeseries

Things Board Version: V3.4.1 CE
OS: Window
Database: postgreSQL timescale
Queue: Rabbitmq
I discover that the telemetry data unable to pass through things board root rule chain with the node of the name save timeseries, i am not sure what is happening, i confirm there should be no problem on the connection between thingsboard and also postgreSQL...
I can see debug from here to know the problem is because failed to save to timeseries data....
2022-11-02 09:04:27,148 [sql-queue-2-ts timescale-11-thread-1] ERROR o.t.s.dao.sql.TbSqlBlockingQueue - [TS Timescale] Failed to save 2 entities
org.springframework.transaction.TransactionSystemException: Could not roll back JPA transaction; nested exception is org.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:593)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:835)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:809)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:672)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:392)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at org.thingsboard.server.dao.sqlts.insert.timescale.TimescaleInsertTsRepository$$EnhancerBySpringCGLIB$$693764a7.saveOrUpdate()
at org.thingsboard.server.dao.sqlts.timescale.TimescaleTimeseriesDao.lambda$init$1(TimescaleTimeseriesDao.java:89)
at org.thingsboard.server.dao.sql.TbSqlBlockingQueue.lambda$init$2(TbSqlBlockingQueue.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:127)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordinatorImpl.java:304)
at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:142)
at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:589)
... 16 common frames omitted
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:515)
at com.sun.proxy.$Proxy153.rollback(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.rollback(ProxyConnection.java:396)
at com.zaxxer.hikari.pool.HikariProxyConnection.rollback(HikariProxyConnection.java)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:121)
... 19 common frames omitted
2022-11-02 09:04:27,148 [tb-rule-engine-consumer-37-thread-35 | QK(Main,TB_RULE_ENGINE,system)-10] INFO o.t.s.s.q.DefaultTbRuleEngineConsumerService - Failed to process 1 messages
2022-11-02 09:04:27,148 [tb-rule-engine-consumer-37-thread-35 | QK(Main,TB_RULE_ENGINE,system)-10] INFO o.t.s.s.q.DefaultTbRuleEngineConsumerService - [c1737420-58eb-11eb-808a-dfdc947dc52b] Failed to process message: TbMsg(queueName=Main, id=1318aa98-0755-49b0-9685-a71a2326ff7d, ts=1667351067141, type=POST_TELEMETRY_REQUEST, originator=354d8300-aa84-11ec-9a47-4727b3504d5d, customerId=d7094170-5c4c-11eb-b06a-c93fc5e45132, metaData=TbMsgMetaData(data={deviceType=Sensor, deviceName=RMS Voltage Sensor, ts=1667351067141}), dataType=JSON, data={"timestamp":1667351069011,"values":[{"id":"CnB Prai Gateway.RMS Shearline.Sensor5_Active","v":true,"t":1667291491472},{"id":"CnB Prai Gateway.RMS Shearline.Sensor5_Battery","v":296,"t":1667342191745},{"id":"CnB Prai Gateway.RMS Shearline.Sensor5_Signal","v":65478,"t":1667350910940},{"id":"CnB Prai Gateway.RMS Shearline.Sensor5_Voltage","v":0,"t":1667351068948}]}, ruleChainId=c1c082b0-58eb-11eb-808a-dfdc947dc52b, ruleNodeId=null, ctx=org.thingsboard.server.common.msg.TbMsgProcessingCtx#4c99aecc, callback=org.thingsboard.server.common.msg.queue.TbMsgCallback$1#415dca17), Last Rule Node: [RuleChain: Root Rule Chain|RuleNode: Save raw telemetry(71b87e70-177d-11ec-9530-3197ec48e7c5)]
I would suggest to use generator node to test where the problem is.
First you should test if you can save basic message (like the one you get when you open generator node). With this you will confirm that you can save data to database.
After that you should configure generator node to act as your device, and have same data and metadata as you would get from you device/integration.
Reach out back here with your findings from that.
Generator rule node ref: https://thingsboard.io/docs/user-guide/rule-engine-2-0/action-nodes/#generator-node

Debezium: Replicating data from Oracle read-only database

My use case is to capture CDC data from Oracle read-only PDB database using Debezium. When I tried installing and run Debezium, it's throwing below error message
Can someone please help with the correct Debezium config?
ORA-16000: database or pluggable database open for read-only access
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:628)
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:562)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1145)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:726)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:291)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:492)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:144)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1034)
at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1507)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1287)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3735)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3933)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4279)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010)
at io.debezium.connector.oracle.logminer.LogMinerHelper.executeCallableStatement(LogMinerHelper.java:701)
at io.debezium.connector.oracle.logminer.LogMinerHelper.createFlushTable(LogMinerHelper.java:105)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:122)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:63)
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:159)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:122)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: Error : 604, Position : 0, Sql = CREATE TABLE LOG_MINING_FLUSH(LAST_SCN NUMBER(19,0)), OriginalSql = CREATE TABLE LOG_MINING_FLUSH(LAST_SCN NUMBER(19,0)), Error Msg = ORA-00604: error occurred at recursive SQL level 1
ORA-16000: database or pluggable database open for read-only access
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:632)
... 24 more

Table not found on postresql connection from spring boot 2

I am trying to perform a basic connection between to a local postgresql DB from a very basic SpringBootProject (for training purpose).
# configure postgresql
spring.datasource.platform=postgres
spring.datasource.jdbc-url=jdbc:postgresql://localhost:5432/db
spring.datasource.username=postgres
spring.datasource.password=
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.format_sql=true
spring.jpa.database-platform = org.hibernate.dialect.PostgreSQL94Dialect
spring.jpa.hibernate.ddl-auto=none
So far, I have managed to have it compile and run... but when I am trying to access my DB, I get a "Table "FILMS" not found" error (org.h2.jdbc.JdbcSQLException). When I insert data from a data.sql, I get the inserted movies. But without this initializer, I cannot get data from my db...
Table "FILMS" not found; SQL statement:
select film0_.code as
code1_0_, film0_.date_prod as date_pro2_0_, film0_.did as did3_0_,
film0_.kind as kind4_0_, film0_.title as title5_0_ from films film0_
[42102-197]
2018-07-26 12:24:18.151 ERROR 10376 --- [nio-9000-exec-1]
o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for
servlet [dispatcherServlet] in context with path [] threw exception
[Request processing failed; nested exception is
org.springframework.dao.InvalidDataAccessResourceUsageException: could
not prepare statement; SQL [select film0_.code as code1_0_,
film0_.date_prod as date_pro2_0_, film0_.did as did3_0_, film0_.kind
as kind4_0_, film0_.title as title5_0_ from films film0_]; nested
exception is org.hibernate.exception.SQLGrammarException: could not
prepare statement] with root cause
org.h2.jdbc.JdbcSQLException: Table "FILMS" non trouvée
Spring boot is in version 2.03 RELEASE; both postgresql connector AND server's ones are 9.4.
I am using spring boot parent starter, as well as data-jpa-starter (with no extra hibernate/jpa depencies defined).
The source code is here
My postresql server is up, the database "db" exists, and its table public."FILMS" is full of data...
Any idea on what I am doing (or undestanding) wrong?
In your application properties file try
spring.jpa.hibernate.ddl-auto=validate
It will try to check the table in your db

Quartz jobPersistenceException with hsqldb

I'm trying to run a scheduler using quartz. I use HSQLDB (version 2.2.7 ). But I get following exception;
Caused by: org.quartz.JobPersistenceException: Couldn't store job: data exception: string data, right truncation [See nested exception: java.sql.SQLDataException: data exception: string data, right truncation]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1132)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.impl.jdbcjobstore.JobStoreSupport$3.execute(JobStoreSupport.java:1071)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3716)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3712)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1059)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:822)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at com.aepona.ase.services.terminalstatus.utils.SimpleTriggerExample.afterPropertiesSet(SimpleTriggerExample.java:32)[320:com.aepona.ase.services.terminalstatus:3.0.7.VFB-SNAPSHOT]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1514)[439:org.springframework.beans:3.1.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452)[439:org.springframework.beans:3.1.4.RELEASE]
... 14 more
Caused by: java.sql.SQLDataException: data exception: string data, right truncation
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)[315:org.hsqldb.hsqldb:2.2.7]
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)[315:org.hsqldb.hsqldb:2.2.7]
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)[315:org.hsqldb.hsqldb:2.2.7]
at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source)[315:org.hsqldb.hsqldb:2.2.7]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)[301:org.apache.servicemix.bundles.commons-dbcp:1.2.2.7]
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.insertJobDetail(StdJDBCDelegate.java:530)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJob(JobStoreSupport.java:1126)[475:org.apache.servicemix.bundles.quartz:1.8.6.1]
anyone familiar with this issue?
I am using quartz-scheduler 1.8.6. The DB script comes with it for HSQLDB uses BINARY data type in qrtz_job_details, qrtz_triggers, qrtz_calendars and qrtz_blob_triggers tables. I just change BINARY to BLOB and problem solved.

PostgreSQL on cloudfoundry throws PSQLException: FATAL: terminating connection due to administrator command

I have a grails application successfully running in cloudfoundry with H2 as database. I now want to switch to postgresql. Everything works as expected when I run the application locally against postgreSQL.
On the cloudfoundry instance I get an exception during one rather long running analysis, which I perform as a background task (using grails executor plugin). Within this asynchronous task
def future = executorService.submit({
return analysisService.analyzeProject(model, project)
})
I get the following exception from the database:
2012-11-26 10:27:38,319 [pool-2-thread-1] ERROR interceptor.TransactionInterceptor - Application exception overridden by rollback exception
org.springframework.dao.DataAccessResourceFailureException: Hibernate operation: could not execute query; SQL [select this_.id as id8_0_, this_.version as version8_0_, this_.language as language8_0_, this_.url as url8_0_ from sonar_adapter_configuration this_]; FATAL: terminating connection due to administrator command; nested exception is org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at myapp.adapters.sonar.SonarAdapterService.loadSonarConfig(SonarAdapterService.groovy:184)
at myapp.adapters.sonar.SonarAdapterService.determineArtefactSizes(SonarAdapterService.groovy:145)
at myapp.project.AnalysisService.analyzeProject(AnalysisService.groovy:46)
at myapp.project.ProjectController$_analyzeProject_closure2.doCall(ProjectController.groovy:69)
at grails.plugin.executor.PersistenceContextRunnableWrapper$_run_closure1.doCall(PersistenceContextRunnableWrapper.groovy:34)
at grails.plugin.executor.PersistenceContextWrapper.wrap(PersistenceContextWrapper.groovy:35)
at grails.plugin.executor.PersistenceContextRunnableWrapper.run(PersistenceContextRunnableWrapper.groovy:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
... 13 more
Any ideas what might cause this?
Chances are you are hitting the configured timeout for long running connections (which can't be changed if I'm not mistaken, as it is here to protect the server from ill formed queries that would consume all cpu/io)
What you may want to try is to slice your long running query into smaller queries and sum up the results (if it's a select)