Make PostgreSQL transaction management like Oracle - postgresql

I'm looking at porting a Java-Spring application from Oracle to PostgreSQL. I'm running into a big difference between the way the transactions are handled. I'm having a problem with this statement:
public void upsert(String username, String pName, String pValue) {
Object[] args = { pValue, pName, username };
try {
jdbcTemplate.update(INSERT_USER_PREFERENCE, args);
} catch (DuplicateKeyException e) {
jdbcTemplate.update(UPDATE_USER_PREFERENCE, args);
}
}
This works fine in Oracle but in PostgreSQL this fails because PostgreSQL does not allow further commands in a transaction if there is a failure and it wants you to rollback or commit
(SQL state [25P02]; error code [0]; ERROR: current transaction is aborted, commands ignored until end of transaction block; nested exception is org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block)
This seems pretty unique to PostgreSQL. Is there a way to make PostgreSQL behave in the same way as Oracle (and other databases)?
Here is my datasource setup:
<bean id="portalDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="org.postgresql.Driver"/>
<property name="url" value="${test.database.url}"/>
<property name="username" value="${test.database.username}"/>
<property name="password" value="${test.database.password}"/>
</bean>
<bean id="txManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="portalDataSource"/>
</bean>

Related

Jmeter test interface of PostgreSql, but often happens "idle in transaction"

I use Jmeter test the springboot system with druid pool, there are almost 50 interfaces need to be tested. When tested a half (Serial execution), the Jmeter was blocked, then I use sql
select *
from pg_stat_activity
where pid in (select pid
from pg_locks l
join pg_class t on l.relation = t.oid and t.relkind = 'r' );
found that a query sql state is "idle in transaction", and a update sql wait_event_type is "Lock", wait_event is "transactionid".
And I also found if cancel the DataSourceTransactionManager config, run Jmeter OK, there won't happen "idle in transaction". But If config DataSourceTransactionManager, run Jmeter always happend "idle in transaction", whatever there's #Transactional annotation on method
How to avoid "idle in transaction" in this springboot system?
The TransactionManager config like this:
<bean id="txManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager" primary="true">
<property name="dataSource" ref="wasDataSource"/>
</bean>
The druid config like this:
<property name="maxActive" value="100"/>
<property name="initialSize" value="10"/>
<property name="minIdle" value="10"/>
<property name="maxWait" value="60000"/>
<property name="timeBetweenEvictionRunsMillis" value="60000"/>
<property name="minEvictableIdleTimeMillis" value="250000"/>
<property name="maxEvictableIdleTimeMillis" value="300000"/>
<property name="validationQuery" value="select 1"/>
<property name="testWhileIdle" value="true"/>
<property name="testOnBorrow" value="false"/>
<property name="testOnReturn" value="false"/>
<property name="poolPreparedStatements" value="true"/>
<property name="maxPoolPreparedStatementPerConnectionSize" value="20"/>
The Postgres version is 12.6, springboot is 2.6.6, spring is 5.3.19

postgres: Error querying database

Project run for some time after the database connection is not found, the error message is as follows:
2017-05-02 10:02:17,224 ERROR [main] (line:com.unis.license.agent.management.aop.ManagementServicesAop.afterThrowing(ManagementServicesAop.java:47)) - BeforeMethod:com.unis.license.agent.management.service.impl.LicenseServiceImpl.loadLicenseInfo Params: []
2017-05-02 10:02:17,225 ERROR [main] (line:com.unis.license.agent.management.aop.ManagementServicesAop.afterThrowing(ManagementServicesAop.java:50)) - methodException:com.unis.license.agent.management.service.impl.LicenseServiceImpl.loadLicenseInfo Exception:org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException:
Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (FATAL: the database system is starting up)
The error may exist in licenseAgent/orm/LicenseInfo.xml
The error may involve com.unis.license.agent.management.dao.ILicenseInfoOper.getAll
The error occurred while executing a query
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (FATAL: the database system is starting up)
2017-05-02 10:02:17,226 ERROR [main] (line:org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:318)) - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.unis.license.agent.management.filter.Initialization#0' defined in class path resource [spring-context.xml]: Invocation of init method failed; nested exception is org.springframework.transaction.NoTransactionException: No transaction aspect-managed TransactionStatus in scope
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:700)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:381)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:293)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:799)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:446)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:296)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1347)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:743)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:492)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:117)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:99)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:154)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
My DataSource configuration information is as follows:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="org.postgresql.Driver"></property>
<property name="url" value="jdbc:postgresql://127.0.0.1:5432/ucsm"></property>
<property name="username" value="uqdm"></property>
<property name="password" value="unis123"></property>
<property name="maxActive" value="100"></property>
<property name="maxIdle" value="30"></property>
<property name="maxWait" value="500"></property>
<property name="defaultAutoCommit" value="true"></property>
</bean>
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<tx:annotation-driven transaction-manager="transactionManager" />
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="configLocation" value="classpath:licenseAgent/mybatiscfg.xml"></property>
<property name="dataSource" ref="dataSource" />
</bean>
<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
<property name="basePackage" value="com.unis.license.agent.management.dao" />
</bean>
This is the key part of your stack trace:
FATAL: the database system is starting up
It means that the PostgreSQL server is starting or restarting and not ready to accept connections yet.
You should check the PostgreSQL server log to find the cause of this unexpected start.
Chances are that it is a restart caused by a crashing PostgreSQL server process. Such crashes can be caused (in rough order of likelihood) by buggy server extensions, buggy hardware or PostgreSQL bugs.

propagation NOT_SUPPORTED creates failing transaction in JDBC when using ChainedTransactionManager (JDBC + JPA)

I'm working on an existing Spring application that uses JDBC (DAO's extend NamedParameterJdbcDaoSupport). There were four datasources configured, each with it's own DataSourceTransactionManager. (though only one was registered with tx:annotation-driven for some reason)
I've recently added JPA (Spring-data-JPA) into the application and configured two entityManagerFactories (for now I don't need the other two datasources). I also configured two JpaTransactionManagers and removed the corresponding DataSourceTransactionManagers for these dataSources, since the JpaTransactionManagers can also be used for JDBC transactions. (correct me if I'm wrong)
It appears I need to be able to have distributed transactions, since the two datasources (to two different databases) need to be accessed (through JPA) in one service method. Since I did not have all I need to set up JTA (missing XA-driver for one of the databases) I've decided to give the Spring ChainedTransactionManager a try. Sadly this didn't work out as expected. All works fine if I just call a service method that only uses JPA.
Though when I call an existing service method that uses a JDBC find that has a class level #transactional annotation with it's propagation set to NOT_SUPPORTED and call another service method after that with a JPA call and a #transactional, I get an exception:
Caused by: java.lang.IllegalStateException: Already value [org.springframework.jdbc.datasource.ConnectionHolder#462cf9d9] for key [org.jboss.jca.adapters.jdbc.WrapperDataSource#3fbb4c32] bound to thread [http-/127.0.0.1:8080-5]
at org.springframework.transaction.support.TransactionSynchronizationManager.bindResource(TransactionSynchronizationManager.java:189) [spring-tx-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:403) [spring-orm-3.2.5.RELEASE.jar:3.2.5.RELEASE]
After some debugging, I found out that the transactions in Spring get added to a map on a ThreadLocal in the "TransactionSynchronizationManager.bindResource" method. The problem is that when using a JDBC call with #transactional and propogation NOT_SUPPORTED, a transaction is made anyway and registered through that method. When the JpaTransactionManager tries to bind it's resource, it is already on the map (and not marked as void) which causes the error to occur.
Changing the propagation to the default "REQUIRED" for the service call that encapsulates the JDBC call fixes the problem.
I have no idea why Spring is still creating that transaction when the transactional annotation is NOT_SUPPORTED. And if it creates that transaction, it should not bypass the JpaTransactionManager.
So what I'd like to know is if there is some way to tell Spring to use the JpaTransactionManager also when it creates a transaction itself inside the NamedParameterJdbcDaoSupport. (Well actually the JdbcDaoSupport... Well actually the DataSourceUtils)
We're using Spring 3.2.5, spring-data-jpa 1.6.0 and I've used Hibernate 4.2.0 as JpaVendor.
This problem doesn't occur without the ChainedTransactionManager.
Datasources:
<bean id="dataSourceCompta" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/comptaDS"/>
</bean>
<bean id="dataSourceUnisys" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/insoverDS"/>
</bean>
<bean id="dataSourceInsoverwebMysql" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/insoverWebDS"/>
</bean>
<bean id="dataSourceBatch" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/batchDS"/>
</bean>
Single remaining JDBC transaction manager (no JPA counterpart):
<bean id="transactionManagerBatch" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSourceBatch"/>
</bean>
JPA Transaction Managers:
<bean id="jpaUnisysTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactoryUnisys"/>
<qualifier value="unisys" />
</bean>
<bean id="jpaMysqlTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactoryMysql"/>
<qualifier value="mysql" />
</bean>
My ChainedTransactionManager:
<bean id="chainedTransactionManager" class="org.springframework.data.transaction.ChainedTransactionManager">
<constructor-arg>
<list>
<ref bean="jpaUnisysTransactionManager" />
<ref bean="jpaMysqlTransactionManager" />
</list>
</constructor-arg>
</bean>
JPA Entity manager factories:
<bean name="jpaVendorAdapter" class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
<bean id="entityManagerFactoryUnisys" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceXmlLocation" value="classpath:META-INF/some-persistence.xml"/>
<property name="dataSource" ref="dataSourceUnisys"/>
<property name="persistenceUnitName" value="unisysPU"/>
<property name="jpaVendorAdapter" ref="jpaVendorAdapter"/>
<property name="jpaProperties">
<!-- properties -->
</property>
</bean>
<bean id="entityManagerFactoryMysql" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceXmlLocation" value="classpath:META-INF/some-persistence.xml"/>
<property name="dataSource" ref="dataSourceCompta"/>
<property name="persistenceUnitName" value="mysqlPU"/>
<property name="jpaVendorAdapter" ref="jpaVendorAdapter"/>
<property name="jpaProperties">
<!-- properties -->
</property>
</bean>
For now I've "fixed" this, by changing all the class-level transactional annotations to have propagation.REQUIRED (default) instead of NOT_SUPPORTED. Though I do not really like this solutions, since it might be somebody set those propagations to NOT_SUPPORTED with a good reason. I've also tried SUPPORTED, but using that had the same issue as NOT_SUPPORTED: a transaction was being made anyway by the Spring DataSourceUtils when the query was being executed by the NamedParameterJdbcDaoSupport DAO.
When no transactional annotation is set on the service, all works well too.

OrientDB - JDBC Driver controlled by a Connection Pool

We are in the process of changing some of the applications that use a relation database, to OrientDB (2.1.9).
This transaction will be made in small steps, we will first start with the JDBC driver approach, and maybe we will switch to a native implementation later.
We have an application that is based on Spring Integration, where we changed from an Oracle Database to OrientDB, using the JDBC driver.
The database connections are managed by Tomcat 8 Connection Pool, and the transactional behavior is handled by Spring's Transaction Manager:
<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource" destroy-method="close">
<property name="driverClassName" value="com.orientechnologies.orient.jdbc.OrientJdbcDriver" />
...
</bean>
<bean id="txManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
After changing the database to OrientDB, we encountered the following error:
com.orientechnologies.orient.core.exception.ODatabaseException: Database instance is not set in current thread. Assure to set it with: ODatabaseRecordThreadLocal.INSTANCE.set(db);
The OrientDB documentation, on Multi threading, states:
OrientDB supports multi-threads access to the database. ODatabase* and OrientGraph* instances are not thread-safe, so you've to get an instance per thread and each database instance can be used only in one thread per time. For more information about how concurrency is managed by OrientDB look at Concurrency.
Since v2.1 OrientDB doesn't allow implicit usage of multiple database instances from the same thread. Any attempt to manage multiple instances in the same thread must explicitly call the method db.activateOnCurrentThread() against the database instance BFORE you use it.
This posed a problem, as all the database connection management is handled by the connection pool (as it should be).
So, in order to implement the solution proposed by OrientDB, I had to have access to the underlying ODatabaseDocumentTx database, in order to apply the activateOnCurrentThread() method.
To be able to do it, I had to create a wrapper from both the JDBC driver class, and the OrientJdbcConnection.
In the OrientJdbcConnection, I obtained the underlying database by the use of Java Reflection:
public class OrientJdbcConnectionWrapper extends OrientJdbcConnection {
#Override
public void commit() throws SQLException {
final ODatabaseDocumentTx database = getPrivateDatabaseFieldByReflection();
database.activateOnCurrentThread();
super.commit();
}
And the JDBC Driver class wrapper, now returns a wrapped connection:
public class OrientJdbcDriverWrapper extends OrientJdbcDriver {
public Connection connect(final String url, final Properties info) throws SQLException {
if (!acceptsURL(url)) {
throw new SQLException(DO_NOT_ACCEPT_URL_ERROR + url);
}
return new OrientJdbcConnectionWrapper(url, info);
}
}
Finally, the configuration of the Connection pool now uses the wrapped driver:
<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource" destroy-method="close">
<property name="driverClassName" value="x.y.z.driver.orientdb.OrientJdbcDriverWrapper" />
</bean>
As expected, this works perfectly, but i'm not entirely sure that this is the correct way to do it as i haven't see anyone with this issue.
On the other hand, the OrientDB documentation does suggest the use of activateOnCurrentThread() on a multi-threading environment.
OrientDB has its own database pool, that can be easily configured:
<bean id="dataSource" class="com.orientechnologies.orient.jdbc.OrientDataSource">
<constructor-arg name="url" value="jdbc:orient:plocal:xx\\databases\\test" />
<constructor-arg name="username" value="xx"/>
<constructor-arg name="password" value="xx"/>
<constructor-arg name="info" ref="databaseProperties" />
</bean>
<util:map id="databaseProperties" value-type="java.lang.String">
<entry key="db.usePool" value="true" />
<entry key="db.pool.min" value="1" />
<entry key="db.pool.max" value="10" />
Note: I'm using OrientDB through its orient-jdbc driver.

Connecting to database which is different from I specified in spring XML

<bean
id="DBDataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource"
lazy-init="true">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc\:mysql\://pc123456\:3306/reven"/>
<property name="username" value="reven"/>
<property name="password" value="23454"/>
</bean>
this is my code but the eclipse connecting some other database and giving below error
java.lang.AssertionError: Failed due to Error:
org.springframework.jdbc.UncategorizedSQLException: Hibernate
operation: Cannot open connection; uncategorized SQLException for SQL
[???]; SQL state [28000]; error code [1045]; Access denied for user
'ules'#'PC234333' (using password: YES); nested exception is
java.sql.SQLException: Access denied for user 'ules'#'PC234333' (using
password: YES)
It seems that you have wrong url, put the url like this
<property name="url" value="jdbc:mysql://pc123456:3306/xxx"/>
Where xxx is the db name of MySql.
And make sure with the given user name you are able to login in the MySql db