Jmeter test interface of PostgreSql, but often happens "idle in transaction" - postgresql

I use Jmeter test the springboot system with druid pool, there are almost 50 interfaces need to be tested. When tested a half (Serial execution), the Jmeter was blocked, then I use sql
select *
from pg_stat_activity
where pid in (select pid
from pg_locks l
join pg_class t on l.relation = t.oid and t.relkind = 'r' );
found that a query sql state is "idle in transaction", and a update sql wait_event_type is "Lock", wait_event is "transactionid".
And I also found if cancel the DataSourceTransactionManager config, run Jmeter OK, there won't happen "idle in transaction". But If config DataSourceTransactionManager, run Jmeter always happend "idle in transaction", whatever there's #Transactional annotation on method
How to avoid "idle in transaction" in this springboot system?
The TransactionManager config like this:
<bean id="txManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager" primary="true">
<property name="dataSource" ref="wasDataSource"/>
</bean>
The druid config like this:
<property name="maxActive" value="100"/>
<property name="initialSize" value="10"/>
<property name="minIdle" value="10"/>
<property name="maxWait" value="60000"/>
<property name="timeBetweenEvictionRunsMillis" value="60000"/>
<property name="minEvictableIdleTimeMillis" value="250000"/>
<property name="maxEvictableIdleTimeMillis" value="300000"/>
<property name="validationQuery" value="select 1"/>
<property name="testWhileIdle" value="true"/>
<property name="testOnBorrow" value="false"/>
<property name="testOnReturn" value="false"/>
<property name="poolPreparedStatements" value="true"/>
<property name="maxPoolPreparedStatementPerConnectionSize" value="20"/>
The Postgres version is 12.6, springboot is 2.6.6, spring is 5.3.19

Related

Connections remain in idle status and increase untill reached max connections limit

I have a web-app using apache-camel to submit routes which execute some postgresql select and insert.
I'm not using any DAO, so I haven't a code where begin and close connections, I believed that connections life-cycle was managed by Spring but it seems not working.
The problem is that everytime my route executes, I see one more connection which remains IDLE, so previous IDLE connections are not being reused, this takes to the "too many client connections problem"
In my route I have:
<bean id="configLocation" class="org.springframework.core.io.FileSystemResource">
<constructor-arg type="java.lang.String" value="..../src/main/resources/config/test.xml" />
</bean>
<bean id="dataSourcePostgres" class="org.apache.ibatis.datasource.pooled.PooledDataSource">
<property name="driver" value="org.postgresql.Driver" />
<property name="url" value="jdbc:postgresql://localhost:5432/postgres" />
<property name="username" value="postgres" />
<property name="password" value="postgres" />
</bean>
<bean id="postgresTrivenetaSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="dataSource" ref="dataSourcePostgres" />
<property name="configLocation" ref="configLocation" />
</bean>
Here they are some sample queries:
<select id="selectTest" resultType="java.util.LinkedHashMap">
select * from test;
</select>
<insert id="insertTest" parameterType="java.util.LinkedHashMap" useGeneratedKeys="true" keyProperty="id" keyColumn="id">
INSERT INTO test(note,regop_id)
VALUES (#{note},#{idKey});
</insert>
I tried even adding this:
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSourcePostgresTriveneta" />
</bean>
At last I found the problem, it was that the DataSource is never closed automatically at the end of a Camel route.
So, each time that Camel route executed, it left an open Datasource, then all the created IDLE connections (their number obviously depends from the DataSource configuration and its usage) remained and accumulate over and over.
The final solution was to add a bean created ad hoc at the end of the Camel route, taking the DataSource as argument and closing it, that's all.

Wso2 cant write groups in external LDAP

I am configuring Wso2 with a read write external LDAP.
I am using the SOAP Api provided by Wso2 for managing users n groups.
However when trying to add a user to an internal group everything works fine but when i try to add a user to an external LDAP group i get that the role does not exist. Which is weird cause i can see the roles in Wso2. I even can i add a user to an external group from there but after it is added its not showed as added even it is actually added in the LDAP.
This is the ldap connection configuration
<UserStoreManager class="org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager">
<Property name="TenantManager">org.wso2.carbon.user.core.tenant.CommonHybridLDAPTenantManager</Property>
<Property name="ConnectionURL">ldaps://some-url:636</Property>
<Property name="ConnectionName">cn=admin,ou=adminGroups,dc=userstore,dc=gk</Property>
<Property name="ConnectionPassword">someAdminPass</Property>
<Property name="AnonymousBind">false</Property>
<Property name="UserSearchBase">ou=users,dc=userstore,dc=gk</Property>
<Property name="UserEntryObjectClass">inetOrgPerson</Property>
<Property name="UserNameAttribute">uid</Property>
<Property name="UserNameSearchFilter">(&(objectClass=posixAccount)(uid=?))</Property>
<Property name="UserNameListFilter">(objectClass=posixAccount)</Property>
<Property name="DisplayNameAttribute"/>
<Property name="ReadGroups">true</Property>
<Property name="WriteGroups">true</Property>
<Property name="GroupSearchBase">ou=groups,dc=userstore,dc=gk</Property>
<Property name="GroupEntryObjectClass">posixGroup</Property>
<Property name="GroupNameAttribute">cn</Property>
<Property name="GroupNameSearchFilter">(&(objectClass=posixGroup)(cn=?))</Property>
<Property name="GroupNameListFilter">(objectClass=posixGroup)</Property>
<Property name="MembershipAttribute">memberOf</Property>
<Property name="BackLinksEnabled">false</Property>
<Property name="UsernameJavaRegEx">someregex stuff</Property>
<Property name="UsernameJavaScriptRegEx">someregex stuff</Property>
<Property name="UsernameJavaRegExViolationErrorMsg">Username pattern policy violated</Property>
<Property name="PasswordJavaRegEx">someregex stuff</Property>
<Property name="PasswordJavaScriptRegEx">someregex stuff</Property>
<Property name="PasswordJavaRegExViolationErrorMsg">Password length should be within 5 to 30 characters</Property>
<Property name="RolenameJavaRegEx">someregex stuff</Property>
<Property name="RolenameJavaScriptRegEx">someregex stuff</Property>
<Property name="SCIMEnabled">true</Property>
<Property name="IsBulkImportSupported">false</Property>
<Property name="EmptyRolesAllowed">true</Property>
<Property name="PasswordHashMethod">PLAIN_TEXT</Property>
<Property name="MultiAttributeSeparator">,</Property>
<Property name="MaxUserNameListLength">100</Property>
<Property name="MaxRoleNameListLength">100</Property>
<Property name="kdcEnabled">false</Property>
<Property name="defaultRealmName">WSO2.ORG</Property>
<Property name="UserRolesCacheEnabled">true</Property>
<Property name="ConnectionPoolingEnabled">false</Property>
<Property name="LDAPConnectionTimeout">5000</Property>
<Property name="ReadTimeout"/>
<Property name="RetryAttempts"/>
</UserStoreManager>
I am using SoapUi to make this call having this request body
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:ser="http://service.ws.um.carbon.wso2.org">
<soap:Header/>
<soap:Body>
<ser:updateRoleListOfUser>
<!--Optional:-->
<ser:userName>username</ser:userName>
<!--Zero or more repetitions:-->
<ser:deletedRoles>?</ser:deletedRoles>
<!--Zero or more repetitions:-->
<ser:newRoles>Users</ser:newRoles>
</ser:updateRoleListOfUser>
</soap:Body>
</soap:Envelope>
Seems that some configuration has gone wrong.Is there some special configuration needed to to that ?
This was a silly mistake. I am passing <ser:deletedRoles>?</ser:deletedRoles> and that is what actually throws the error saying that group does not exist.
About the part that i say
after it is added its not showed as added even it is actually added in
the LDAP
This has to do with a different setup that i use in my LDAP server. I have created an additional thread for this configuration issue.

Getting exceptions (Missing descriptor / No [EntityType] was found for the key class) on the below scenario

Getting exceptions (Missing descriptor / No [EntityType] was found for the key class) on below scenario.
Start the weblogic server
Deploy the WAR
Test the application ā€˜nā€™ of times
Delete the WAR from web logic admin console
Re deploy the same WAR
Basically, if we do redeploy we will got the above exception. This issue will get resolved if you restart the server.
Persistence.xml
<provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
<jta-data-source>DEV</jta-data-source>
<class>com.Class1</class>
<properties>
<property name="hibernate.show_sql" value="true"/>
<property name="eclipselink.allow-zero-id" value="true"/>
<property name="eclipselink.logging.level" value="FINE" />
<property name="eclipselink.logging.parameters" value="true" />
<property name="eclipselink.target-server" value="WebLogic"/>
<property name="javax.persistence.query.timeout" value="120000"/>
<property name="javax.persistence.lock.timeout" value="120000"/>
</properties>
</persistence-unit>
Bean Configuration to handle spring transaction
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceXmlLocation" value="classpath*:META-INF/persistence.xml"/>
<property name="persistenceUnitManager" ref="persistenceUnitManager" />
</bean>
<bean id="persistenceUnitManager" class="org.springframework.orm.jpa.persistenceunit.DefaultPersistenceUnitManager">
<property name="persistenceXmlLocations">
<list>
<value>classpath*:META-INF/persistence.xml</value>
</list>
</property>
<property name="loadTimeWeaver" ref="loadTimeWeaver"/>
</bean>
<bean id="loadTimeWeaver" class="org.springframework.instrument.classloading.weblogic.WebLogicLoadTimeWeaver"/>
<bean id="em" class="org.springframework.orm.jpa.support.SharedEntityManagerBean">
<property name="entityManagerFactory" ref="entityManagerFactory"/>
</bean>
The WL Server throwing this error because of the ServerSession is cached in the EntityManagerSetupImpl static, so on createEntityManagerFactory() the old one is used with the old descriptors/class before the redeployment.
**
We managed to take the EntityManagerFactory Object and closed the same
in the destroy method in one of the InitializerServlet.
**

Heroku's Spring MVC Hibernate template application not connecting to DB

I'm trying to get the sample application running, but getting the following error when it tries to connect to the db:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:894)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
I haven't changed the applicationContext.xml, and the particular portion is:
<beans profile="default">
<jdbc:embedded-database id="dataSource"/>
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.HSQLDialect</prop>
<prop key="hibernate.hbm2ddl.auto">create</prop>
</props>
</property>
</bean>
</beans>
<beans profile="prod">
<bean class="java.net.URI" id="dbUrl">
<constructor-arg value="#{systemEnvironment['DATABASE_URL']}"/>
</bean>
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="url" value="#{ 'jdbc:postgresql://' + #dbUrl.getHost() + #dbUrl.getPath() }"/>
<property name="username" value="#{ #dbUrl.getUserInfo().split(':')[0] }"/>
<property name="password" value="#{ #dbUrl.getUserInfo().split(':')[1] }"/>
</bean>
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<!-- change this to 'verify' before running as a production app -->
<prop key="hibernate.hbm2ddl.auto">update</prop>
</props>
</property>
</bean>
</beans>
I am able to connect to the db using pgAdmin III from my laptop.
Also, I am learning Spring, and I see some beans are wrapped in the profile "prod", but I cannot tell anywhere in code or web.xml that uses a particular profile.
Does the application server (Heroku?) need to start in a particular mode/profile, could that be why the db connection is not opening?
I'm learning Heroku as well.
Are you trying to run application on local machine? To be able to run this sample project on your local machine, you need to have database created. It's not described in the tutorial but if you try to use sample DATABASE_URL (postgres://scott:tiger#localhost/myapp) you need to create user scott with password tiger and create database myapp and grant scott required privileges. What I did, I've created sampledb database with existing postgres user, since it's database admin, I don't need to bother with grants and just changed url to
export DATABASE_URL=postgres://postrges:<password>#localhost/sampledb

Invoking Stored Procedure using Spring JdbcBatchItemWriter

I would like to execute a Stored Procedure using spring JdbcBatchItemWriter. My current code looks like :
<bean id="xyzWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
......
<property name="sql" value="update abc where x=:paramX" />
......
</bean>
I would like to replace this update sql query with a Stored Proc call. I would like to handle it in the xml file itself. Any help is really appreciated.
Thanks
Did you tried running SP through JdbcBatchItemWriter?
because I also had same requirement and i just tried and it worked for me
<bean id="trackItemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="mySQLDatasource"/>
<property name="itemPreparedStatementSetter">
<bean class="com.MyDataPreparedStatmentSetter"/>
</property>
<property name="assertUpdates" value="false" />
<property name="sql" value="Call my_Stored_Proc (?,?,?,?)"/>
</bean>
Hope it helps.