Connections remain in idle status and increase untill reached max connections limit - postgresql

I have a web-app using apache-camel to submit routes which execute some postgresql select and insert.
I'm not using any DAO, so I haven't a code where begin and close connections, I believed that connections life-cycle was managed by Spring but it seems not working.
The problem is that everytime my route executes, I see one more connection which remains IDLE, so previous IDLE connections are not being reused, this takes to the "too many client connections problem"
In my route I have:
<bean id="configLocation" class="org.springframework.core.io.FileSystemResource">
<constructor-arg type="java.lang.String" value="..../src/main/resources/config/test.xml" />
</bean>
<bean id="dataSourcePostgres" class="org.apache.ibatis.datasource.pooled.PooledDataSource">
<property name="driver" value="org.postgresql.Driver" />
<property name="url" value="jdbc:postgresql://localhost:5432/postgres" />
<property name="username" value="postgres" />
<property name="password" value="postgres" />
</bean>
<bean id="postgresTrivenetaSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="dataSource" ref="dataSourcePostgres" />
<property name="configLocation" ref="configLocation" />
</bean>
Here they are some sample queries:
<select id="selectTest" resultType="java.util.LinkedHashMap">
select * from test;
</select>
<insert id="insertTest" parameterType="java.util.LinkedHashMap" useGeneratedKeys="true" keyProperty="id" keyColumn="id">
INSERT INTO test(note,regop_id)
VALUES (#{note},#{idKey});
</insert>
I tried even adding this:
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSourcePostgresTriveneta" />
</bean>

At last I found the problem, it was that the DataSource is never closed automatically at the end of a Camel route.
So, each time that Camel route executed, it left an open Datasource, then all the created IDLE connections (their number obviously depends from the DataSource configuration and its usage) remained and accumulate over and over.
The final solution was to add a bean created ad hoc at the end of the Camel route, taking the DataSource as argument and closing it, that's all.

Related

Camel The application attempted to use a JMS session after it had closed the session

I am new to camel and I am attempting to write an app that bridges Websphere MQ and Active MQ on JBoss EAP 7. The app deploys successfully works, I can drop messages on the Websphere queue, and it gets picked up by Active MQ. However I see error messages in the log showing it is attempting to use a connection after it is open.
15:48:57,814 ERROR [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (Camel (camel) thread #1 - JmsConsumer[I0_TEST]) IJ000315: Pool IbmMQQueueFactory has 1 active handles
15:48:57,819 INFO [org.jboss.as.connector.deployers.RaXmlDeployer] (Camel (camel) thread #1 - JmsConsumer[I0_TEST]) wmq.jmsra.rar: MQJCA4016:Unregistered connection handle being closed: 'com.ibm.mq.connector.outbound.ConnectionWrapper#214da401'.
15:49:02,819 WARN [org.apache.camel.component.jms.DefaultJmsMessageListenerContainer] (Camel (camel) thread #1 - JmsConsumer[I0_TEST]) Setup of JMS message listener invoker failed for destination 'I0_TEST' - trying to recover. Cause: Local JMS transaction failed to commit; nested exception is com.ibm.msg.client.jms.DetailedIllegalStateException: MQJCA1020: The session is closed.
The application attempted to use a JMS session after it had closed the session.
Modify the application so that it closes the JMS session only after it has finished using the session.
Here is my applicationContext.xml
<bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:/ConnectionFactory" />
<property name="lookupOnStartup" value="false" />
<property name="cache" value="true" />
<property name="proxyInterface" value="javax.jms.ConnectionFactory" />
</bean>
<bean id="jmsTransactionManager"
class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManagerName" value="java:/TransactionManager" />
</bean>
<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="transacted" value="true" />
<property name="transactionManager" ref="jmsTransactionManager" />
</bean>
<bean id="wmqConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:/jms/IbmMQMsgQCF" />
<property name="lookupOnStartup" value="false" />
<property name="cache" value="true" />
<property name="proxyInterface" value="javax.jms.ConnectionFactory" />
</bean>
<bean id="wmqTransactionManager"
class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManagerName" value="java:/TransactionManager" />
</bean>
<bean id="wmq" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="wmqConnectionFactory" />
<property name="transacted" value="true" />
<property name="transactionManager" ref="wmqTransactionManager" />
</bean>
<bean id="routerlogger" class="org.jboss.as.quickstarts.mdb.RoutLogger" />
<camelContext trace="true" id="camel"
xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="wmq:websphereQueue"/>
<setExchangePattern pattern="InOnly"/>
<to uri="jms:activeQueue" pattern="InOnly" />
</route>
</camelContext>
Its a simple app, trying to determine what I'm missing.
I found this JBossDeveloper bug "JBEAP-2344: UserTransaction commit(), rollback() closes connection in Websphere MQ 7.5" which looks like it describes your issue and has comments pointing to documentation update "JBEAP-3535: Documentation: Add note about connection close on commit() and rollback() to Deploy the WebSphere MQ Resource Adapter subchapter".
Could you please add a note, that setting tracking="false", solves
problem with WebSphere MQ 7.5 and 8, where method commit() or
rollback() on UserTransaction closes any JMS connections which was
part of this transaction. This part is related to documenting known
limitation of WebSphere MQ in
JBEAP-3142.

Spring Integration: Mail error and HTTP gateway response

My use case is simple. I want to handle an exception caused by a system being unreachable, perform a retry based upon a configured retry policy, send an email when the retry threshold has been met, and return a custom response back the caller.
The challenges I am facing is that I cannot both send an email and return a response back to the caller. Since I was using a int-mail:outbound-channel-adapter initially, I would expect this behavior since this is a one-way component:
<int:chain input-channel="defaultErrorChannel">
<int:service-activator id="mailMessageActivator" expression="#mailHandler.process(payload)" />
<int-mail:outbound-channel-adapter mail-sender="mailSender" />
</int:chain>
However, if I introduce a int-amqp:outbound-gateway in front of the int-mail:outbound-channel-adapter (see the Error Handling config below), I would expect to be able to invoke a int:service-activator to construct and return a response to the caller.
Am I thinking about this the wrong way? I see that someone else had a similar question which is still unanswered. Both of the configurations I mentioned send emails, but always block from the caller without receiving a response upon timeout.
Here are the relevant parts of my configuration:
Gateway
<int:gateway id="customerGateway" service-interface="com.uscs.crm.integration.CustomerGateway"
default-request-channel="syncCustomers" default-reply-channel="replySyncCustomers" default-reply-timeout="30000">
</int:gateway>
<int:object-to-json-transformer input-channel="syncCustomers" output-channel="outboundRequestChannel" />
<int-http:outbound-gateway request-channel="outboundRequestChannel" reply-channel="replySyncCustomers"
url="http://voorhees148.uscold.com:9595/web/customerSync/createCustomer"
http-method="POST"
rest-template="restTemplate"
expected-response-type="com.uscs.crm.model.CustSyncResponseVO"
mapped-request-headers="Authorization, HTTP_REQUEST_HEADERS">
<int-http:request-handler-advice-chain>
<ref bean="retryWithBackoffAdviceSession" />
</int-http:request-handler-advice-chain>
</int-http:outbound-gateway>
Error Handling
<int:channel id="defaultErrorChannel"/>
<int:channel id="errorResponses"/>
<!--
ExponentialBackOffPolicy.multipler is applied to wait time over each retry attempt
with a ExponentialBackOffPolicy.maximum configured.
-->
<bean id="retryWithBackoffAdviceSession" class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice">
<property name="retryTemplate">
<bean class="org.springframework.retry.support.RetryTemplate">
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="2000" />
<property name="multiplier" value="2" />
<property name="maxInterval" value="30000"/>
</bean>
</property>
<property name="retryPolicy">
<bean class="org.springframework.retry.policy.SimpleRetryPolicy">
<property name="maxAttempts" value="3"/>
</bean>
</property>
</bean>
</property>
<property name="recoveryCallback">
<bean class="org.springframework.integration.handler.advice.ErrorMessageSendingRecoverer">
<constructor-arg ref="defaultErrorChannel"/>
</bean>
</property>
</bean>
<bean id="custSyncResponseHandler" class="com.uscs.crm.integration.handler.CustSyncResponseHandler"></bean>
<int:chain input-channel="defaultErrorChannel" output-channel="replySyncCustomers">
<int:service-activator id="mailMessageActivator" expression="#mailHandler.process(payload)" />
<int:header-enricher>
<int:header name="ERROR_ID" expression="T(java.lang.System).currentTimeMillis()"/>
</int:header-enricher>
<int-amqp:outbound-gateway
exchange-name="error-responses-exchange"
routing-key-expression="'error.response.'+headers.ERROR_ID"
amqp-template="amqpTemplate" />
<!-- Will this service-activator return a response to the caller (int:gateway) using channel `replySyncCustomers`? -->
<int:service-activator id="custSyncResponseActivator" expression="#custSyncResponseHandler.process(payload)" />
</int:chain>
<int-amqp:inbound-gateway queue-names="error-responses" request-channel="errorResponses"
connection-factory="rabbitConnectionFactory" acknowledge-mode="AUTO" />
<int-mail:outbound-channel-adapter channel="errorResponses" mail-sender="mailSender" />
<!-- (Outbound Channel Adapter/Gateway) rabbit exchanges, queues, and bindings used by this app -->
<rabbit:topic-exchange name="error-responses-exchange" auto-delete="false" durable="true">
<rabbit:bindings>
<rabbit:binding queue="error-responses" pattern="error.response.*"/>
</rabbit:bindings>
</rabbit:topic-exchange>
<rabbit:queue name="error-responses" auto-delete="false" durable="true"/>
SOLUTION: I was able to get this working with help from #Artem. Below are the changes I made.
Service Activator Implementation (handling ErrorMessage)
The key is the line which returns the reconstructed Message with all of the header information from the ErrorMessage.
#Override
public Message<CustSyncResponseVO> process(Message<MessagingException> errorMessage) {
MessagingException errorException = errorMessage.getPayload();
CustSyncResponseVO custSyncResponse = new CustSyncResponseVO();
custSyncResponse.setResponseMessage(ExceptionUtils
.convertToString(errorMessage.getPayload()));
return MessageBuilder.withPayload(custSyncResponse)
.copyHeaders(errorMessage.getHeaders())
.copyHeadersIfAbsent(errorException.getFailedMessage().getHeaders()).build();
}
Service Activator Config
Used SpEL to reference the #root context to retrieve the ErrorMessage instead of the default which would be MessagingException (payload) and passed it to my process method on the POJO.
<bean id="custSyncResponseHandler" class="com.uscs.crm.integration.handler.CustSyncResponseHandler" />
<int:chain id="errorGatewayResponseChain" input-channel="defaultErrorChannel" output-channel="replySyncCustomers">
<int:service-activator id="custSyncResponseActivator" expression="#custSyncResponseHandler.process(#root)" />
</int:chain>
I don't see reason to introduce the AMQP middleware complexity there just for sending email in the end.
What only you need is <publish-subscribe-channel id="defaultErrorChannel"> with to endpoints as subscribers to it.
The first one is one-way email sending <chain> and the second one is custSyncResponseActivator to reply something to your <int-http:outbound-gateway>.
You can find more info on the matter in the Spring Integration Reference Manual.

Activiti BPM Platform - Select a specific schema for creating tables?

I've just started working with activity and integrated it in my project (postgres based) in an embedded way (sample spring configuration file snip)
(...)
<!-- Activiti components -->
<bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
<property name="dataSource" ref="dataSource" />
<property name="transactionManager" ref="transactionManager" />
<property name="databaseSchemaUpdate" value="true" />
<property name="jobExecutorActivate" value="false" />
</bean>
<bean id="processEngine" class="org.activiti.spring.ProcessEngineFactoryBean">
<property name="processEngineConfiguration" ref="processEngineConfiguration" />
</bean>
<bean id="repositoryService" factory-bean="processEngine" factory-method="getRepositoryService" />
<bean id="runtimeService" factory-bean="processEngine" factory-method="getRuntimeService" />
<bean id="taskService" factory-bean="processEngine" factory-method="getTaskService" />
<bean id="historyService" factory-bean="processEngine" factory-method="getHistoryService" />
<bean id="managementService" factory-bean="processEngine" factory-method="getManagementService" />
(...)
It works well and create a lot of tables on my application schema at startup.
My problem is : tables are created in the 'public' schema in my postgres database. I would have preferred to put those tables in a separate schema, say 'activity'.
Fact is that after browsing the documentation / the net for almost two hours, I didn't found any way to change the default schema target creation behavior.
Any help... greatly appreciated ! ;)
Since Postgres 9.4 JDBC driver you can specify the default schema in the JDBC url like this:
jdbc:postgresql://localhost:5432/mydatabase?currentSchema=myschema
With this URL, all Activiti tables are created in the myschema schema instead of the default one in the search path, usually public.
Sources: this response on Stack Overflow and the latest documentation.

how partition reader from database, writer in different files and optimize thread load

EDIT:
I think that there are something wrong with this clause:
I tried to run my first test that runs single thread and take about 35 minutes with this whereCause and the execution is terribly slow. When I just do an select * from table, whitout whereClause the process happens normally.
I trying to use Step Partitioning in a Job with Spring Batch, but I dont realize if is it's
appropriate to my case:
I have read from a database with ~30 million records. In the record, I have a column bank_id and there is about 23 differents banks.
I have to read the value from this column and separate the records from each bank into different txt files.
I want my job parallelize the work in 4 or 8 threads, in a first moment I try to use step partitioning and I split the job in 4 slaves and set the id_bank that I process in a parameter for a query in SqlPagingQueryProviderFactoryBean and I use only 4 different Ids. But the amount of records from one bank_id to another varies widely resulting in a slave finish they job before anothers.
I want that when the slave finish they work, he begin to process another bank_id.
I need a help to do anything like this in spring batch. I use the 2.1 version of spring batch.
here is my files:
<bean id="arquivoWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter"
scope="step">
<property name="encoding" value="ISO-8859-1" />
<property name="lineAggregator">
<bean
class="org.springframework.batch.item.file.transform.FormatterLineAggregator">
<property name="fieldExtractor">
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names"
value="name_bank, id_bank, etc" />
</bean>
</property>
<property name="format"
value="..." />
</bean>
</property>
<property name="resource"
value="file:./arquivos/#{stepExecutionContext[faixa]}.txt" />
</bean>
<job id="partitionJob" xmlns="http://www.springframework.org/schema/batch">
<step id="masterStep">
<partition step="slave" partitioner="rangePartitioner">
<handler task-executor="taskExecutor" />
</partition>
</step>
</job>
<step id="slave" xmlns="http://www.springframework.org/schema/batch">
<tasklet>
<chunk reader="pagingReader" writer="arquivoWriter"
commit-interval="#{jobParameters['commit.interval']}" />
<listeners>
<listener ref="myChunkListener"></listener>
</listeners>
</tasklet>
</step>
<bean id="rangePartitioner" class="....RangePartitioner" />
<bean id="pagingReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="fetchSize" value="#{jobParameters['fetch.size']}"></property>
<property name="queryProvider">
<bean
class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="selectClause">
<value>
<![CDATA[
SELECT ...
]]>
</value>
</property>
<property name="fromClause" value="FROM my_table" />
<property name="whereClause" value="where id_bank = :id_op" />
</bean>
</property>
<property name="parameterValues">
<map>
<entry key="id_op" value="#{stepExecutionContext[id_op]}" />
</map>
</property>
<property name="maxItemCount" value="#{jobParameters['max.rows']}"></property>
<property name="rowMapper">
<bean class="....reader.MyRowMapper" />
</property>
</bean>
The range partitioner:
public class RangePartitioner implements Partitioner {
#Autowired
BancoDao bancoDao;
final Map<String, ExecutionContext> result = new HashMap<String, ExecutionContext> ();
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
List<OrgaoPagadorQuantidadeRegistrosTO> lista = bancoDao.findIdsOps();
for (OrgaoPagadorQuantidadeRegistrosTO op:lista){
String name = String.valueOf(op.getIdOrgaoPagador());
ExecutionContext ex = new ExecutionContext();
ex.putLong("id_op", op.getIdBank());
ex.putString ("faixa", name);
result.put("p"+name, ex);
}
return result;
}
}
What you're asking for should work assuming that you have enough work for each of the slaves to work on. For example, if you have 23 banks but one has 20 million records and the others each have 100,000, the slaves not working on the big bank will free up quickly.
Are you creating a StepExecution per bank or per thread? I'd recommend doing it per bank. This would allow threads to pick up work as they finish. Otherwise, you end up being responsible for that load balancing by implementing a Partitioner that does this normalization.

In-memory Job-Explorer definition in Spring batch

I was trying to share My in-memory jobRepository to the jobExplorer. But it throws an error as,
Nested exception is
org.springframework.beans.ConversionNotSupportedException:
Failed to convert property value of type '$Proxy1 implementing
org.springframework.batch.core.repository.JobRepository,org.
springframework.aop.SpringProxy,org.springframework.aop.framework.Advised'
to required type
Even i tried putting '&' sign before jobRepository when passing to jobExplorer for sharing.But attempt end in vain.
I am using Spring Batch 2.2.1
Is the dependency for jobExplorer is only database not in-memory?
Definition is,
<bean id="jobRepository"
class="com.test.repository.BatchRepositoryFactoryBean">
<property name="cache" ref="cache" />
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="jobOperator" class="test.batch.LauncherTest.TestBatchOperator">
<property name="jobExplorer" ref="jobExplorer" />
<property name="jobRepository" ref="jobRepository" />
<property name="jobRegistry" ref="jobRegistry" />
<property name="jobLauncher" ref="jobLauncher" />
</bean>
<bean id="jobExplorer" class="test.batch.LauncherTest.TestBatchExplorerFactoryBean">
<property name="repositoryFactory" ref="&jobRepository" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher" class="com.scb.smartbatch.core.BatchLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- To store Batch details -->
<bean id="jobRegistry" class="com.scb.smartbatch.repository.SmartBatchRegistry" />
<bean id="jobRegistryBeanPostProcessor"
class="org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor">
<property name="jobRegistry" ref="jobRegistry" />
</bean>
<!--Runtime cache of batch executions -->
<bean id="cache" class="com.scb.cache.TCRuntimeCache" />
thanks for your valuable inputs.
But I used '&' before the job repository reference, which allowed me to use it for my job explorer as a shared resource.
problem solved.
kudos.
Usually you have to wire interface instead of implementation.
Else, probably, you have to add <aop:config proxy-target-class="true"> to create CGLIB-based proxy instead of standard Java-based proxy.
Read Spring official documentation about that