We have a jboss esb server which is reading files from the file system in a scheduled way (schedule frequency of 20sec) and convert them into the esb message then we parse the message.
There are some other providers/listeners (jms) and services configured on the esb servers. When there is an error in one of the services it effects the above process. File system provider (gateway) is working fine but the jms-listener who takes the gateway messages are not working and lots of messages are accumulated in the jbm queue (jbm_msg Oracle DB table).
Here is the problem, when my server is restarted messages in the jbm-queue is parsed in the esb for just 20 seconds which is the scheduled frequency of fs-provider, never process messages again and cpu usage goes up to 100% and stays there. We believe somehow fs-providers interrupts the jms-provider.
Is there any configuration we have been missing out.
Here are the configuration files that we have:
jboss-esb.xml
<?xml version = "1.0" encoding = "UTF-8"?>
<jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd" parameterReloadSecs="5">
<providers>
<fs-provider name="SitaIstProvider">
<fs-bus busid="gw_sita_ist" >
<fs-message-filter
directory="/ikarussita/IST/IN"
input-suffix=".RCV"
work-suffix=".lck"
post-delete="false"
post-directory="/ikarussita/IST/OK"
post-suffix=".ok"
error-delete="false"
error-directory="/ikarussita/IST/ERR"
error-suffix=".err"/>
</fs-bus>
</fs-provider>
<jms-provider name="SitaESBQueue" connection-factory="ConnectionFactory">
<jms-bus busid="esb_sita_queue">
<jms-message-filter dest-type="QUEUE" dest-name="queue/esb_sita_queue"/>
</jms-bus>
</jms-provider>
</providers>
<services>
<service category="SITA" name="SITA_IST" description="SITA Daemon For ISTCOXH">
<listeners>
<fs-listener name="Sita_Ist_Gateway" busidref="gw_sita_ist" is-gateway="true" schedule-frequency="20" />
<jms-listener name="Jms_Sita_EsbAware" busidref="esb_sita_queue" />
</listeners>
<actions mep="OneWay">
<action name="parse_msg" class="com.celebi.integration.action.sita.inbound.SitaHandler" process="parseMessage" />
<action name="send_ikarus" class="com.celebi.integration.action.ikarus.outbound.fis.FlightJmsSender" />
</actions>
</service>
</services>
</jbossesb>
jbm-queue-service.xml
<?xml version="1.0" encoding="UTF-8"?>
<server>
<mbean code="org.jboss.jms.server.destination.QueueService"
name="jboss.messaging.destination:service=Queue,name=esb_sita_queue"
xmbean-dd="xmdesc/Queue-xmbean.xml">
<depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
<depends>jboss.messaging:service=PostOffice</depends>
</mbean>
<server>
deployment.xml
<jbossesb-deployment>
<depends>jboss.messaging.destination:service=Queue,name=esb_sita_queue</depends>
</jbossesb-deployment>
Thanx
Split the service into 2 separate services, one handling the JMS queue, the other the file poller. Specify the same action pipeline. That way you get the same functionality but without the threading issue. Also use max-threads attr on the listener to specify the number of reading threads.
Related
I am developing mule flow wherein mule process has to upload file/files (from source directory) to SFTP server.
IN Folder (This folder holds file to transfer to SFTP)
I have below mule flow
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/sftp http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd">
<flow name="sftpFlow1">
<poll doc:name="Poll">
<fixed-frequency-scheduler frequency="30" startDelay="10" timeUnit="SECONDS"/>
<logger message="Hi" level="INFO" doc:name="Logger"/>
</poll>
<flow-ref name="sftpSub_Flow" doc:name="Flow Reference"/>
</flow>
<sub-flow name="sftpSub_Flow">
<file:outbound-endpoint path="D:\IN" responseTimeout="10000" doc:name="File"/>
<sftp:outbound-endpoint exchange-pattern="one-way" host="host" port="port" path="sftppath" user="user" password="password" responseTimeout="10000" doc:name="SFTP"/>
</sub-flow>
</mule>
Issues:
I don't see any error or exceptions but expected files from IN folder are not getting transfer SFTP server. I see ****.dat file every time sub-flow executes.
Any suggestion?
Here you have used both outbound end points. You need file inbound end point component to pick the file from source location.
refer below sample for further
https://dzone.com/articles/anypoint-file-connector-with-mulesoft
you can use the outputPattern property in sftpoutbound endpoint to change the file name with any extension
Because the application doesn't actually show any input, I'll assume that the sample is flawed but the real application has some inbound endpoint (for example a file inbound endpoing) that is reading in streaming mode. The subflow is trying to consume two time the input stream to output to the two outbound endpoints. Because the first one consumes the stream the second one, the SFTP, gets a consumed empty stream as input so it doesn't output anything. A solution is to convert the input to something in memory, like using the object to byte array transformer in the middle. Be warned that for large files you might run out of memory doing this.
all
I've been using AutoHotkey to perform a number of batch processes at work. It's going pretty well, but I keep running into an issue. The scripts (more than one, doing very different things) will all crash after about 4 hours of running. It's pretty consistent with the time, but only kind of consistent with the location in the program where it's running.
I'm not getting an error message. The script will just stop, and when I try to open it up to see the last line, the program closes without any info.
I know it's vague, but any ideas what may be happening?
Edit: added Event Viewer data. Two other crash events were almost identical.
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2013-10-11T04:20:51.000000000Z" />
<EventRecordID>11519</EventRecordID>
<Channel>Application</Channel>
<Computer>My Computer</Computer>
<Security />
</System>
- <EventData>
<Data>CDR to CGM-PDF 3-0-1.exe</Data>
<Data>1.1.11.2</Data>
<Data>51f4589f</Data>
<Data>CDR to CGM-PDF 3-0-1.exe</Data>
<Data>1.1.11.2</Data>
<Data>51f4589f</Data>
<Data>c00000fd</Data>
<Data>00000000000ca117</Data>
<Data>1c44</Data>
<Data>01cec61d2e612892</Data>
<Data>C:\CDR to CGM-PDF 3-0-1.exe</Data>
<Data>C:\CDR to CGM-PDF 3-0-1.exe</Data>
<Data>82958232-322c-11e3-ab1f-d067e5491d83</Data>
</EventData>
</Event>
I am using jboss-5.1 to deploy message driven bean which is used to subscribe messages from a third party queue.
Around 16 messages were posted to that queue but they remained pending in our subscriber queue. I restarted the server and the messages were readily picked.
As much as I have analysed, I think maxsize and maxsession could have affected it, as both are 15. But I do not understand if there was some real issue, how it got solved by just restarting.
The logs were in error mode. I did not get the full stack trace.
This is the snippet of that error log.
[2012-10-30 17:01:00,228] [MQQueueAgent (GQH1_PLANNING_MDM_001)]
[ERROR] STDERR: 2012.10.30 17:01:00 MQJMS1023E rollback failed
[2012-10-30 17:01:00,228] [exceptionDelivery0] [WARN ]
org.jboss.resource.adapter.jms.inflow.JmsActivation: Failure in jms activation
org.jboss.resource.adapter.jms.inflow.JmsActivationSpec#85d0d(ra=org.jboss.resource.adapter.jms.JmsResourceAdapter#b21aae
destination=remotewsmq/NOTIFICATION_PLANNING_MDM_001.SUBQ
destinationType=javax.jms.Queue tx=true durable=false reconnect=10 provider=RemoteWSMQJMSProvider
user=null maxMessages=1 minSession=1 maxSession=5 keepAlive=60000 useDLQ=false)
GQH1_PLANNING_MDM_001: The name of the queue used for subscribing.
The files that I use to configure the properties of the MDBs are as follows.
1.ejb3-interceptors-aop.xml
<domain name="Message Driven Bean" extends="Intercepted Bean" inheritBindings="true">
<bind pointcut="execution(public * *->*(..))">
<interceptor-ref name="org.jboss.ejb3.security.AuthenticationInterceptorFactory"/>
<interceptor-ref name="org.jboss.ejb3.security.RunAsSecurityInterceptorFactory"/>
</bind>
<!-- TODO: Authorization? -->
<bind pointcut="execution(public * *->*(..))">
<interceptor-ref name="org.jboss.ejb3.tx.CMTTxInterceptorFactory"/>
<interceptor-ref name="org.jboss.ejb3.stateless.StatelessInstanceInterceptor"/>
<interceptor-ref name="org.jboss.ejb3.tx.BMTTxInterceptorFactory"/>
<interceptor-ref name="org.jboss.ejb3.AllowedOperationsInterceptor"/>
<interceptor-ref name="org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor"/>
<!-- interceptor-ref name="org.jboss.ejb3.interceptor.EJB3InterceptorsFactory"/ -->
<stack-ref name="EJBInterceptors"/>
</bind>
<annotation expr="class(*) AND !class(#org.jboss.ejb3.annotation.Pool)">
#org.jboss.ejb3.annotation.Pool (value="StrictMaxPool", maxSize=15, timeout=10000)
</annotation>
</domain>
2.standardjboss.xml
<invoker-proxy-binding>
<name>message-driven-bean</name>
<invoker-mbean>default</invoker-mbean>
<proxy-factory>org.jboss.ejb.plugins.jms.JMSContainerInvoker</proxy-factory>
<proxy-factory-config>
<JMSProviderAdapterJNDI>DefaultJMSProvider</JMSProviderAdapterJNDI>
<ServerSessionPoolFactoryJNDI>StdJMSPool</ServerSessionPoolFactoryJNDI>
<CreateJBossMQDestination>false</CreateJBossMQDestination>
<!-- WARN: Don't set this to zero until a bug in the pooled executor is fixed -->
<MinimumSize>1</MinimumSize>
<MaximumSize>15</MaximumSize>
<KeepAliveMillis>30000</KeepAliveMillis>
<MaxMessages>1</MaxMessages>
<MDBConfig>
<ReconnectIntervalSec>10</ReconnectIntervalSec>
<DLQConfig>
<DestinationQueue>queue/DLQ</DestinationQueue>
<MaxTimesRedelivered>10</MaxTimesRedelivered>
<TimeToLive>0</TimeToLive>
</DLQConfig>
</MDBConfig>
</proxy-factory-config>
</invoker-proxy-binding>
<activation-config-property>
<activation-config-property-name>maxSession</activation-config-property-name>
<activation-config-property-value>15</activation-config-property-value>
</activation-config-property>
3.jms-ds.xml
<?xml version="1.0" encoding="UTF-8"?>
<connection-factories>
<!-- ==================================================================== -->
<!-- JMS Stuff -->
<!-- ==================================================================== -->
<!--
The JMS provider loader. Currently pointing to a non-clustered ConnectionFactory. Need to
be replaced with a clustered non-load-balanced ConnectionFactory when it becomes available.
See http://jira.jboss.org/jira/browse/JBMESSAGING-843.
-->
<mbean code="org.jboss.jms.jndi.JMSProviderLoader"
name="jboss.messaging:service=JMSProviderLoader,name=JMSProvider">
<attribute name="ProviderName">DefaultJMSProvider</attribute>
<attribute name="ProviderAdapterClass">org.jboss.jms.jndi.JNDIProviderAdapter</attribute>
<attribute name="FactoryRef">java:/XAConnectionFactory</attribute>
<attribute name="QueueFactoryRef">java:/XAConnectionFactory</attribute>
<attribute name="TopicFactoryRef">java:/XAConnectionFactory</attribute>
</mbean>
<!-- JMS XA Resource adapter, use this to get transacted JMS in beans -->
<tx-connection-factory>
<jndi-name>JmsXA</jndi-name>
<xa-transaction/>
<rar-name>jms-ra.rar</rar-name>
<connection-definition>org.jboss.resource.adapter.jms.JmsConnectionFactory</connection-definition>
<config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>
<config-property name="JmsProviderAdapterJNDI" type="java.lang.String">java:/DefaultJMSProvider</config-property>
<max-pool-size>20</max-pool-size>
<security-domain-and-application>JmsXARealm</security-domain-and-application>
<depends>jboss.messaging:service=ServerPeer</depends>
</tx-connection-factory>
</connection-factories>
Please help.
If the listener did not try to reconnect, then it might be the messages pending which caused it to fail.
According to the error, a transaction ROLLBACK call failed. After the failure, the queue manager probably held those messages in an outstanding unit of work. Restarting the server would have closed the connection at which point the queue manager will have rolled back the transaction on behalf of the application. On restart, the application will create a new UOW and retrieve the messages.
Look in WebSphere MQ's queue manager error logs and global error logs to determine whether the error was caused by a resource shortage. It may be necessary to increase the size of the queue manager transaction logs or to tune transaction parameters such as MAXUOW.
You may also need to update the MQ client version or Queue Manager version. According to this Technote, WebSphere MQ JMS classes were updated as of 6.0.2.3 to fix a bug that resulted in MQJMS1023E errors. If you need to update the client version, it is available as a free download as SupportPac MQC75. A new client is able to run with any back level queue manager. After upgrading, the app benefits from the bug fixes and performance enhancements of the new client code and provides API functionality appropriate for the version of Queue Manager to which it connects. What version of WebSphere MQ JMS client is currently installed? What version of WebSphere MQ queue manager is currently installed?
We are trying to build simple transacted jms-to-jms router using Mule ESB and JBoss Messaging. When we run Mule ESB with application configured as below, we observe strange behaviour.
Approximately 10 messages are routed from queue test1 to test2
Nothing happens for ~40 seconds.
goto 1
Queue test1 is filled with around 500 messages when we start test. We use Mule 3.2 and JBoss 5.1.
If I remove transactions from code below everything works fine, all messages are sent to queue test2 instantly. Also, everything is fine if I change transactions from xa to jms -- by replacing xa-transaction tags with jms:transaction.
I don't know what causes this pause in message processing on ESB, probably transaction commit is delayed.
My question is: what should I do to have xa transactions working correctly?
I'll provide more details if needed. I asked this question on Mule ESB forum before with no answer http://forum.mulesoft.org/mulesoft/topics/transaction_commit_delay_when_routing_message_from_one_jms_queue_to_another
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:jms="http://www.mulesoft.org/schema/mule/jms" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" xmlns:core="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jbossts="http://www.mulesoft.org/schema/mule/jbossts" version="CE-3.2.1" xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/jbossts http://www.mulesoft.org/schema/mule/jbossts/current/mule-jbossts.xsd ">
<jbossts:transaction-manager> </jbossts:transaction-manager>
<configuration>
<default-threading-profile maxThreadsActive="30" maxThreadsIdle="5"/>
<default-receiver-threading-profile maxThreadsActive="10" maxThreadsIdle="5"/>
</configuration>
<spring:beans>
<spring:bean id="jmsJndiTemplate" class="org.springframework.jndi.JndiTemplate" doc:name="Bean">
<spring:property name="environment">
<spring:props>
<spring:prop key="java.naming.factory.url.pkgs">org.jboss.naming:org.jnp.interfaces</spring:prop>
<spring:prop key="jnp.disableDiscovery">true</spring:prop>
<spring:prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</spring:prop>
<spring:prop key="java.naming.provider.url">localhost:1099</spring:prop>
</spring:props>
</spring:property>
</spring:bean>
<spring:bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean" doc:name="Bean">
<spring:property name="jndiTemplate">
<spring:ref bean="jmsJndiTemplate"/>
</spring:property>
<spring:property name="jndiName">
<spring:value>XAConnectionFactory</spring:value>
</spring:property>
</spring:bean>
</spring:beans>
<jms:connector name="JMS" specification="1.1" numberOfConsumers="10" connectionFactory-ref="jmsConnectionFactory" doc:name="JMS"/>
<flow name="flow" doc:name="flow">
<jms:inbound-endpoint queue="test1" connector-ref="JMS" doc:name="qt1">
<xa-transaction action="ALWAYS_BEGIN"/>
</jms:inbound-endpoint>
<echo-component doc:name="Echo"/>
<jms:outbound-endpoint queue="test2" connector-ref="JMS" doc:name="qt2">
<xa-transaction action="ALWAYS_JOIN"/>
</jms:outbound-endpoint>
<echo-component doc:name="Echo"/>
</flow>
</mule>
Here you can find log fragment for 1 message interaction. Please note that in this case there was no delay.
And here is log fragment for 11 messages. All of them were in queue test1 when app started, as you can see 10 messages are routed instantly and one is delayed by 1 minute.
I've found root of my problem: my queues were defined with following attribute:
<attribute name="RedeliveryDelay">60000</attribute>
Removing it or setting low value solves my problem with delays. Problem is, I don't know why :)
I always thought that redelivery delay is used when delivery fails, which was not the case in my app.
Scenario:
I'm using NServiceBus with MSMQ Transport.
I have messages from an app dropped into QUEUE-A.
I have a NServiceBus Gateway grab messages from QUEUE-A and send them to another Gateway
I the Destination Gateway receives the messages and places them into the respective queue.
Source Gateway throws a Null Reference Exception and send the messages 5 total times (max retries).
Destination Gateway receives all messages and shows that it's Sending the HTTP response in the log.
The error that the Destination Gateway throws is:
WARN NServiceBus.Unitcast.Transport.Msmq.MsmqTransport [(null)] <(null)> - Failed raising 'transport message received' event for message with ID=GUID
System.NullReferenceException: Object reference not set to an instance of an object.
This error initially was setup on 2 different servers though it occurs when I run the scenario locally, under admin rights, on my local computer (just listening on different ports). I'm setting the SetHttpToHeader() of the message to the proper destination address.
The gateway configs are as follows:
Gateway1:
<appSettings>
<add key="NumberOfWorkerThreads" value="10"/>
<add key="InputQueue" value="Gateway1Pickup"/>
<add key="ErrorQueue" value="Gateway1Error"/>
<add key="ForwardReceivedMessageTo" value="audit"/>
<add key="OutputQueue" value="Gateway1Output"/>
<add key="ListenUrl" value="http://address:6768/Gateway/"/>
<add key="RequireMD5FromClient" value="true"/>
</appSettings>
Gateway2:
<appSettings>
<add key="NumberOfWorkerThreads" value="10"/>
<add key="InputQueue" value="Gateway2Pickup"/>
<add key="ErrorQueue" value="Gateway2Error"/>
<add key="ForwardReceivedMessageTo" value="audit"/>
<add key="OutputQueue" value="Gateway2Output"/>
<add key="ListenUrl" value="http://address:6768/Gateway/"/>
<add key="RequireMD5FromClient" value="true"/>
</appSettings>
Does anyone know where I'm going wrong on this?
At first I got a solid repro and then I realized that NSB does not handle creating the Audit queues for you. If you remove them from the config or create the Audit queues, you should be all set.
I found this above problem showed up when I didn't have some dependent assemblies in the service directory. I found the error attached to the message in the MSMQ queue. I used Queue Explorer, but I presume other tools would show the same.