AMQPConnector on Mule - SocketException: Too many open files - sockets

I'm working a long time with Mule application and AMQPConnector.
Before some days I saw an ERROR: Too many open files, crash, and can't send request till I'm restarted the Mule application.
Mule code is something like this: (critical parts)
<amqp:connector name="AMQPConnector" validateConnections="true"
doc:name="AMQPConnector" host="x.x.x.x" port="5672"
password="xxxxx" username="xxx">
<reconnect-forever frequency="1000" blocking="false" />
From flows:
<flow name="first">
<http:listener config-ref="HTTP_Listener_Configuration" path="order/submit" doc:name="HTTP"/>
<byte-array-to-object-transformer doc:name="Byte Array to Object"/>
<json:object-to-json-transformer doc:name="Object to JSON"/>
<json:json-to-object-transformer returnClass="java.lang.Object" doc:name="JSON to Object"/>
<set-session-variable value="#[payload]" variableName="order" doc:name="Session Variable" />
<set-payload doc:name="order" value="#[payload.info]" />
<amqp:outbound-endpoint queueName="xxxx"
responseTimeout="100000" exchange-pattern="request-response"
connector-ref="AMQPConnector" doc:name="XXXX" />
<object-to-string-transformer doc:name="Object to String"/>
<logger message="response from queue #[payload]"
level="INFO" doc:name="Logger" />
</flow>
<flow name="second">
<amqp:inbound-endpoint queueName="xxxx"
responseTimeout="10000" exchange-pattern="request-response"
connector-ref="AMQPConnector" doc:name="AMQP-0-9" />
<byte-array-to-object-transformer doc:name="Byte Array to Object" />
<json:object-to-json-transformer doc:name="Object to JSON" />
<http:request config-ref="HTTP_Request_XXXX" host="#[sessionVars['partUrl']]" path="#[sessionVars['path']]" method="#[sessionVars['method']]" doc:name="HTTP">
<http:request-builder>
<http:header headerName="#[sessionVars['key']]" value="#[sessionVars['value']]"/>
</http:request-builder>
</http:request>
</flow>
The error exception log is:
2016-11-27 04:30:12,272 [amqpReceiver.1018] ERROR org.mule.exception.CatchMessagingExceptionStrategy -
Message : Error sending HTTP request. Message payload is of type: String
Code : MULE_ERROR--2
Exception stack is:
1. Too many open files (java.net.SocketException) sun.nio.ch.Net:-2 (null)
2. java.net.SocketException: Too many open files (java.util.concurrent.ExecutionException)
org.glassfish.grizzly.impl.SafeFutureImpl$Sync:349 (null)
3. java.util.concurrent.ExecutionException: java.net.SocketException: Too many open files (java.io.IOException)
org.mule.module.http.internal.request.grizzly.GrizzlyHttpClient:223 (null)
4. Error sending HTTP request. Message payload is of type: String (org.mule.api.MessagingException)
org.mule.module.http.internal.request.DefaultHttpRequester:287 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
Root Exception stack trace:
java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:411)
at sun.nio.ch.Net.socket(Net.java:404)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
Request was accepted by http-listner and system stopped.
Maybe miss one configuration, or close connection....
(I saw same questions with solutions to increase system or add code in java class etc... - it wasn't help me, I don't have java classes. )
Someone can help me?

This is an issue with Anypoint Studio. Unfortunately your operating system can only open so many files. Depending on your operating system this limit varies, however I experienced this same issue in both Ubuntu and macOS using the latest Anypoint Studio (5 to 6).
The best thing I can suggest is to restart and try and keep your open files to a minimum at any one point whilst using Anypoint. When I'm only working on 4-5 files at a time I can usually go an entire day without needing to restart.
With heavy usage I end up needing to restart within 5 hours.

Related

Mule SFTP Component component

I am developing mule flow wherein mule process has to upload file/files (from source directory) to SFTP server.
IN Folder (This folder holds file to transfer to SFTP)
I have below mule flow
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/sftp http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd">
<flow name="sftpFlow1">
<poll doc:name="Poll">
<fixed-frequency-scheduler frequency="30" startDelay="10" timeUnit="SECONDS"/>
<logger message="Hi" level="INFO" doc:name="Logger"/>
</poll>
<flow-ref name="sftpSub_Flow" doc:name="Flow Reference"/>
</flow>
<sub-flow name="sftpSub_Flow">
<file:outbound-endpoint path="D:\IN" responseTimeout="10000" doc:name="File"/>
<sftp:outbound-endpoint exchange-pattern="one-way" host="host" port="port" path="sftppath" user="user" password="password" responseTimeout="10000" doc:name="SFTP"/>
</sub-flow>
</mule>
Issues:
I don't see any error or exceptions but expected files from IN folder are not getting transfer SFTP server. I see ****.dat file every time sub-flow executes.
Any suggestion?
Here you have used both outbound end points. You need file inbound end point component to pick the file from source location.
refer below sample for further
https://dzone.com/articles/anypoint-file-connector-with-mulesoft
you can use the outputPattern property in sftpoutbound endpoint to change the file name with any extension
Because the application doesn't actually show any input, I'll assume that the sample is flawed but the real application has some inbound endpoint (for example a file inbound endpoing) that is reading in streaming mode. The subflow is trying to consume two time the input stream to output to the two outbound endpoints. Because the first one consumes the stream the second one, the SFTP, gets a consumed empty stream as input so it doesn't output anything. A solution is to convert the input to something in memory, like using the object to byte array transformer in the middle. Be warned that for large files you might run out of memory doing this.

Configuring MongoDB connector in MuleStudio when authentication is disabled - "failed to connect" error

I'm writing some flows in MuleStudio that save to and query a MongoDB database. The flows work fine when pointed at a database that has one user with dbAdmin and userAdmin roles and is configured with that user's credentials. However, when I point it to a different database with no users added, I get the error below. I kept the username and password in the MongoDB connector config in Mule, as dummy text as is suggested in the documentation, then took them out, but to no avail. I'm using 3.3.1 CE runtime.
Error message with dummy credentials:
ERROR 2013-10-10 14:48:08,176 [[sso-mds].connector.http.mule.default.receiver.02] org.mule.retry.notifiers.ConnectNotifier: Failed to connect/reconnect: Work Descriptor. Root Exception was: Could not create a validated object, cause: Couldn't connect with the given credentials. Type: class java.util.NoSuchElementException
ERROR 2013-10-10 14:48:08,181 [[sso-mds].connector.http.mule.default.receiver.02] org.mule.exception.DefaultMessagingExceptionStrategy:
********************************************************************************
Message : Failed to invoke findObjectsUsingQueryMap. Message payload is of type: LinkedHashMap
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. Could not create a validated object, cause: Couldn't connect with the given credentials (java.util.NoSuchElementException)
org.apache.commons.pool.impl.GenericKeyedObjectPool:1219 (null)
2. Failed to invoke findObjectsUsingQueryMap. Message payload is of type: LinkedHashMap (org.mule.api.MessagingException)
org.mule.module.mongo.processors.FindObjectsUsingQueryMapMessageProcessor:177 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.util.NoSuchElementException: Could not create a validated object, cause: Couldn't connect with the given credentials
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1219)
at org.mule.module.mongo.connectivity.MongoCloudConnectorConnectionManager.acquireConnection(MongoCloudConnectorConnectionManager.java:437)
at org.mule.module.mongo.connectivity.MongoCloudConnectorConnectionManager.acquireConnection(MongoCloudConnectorConnectionManager.java:27)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
Error message with no credentials:
ERROR 2013-10-10 14:19:48,572 [[sso-mds].connector.http.mule.default.receiver.02] org.mule.retry.notifiers.ConnectNotifier: Failed to connect/reconnect: Work Descriptor. Root Exception was: null. Type: class java.lang.NullPointerException
ERROR 2013-10-10 14:19:48,574 [[sso-mds].connector.http.mule.default.receiver.02] org.mule.exception.DefaultMessagingExceptionStrategy:
********************************************************************************
Message : Failed to invoke findObjectsUsingQueryMap. Message payload is of type: LinkedHashMap
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. null (java.lang.NullPointerException)
org.mule.module.mongo.connectivity.MongoCloudConnectorConnectionKey:86 (null)
2. Failed to invoke findObjectsUsingQueryMap. Message payload is of type: LinkedHashMap (org.mule.api.MessagingException)
org.mule.module.mongo.processors.FindObjectsUsingQueryMapMessageProcessor:177 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.lang.NullPointerException
at org.mule.module.mongo.connectivity.MongoCloudConnectorConnectionKey.hashCode(MongoCloudConnectorConnectionKey.java:86)
at java.util.HashMap.hash(Unknown Source)
at java.util.HashMap.getEntry(Unknown Source)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
Mule config:
<mongo:config name="Mongo_DB" doc:name="Mongo DB" database="${mongo.db}" host="${mongo.host}" port="${mongo.port}" password="${db.password}" username="${db.username}>
<mongo:connection-pooling-profile initialisationPolicy="INITIALISE_ONE" exhaustedAction="WHEN_EXHAUSTED_GROW"/>
</mongo:config>
<flow name="user_gets_simple" doc:name="user_gets_simple">
<composite-source doc:name="Composite Source">
<http:inbound-endpoint exchange-pattern="request-response" host="${my.host}" port="${my.ID}" path="user/getAll" doc:name="HTTP /user/getAll"/>
<http:inbound-endpoint exchange-pattern="request-response" host="${my.host}" port="${my.ID}" path="user/getByUsername" doc:name="HTTP /user/getByUsername"/>
<http:inbound-endpoint exchange-pattern="request-response" host="${my.host}" port="${my.ID}" path="user/getById" doc:name="HTTP /user/getById"/>
<http:inbound-endpoint exchange-pattern="request-response" host="${my.host}" port="${my.ID}" path="user/getByEmail" doc:name="HTTP /user/getByEmail"/>
</composite-source>
<flow-ref name="get_from_MongoDB" doc:name="Get from MongoDB subflow"/>
</flow>
<flow name="user_save_flow" doc:name="user_save_flow">
<http:inbound-endpoint exchange-pattern="request-response" host="${my.host}" port="${my.ID}" path="user/save" doc:name="HTTP /user/save" />
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<json:json-to-object-transformer doc:name="JSON to Object" returnClass="java.util.Map"/>
<mongo:insert-object-from-map config-ref="Mongo_DB" collection="${mongo.collection}" doc:name="Insert in Mongo DB" >
<mongo:element-attributes>
<mongo:element-attribute key="firstName">#[payload.firstName]</mongo:element-attribute>
<mongo:element-attribute key="lastName">#[payload.lastName]</mongo:element-attribute>
<mongo:element-attribute key="location">#[payload.location]</mongo:element-attribute>
</mongo:element-attributes>
</mongo:insert-object-from-map>
<set-payload value="Saved successfully" doc:name="Set Payload"/>
</flow>
<sub-flow name="get_from_MongoDB" doc:name="get_from_MongoDB">
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<json:json-to-object-transformer returnClass="java.util.Map" doc:name="JSON to Object"/>
<mongo:find-objects-using-query-map config-ref="Mongo_DB" collection="${mongo.collection}" doc:name="Find objects in Mongo DB" limit="5">
<mongo:query-attributes ref="#[payload]"/>
</mongo:find-objects-using-query-map>
<mongo:mongo-collection-to-json doc:name="Mongo collection to JSON"/>
</sub-flow>
Any suggestions on how I can connect to this database without adding a user?
Not sure how putting any dummy value would work as the connector will pass that as the username and pwd. But I tried this trick and it works. ${db.user} and in mule-app.properties just have an empty entry db.user= and that works fine.
Hi that is an issue as the Mule ESB doesnt work without authentication...
The user and the password fields are not "#optional".
for creating users in mongo and running mongo in authentication mode
http://www.mkyong.com/mongodb/mongodb-authentication-example/
and then you can put it the credentials in Mule ESB and you are good to go
....
Just add mongo-java-driver: 2.13.3 dependency. Check the maven dependency for the conflicts.

Mule Persistent storage

I am trying to use persistent queue storage to recover from unexpected failure. My mule version is 3.3.1
I pick up messages from a queue and enter "until successful" loop. If mule stops for some reason, I would like the message to be persistent.
Here is my relevant code
<spring:bean id="outboundStore" class="org.mule.util.store.QueuePersistenceObjectStore" />
<until-successful objectStore-ref="outboundStore"
I do not see the messages in .mule directory. What am I doing wrong?
Sorry if the question is not clear.
Adding Flows as requested:
<flow name="InitialFlow" processingStrategy="synchronous">
<inbound-endpoint ref="firstQueue"/>
<until-successful objectStore-ref="outboundStore" maxRetries="6" secondsBetweenRetries="5" deadLetterQueue-ref="secondQueue" failureExpression="groovy:message.getInvocationProperty('soapResponse') == 'BAD'">
<flow-ref name="somSubFlow" />
</until-successful>
</flow>
<sub-flow name="someSubFlow">
<http:outbound-endpoint ref="someEndpoint" exchange-pattern="request-response" method="GET" />
</sub-flow>
Please do let me know if you need more information.
Using a configuration very similar to yours, I can totally see messages pending delivery written in the .mule/queuestore/queuestore directory.
The only thing I can think of is an issue with this expression groovy:message.getInvocationProperty('soapResponse') == 'BAD' that would somehow mess with the processing.
Is this expression correct? Why not using MEL?

Transaction commit delay when routing message from one jms queue to another

We are trying to build simple transacted jms-to-jms router using Mule ESB and JBoss Messaging. When we run Mule ESB with application configured as below, we observe strange behaviour.
Approximately 10 messages are routed from queue test1 to test2
Nothing happens for ~40 seconds.
goto 1
Queue test1 is filled with around 500 messages when we start test. We use Mule 3.2 and JBoss 5.1.
If I remove transactions from code below everything works fine, all messages are sent to queue test2 instantly. Also, everything is fine if I change transactions from xa to jms -- by replacing xa-transaction tags with jms:transaction.
I don't know what causes this pause in message processing on ESB, probably transaction commit is delayed.
My question is: what should I do to have xa transactions working correctly?
I'll provide more details if needed. I asked this question on Mule ESB forum before with no answer http://forum.mulesoft.org/mulesoft/topics/transaction_commit_delay_when_routing_message_from_one_jms_queue_to_another
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:jms="http://www.mulesoft.org/schema/mule/jms" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" xmlns:core="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jbossts="http://www.mulesoft.org/schema/mule/jbossts" version="CE-3.2.1" xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/jbossts http://www.mulesoft.org/schema/mule/jbossts/current/mule-jbossts.xsd ">
<jbossts:transaction-manager> </jbossts:transaction-manager>
<configuration>
<default-threading-profile maxThreadsActive="30" maxThreadsIdle="5"/>
<default-receiver-threading-profile maxThreadsActive="10" maxThreadsIdle="5"/>
</configuration>
<spring:beans>
<spring:bean id="jmsJndiTemplate" class="org.springframework.jndi.JndiTemplate" doc:name="Bean">
<spring:property name="environment">
<spring:props>
<spring:prop key="java.naming.factory.url.pkgs">org.jboss.naming:org.jnp.interfaces</spring:prop>
<spring:prop key="jnp.disableDiscovery">true</spring:prop>
<spring:prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</spring:prop>
<spring:prop key="java.naming.provider.url">localhost:1099</spring:prop>
</spring:props>
</spring:property>
</spring:bean>
<spring:bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean" doc:name="Bean">
<spring:property name="jndiTemplate">
<spring:ref bean="jmsJndiTemplate"/>
</spring:property>
<spring:property name="jndiName">
<spring:value>XAConnectionFactory</spring:value>
</spring:property>
</spring:bean>
</spring:beans>
<jms:connector name="JMS" specification="1.1" numberOfConsumers="10" connectionFactory-ref="jmsConnectionFactory" doc:name="JMS"/>
<flow name="flow" doc:name="flow">
<jms:inbound-endpoint queue="test1" connector-ref="JMS" doc:name="qt1">
<xa-transaction action="ALWAYS_BEGIN"/>
</jms:inbound-endpoint>
<echo-component doc:name="Echo"/>
<jms:outbound-endpoint queue="test2" connector-ref="JMS" doc:name="qt2">
<xa-transaction action="ALWAYS_JOIN"/>
</jms:outbound-endpoint>
<echo-component doc:name="Echo"/>
</flow>
</mule>
Here you can find log fragment for 1 message interaction. Please note that in this case there was no delay.
And here is log fragment for 11 messages. All of them were in queue test1 when app started, as you can see 10 messages are routed instantly and one is delayed by 1 minute.
I've found root of my problem: my queues were defined with following attribute:
<attribute name="RedeliveryDelay">60000</attribute>
Removing it or setting low value solves my problem with delays. Problem is, I don't know why :)
I always thought that redelivery delay is used when delivery fails, which was not the case in my app.

jboss-esb fs-listener jbm message queue overflow

We have a jboss esb server which is reading files from the file system in a scheduled way (schedule frequency of 20sec) and convert them into the esb message then we parse the message.
There are some other providers/listeners (jms) and services configured on the esb servers. When there is an error in one of the services it effects the above process. File system provider (gateway) is working fine but the jms-listener who takes the gateway messages are not working and lots of messages are accumulated in the jbm queue (jbm_msg Oracle DB table).
Here is the problem, when my server is restarted messages in the jbm-queue is parsed in the esb for just 20 seconds which is the scheduled frequency of fs-provider, never process messages again and cpu usage goes up to 100% and stays there. We believe somehow fs-providers interrupts the jms-provider.
Is there any configuration we have been missing out.
Here are the configuration files that we have:
jboss-esb.xml
<?xml version = "1.0" encoding = "UTF-8"?>
<jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd" parameterReloadSecs="5">
<providers>
<fs-provider name="SitaIstProvider">
<fs-bus busid="gw_sita_ist" >
<fs-message-filter
directory="/ikarussita/IST/IN"
input-suffix=".RCV"
work-suffix=".lck"
post-delete="false"
post-directory="/ikarussita/IST/OK"
post-suffix=".ok"
error-delete="false"
error-directory="/ikarussita/IST/ERR"
error-suffix=".err"/>
</fs-bus>
</fs-provider>
<jms-provider name="SitaESBQueue" connection-factory="ConnectionFactory">
<jms-bus busid="esb_sita_queue">
<jms-message-filter dest-type="QUEUE" dest-name="queue/esb_sita_queue"/>
</jms-bus>
</jms-provider>
</providers>
<services>
<service category="SITA" name="SITA_IST" description="SITA Daemon For ISTCOXH">
<listeners>
<fs-listener name="Sita_Ist_Gateway" busidref="gw_sita_ist" is-gateway="true" schedule-frequency="20" />
<jms-listener name="Jms_Sita_EsbAware" busidref="esb_sita_queue" />
</listeners>
<actions mep="OneWay">
<action name="parse_msg" class="com.celebi.integration.action.sita.inbound.SitaHandler" process="parseMessage" />
<action name="send_ikarus" class="com.celebi.integration.action.ikarus.outbound.fis.FlightJmsSender" />
</actions>
</service>
</services>
</jbossesb>
jbm-queue-service.xml
<?xml version="1.0" encoding="UTF-8"?>
<server>
<mbean code="org.jboss.jms.server.destination.QueueService"
name="jboss.messaging.destination:service=Queue,name=esb_sita_queue"
xmbean-dd="xmdesc/Queue-xmbean.xml">
<depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
<depends>jboss.messaging:service=PostOffice</depends>
</mbean>
<server>
deployment.xml
<jbossesb-deployment>
<depends>jboss.messaging.destination:service=Queue,name=esb_sita_queue</depends>
</jbossesb-deployment>
Thanx
Split the service into 2 separate services, one handling the JMS queue, the other the file poller. Specify the same action pipeline. That way you get the same functionality but without the threading issue. Also use max-threads attr on the listener to specify the number of reading threads.