Mule SFTP Component component - mule-studio

I am developing mule flow wherein mule process has to upload file/files (from source directory) to SFTP server.
IN Folder (This folder holds file to transfer to SFTP)
I have below mule flow
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/sftp http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd">
<flow name="sftpFlow1">
<poll doc:name="Poll">
<fixed-frequency-scheduler frequency="30" startDelay="10" timeUnit="SECONDS"/>
<logger message="Hi" level="INFO" doc:name="Logger"/>
</poll>
<flow-ref name="sftpSub_Flow" doc:name="Flow Reference"/>
</flow>
<sub-flow name="sftpSub_Flow">
<file:outbound-endpoint path="D:\IN" responseTimeout="10000" doc:name="File"/>
<sftp:outbound-endpoint exchange-pattern="one-way" host="host" port="port" path="sftppath" user="user" password="password" responseTimeout="10000" doc:name="SFTP"/>
</sub-flow>
</mule>
Issues:
I don't see any error or exceptions but expected files from IN folder are not getting transfer SFTP server. I see ****.dat file every time sub-flow executes.
Any suggestion?

Here you have used both outbound end points. You need file inbound end point component to pick the file from source location.
refer below sample for further
https://dzone.com/articles/anypoint-file-connector-with-mulesoft
you can use the outputPattern property in sftpoutbound endpoint to change the file name with any extension

Because the application doesn't actually show any input, I'll assume that the sample is flawed but the real application has some inbound endpoint (for example a file inbound endpoing) that is reading in streaming mode. The subflow is trying to consume two time the input stream to output to the two outbound endpoints. Because the first one consumes the stream the second one, the SFTP, gets a consumed empty stream as input so it doesn't output anything. A solution is to convert the input to something in memory, like using the object to byte array transformer in the middle. Be warned that for large files you might run out of memory doing this.

Related

How do I limit the size of log file generated by druid while using imply?

I'm using imply to handle druid's cluster. But my logs files have increased to hundreds of gigabytes of storage. I'm talking about logs files present in imply/var/sv/ directory in which there are these 7 log files, broker.log, historical.log, middleManager.log zk.log, coordinator.log, imply-ui.log, and overlord.log.
Among them, this particular file called coordinator.log has increased to the really massive size of about 560 GBs in a matter of a few months. I have read all those logs, and they don't bother me much. What I'm concerned about is the size of the file which is eating my entire storage. I have tried finding ways to limit the size of those log files but believe me nothing worked for me.
I read in many places that druid uses log4j2 logger so we can limit the size using its configuration from log4j2.xml file. But again a big confusion there are four log4j2.xml files which one shall I modify?
I tried modifying all of them, but still, it didn't work. I'm kind of a fool while handling it seems like... Well so this is my request if anybody could point me in the right direction in limiting the size of these log files
You can setup a simple cron process to truncate these files periodically using truncate -s 0 imply/var/sv/*.log
The default log level in imply distribution is set to info which generates a lots of logs. If they don't bother you much, you can set the log level to error so that logs will be generated only when there is any error during system runtime. To set that, you need to modify logger level inside conf/druid/_common/log4j2.xml file.
<?xml version="1.0" encoding="UTF-8" ?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
And even after do so, you should periodically truncate log files as #mdeora suggested.

Not create new log file once start my service with using log4j2

following is my configration of log4j2:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="trace" name="MyApp" packages="com.swimap.base.launcher.log">
<Appenders>
<RollingFile name="RollingFile" fileName="logs/app-${date:MM-dd-yyyy-HH-mm-ss-SSS}.log"
filePattern="logs/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="1 KB"/>
</Policies>
<DefaultRolloverStrategy max="3"/>
</RollingFile>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>
the issue is that each time when start up my service, a new log will be created even the old one has not reached the specific size. If the program restart frequently, i will got many log files end with ‘.log’ which never be compressed.
the logs i got like this:
/log4j2/logs
/log4j2/logs/2017-07
/log4j2/logs/2017-07/app-07-18-2017-1.log.gz
/log4j2/logs/2017-07/app-07-18-2017-2.log.gz
/log4j2/logs/2017-07/app-07-18-2017-3.log.gz
/log4j2/logs/app-07-18-2017-20-42-06-173.log
/log4j2/logs/app-07-18-2017-20-42-12-284.log
/log4j2/logs/app-07-18-2017-20-42-16-797.log
/log4j2/logs/app-07-18-2017-20-42-21-269.log
someone can tell me how can i append log to the exists log file when i start up my program? much thanks whether u can help me closer to the answer!!
I suppose that your problem it that you have fileName="logs/app-${date:MM-dd-yyyy-HH-mm-ss-SSS}.log in your log4j2 configuration file.
This fileName template means that log4j2 will create log file with name that contains current date + hours + minutes + seconds + milliseconds in its name.
You should probably remove HH-mm-ss-SSS section and this will allow you to have daily rolling file and to not create new file every app restart.
You can play with template and choose format that you need.
If you want only one log file forever - then create constant fileName, like fileName=app.log
It's not hard to implement this. There is a interface DirectFileRolloverStrategy, implement below method:
public String getCurrentFileName(RollingFileManager manager)
Mybe someone met same problem and this can help him.

Spring XD - Mail source configuration - how to provide password

I created a mailstream with following command -------
stream create --name mailstream --definition "mail --host=imap.gmail.com --username=yyyyyyyy12#gmail.com --password=my password | file --dir=/tmp/gmailData" --deploy
Refer -http://docs.spring.io/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#modules
But in the xd-singletone console I get -
Caused by: javax.mail.AuthenticationFailedException: failed to connect, no password specified?
How to resolve this issue.
Also --password=secret - how to keep my password invisible or secret in the XD shell
/shankha
You need to escape "#" with "%40" for both username and password and to specify --port=993 for gmail. Also, it may be possible not to work with the default settings as GMail requires SSL for imap and this needs to be configured as well.
So, I would suggest the following (basically, creating a new source module):
Go to spring-xd-1.0.0.M6\xd\modules\source and make a copy of mail folder and name this copy gmail
Go to spring-xd-1.0.0.M6\xd\modules\source\gmail\config and rename both mail.properties and mail.xml to gmail.properties and gmail.xml respectively
Inside gmail.xml replace everything with:
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/integration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:beans="http://www.springframework.org/schema/beans"
xmlns:file="http://www.springframework.org/schema/integration/file"
xmlns:int-mail="http://www.springframework.org/schema/integration/mail"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/integration/mail http://www.springframework.org/schema/integration/mail/spring-integration-mail.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<channel id="output" />
<int-mail:mail-to-string-transformer
charset="${charset}" input-channel="transform" output-channel="output" />
<beans:beans profile="use-polling">
<int-mail:inbound-channel-adapter
store-uri="${protocol}://${username:}:${password:}#${host}:${port}/${folder}"
channel="transform" should-mark-messages-as-read="${markAsRead}"
should-delete-messages="${delete}" java-mail-properties="javaMailProperties">
<poller fixed-delay="${fixedDelay}" time-unit="SECONDS">
<advice-chain>
<beans:bean
class="org.springframework.xd.dirt.module.support.ThreadContextClassLoaderSetterAdvice" />
</advice-chain>
</poller>
</int-mail:inbound-channel-adapter>
</beans:beans>
<beans:beans profile="use-idle">
<int-mail:imap-idle-channel-adapter
store-uri="${protocol}://${username:}:${password:}#${host}:${port}/${folder}"
channel="transform" auto-startup="true" mail-filter-expression="${expression}"
should-mark-messages-as-read="${markAsRead}"
should-delete-messages="${delete}" java-mail-properties="javaMailProperties">
</int-mail:imap-idle-channel-adapter>
</beans:beans>
<beans:beans profile="default">
<util:properties id="javaMailProperties">
<beans:prop key="mail.imap.socketFactory.class">javax.net.ssl.SSLSocketFactory</beans:prop>
<beans:prop key="mail.imap.socketFactory.fallback">false</beans:prop>
<beans:prop key="mail.store.protocol">imaps</beans:prop>
<beans:prop key="mail.debug">false</beans:prop>
</util:properties>
</beans:beans>
</beans:beans>
4. In XD shell now you will use something like the following to create your stream:
stream create --name myGmailStream --definition "gmail --host=imap.gmail.com --username=yyyyyyyy12%40gmail.com --password=my_password --port=993 | file --dir=/tmp/gmailData" --deploy
Here, please note the following:
I added --port=993
the username contains "%40" instead of "#"
the definition of the stream starts with "gmail
if your password contains "#" you need to replace that with "%40" as well
What I've done above is to, basically, create a new custom module (a source) which is kind of easy (more details about this you can find in the documentation). The XD single node or the XD Shell doesn't even need to be restarted. Give it a try and let me know how it goes.
Regarding the password that you don't want to appear as part of the stream definition, you can provide it as part of your mail module options, as described here: http://docs.spring.io/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#_module_configuration
ie
<xd_home>/config/modules/source/mail/mail.properties:
password: yourpassword

Transaction commit delay when routing message from one jms queue to another

We are trying to build simple transacted jms-to-jms router using Mule ESB and JBoss Messaging. When we run Mule ESB with application configured as below, we observe strange behaviour.
Approximately 10 messages are routed from queue test1 to test2
Nothing happens for ~40 seconds.
goto 1
Queue test1 is filled with around 500 messages when we start test. We use Mule 3.2 and JBoss 5.1.
If I remove transactions from code below everything works fine, all messages are sent to queue test2 instantly. Also, everything is fine if I change transactions from xa to jms -- by replacing xa-transaction tags with jms:transaction.
I don't know what causes this pause in message processing on ESB, probably transaction commit is delayed.
My question is: what should I do to have xa transactions working correctly?
I'll provide more details if needed. I asked this question on Mule ESB forum before with no answer http://forum.mulesoft.org/mulesoft/topics/transaction_commit_delay_when_routing_message_from_one_jms_queue_to_another
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:jms="http://www.mulesoft.org/schema/mule/jms" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" xmlns:core="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jbossts="http://www.mulesoft.org/schema/mule/jbossts" version="CE-3.2.1" xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/jbossts http://www.mulesoft.org/schema/mule/jbossts/current/mule-jbossts.xsd ">
<jbossts:transaction-manager> </jbossts:transaction-manager>
<configuration>
<default-threading-profile maxThreadsActive="30" maxThreadsIdle="5"/>
<default-receiver-threading-profile maxThreadsActive="10" maxThreadsIdle="5"/>
</configuration>
<spring:beans>
<spring:bean id="jmsJndiTemplate" class="org.springframework.jndi.JndiTemplate" doc:name="Bean">
<spring:property name="environment">
<spring:props>
<spring:prop key="java.naming.factory.url.pkgs">org.jboss.naming:org.jnp.interfaces</spring:prop>
<spring:prop key="jnp.disableDiscovery">true</spring:prop>
<spring:prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</spring:prop>
<spring:prop key="java.naming.provider.url">localhost:1099</spring:prop>
</spring:props>
</spring:property>
</spring:bean>
<spring:bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean" doc:name="Bean">
<spring:property name="jndiTemplate">
<spring:ref bean="jmsJndiTemplate"/>
</spring:property>
<spring:property name="jndiName">
<spring:value>XAConnectionFactory</spring:value>
</spring:property>
</spring:bean>
</spring:beans>
<jms:connector name="JMS" specification="1.1" numberOfConsumers="10" connectionFactory-ref="jmsConnectionFactory" doc:name="JMS"/>
<flow name="flow" doc:name="flow">
<jms:inbound-endpoint queue="test1" connector-ref="JMS" doc:name="qt1">
<xa-transaction action="ALWAYS_BEGIN"/>
</jms:inbound-endpoint>
<echo-component doc:name="Echo"/>
<jms:outbound-endpoint queue="test2" connector-ref="JMS" doc:name="qt2">
<xa-transaction action="ALWAYS_JOIN"/>
</jms:outbound-endpoint>
<echo-component doc:name="Echo"/>
</flow>
</mule>
Here you can find log fragment for 1 message interaction. Please note that in this case there was no delay.
And here is log fragment for 11 messages. All of them were in queue test1 when app started, as you can see 10 messages are routed instantly and one is delayed by 1 minute.
I've found root of my problem: my queues were defined with following attribute:
<attribute name="RedeliveryDelay">60000</attribute>
Removing it or setting low value solves my problem with delays. Problem is, I don't know why :)
I always thought that redelivery delay is used when delivery fails, which was not the case in my app.

jboss-esb fs-listener jbm message queue overflow

We have a jboss esb server which is reading files from the file system in a scheduled way (schedule frequency of 20sec) and convert them into the esb message then we parse the message.
There are some other providers/listeners (jms) and services configured on the esb servers. When there is an error in one of the services it effects the above process. File system provider (gateway) is working fine but the jms-listener who takes the gateway messages are not working and lots of messages are accumulated in the jbm queue (jbm_msg Oracle DB table).
Here is the problem, when my server is restarted messages in the jbm-queue is parsed in the esb for just 20 seconds which is the scheduled frequency of fs-provider, never process messages again and cpu usage goes up to 100% and stays there. We believe somehow fs-providers interrupts the jms-provider.
Is there any configuration we have been missing out.
Here are the configuration files that we have:
jboss-esb.xml
<?xml version = "1.0" encoding = "UTF-8"?>
<jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd" parameterReloadSecs="5">
<providers>
<fs-provider name="SitaIstProvider">
<fs-bus busid="gw_sita_ist" >
<fs-message-filter
directory="/ikarussita/IST/IN"
input-suffix=".RCV"
work-suffix=".lck"
post-delete="false"
post-directory="/ikarussita/IST/OK"
post-suffix=".ok"
error-delete="false"
error-directory="/ikarussita/IST/ERR"
error-suffix=".err"/>
</fs-bus>
</fs-provider>
<jms-provider name="SitaESBQueue" connection-factory="ConnectionFactory">
<jms-bus busid="esb_sita_queue">
<jms-message-filter dest-type="QUEUE" dest-name="queue/esb_sita_queue"/>
</jms-bus>
</jms-provider>
</providers>
<services>
<service category="SITA" name="SITA_IST" description="SITA Daemon For ISTCOXH">
<listeners>
<fs-listener name="Sita_Ist_Gateway" busidref="gw_sita_ist" is-gateway="true" schedule-frequency="20" />
<jms-listener name="Jms_Sita_EsbAware" busidref="esb_sita_queue" />
</listeners>
<actions mep="OneWay">
<action name="parse_msg" class="com.celebi.integration.action.sita.inbound.SitaHandler" process="parseMessage" />
<action name="send_ikarus" class="com.celebi.integration.action.ikarus.outbound.fis.FlightJmsSender" />
</actions>
</service>
</services>
</jbossesb>
jbm-queue-service.xml
<?xml version="1.0" encoding="UTF-8"?>
<server>
<mbean code="org.jboss.jms.server.destination.QueueService"
name="jboss.messaging.destination:service=Queue,name=esb_sita_queue"
xmbean-dd="xmdesc/Queue-xmbean.xml">
<depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
<depends>jboss.messaging:service=PostOffice</depends>
</mbean>
<server>
deployment.xml
<jbossesb-deployment>
<depends>jboss.messaging.destination:service=Queue,name=esb_sita_queue</depends>
</jbossesb-deployment>
Thanx
Split the service into 2 separate services, one handling the JMS queue, the other the file poller. Specify the same action pipeline. That way you get the same functionality but without the threading issue. Also use max-threads attr on the listener to specify the number of reading threads.