Spring XD - Mail source configuration - how to provide password - email

I created a mailstream with following command -------
stream create --name mailstream --definition "mail --host=imap.gmail.com --username=yyyyyyyy12#gmail.com --password=my password | file --dir=/tmp/gmailData" --deploy
Refer -http://docs.spring.io/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#modules
But in the xd-singletone console I get -
Caused by: javax.mail.AuthenticationFailedException: failed to connect, no password specified?
How to resolve this issue.
Also --password=secret - how to keep my password invisible or secret in the XD shell
/shankha

You need to escape "#" with "%40" for both username and password and to specify --port=993 for gmail. Also, it may be possible not to work with the default settings as GMail requires SSL for imap and this needs to be configured as well.
So, I would suggest the following (basically, creating a new source module):
Go to spring-xd-1.0.0.M6\xd\modules\source and make a copy of mail folder and name this copy gmail
Go to spring-xd-1.0.0.M6\xd\modules\source\gmail\config and rename both mail.properties and mail.xml to gmail.properties and gmail.xml respectively
Inside gmail.xml replace everything with:
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/integration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:beans="http://www.springframework.org/schema/beans"
xmlns:file="http://www.springframework.org/schema/integration/file"
xmlns:int-mail="http://www.springframework.org/schema/integration/mail"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/integration/mail http://www.springframework.org/schema/integration/mail/spring-integration-mail.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<channel id="output" />
<int-mail:mail-to-string-transformer
charset="${charset}" input-channel="transform" output-channel="output" />
<beans:beans profile="use-polling">
<int-mail:inbound-channel-adapter
store-uri="${protocol}://${username:}:${password:}#${host}:${port}/${folder}"
channel="transform" should-mark-messages-as-read="${markAsRead}"
should-delete-messages="${delete}" java-mail-properties="javaMailProperties">
<poller fixed-delay="${fixedDelay}" time-unit="SECONDS">
<advice-chain>
<beans:bean
class="org.springframework.xd.dirt.module.support.ThreadContextClassLoaderSetterAdvice" />
</advice-chain>
</poller>
</int-mail:inbound-channel-adapter>
</beans:beans>
<beans:beans profile="use-idle">
<int-mail:imap-idle-channel-adapter
store-uri="${protocol}://${username:}:${password:}#${host}:${port}/${folder}"
channel="transform" auto-startup="true" mail-filter-expression="${expression}"
should-mark-messages-as-read="${markAsRead}"
should-delete-messages="${delete}" java-mail-properties="javaMailProperties">
</int-mail:imap-idle-channel-adapter>
</beans:beans>
<beans:beans profile="default">
<util:properties id="javaMailProperties">
<beans:prop key="mail.imap.socketFactory.class">javax.net.ssl.SSLSocketFactory</beans:prop>
<beans:prop key="mail.imap.socketFactory.fallback">false</beans:prop>
<beans:prop key="mail.store.protocol">imaps</beans:prop>
<beans:prop key="mail.debug">false</beans:prop>
</util:properties>
</beans:beans>
</beans:beans>
4. In XD shell now you will use something like the following to create your stream:
stream create --name myGmailStream --definition "gmail --host=imap.gmail.com --username=yyyyyyyy12%40gmail.com --password=my_password --port=993 | file --dir=/tmp/gmailData" --deploy
Here, please note the following:
I added --port=993
the username contains "%40" instead of "#"
the definition of the stream starts with "gmail
if your password contains "#" you need to replace that with "%40" as well
What I've done above is to, basically, create a new custom module (a source) which is kind of easy (more details about this you can find in the documentation). The XD single node or the XD Shell doesn't even need to be restarted. Give it a try and let me know how it goes.

Regarding the password that you don't want to appear as part of the stream definition, you can provide it as part of your mail module options, as described here: http://docs.spring.io/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#_module_configuration
ie
<xd_home>/config/modules/source/mail/mail.properties:
password: yourpassword

Related

Mule SFTP Component component

I am developing mule flow wherein mule process has to upload file/files (from source directory) to SFTP server.
IN Folder (This folder holds file to transfer to SFTP)
I have below mule flow
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/sftp http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd">
<flow name="sftpFlow1">
<poll doc:name="Poll">
<fixed-frequency-scheduler frequency="30" startDelay="10" timeUnit="SECONDS"/>
<logger message="Hi" level="INFO" doc:name="Logger"/>
</poll>
<flow-ref name="sftpSub_Flow" doc:name="Flow Reference"/>
</flow>
<sub-flow name="sftpSub_Flow">
<file:outbound-endpoint path="D:\IN" responseTimeout="10000" doc:name="File"/>
<sftp:outbound-endpoint exchange-pattern="one-way" host="host" port="port" path="sftppath" user="user" password="password" responseTimeout="10000" doc:name="SFTP"/>
</sub-flow>
</mule>
Issues:
I don't see any error or exceptions but expected files from IN folder are not getting transfer SFTP server. I see ****.dat file every time sub-flow executes.
Any suggestion?
Here you have used both outbound end points. You need file inbound end point component to pick the file from source location.
refer below sample for further
https://dzone.com/articles/anypoint-file-connector-with-mulesoft
you can use the outputPattern property in sftpoutbound endpoint to change the file name with any extension
Because the application doesn't actually show any input, I'll assume that the sample is flawed but the real application has some inbound endpoint (for example a file inbound endpoing) that is reading in streaming mode. The subflow is trying to consume two time the input stream to output to the two outbound endpoints. Because the first one consumes the stream the second one, the SFTP, gets a consumed empty stream as input so it doesn't output anything. A solution is to convert the input to something in memory, like using the object to byte array transformer in the middle. Be warned that for large files you might run out of memory doing this.

Not create new log file once start my service with using log4j2

following is my configration of log4j2:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="trace" name="MyApp" packages="com.swimap.base.launcher.log">
<Appenders>
<RollingFile name="RollingFile" fileName="logs/app-${date:MM-dd-yyyy-HH-mm-ss-SSS}.log"
filePattern="logs/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="1 KB"/>
</Policies>
<DefaultRolloverStrategy max="3"/>
</RollingFile>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>
the issue is that each time when start up my service, a new log will be created even the old one has not reached the specific size. If the program restart frequently, i will got many log files end with ‘.log’ which never be compressed.
the logs i got like this:
/log4j2/logs
/log4j2/logs/2017-07
/log4j2/logs/2017-07/app-07-18-2017-1.log.gz
/log4j2/logs/2017-07/app-07-18-2017-2.log.gz
/log4j2/logs/2017-07/app-07-18-2017-3.log.gz
/log4j2/logs/app-07-18-2017-20-42-06-173.log
/log4j2/logs/app-07-18-2017-20-42-12-284.log
/log4j2/logs/app-07-18-2017-20-42-16-797.log
/log4j2/logs/app-07-18-2017-20-42-21-269.log
someone can tell me how can i append log to the exists log file when i start up my program? much thanks whether u can help me closer to the answer!!
I suppose that your problem it that you have fileName="logs/app-${date:MM-dd-yyyy-HH-mm-ss-SSS}.log in your log4j2 configuration file.
This fileName template means that log4j2 will create log file with name that contains current date + hours + minutes + seconds + milliseconds in its name.
You should probably remove HH-mm-ss-SSS section and this will allow you to have daily rolling file and to not create new file every app restart.
You can play with template and choose format that you need.
If you want only one log file forever - then create constant fileName, like fileName=app.log
It's not hard to implement this. There is a interface DirectFileRolloverStrategy, implement below method:
public String getCurrentFileName(RollingFileManager manager)
Mybe someone met same problem and this can help him.

How to add the description of job in spring batch admin user interface?

Is there any way to add the description of job at the user interface of spring batch admin?
Although, I tried to added the description of the job, spring batch admin cannot support it.
I would like to know that whether spring batch admin does not support it or not.
I know i'm late to the party but I figured it out and it works flawlessly for me. All you have to do is :
Add a messages.properties file in your classpath (under
src/main/resources).
Add yourJobName.description=Your description goes here in that file.
Override manager-context.xml for SBA by creating a file on path src/main/resources/META-INF/spring/batch/servlet/override/manager-context.xml
The content of the above created file should be :
`
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
<!-- Override messageSource bean in order to provide custom text content -->
<bean id="messageSource" class="org.springframework.context.support.ResourceBundleMessageSource">
<property name="basename" value="messages" />
</bean>
</beans>
`
That's it. Your custom description shows up in SBA. Hope this helps someone who's looking for it.
There isn't the ability out of the box to display the job's description. That is only contained in the XML and the data seen in the UI comes from the JobRepository. You'd have to extend the UI to add that functionality.

Change service location on Mule

my first try was to simple proxy a service from one location to another, and it work just fine, right now i need some help in how to change part of the service location, for example, the retrieved WSDL point 4 services to a machine, i need to change 1 of those services for another server, is that even possible? If so, how do i do it?
Mule Version CE 3.4.
my code atm is as follow:
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:ssl="http://www.mulesoft.org/schema/mule/ssl" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
xmlns:mulexml="http://www.mulesoft.org/schema/mule/xml" xmlns:https="http://www.mulesoft.org/schema/mule/https"
xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:spring="http://www.springframework.org/schema/beans" xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:pattern="http://www.mulesoft.org/schema/mule/pattern"
xmlns:mule-ss="http://www.mulesoft.org/schema/mule/spring-security"
xmlns:ss="http://www.springframework.org/schema/security"
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd
http://www.mulesoft.org/schema/mule/xml http://www.mulesoft.org/schema/mule/xml/current/mule-xml.xsd
http://www.mulesoft.org/schema/mule/https http://www.mulesoft.org/schema/mule/https/current/mule-https.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/3.4/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/3.4/mule-http.xsd
http://www.mulesoft.org/schema/mule/pattern http://www.mulesoft.org/schema/mule/pattern/3.4/mule-pattern.xsd
http://www.mulesoft.org/schema/mule/spring-security http://www.mulesoft.org/schema/mule/spring-security/3.4/mule-spring-security.xsd
http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-3.1.xsd
http://www.mulesoft.org/schema/mule/ssl http://www.mulesoft.org/schema/mule/ssl/current/mule-ssl.xsd" version="EE-3.4.0">
<mule-ss:security-manager>
<mule-ss:delegate-security-provider
name="memory-dao" delegate-ref="authenticationManager" />
</mule-ss:security-manager>
<spring:beans>
<ss:authentication-manager alias="authenticationManager">
<ss:authentication-provider>
<ss:user-service id="userService">
<ss:user name="asd" password="asd" authorities="ROLE_ADMIN" />
</ss:user-service>
</ss:authentication-provider>
</ss:authentication-manager>
</spring:beans>
<https:connector name="httpsConnector">
<https:tls-key-store path="${mule.home}/conf/keystore.jks"
keyPassword="1234567" storePassword="1234567" />
</https:connector>
<pattern:web-service-proxy name="Service"
inboundAddress="https://LocalAdress.com:443/services/Service"
outboundAddress="http://RemoteAddress.com/services/Service.svc"
wsdlLocation="http://RemoteAddress.com/services/Service.svc?singleWSDL"/>
Mule doesn't offer a mechanism to realize this type of WSDL customization. What you have to do is:
download "http://RemoteAddress.com/services/Service.svc?singleWSDL",
customize it by hand,
embed the customized version in your project (say in "src/main/resources"),
serve it with wsdlFile="yourCustom.wsdl"

Transaction commit delay when routing message from one jms queue to another

We are trying to build simple transacted jms-to-jms router using Mule ESB and JBoss Messaging. When we run Mule ESB with application configured as below, we observe strange behaviour.
Approximately 10 messages are routed from queue test1 to test2
Nothing happens for ~40 seconds.
goto 1
Queue test1 is filled with around 500 messages when we start test. We use Mule 3.2 and JBoss 5.1.
If I remove transactions from code below everything works fine, all messages are sent to queue test2 instantly. Also, everything is fine if I change transactions from xa to jms -- by replacing xa-transaction tags with jms:transaction.
I don't know what causes this pause in message processing on ESB, probably transaction commit is delayed.
My question is: what should I do to have xa transactions working correctly?
I'll provide more details if needed. I asked this question on Mule ESB forum before with no answer http://forum.mulesoft.org/mulesoft/topics/transaction_commit_delay_when_routing_message_from_one_jms_queue_to_another
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:jms="http://www.mulesoft.org/schema/mule/jms" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" xmlns:core="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jbossts="http://www.mulesoft.org/schema/mule/jbossts" version="CE-3.2.1" xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/jbossts http://www.mulesoft.org/schema/mule/jbossts/current/mule-jbossts.xsd ">
<jbossts:transaction-manager> </jbossts:transaction-manager>
<configuration>
<default-threading-profile maxThreadsActive="30" maxThreadsIdle="5"/>
<default-receiver-threading-profile maxThreadsActive="10" maxThreadsIdle="5"/>
</configuration>
<spring:beans>
<spring:bean id="jmsJndiTemplate" class="org.springframework.jndi.JndiTemplate" doc:name="Bean">
<spring:property name="environment">
<spring:props>
<spring:prop key="java.naming.factory.url.pkgs">org.jboss.naming:org.jnp.interfaces</spring:prop>
<spring:prop key="jnp.disableDiscovery">true</spring:prop>
<spring:prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</spring:prop>
<spring:prop key="java.naming.provider.url">localhost:1099</spring:prop>
</spring:props>
</spring:property>
</spring:bean>
<spring:bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean" doc:name="Bean">
<spring:property name="jndiTemplate">
<spring:ref bean="jmsJndiTemplate"/>
</spring:property>
<spring:property name="jndiName">
<spring:value>XAConnectionFactory</spring:value>
</spring:property>
</spring:bean>
</spring:beans>
<jms:connector name="JMS" specification="1.1" numberOfConsumers="10" connectionFactory-ref="jmsConnectionFactory" doc:name="JMS"/>
<flow name="flow" doc:name="flow">
<jms:inbound-endpoint queue="test1" connector-ref="JMS" doc:name="qt1">
<xa-transaction action="ALWAYS_BEGIN"/>
</jms:inbound-endpoint>
<echo-component doc:name="Echo"/>
<jms:outbound-endpoint queue="test2" connector-ref="JMS" doc:name="qt2">
<xa-transaction action="ALWAYS_JOIN"/>
</jms:outbound-endpoint>
<echo-component doc:name="Echo"/>
</flow>
</mule>
Here you can find log fragment for 1 message interaction. Please note that in this case there was no delay.
And here is log fragment for 11 messages. All of them were in queue test1 when app started, as you can see 10 messages are routed instantly and one is delayed by 1 minute.
I've found root of my problem: my queues were defined with following attribute:
<attribute name="RedeliveryDelay">60000</attribute>
Removing it or setting low value solves my problem with delays. Problem is, I don't know why :)
I always thought that redelivery delay is used when delivery fails, which was not the case in my app.