WSO2 Kafka Inbound Endpoint Retry on Error - apache-kafka

We try to use Kafka with WSO2 ESB server.
We've implemented and API which puts incoming messages to Kafka.
Then we've implemented an inbound endpoint that retrieves messages from Kafka and transfer these messages to other external sytems.
Everything works very well in happy path but when we test the "external systems down" scenario failed messages are not delivered when external systems up again.
How can we send failed messages to external systems?
API config:
<?xml version="1.0" encoding="UTF-8"?>
<api context="/api/event" name="EventAPI" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="POST">
<inSequence>
<log category="DEBUG" description="" level="full"/>
<kafkaTransport.init>
<bootstrapServers>localhost:9092</bootstrapServers>
<keySerializerClass>org.apache.kafka.common.serialization.StringSerializer</keySerializerClass>
<valueSerializerClass>org.apache.kafka.common.serialization.StringSerializer</valueSerializerClass>
<maxPoolSize>20</maxPoolSize>
</kafkaTransport.init>
<kafkaTransport.publishMessages>
<topic>event_topic</topic>
</kafkaTransport.publishMessages>
<loopback/>
</inSequence>
<outSequence>
<payloadFactory media-type="json">
<format>{"result" : "OK"}</format>
</payloadFactory>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<send/>
</outSequence>
</resource>
</api>
Inbound config:
<?xml version="1.0" encoding="UTF-8"?>
<inboundEndpoint name="EventTransmitter" protocol="kafka"
sequence="transmit_sequence" suspend="false" onError="fault"
xmlns="http://ws.apache.org/ns/synapse">
<parameters>
<parameter name="interval">10</parameter>
<parameter name="coordination">true</parameter>
<parameter name="sequential">true</parameter>
<parameter name="zookeeper.connect">localhost:2181</parameter>
<parameter name="consumer.type">highlevel</parameter>
<parameter name="content.type">application/json</parameter>
<parameter name="topics">event_topic</parameter>
<parameter name="group.id">myconsumer</parameter>
<parameter name="consumer.id">myconsumer</parameter>
<parameter name="dual.commit.enabled">true</parameter>
<parameter name="auto.offset.reset">largest</parameter>
</parameters>
</inboundEndpoint>
Sequence:
<?xml version="1.0" encoding="UTF-8"?>
<sequence name="transmit_sequence" onError="fault" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<send receive="event_transmit_out_sequence">
<endpoint key="gov:endpoints/HandlerEndpoint.xml"/>
</send>
</sequence>

Related

REST WSO2 Kafka

I've installed WSO2 Integration Studio version 6.5.0 in my Windows workstation and created a project using the Kafka Consumer and Producer built-in template.
api.xml:
<?xml version="1.0" encoding="UTF-8"?>
<api context="/publishweatherdata" name="WeatherDataPublishAPI" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="POST">
<inSequence>
<kafkaTransport.init>
<bootstrapServers>localhost:9092</bootstrapServers>
<keySerializerClass>org.apache.kafka.common.serialization.StringSerializer</keySerializerClass>
<valueSerializerClass>org.apache.kafka.common.serialization.StringSerializer</valueSerializerClass>
<acks>all</acks>
<requestTimeout>10000</requestTimeout>
<timeout>8000</timeout>
<metadataFetchTimeout>5000</metadataFetchTimeout>
<maxPoolSize>50</maxPoolSize>
</kafkaTransport.init>
<kafkaTransport.publishMessages>
<topic>weatherdatatopic</topic>
</kafkaTransport.publishMessages>
<payloadFactory media-type="json">
<format>
{"topic":"$1", "partition":"$2", "offset":"$3"}
</format>
<args>
<arg evaluator="xml" expression="$ctx:topic"/>
<arg evaluator="xml" expression="$ctx:partition"/>
<arg evaluator="xml" expression="$ctx:offset"/>
</args>
</payloadFactory>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<respond/>
</inSequence>
<outSequence/>
<faultSequence/>
</resource>
</api>
weatherdatatransmitinboundEP.xml:
<?xml version="1.0" encoding="UTF-8"?>
<inboundEndpoint class="org.wso2.carbon.inbound.kafka.KafkaMessageConsumer" name="WeatherDataTransmitInboundEP" onError="WeatherDataErrorSeq" sequence="WeatherDataProcessSeq" suspend="false" xmlns="http://ws.apache.org/ns/synapse">
<parameters>
<parameter name="sequential">true</parameter>
<parameter name="interval">10</parameter>
<parameter name="coordination">true</parameter>
<parameter name="inbound.behavior">polling</parameter>
<parameter name="value.deserializer">org.apache.kafka.common.serialization.StringDeserializer</parameter>
<parameter name="topic.name">weatherdatatopic</parameter>
<parameter name="poll.timeout">100</parameter>
<parameter name="bootstrap.servers">localhost:9092</parameter>
<parameter name="group.id">hello</parameter>
<parameter name="contentType">application/json</parameter>
<parameter name="key.deserializer">org.apache.kafka.common.serialization.StringDeserializer</parameter>
<parameter name="class">org.wso2.carbon.inbound.kafka.KafkaMessageConsumer</parameter>
</parameters>
</inboundEndpoint>
WeatherDataPublishService.xml:
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="WeatherDataPublishService" startOnLoad="true" transports="http https" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<kafkaTransport.init>
<bootstrapServers>localhost:9092</bootstrapServers>
<keySerializerClass>org.apache.kafka.common.serialization.StringSerializer</keySerializerClass>
<valueSerializerClass>org.apache.kafka.common.serialization.StringSerializer</valueSerializerClass>
<acks>all</acks>
<requestTimeout>10000</requestTimeout>
<timeout>8000</timeout>
<metadataFetchTimeout>5000</metadataFetchTimeout>
<maxPoolSize>50</maxPoolSize>
</kafkaTransport.init>
<kafkaTransport.publishMessages>
<topic>weatherdatatopic</topic>
</kafkaTransport.publishMessages>
<payloadFactory media-type="json">
<format>
{"topic":"$1", "partition":"$2", "offset":"$3"}
</format>
<args>
<arg evaluator="xml" expression="$ctx:topic"/>
<arg evaluator="xml" expression="$ctx:partition"/>
<arg evaluator="xml" expression="$ctx:offset"/>
</args>
</payloadFactory>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<respond/>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
</proxy>
now I can send post request to http://localhost:8290/publishweatherdata and get it in the kafka topic. Also I can receive message produced by kafka in wso2. How can I send message to external service from wso2? I think I should to use
<endpoint [name="string"] [key="string"]>
address-endpoint | default-endpoint | wsdl-endpoint | load-balanced-endpoint | fail-over-endpoint
</endpoint>
but I have no idea where it must be added and how configured
You can use a call mediator [1] or a send mediator [2] to achieve your use case. Within the mediator, you can define the desired endpoint you want to invoke. Please refer to the following sample configuration. Here we have used a call mediator to invoke the external endpoint http://run.mocky.io/v3/9cf4b844-57c1-4fa5-a101-881dc36385bd.
<call>
<endpoint>
<address uri="http://run.mocky.io/v3/9cf4b844-57c1-4fa5-a101-881dc36385bd"/>
</endpoint>
</call>
In your use case, if you have finish building the desired payload after the payload factory mediator, you can use the call mediator after the payload factory mediator to invoke the external endpoint. Here the payload build by the payload factory mediator will be used to invoke the external endpoint.
<payloadFactory media-type="json">
<format>
{"topic":"$1", "partition":"$2", "offset":"$3"}
</format>
<args>
<arg evaluator="xml" expression="$ctx:topic"/>
<arg evaluator="xml" expression="$ctx:partition"/>
<arg evaluator="xml" expression="$ctx:offset"/>
</args>
</payloadFactory>
<call>
<endpoint>
<address uri="http://run.mocky.io/v3/9cf4b844-57c1-4fa5-a101-881dc36385bd"/>
</endpoint>
</call>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<respond/>
Further, in the case of WeatherDataTransmitInboundEP(inbound endpoint), it will read the messages published to Kafka, and then the message is sent to the sequence defined in the inbound endpoint. If you want to send the messages consumed by WeatherDataTransmitInboundEP to an external endpoint you have to follow a different approach.
In your case, WeatherDataProcessSeq is invoked after reading messages from Kafka. So if your requirement is to send the messages in Kafka you will need to define the call or the send mediator in the WeatherDataProcessSeq.
If you want further clarification regarding the call/send mediator please refer to the blog post [3].
[1]-https://docs.wso2.com/display/EI6xx/Call+Mediator
[2]-https://docs.wso2.com/display/EI600/Send+Mediator
[3]-https://www.yenlo.com/blog/wso2torial-to-send-or-not-to-send-that-is-your-choice
I just add
<send>
<endpoint>
<http method="post" statistics="enable" trace="enable" uri-template="http://localhost:8081/api">
<property name="name" scope="axis2" value="messageValue"/>
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>-1</progressionFactor>
<maximumDuration>0</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</http>
</endpoint>
</send>
to sequence and get what I want

Error when sending Email in WSO2 ESB

I send Mail with WSO2 ESB 5.0.0
1. I have uncommented the following line in Axis2.xml file
<transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">
<parameter name="mail.smtp.host">smtp.gmail.com</parameter>
<parameter name="mail.smtp.port">587</parameter>
<parameter name="mail.smtp.starttls.enable">true</parameter>
<parameter name="mail.smtp.auth">true</parameter>
<parameter name="mail.smtp.user">lmphuong</parameter>
<parameter name="mail.smtp.password">password</parameter>
<parameter name="mail.smtp.from">lmphuong#gmail.com</parameter>
</transportSender>
<transportReceiver name="mailto" class="org.apache.axis2.transport.mail.MailTransportListener">
<!-- configure any optional POP3/IMAP properties
check com.sun.mail.pop3 and com.sun.mail.imap package documentation for more details-->
</transportReceiver>
2. Add content at messageFormatters in axis2.xml
<messageFormatter contentType="text/html" class="org.apache.axis2.transport.http.ApplicationXMLFormatter"/>
3. I have create Proxy Service in WSO2 ESB
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="EmailSender"
transports="http https"
startOnLoad="true">
<description/>
<target>
<inSequence>
<log/>
<property name="messageType"
value="text/html"
scope="axis2"
type="STRING"/>
<property name="ContentType" value="text/html" scope="axis2"/>
<property name="Subject" value="Testing ESB" scope="transport"/>
<property name="OUT_ONLY" value="true"/>
<property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
<payloadFactory media-type="xml">
<format>
<ns:text xmlns:ns="http://ws.apache.org/commons/ns/payload">$1</ns:text>
</format>
<args>
<arg value="Hello WSO2 ESB.....!"/>
</args>
</payloadFactory>
<log level="full"/>
<send>
<endpoint>
<address uri="mailto:ledung123#gmail.com"/>
</endpoint>
</send>
</inSequence>
<outSequence/>
</target>
</proxy>
4. I recieved error
ERROR - MailTransportSender Error creating mail message or sending it to the configured server
javax.mail.AuthenticationFailedException
at javax.mail.Service.connect(Service.java:306)
...
[2017-09-15 08:04:05,945] ERROR - MailTransportSender Error generating mail message
...
Please help me out how to soved this error
Looks like thereĀ“s something wrong with your credentials (AuthenticationFailedException). Do you have any special chracters in your credentials? Or is there are proxy server between esb and gmail?
What you could do is start esb in debug/enable wire log to see the complete traffic. More info can be found here.
https://docs.wso2.com/display/ESB500/Debugging+Mediation#DebuggingMediation-Viewingwirelogs
Another option might be to use the GMail connector which can be found here.
https://docs.wso2.com/display/ESBCONNECTORS/Gmail+Connector
https://store.wso2.com/store/pages/top-assets?q=%22_default%22%3A%22gmail%22

Restore task class in WSO2 ESB

When i use a task in WSO2 ESB it always returns the same error with every web service:
"Unable to handle request. The action '(mySoapAction)' was not recognized"
where (mySoapAction) is every SOAP action used, for EVERY action, for EVERY proxy service i use for task implementation.
What could i do in order to fix this error? I thought a task class error in org.apache.synapse.startup.tasks.MessageInjector.
Obvoiusly the task implementation is correct, because the same tasks some day ago were perfectly working. Suggestions?
Here come a sample with the weather webservice (http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL)
Proxy service :
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="TestSOF"
transports="https http"
startOnLoad="true"
trace="disable">
<target>
<endpoint>
<wsdl service="Weather"
port="WeatherSoap12"
uri="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL"/>
</endpoint>
<outSequence>
<log level="full"/>
<property name="OUT_ONLY" value="true"/>
<property name="transport.vfs.ReplyFileName" value="weather.xml" scope="transport"/>
<send>
<endpoint>
<address uri="vfs:file:///E:/temp"/>
</endpoint>
</send>
</outSequence>
</target>
<publishWSDL uri="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL"/>
</proxy>
Task :
<?xml version="1.0" encoding="UTF-8"?>
<task xmlns="http://ws.apache.org/ns/synapse"
name="TestSOFTask"
class="org.apache.synapse.startup.tasks.MessageInjector"
group="synapse.simple.quartz">
<trigger count="1" interval="1"/>
<property xmlns:task="http://www.wso2.org/products/wso2commons/tasks"
name="proxyName"
value="TestSOF"/>
<property xmlns:task="http://www.wso2.org/products/wso2commons/tasks"
name="soapAction"
value="http://ws.cdyne.com/WeatherWS/GetCityWeatherByZIP"/>
<property xmlns:task="http://www.wso2.org/products/wso2commons/tasks" name="message">
<weat:GetCityWeatherByZIP xmlns:weat="http://ws.cdyne.com/WeatherWS/">
<weat:ZIP>11010</weat:ZIP>
</weat:GetCityWeatherByZIP>
</property>
<property xmlns:task="http://www.wso2.org/products/wso2commons/tasks"
name="format"
value="soap12"/>
<property xmlns:task="http://www.wso2.org/products/wso2commons/tasks"
name="injectTo"
value="proxy"/>
</task>
Schedule the task and you will find the service response in a file name weather.xml
If you want to change from soap12 to soap11 :
Change the value of property "format" in the task def to : soap11
Change the endpoint def in the proxy service using port "WeatherSoap" rather than "WeatherSoap12"
Hope it will help you to find what is going wrong with your conf...

WSO2 ESB How to send an email from a JMS message to a custom email address?

I'm wondering if someone can help me with the following setup.
I want to send a message from my application via JMS to WSO2 ESB so the ESB can send it as en email. I'm using ActiveMQ as queue. Until now, when I send a message via the ActiveMQ interface to the queue, wso2 esb gets it. Then, wso2 esb send the message as email to a specific email address.
So I could configure ActiveMQ and WSO2 esb to send the JMS message to a specific email address (eg. specificaddress#test.com).
And here is my question. How can I modify the receiver address for the email? In the ESB sequence configuration, I currently use a specific address. But the address is dependant on the user that uses my application. So I have to change the "To" property, dependant on the user that has to receive the email.
So how can I pass the values for the properties "To", but also for "Subject", through a JMS message to WSO2 esb sequence?
That's the configuration of the sequence I have:
<sequence xmlns="http://ws.apache.org/ns/synapse" name="sendMail">
<property name="messageType" value="text/html" scope="axis2" type="STRING"></property>
<property name="ContentType" value="text/html" scope="axis2"></property>
<property name="Subject" value="This is the subject." scope="transport"></property>
<property name="To" value="specificaddress#test.com" scope="transport"></property>
<property name="OUT_ONLY" value="true" scope="default" type="STRING"></property>
<log level="full"></log>
<send>
<endpoint>
<address uri="mailto:"></address>
</endpoint>
</send>
</sequence>
And this is my proxy:
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="sendToMailIn"
transports="jms"
statistics="disable"
trace="disable"
startOnLoad="true">
<target inSequence="sendMail"/>
<description/>
</proxy>
I hope someone has a clue.
UPDATE
I think I have the solution!!! Wow :-) Maybe, at first, I was stupid, but here it is ...
What you can do is sending a SOAP envelop through a JMS message to WSO2 ESB. And then, with an XPath expression, you can get the passed values. A little bit has to changed at the proxy and the sequence.
This is the new sequence:
<sequence xmlns="http://ws.apache.org/ns/synapse" name="sendMail">
<property name="messageType" value="text/html" scope="axis2" type="STRING"></property>
<property name="ContentType" value="text/html" scope="axis2"></property>
<property xmlns:ns="http://org.apache.synapse/xsd" name="Subject" expression="$body/subject" scope="transport"></property>
<property xmlns:ns="http://org.apache.synapse/xsd" name="To" expression="$body/to" scope="transport"></property>
<property name="OUT_ONLY" value="true" scope="default" type="STRING"></property>
<log level="full"></log>
<send>
<endpoint>
<address uri="mailto:"></address>
</endpoint>
</send>
</sequence>
And this is the new proxy:
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="sendToMailIn"
transports="jms"
statistics="disable"
trace="disable"
startOnLoad="true">
<target inSequence="sendMail"/>
<parameter name="transport.jms.ContentType">
<rules>
<jmsProperty>contentType</jmsProperty>
<default>text/xml</default>
</rules>
</parameter>
<description/>
</proxy>
And this was my SOAP Envelop that WSO2 ESB receives from my ActiMQ queue as JMS message:
<?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org /soap/envelope/">
<soapenv:Body>
<subject>Email subject comes here.</subject>
<to>address#test.com</to>
</soapenv:Body>
</soapenv:Envelope>
you have couple of options here.
you can use http headers and send the "to", "subject" values to ESB.
Send it as a payload value and extract using XPath expression
I solved it with using JSON in a JMS message. Here is my setup that works for me.
This is my JSON message:
{"to":"mail#test.com","subject":"TestSubject","mailbody":"Some body text ..."}
This is my proxy:
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="sendToMailIn"
transports="jms"
statistics="disable"
trace="disable"
startOnLoad="true">
<target inSequence="sendMail"/>
<parameter name="transport.jms.ContentType">
<rules>
<jmsProperty>contentType</jmsProperty>
<default>application/json</default>
</rules>
</parameter>
<parameter name="transport.mail.ContentType">application/xml</parameter>
<description/>
</proxy>
And this is my sequence:
<sequence xmlns="http://ws.apache.org/ns/synapse" name="sendMail">
<property name="messageType" value="text/plain" scope="axis2" type="STRING"></property>
<property name="ContentType" value="text/plain" scope="axis2"></property>
<property xmlns:ns="http://org.apache.synapse/xsd" name="Subject" expression="json-eval($.subject)" scope="transport"></property>
<property xmlns:ns="http://org.apache.synapse/xsd" name="To" expression="json-eval($.to)" scope="transport"></property>
<property name="OUT_ONLY" value="true" scope="default" type="STRING"></property>
<script language="js">
<![CDATA[var mailbody = mc.getPayloadJSON().mailbody.toString(); mc.setPayloadXML(
<ns:text xmlns:ns="http://ws.apache.org/commons/ns/payload">{mailbody}</ns:text>);]]>
</script>
<log level="full"></log>
<send>
<endpoint>
<address uri="mailto:"></address>
</endpoint>
</send>
</sequence>

Transaction mediator in wso2esb 4.8.0

I'm working with wso2esb 4.8.0 and i go through this scenario which is given in wso2esb documentation.I want to copy the records from first database to second database as soon as i delete them from first database.but when i put the duplicate record it should neither delete from first nor insert into second.
my proxy service :
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="Transaction_mediator"
transports="https,http"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<property name="name"
expression="//name/text()"
scope="default"
type="STRING"/>
<property name="id"
expression="//id/text()"
scope="default"
type="STRING"/>
<property name="price"
expression="//price/text()"
scope="default"
type="STRING"/>
<transaction action="new"/>
<log level="full">
<property name="text" value="Reporting to the DB1"/>
</log>
<dbreport useTransaction="true">
<connection>
<pool>
<dsName>DS1</dsName>
</pool>
</connection>
<statement>
<sql>
delete from c1 where name=?</sql>
<parameter expression="get-property('name')" type="VARCHAR"/>
</statement>
</dbreport>
<log level="full">
<property name="text" value="Reporting to the DB2"/>
</log>
<dbreport useTransaction="true">
<connection>
<pool>
<dsName>DS2</dsName>
</pool>
</connection>
<statement>
<sql>
INSERT into c1 values (?,?,?)</sql>
<parameter expression="get-property('name')" type="VARCHAR"/>
<parameter expression="get-property('id')" type="INTEGER"/>
<parameter expression="get-property('price')" type="INTEGER"/>
</statement>
</dbreport>
<transaction action="commit"/>
<send/>
</inSequence>
<outSequence>
<log level="full"/>
<send/>
</outSequence>
</target>
<description/>
</proxy>
It's working but error occurred when i put duplicate records.At that time It deleting records from first but not inserting into second database.What should i do?is their any solution?let me know.
There is an issue with transaction mediator with JDBC and it has been reported in the wso2 esb issue tracker. Therefore, this may be fixed in feature releases.