I'm using wso2 cep 4.1
I created receiver to catch some json data from my source. Then I process this data internally and should give the response with additional data. My response should be through the same point as data come in to CEP. It is classical rest API. Is it possible and how can I make it?
Or, I need websocket (websocket-local) for similar purposes?
Hope you are still trying to understand functionalities of WSO2 CEP. Let me explain the basic overview of CEP before addressing your question. If you look at below diagram you will understand what is happening under the hood at a high level. . I will explain what these components suppose to do in the context of event processing.
Event receivers :-
Event receivers receive events that are coming to the CEP. WSO2 CEP supports the most common adapter implementations by default. For specific use cases, you can also plug custom adapters. For more information, see Configuring Event Receivers.
Event streams :-
Event streams contain unique sets of attributes of specific types that provide a structure based on which the events processed by the relevant event flow are selected. Event streams are stored as stream definitions in the file system via the data bridge stream definition store.
Event processors :-
Event processor handles actual event processing. It is the core event processing unit of the CEP. It manages different execution plans and processes events based on logic, with the help of different Siddhi queries. Event Processor gets a set of event streams from the Event Stream Manager, processes them using Siddhi engine, and triggers new events on different event streams back to the Event Stream Manager. For more information, see Creating a Standalone Execution Plan.
Event publishers
Event publishers:- publish events to external systems and store data to databases for future analysis. Like the event receivers, this component also has different adapter implementations. The most common ones are available by default in the CEP. You can implement custom adapters for specific use cases. For more information, see Configuring CEP to Create Alerts.
According to your requirement, you should have HTTP receiver as well as HTTP publisher where the receiver receives a request from a third party API and hand message over to event processors so as to perform some pre-defined tasks.This may compose with several event streams and execution plans. Once processing is done event publishers can be used to publish result to required third-party API as you pointed out.
OOB CEP provides HTTP receiver and HTTP publisher adapters[1-2] which you can try it out.There are some limitations which might not suit for your scenario. You are required to implement your own custom HTTP receiver and publisher[3-4] which does what you intended to do.
Since you need to publish a response to difference endpoints,you can achieve this defining REST API endpoint,user credentials(if required) and HTTP verbs and other information which required to send a message in the event stream[5] as meta information. Then that information you can read from the stream itself and push to desired third-party API as you require.
I need websocket (websocket-local) for similar purposes?
This isn't clear what exactly is to be done. Please raise an another question and ask it again.
https://docs.wso2.com/display/CEP410/HTTP+Event+Receiver
https://docs.wso2.com/display/CEP410/HTTP+Event+Publisher
https://docs.wso2.com/display/CEP410/Building+Custom+Event+Receivers
https://docs.wso2.com/display/CEP410/Building+Custom+Event+Publishers
https://docs.wso2.com/display/CEP410/Understanding+Event+Streams
The feature you are looking for doesn't come OOTB with CEP. However, you can try something similar to below;
Implement a REST API. Probably using Apache CXF since CXF dependencies are present in WSO2 servers by default. You can follow this guide if you are using a swagger based approach to develop the REST API.
Within that custom REST implementation, you need to read the HTTP request, send it to CEP (step 3), wait for an output from CEP (step 4) and then send back that details as HTTP response inside the method which represents your operation.
To send an event to CEP you can use WSO2 Event receiver. Create a receiver at CEP side and then send events to the receiver using DataPublisher client. Make sure you have the same stream definition that you set in CEP receiver in the DataPublisher.publish() method and object array you send adhere to that definition. Also, you might need to set truststore and keystore params here.
After publishing your events successfully you need to block the request thread till you receive a response from CEP. You can use a java object like CountDownLatch for this purpose.
To receive a response you need to consume events though EventStreamService For this you need to implement a WSO2EventConsumer and subscribe to EventStreamService. After successfully subscribing, events coming to stream id mentioned in your event consumer will be forwarded to receive method of your Consumer. From there you can extract the results, unblock the initial request thread and return with those results. To access the EventStreamService from within your web app you can use below code snippet.
EventStreamService eventStreamService = (EventStreamService) PrivilegedCarbonContext.getThreadLocalCarbonContext().getOSGiService(EventStreamService.class, null);
Hope this helped.
Related
I am looking for a way to expose an existing event processing system to the external world using a REST interface. I have existing system design where we have RabbitMQ message queues where a publisher could post a message and then wait for the message processed results on a separate queue. Message ID is used to track the output to the original message on the output queue.
Now I want this to be exposed to the external consumers but we don't want to expose our RabbitMQ endpoint for this, so I was wondering if anyone has managed to achieve something similar to this using ExpressJS. Above diagram shows the current thought process
Main challenge I am facing here is that; some of this message processing could take more than couple of minutes, so was not sure how best to develop a API like this. Choices like should I create a polling interface for client here or is there a technology these days that help eliminate the polling on the client API to verify if the message is processed and get the result.
Can someone please help me with a good approach to manage these sort of requirement.
I finally ended up going the webhook way. Now when the REST API service receives a request, the client need to also provide a webhook and this will be registered with the client request and server will call it back when the results are available.
I'm trying to develop a project with microservices.
I have some questions on this topic (something is not clear):
1) How to implement microservices communication?
A) HTTP : Every microservice expose HTTP API , an API GATEWAY broadcast requests.
B) MQTT : every microservice pub/sub to a broker
C) BOTH : but how to understand when one is better than the other ?
Have I to use pub/sub protocol as a standard even for classic operations usually performed over HTTP ? For example I have two microservices:
web-management and product-service. web-management is a panel that lets the administrator to add, modify, ... products in its ecommerce digital shop. Let's say we want to implement createProduct operation. It's a command (according to event /command distinction), a one-to-one communication.
I can open an API in product-service, let's say (POST, "/product") that add the new product. I also can implement this transforming the command in a productCreationRequest event. In this case: web-managemnet publish this event. product-service listen to productCreationRequest events (and also productUpdateRequest, productGetEvents, ...) once it is notified it performs the operation and emits productCreated event.
I find this case borderline. For example a last-occasion-service may listen to productCreated and immediately send a message (email or push notification) to customers. What do you think about this use case?
2) Which may be a valid broker (I will use docker-compose or kubernetes to orchestrate containerized microservices: language adopted probably java, javascript, python)?
Both is definitely a possibility! Choose a broker that allows you to easily mix-and-match between HTTP (synchronous) communication, and more async event-driven pub/sub. It should allow you to migrate your microservices between the two options as required.
HTTP APIs are great at the edge of your distributed application, where a customer wants to submit an order or something, and block waiting for a response (200 OK).
But internally within your application between microservices, a lot of them don't need a response... async, eventually consistent. And using pub/sub (like MQTT) allows for multiple downstream consumers easily. Another great use for MQTT is streaming updates to downstream consumers... like a data-feed from a bus or airline company or something, rather than having to poll a REST API for updates.
For your use-case and similar ones, I would almost always recommend using pub/sub communication, even if today it's a simple request-reply interaction with a single backend process. REST over HTTP is point-to-point, and perhaps in the future you want another process to be able to see/consume/monitor that event or interaction. If you're already using publish-subscribe, adding that 2nd (or more) consumer of that data flow is trivial. Harder with REST/HTTP.
In terms of performance, I would highly doubt a blocking protocol like HTTP is going to outperform something that is asynchronous and bidirectional, like MQTT which uses WebSockets for web communication.
As for a broker to glue all this together, check out the standard edition Solace PubSub+ event broker... can do both (and translate between) MQTT and HTTP. I even wrote a CodeLab for this (almost) exact use case haha!
(BTW, I work for Solace! FYI.)
Consider using SMF framework for Javascript/Node.js, it helps prototype pub/sub communications via a message broker (RabbitMQ) between microservices out of the box:
https://medium.com/#krawa76/bootstrap-node-js-microservice-stack-4a348db38e51
As for the message broker routes, use an event-driven naming convention, e.g. post a "web.new-product", where "web" is the sub-system name, "new-product" - event name.
I'm designing a system that uses a microservices architecture with event-based communication (using Google Cloud Pub/Sub).
Each of the services is listening and publishing messages so between the services everything is excellent.
On top of that, I want to provide a REST API that users can use without breaking the event-based approach. However, if I have an endpoint that triggers event X, how will I send the response to the user? Does it make sense to create a subscriber for a "ProcessXComplete" event and than return 200 OK?
For example:
I have the following microservices:
Service A
Service B
Frontend Service - REST Endpoints
I'm want to send this request "POST /posts" - this request sent to the frontend service.
The frontend service should trigger "NewPostEvent."
Both Service A and Service B will listen to this event and do something.
So far, so good, but here is where things are starting to get messy for me.
Now I want to return the user that made the request a valid response that the operation completed.
How can I know that all services finished their tasks, and how to create the handler to return this response?
Does it even make sense to go this way or is there a better design to implement both event-based communications between services and providing a REST API
What you're describing is absolutely one of the challenges of event-based programming and how eventual-consistency (and lack of atomicity) coordinates with essentially synchronous UI/UX.
It generally does make sense to have an EventXComplete event. Our microservices publish events on completion of anything that could potentially fail. So, there are lots of ServiceA.EventXSuccess events flowing through the queues. I'm not familiar with Google Cloud PubSub specifically, but in general in Messaging systems there is little extra cost to publishing messages with few (or no) subscribers to require compute power. So, we tend to over-articulate service status by default; it's easy to come back later and tone down messaging as needed. In fact, some of our newer services have Messaging Verbosity configurable via an Admin API.
The Frontend Service (which here is probably considered a Gateway Service or Facade Layer) has taken on the responsibility of being a responsive backing for your UI, so it needs to, in fact, BE responsive. In this example, I'd expect it to persist the User's POST request, return a 200 response and then update its local copy of the request based on events it's subscribed to from ServiceA and ServiceB. It also needs to provide a mechanism (events, email, webhook, gRPC, etc.) to communicate from the Frontend Service back to any UI if failure happens (maybe even if success happens). Which communication you use depends on how important and time-sensitive the notification is. A good example of this is getting an email from Amazon saying billing has failed on an Order you placed. They let you know via email within a few minutes, but they don't make you wait for the ExecuteOrderBilling message to get processed in the UI.
Connecting Microservices to the UI has been one of the most challenging aspects of our particular journey; avoiding tight coupling of models/data structures, UI workflows that are independent of microservice process flows, and perhaps the toughest one for us: authorization. These are the hidden dark-sides of this distributed architecture pattern, but they too can be overcome. Some experimentation with your particular system is likely required.
It really depends on your business case. If the REST svc is dropping message in message queue , then after dropping the message we simply return the reference ID that client can poll to check the progress.
E.g. flight search where your system has to calls 100s of backend services to show you flight deals . Search api will drop the message in the queue and save the same in the database with some reference ID and you return same id to client. Once worker are done with the message they will update the reference in DB with results and meanwhile your client will be polling (or web sockets preferably) to update the UI with results.
The idea is you can't block the request and keep everything async , this will make system scaleable.
We're trying to split our monolithic core into microservices and add some new ones connected with each other using the message system (e.g. Kafka).
The next stage is to create API endpoints for communication between mobile apps and microservices through Api gateway.
What would be a good solution for developing API gateway to transmit data to/from microservices?
use message system as request-reply one (transform requests on
API gateway into message commands, wait for response from message
system with status or necessary data)?
create REST endpoints on necessary microservices (e.g. using REST.li) to send or
get data through gateway; use message system for consistency of data
based on produced events by microservices?
Thanks for advice and some ideas
This depends about the Architecture that you are adopting.
If I understood the question, you already have the broker with the kafka message server.
I think that you can use the architecture publish/subscribe to assyncronous message.
If in the backend architecture the legacy systems to support SLA, in this case you can use the rest endpoints necessary to the integration.
This is the gain of if to utilize API Gateway Pattern in the Architecture.
Thanks a lot.
I would like to say that the second option sounds more reasonable for many cases.
Event-Driven solution mainly fits in the cases in which there are several following processes, so the creating entity could be via Rest endpoint, while processes of that entity could be async via events.
To illustrate, Payment Flow could be like below:
1-) API GW -> Payment Rest Controller -> Payment Service - Create Payment
Payment service creates a payment entity, and then publishes "payment.created" event.
2-)Queue -> Payment Stream Controller -> Service - Update Payment
Payment Stream controller consumes the "payment.created" event, and then checks balances, and updates payment entity as Confirmed. After updating the entity, it could send "payment.confirmed" event.
...
On the other hand, I mean the first option, it would be very hard to maintain a very decoupled system, as you need to know all exchanges or queues.
However, I think combining two solutions could be better for some cases. For example, your API is exposed by a client with very high traffic and the task of the API is quite clear. In that situation, using MQ as a buffer for this API would be perfect.
We have a legacy application that allows our developers to "add" messages via a ThreadLocal in Java.
The current SOAP endpoints will scoop these messages off the thread and then package them up in the response.
The endpoints also catch all exceptions and then marshal those exceptions via this same mechanism to normalize the passing of messages (be they informational, warning, or error).
These messages are rich objects (they have a code, severity, classification, and then the actual message text.)
This is nice in many ways because now we have a standard way to communicate meaningful messages to the user (or calling service) but it also makes using the API more challenging because now the client must pick out the messages from the response AND also pick out the real payload.
Any web service can communicate messages this way...but only a handful do.
I would like to start moving our application towards a REST API but I am struggling on how best to handle the messaging. I am not super keen on adding an envelope to each of our REST responses because this really pollutes the API.
The alternative appears to be adding these messages to custom HTTP headers. Is this the "preferred" approach? Remember I will have a list of one or more of these messages and I will likely have to serialize them as json as well.
Thanks.