There is requirement in project and the windows service needs to be used as subscriber of RabbitMQ (message broker).
Once the event has created, this listener windows service get the event and process the event, while processing, there are some important business logics needs to be incorporated and the data needs to be stored into SQL server DB.
From my perspective, windows service can be just a trigger of any business logic. Like once it subscribed to an event, if any event comes, read the event details and perform the business logic using any of the REST (HTTP based) service.
Please provide your suggestions, it would be more helpful. Thanks in advance.
You need to create a Windows application using c# or you can use NUGET rabbitmq client to consume message and save in dB.
https://www.nuget.org/packages/RabbitMQ.Client
Related
I am looking for a way to expose an existing event processing system to the external world using a REST interface. I have existing system design where we have RabbitMQ message queues where a publisher could post a message and then wait for the message processed results on a separate queue. Message ID is used to track the output to the original message on the output queue.
Now I want this to be exposed to the external consumers but we don't want to expose our RabbitMQ endpoint for this, so I was wondering if anyone has managed to achieve something similar to this using ExpressJS. Above diagram shows the current thought process
Main challenge I am facing here is that; some of this message processing could take more than couple of minutes, so was not sure how best to develop a API like this. Choices like should I create a polling interface for client here or is there a technology these days that help eliminate the polling on the client API to verify if the message is processed and get the result.
Can someone please help me with a good approach to manage these sort of requirement.
I finally ended up going the webhook way. Now when the REST API service receives a request, the client need to also provide a webhook and this will be registered with the client request and server will call it back when the results are available.
I'm designing a system that uses a microservices architecture with event-based communication (using Google Cloud Pub/Sub).
Each of the services is listening and publishing messages so between the services everything is excellent.
On top of that, I want to provide a REST API that users can use without breaking the event-based approach. However, if I have an endpoint that triggers event X, how will I send the response to the user? Does it make sense to create a subscriber for a "ProcessXComplete" event and than return 200 OK?
For example:
I have the following microservices:
Service A
Service B
Frontend Service - REST Endpoints
I'm want to send this request "POST /posts" - this request sent to the frontend service.
The frontend service should trigger "NewPostEvent."
Both Service A and Service B will listen to this event and do something.
So far, so good, but here is where things are starting to get messy for me.
Now I want to return the user that made the request a valid response that the operation completed.
How can I know that all services finished their tasks, and how to create the handler to return this response?
Does it even make sense to go this way or is there a better design to implement both event-based communications between services and providing a REST API
What you're describing is absolutely one of the challenges of event-based programming and how eventual-consistency (and lack of atomicity) coordinates with essentially synchronous UI/UX.
It generally does make sense to have an EventXComplete event. Our microservices publish events on completion of anything that could potentially fail. So, there are lots of ServiceA.EventXSuccess events flowing through the queues. I'm not familiar with Google Cloud PubSub specifically, but in general in Messaging systems there is little extra cost to publishing messages with few (or no) subscribers to require compute power. So, we tend to over-articulate service status by default; it's easy to come back later and tone down messaging as needed. In fact, some of our newer services have Messaging Verbosity configurable via an Admin API.
The Frontend Service (which here is probably considered a Gateway Service or Facade Layer) has taken on the responsibility of being a responsive backing for your UI, so it needs to, in fact, BE responsive. In this example, I'd expect it to persist the User's POST request, return a 200 response and then update its local copy of the request based on events it's subscribed to from ServiceA and ServiceB. It also needs to provide a mechanism (events, email, webhook, gRPC, etc.) to communicate from the Frontend Service back to any UI if failure happens (maybe even if success happens). Which communication you use depends on how important and time-sensitive the notification is. A good example of this is getting an email from Amazon saying billing has failed on an Order you placed. They let you know via email within a few minutes, but they don't make you wait for the ExecuteOrderBilling message to get processed in the UI.
Connecting Microservices to the UI has been one of the most challenging aspects of our particular journey; avoiding tight coupling of models/data structures, UI workflows that are independent of microservice process flows, and perhaps the toughest one for us: authorization. These are the hidden dark-sides of this distributed architecture pattern, but they too can be overcome. Some experimentation with your particular system is likely required.
It really depends on your business case. If the REST svc is dropping message in message queue , then after dropping the message we simply return the reference ID that client can poll to check the progress.
E.g. flight search where your system has to calls 100s of backend services to show you flight deals . Search api will drop the message in the queue and save the same in the database with some reference ID and you return same id to client. Once worker are done with the message they will update the reference in DB with results and meanwhile your client will be polling (or web sockets preferably) to update the UI with results.
The idea is you can't block the request and keep everything async , this will make system scaleable.
I'm using wso2 cep 4.1
I created receiver to catch some json data from my source. Then I process this data internally and should give the response with additional data. My response should be through the same point as data come in to CEP. It is classical rest API. Is it possible and how can I make it?
Or, I need websocket (websocket-local) for similar purposes?
Hope you are still trying to understand functionalities of WSO2 CEP. Let me explain the basic overview of CEP before addressing your question. If you look at below diagram you will understand what is happening under the hood at a high level. . I will explain what these components suppose to do in the context of event processing.
Event receivers :-
Event receivers receive events that are coming to the CEP. WSO2 CEP supports the most common adapter implementations by default. For specific use cases, you can also plug custom adapters. For more information, see Configuring Event Receivers.
Event streams :-
Event streams contain unique sets of attributes of specific types that provide a structure based on which the events processed by the relevant event flow are selected. Event streams are stored as stream definitions in the file system via the data bridge stream definition store.
Event processors :-
Event processor handles actual event processing. It is the core event processing unit of the CEP. It manages different execution plans and processes events based on logic, with the help of different Siddhi queries. Event Processor gets a set of event streams from the Event Stream Manager, processes them using Siddhi engine, and triggers new events on different event streams back to the Event Stream Manager. For more information, see Creating a Standalone Execution Plan.
Event publishers
Event publishers:- publish events to external systems and store data to databases for future analysis. Like the event receivers, this component also has different adapter implementations. The most common ones are available by default in the CEP. You can implement custom adapters for specific use cases. For more information, see Configuring CEP to Create Alerts.
According to your requirement, you should have HTTP receiver as well as HTTP publisher where the receiver receives a request from a third party API and hand message over to event processors so as to perform some pre-defined tasks.This may compose with several event streams and execution plans. Once processing is done event publishers can be used to publish result to required third-party API as you pointed out.
OOB CEP provides HTTP receiver and HTTP publisher adapters[1-2] which you can try it out.There are some limitations which might not suit for your scenario. You are required to implement your own custom HTTP receiver and publisher[3-4] which does what you intended to do.
Since you need to publish a response to difference endpoints,you can achieve this defining REST API endpoint,user credentials(if required) and HTTP verbs and other information which required to send a message in the event stream[5] as meta information. Then that information you can read from the stream itself and push to desired third-party API as you require.
I need websocket (websocket-local) for similar purposes?
This isn't clear what exactly is to be done. Please raise an another question and ask it again.
https://docs.wso2.com/display/CEP410/HTTP+Event+Receiver
https://docs.wso2.com/display/CEP410/HTTP+Event+Publisher
https://docs.wso2.com/display/CEP410/Building+Custom+Event+Receivers
https://docs.wso2.com/display/CEP410/Building+Custom+Event+Publishers
https://docs.wso2.com/display/CEP410/Understanding+Event+Streams
The feature you are looking for doesn't come OOTB with CEP. However, you can try something similar to below;
Implement a REST API. Probably using Apache CXF since CXF dependencies are present in WSO2 servers by default. You can follow this guide if you are using a swagger based approach to develop the REST API.
Within that custom REST implementation, you need to read the HTTP request, send it to CEP (step 3), wait for an output from CEP (step 4) and then send back that details as HTTP response inside the method which represents your operation.
To send an event to CEP you can use WSO2 Event receiver. Create a receiver at CEP side and then send events to the receiver using DataPublisher client. Make sure you have the same stream definition that you set in CEP receiver in the DataPublisher.publish() method and object array you send adhere to that definition. Also, you might need to set truststore and keystore params here.
After publishing your events successfully you need to block the request thread till you receive a response from CEP. You can use a java object like CountDownLatch for this purpose.
To receive a response you need to consume events though EventStreamService For this you need to implement a WSO2EventConsumer and subscribe to EventStreamService. After successfully subscribing, events coming to stream id mentioned in your event consumer will be forwarded to receive method of your Consumer. From there you can extract the results, unblock the initial request thread and return with those results. To access the EventStreamService from within your web app you can use below code snippet.
EventStreamService eventStreamService = (EventStreamService) PrivilegedCarbonContext.getThreadLocalCarbonContext().getOSGiService(EventStreamService.class, null);
Hope this helped.
I would like to send all data received from fuse, in a specific Topic, to a Business Process in BPM Studio. Is there any way?
Example:
I send a value to 'testTopic' in Fuse. Then Fuse send this value to a Business Process (or the Business Process retrieve it), then the Business Process do things based on the value recevied, like sending another value to another topic
Is somithing of this kind possible?
Yes it most definitely is possible, although you would need to route from the 'testTopic' to one of the JMS Queues that jBPM can listen on and transform the message to reflect a valid jBPM command. The generic principle is described in the documentation at http://docs.jboss.org/jbpm/v6.0/userguide/jBPMRemoteAPI.html#d0e12149. The real power becomes clear when you look at all the jBPM commands you can send in the packages
org.drools.core.command.runtime.process (Maven: org.drools:drools-core)
and
org.jbpm.services.task.commands (Maven: org.jbpm:jbpm-human-task-core).
When talking from the outside world, it would typically be necessary to identify a correlationKey in the process which is basically the "Business Key" that can be used to identify a process uniquely e.g. as 'ApplicationNumber' for an application process. This can be used to then identify which process you may want to signal/abort/etc.
Since you are working in Fuse you should probably also consider routing that message to the jBPM Rest API described at http://docs.jboss.org/jbpm/v6.0/userguide/jBPMRemoteAPI.html#d0e10088. This may simplify your code a bit because it is a more synchronous API. The drawback however is the REST over HTTP invocation typically does not respect the local transaction.
I am builing a win application that has user access control against a sql db, all the data is stored in this db as well.
This project is to be installed in one site on 30-40 machines (I mean to say that it's not web, it's all in one place, maximum call it intranet).
I want that while the program is logged on, the logged-in user should be able to chat to the other logged in users.
Any recommended approaches in C# & VB?
I would appreciate any idea, link or tip.
Please share me with your experience
Thanks!
NOTE: The program is in Wpf if it does matter.
Architecturally, it seems like a publisher-subscriber message bus would be a good pattern for you. You would have a centralized server that each client would register with that will distribute notifications from publishers to subscribers.
Each client will register for notification of the client list upon starting. Each client can register interest in being notified when another client publishes a message. Each client would publish messages to the bus to be delivered to any subscribers for that client.
There is a good example of a pub-sub message bus written in WCF in MSDN: WCF ESSENTIALS What You Need To Know About One-Way Calls, Callbacks, And Events. You could get this up and running fairly quickly.