We're looking for some java api to use as middleware/bus to make java processes running on different hosts communicate.
The scenario is the following:
- we have a web application we want to use to monitor the different java processes running on different hosts
- we want all the different processes to be able to communicate with the web application, pushing their own status to the central web app
- we want the web app to be able to poll the different processes in order to receive updates about their status
We were thinking to use something like jgroups, while we wouldn't like to use jms.
Do anyone suggest any alternative? Even better if advantages/disavantages are highlighted..
thanks
It sounds like you need a messaging system such as RabbitMQ http://www.rabbitmq.com/ or ActiveMQ http://activemq.apache.org/ . Rabbit has been gaining a lot of popularity recently and is straightforward to use (see http://www.rabbitmq.com/getstarted.html for example).
Related
I need to integrate 3-5 existing and ready services that are developed by different teams. That is something like integrating several independent monolithic applications.
The very wanted feature is having a central communication component where all requests can be logged (or partially logged in case they have a big payload) so that it can be quickly seen what service sent a request with what payload and when if something goes wrong.
The second task is security. It is needed to protect inter-service communication.
I have been researching this topic for several days. That is what I've come up with so far:
Use Enterprise Service Bus (ESB)
Use MessageBroker (ActiveMQ, RabbitMQ, Kafka, etc)
Simply use REST API communication
I've read about ESB and I am not sure whether this solution is OK to use.
The picture from Wikipedia shows the following:
The problem with ESB that it not only implements communication between separate independent services. It also changes the communication itself from synchronous request-response model to an asynchronous messaging style. Currently, we don't need asynchronous messaging (maybe we will in the future, but not right now).
ESB allows to make request-response communication, but in a very inconvenient and complex way with generating correlation IDs, creating a temporary response-topic and consumer. With this, I have doubts whether it has more advantages to use ESB with its super complex Request-Response messaging style or simply use plain old REST API calls (RPC). However, with REST API it is not possible to have a centralized component that can log all communication between services. A similar problem exists with Message Broker too (because it also involves asynchronous messaging).
Is there any ready solution/pattern to integrate several services (not microservices) with centralized logging, security configuration and having simple to implement synchronous request-response model (with a possibility to add messaging, if needed, later)?
Make your services talk to each other via a proxy that will take care of these concerns (Back in 2007 I called this "edge-component" but the today it known as sidecar pattern)
A common way to do that would be to containerize your services, deploy to Kubernetes and use a service mesh (like Console's or istio, etc)
I am creating a gwt widget library for our internal use. I am asked to fire an event whenever any event occurs on any of the controls in my gwt widget which would then be listened to by the application or project that is using my widget. The concept seemed similar to the event bus that i previously used for my other gwt projects to communicate between different views displayed in different regions on the same page. That worked fine as i was using the same instance of event bus through out the application. But now I would be using a particular instance of the event bus in my project but the application consuming it will not be using the same instance. Do I go ahead and use it or is there an alternate way to fire events between different applications. Please help
First, what do you call "different applications" ? Will they be running in the same browser ? Will they be deployed on the same server ?
You can't use EventBus to communicate between your applications if they don't share a common EventBus.
If your applications are running in the same tab, you can store the EventBus in a static variable, and retrieve it from the other applications in the same tab.
If your applications are running in the same browser (on different tabs), you can use LocalStorage to communicate. You can have a look at :
Inter-tab communication using local storage
Using HTML5 Web Storage for Interprocess Communication
Sharing data across windows using localStorage.
GWT provides an implementation of LocalStorage (and SessionStorage) in com.google.gwt.storage.client.Storage.
If they aren't running in the same browser, you will have to do a round-trip to the server to communicate from one application to an other (using RPC to send the message, and RPC polling to check for new messages - or using server push techniques, see GWTEventService and ServerPushFAQ).
If they aren't deployed on the same server, you will have to do a round-trip to the server and to communicate between your servers (using sockets, RPC, webservices,...).
RabbitMQ to the rescue... it's the canonical solution for this type of problem.
http://www.rabbitmq.com/
It was created as a bus for software, much like a hardware bus is to a computer system.
I especially like the http://manning.com/videla/ book. It goes into detail about the approach, outlines source code and the architecture behind it. Very practical.
Apologies if something similar has been asked in the future but my search didn't return, what I would consider, directly related.
I am trying to implement a service with its backend in AWS EC2/S3 and front-end in iPhone and the service is more or less like a todo-list. This is not a novel idea but will help me in a class I teach about IT infrastructure.
Unfortunately I have access to only my own iPhone and I cannot demonstrate scalability over AWS, etc.
Is there a way/software tool/framework to simulate virtual users for this app that can send requests to the AWS servers pretending to be from different accounts/apps?
The simulator should send requests just like my actual iphone app would send if I were to add an item to the list or delete or edit.
I understand stress testing is a well established topic but here I want to just simulate multiple users and demonstrate scalability instead of trying to push the Web service to its limits. Neither am I sure if this completely overlaps with traffic simulation.
Any help will be deeply appreciated.
You might be able to do it using Apache JMeter. That depends on what you have going on on the backend. But it supports the following server types:
Web - HTTP, HTTPS
SOAP
Database via JDBC
LDAP
JMS
Mail - SMTP(S), POP3(S) and IMAP(S)
Native commands or shell scripts
You should be able to wire something together with that.
http://jmeter.apache.org/
http://www.opensourcetesting.org/performance.php
I've used it at various points to simulate VERY heavy loads for my services running in AWS/EC2.
Apache Benchmark is a very convenient tool for doing HTTP load testing -- you can have it make concurrent requests to simulate multiple users. It's main advantage over other tools is that it's simple and easy to get started with. If your backend listens on HTTP, it might be worth trying ab before investing any time in something more complex.
I'm trying to decide if MSMQ is the right tool for communication between our application and a third party web service we are currently communicating with directly. We're looking to uncouple this such that if the service goes down, life could still go on as normal.
I can't find anything outside the usual MS fluff about it being the greatest thing and solving all your problems etc etc. It would be really useful if I could find some information that was somewhere between marketing fluff and API - like an architecture diagram of the components and how they integrate with each other, and more importantly how I integrate with them.
It's probably that I'm just looking for the information in the wrong places, so if someone could point me in the right direction, I'd appreciate it.
A typical MSMQ architecture would be composed of 3 parts...
Message Queue - This would be on one of your servers. You would have to install the MSMQ bits and create your queue.
Client - Your client would insert messages into the queue. I'm assuming you're using .NET. If so, most of what you want is going to be located in the System.Messaging namespace.
Windows Service - This would also run on a server, probably the same server as your queue. Its job would be to watch the queue, process messages as they come in, handle making sure the external service is available, and probably do some logging.
Here's an article that should go into a little more detail and give you some code samples.
MSMQ is a implementation of a message queue as are websphere mq and a bunch of other systems. When looking into the concepts and high level architecture I would suggest reading up on message queue's and how they are applied in disconnected scenario's. I can highly recommend Patterns of Enterprise Application Architecture. For specific examples on msmq check out Pro MSMQ: Microsoft Message Queue Programming it doesn't contain allot of special information but it does group it allot better then most resources available on the internet. This Hello World with MSMQ article would give you a nice overview of what it entails and it's easily executed on a development system.
If you are calling a remote web service from your application, it makes sense to use a queue to decouple your application processing from the remote system. By abstracting the communicating through messaging and having a gateway service that is responsible for communication to the web service, you isolate your application from the latency of the web service and build fault tolerance into the design by reducing the request/response usage inside your application (since messaging is by default asynchronous - you deal with it up front).
There are frameworks for .NET that can make this much easier (such as MassTransit or NServiceBus).
You can also check out SOA Patterns (by Arnon Rotem-Gal-Oz, Manning Press, in MEAP) and Enterprise Integration Patterns (Hohpe,Woolf), the latter of which is an essential read for anyone building a message-based system.
To cut a long story short, I am working on a project where we are rewriting a large web application for all the usual reasons. The main aim of the rewrite is to separate this large single application running on single server into many smaller decoupled applications, which can be run on many servers.
Ok here's what I would like:
I would like HTTP to be the main transport mechanism. When one application for example the CMS has been updated it will contact the broker via http and say "I've changed", then the broker will send back a 200 OK to say "thanks I got the message".
The broker will then look on its list of other applications who wanted to hear about CMS changes and pass the message to the url that the application left when it told the broker it wanted to hear about the message.
The other applications will return 200 OK when they receive the message, if not the broker keeps the message and queues it up for the next time someone tries to contact that application.
The problem is I don't even know where to start or what I need to make it happen. I've been looking at XMPP, ActiveMQ, RabbitMQ, Mule ESB etc. and can see I could spend the next year going around in circles with this stuff.
Could anyone offer any advice from personal experience as I would quite like to avoid learning lessons the hard way.
I've worked with JMS messaging in various software systems since around 2003. I've got a web app where the clients are effectively JMS topic subscribers. By the mere act of publishing a message into a topic, the message gets server-pushed dissemenated to all the subscribing web clients.
The web client is Flex-based. Our middle-tier stack consist of:
Java 6
Tomcat 6
BlazeDS
Spring-Framework
ActiveMQ (JMS message broker)
BlazeDS has ability to be configured as a bridge to JMS. It's a Tomcat servlet that responds to Flex client remoting calls but can also do message push to the clients when new messages appear in the JMS topic that it is configured to.
BlazeDS implements the Comet Pattern for doing server-side message push:
Asynchronous HTTP and Comet architectures
An introduction to asynchronous, non-blocking HTTP programming
Farata Systems has announced that they have modified BlazeDS to work with the Jetty continuations approach to implementing the Comet Pattern. This enables scaling to thousands of Comet connections against a single physical server.
Farata Systems Achieves Performance Breakthrough with Adobe BlazeDS
We are waiting for Adobe to implement support of Servlet 3.0 in BlazeDS themselves as basically we're fairly wedded to using Tomcat and Spring in combo.
The key to the technique of doing massively scalable Comet pattern is to utilize Java NIO HTTP listeners in conjunction to a thread pool (such as the Executor class in Java 5 Concurrency library). The Servlet 3.0 is an async event-driven model for servlets that can be tied together with such a HTTP listener. Thousands (numbers like 10,000 to 20,000) concurrent Comet connections can then be sustained against a single physical server.
Though in our case we are using Adobe Flex technology to turn web clients into event-driven messaging subscribers, the same could be done for any generic AJAX web app. In AJAX circles the technique of doing server-side message push is often referred to as Reverse AJAX. You may have caught that Comet is a play on words, as in the counterpart to Ajax (both household cleaners). The nice thing for us, though, is we just wire together our pieces and away we go. Generic AJAX web coders will have a lot more programming work to do. (Even a generic web app could play with BlazeDS, though - it just wouldn't have any use for the AMF marshaling that BlazeDS is capable of.)
Finally, Adobe and SpringSource are cooperating on establishing a smoother, out-of-the-box integration of BlazeDS in conjunction to the Spring-Framework:
Adobe Collaborates with SpringSource for Enhanced Integration Between Flash and SpringSource Platforms
First of all, don't worry about ESBs. The situation you've described lies well within the bounds of straightforward message-oriented middleware. You only "need" an ESB if you're doing things like mediations, content-based routing, protocol transformations; things where the middleware does stuff to the message, on top of routing it to the right place.
If you have a diverse set of destination applications that need to speak to each other - and it sounds like you do - you're right that messaging over a language agnostic protocol (like XMPP, STOMP or HTTP) is a neat solution. It basically means you don't have to write and run loads of Java daemons to translate messages into your favourite flavour of JMS.
STOMP is increasingly supported by message brokers, especially by the open-source ones, and there's a number of different client libraries. It is a lightweight protocol, specifically designed for messaging so you get a much richer feature set out of the box than you would with HTTP.
For me, XMPP is a bit of a weak option as it's not so well supported on the server side, although it is fun to be able to IM your broker :)
If you are set on HTTP, OpenMQ is very good, and I've personally used its Universal Message Service - basically a webapp wrapper around JMS destinations. It provides a REST-ful interface, with a similar set of verbs as STOMP provides.
As someone has already said, what your describing is basically the Publisher/Subscribe Model. This is very easily achieved using either an ESB or a message queue. I have had some experience with RabbitMQ. Its very good. Nothing gets lost and it deals with the publish subscribe model very well. I have in the past gone down the route on small scale systems of developing my own Message broker with a bespoke protocol over http. I wouldn't advise this, reason being is that as you start to develop it you keep thinking of ways of how to extend it.
RabbitMQ is developed in Erlang but it has java,net,python etc clients that can hook into it very easily. I have used the .net and python clients, it works well. I chose it for Erlangs reputation for creating solid systems that can cope with multiple things going on at the same time, very well. I would call them threads but I think that its smarter than just threads, I think I remember mutterings of the Actor Model and mailboxes, which I recall were pretty neat.
I was in a similar position as yourself but with very bad experiences of other messaging systems (Biztalk et al.) that were too propriety that tied you into a solution. If you can keep the messages separate from the transport and delivery mechanisms, then you can develop your system to your hearts content. I used JSON in the end as the packet sizes are small. You could use anything you like, some opt for SOAP messages, but I feel that these are way too heavy for most stuff, although it does allow you to nicely give XSD schemas to outsiders so that they can/could develop systems that interop with your system in the future.
http://www.rabbitmq.com/tutorials/tutorial-three-java.html, this is a link to the tutorial on the Publish/Subscribe model and how you would achieve it using a message queue system. Its for rabbitMQ, but to be honest it will work with ESB and any other Messaging queue system out there.
ESB (Enterprise Serial Bus) - Consider this when your application have much interaction with two or more external/separate applications where each of these won't communicate in a similar data format. Ex: Some systems may accept objects, XML, JSON, SMTP, TCP/IP, HTTP, HTTPS etc.
ESB has many features like:
Routing,Addressing,Messaging styles,Transport protocols,Service messaging model.
Consider queue system if the producer - consumer applications follows the same type of data format.
Web services (SOAP / REST) is best if one application need the other application to complete the work flow.
Use Queues if the application need asynchronous data transfer.
You're really talking about publish and subscribe with assured delivery. Most MOM software should easily support your use case.
As it was already said earlier, having an ESB for you current case seems to me like to smash a fly with a hammer.
The ESB software itself will be time consuming and will require maintenance. If you go to open source solution, it might be more time consuming than using a licensed solution (IBM, ORACLE, ...).
Of course an ESB would do the job, and it would be really easy to develop a solution, but setting up an ESB would be way more difficult than doing the solution itself.
If your problem is limited to the case described, I would highly suggest you to build a simple architecture over OpenMQ (or similar), and using it through JMS