I'm working on a messaging/notification system for our products. Basic requirements are:
Fire and forget
Persistent set of messages, possibly updating, to stay there until the sender says to remove them
The libraries will be written in C#. Spring.NET just released a milestone build with lots of nice messaging abstraction, which is great - I plan on using it extensively. My basic question comes down to the question of message brokers. My architecture will look something like app -> message broker queue -> server app that listens, dispatches all messages to where they need to go, and handles the life cycle of those long-lived messages -> message broker queue or topic -> listening apps.
Finally, the question: Which message broker should I use? I am biased towards ActiveMQ - We used it on our last project and loved it. I can't really think of a single strike against it, except that it's Java, and will require java to be installed on a server somewhere, and that might be a hard sell to some of the people that will be using this service. The other option I've been looking at is MSMQ. I am biased against it for some unknown reason, and it also doesn't seem to have great multicast support.
Has anyone used MSMQ for something like this? Any pros or cons, stuff that might sway the vote one way or the other?
One last thing, we are using .NET 2.0.
I'm kinda biased as I work on ActiveMQ but pretty much all of benefits listed for MSMQ above also apply to ActiveMQ really.
Some more benefits of ActiveMQ include
great support for cross language client access and multi protocol support
excellent support for enterprise integration patterns
a ton of advanced features like exclusive queues and message groups
The main downside you mention is that the ActiveMQ broker is written in Java; but you can run it on IKVM as a .net assembly if you really want - or run it as a windows service, or compile it to a DLL/EXE via GCJ. MSMQ may or may not be written in .NET - but it doesn't really matter much how its implemented right?
Irrespective of whether you choose MSMQ or ActiveMQ I'd recommend at least considering using the NMS API which as you say is integrated great into Spring.NET. There is an MSMQ implementation of this API as well as implementations for TibCo, ActiveMQ and STOMP which will support any other JMS provider via StompConnect.
So by choosing NMS as your API you will avoid lockin to any proprietary technology - and you can then easily switch messaging providers at any point in time; rather than locking your code all into a proprietary API
Pros for MSMQ.
It is built into Windows
It supports transactions, it also supports queues with no transactions
It is really easy to setup
AD Integration
It is fast, but you would need to compare ActiveMQ and MSMQ for your traffic to know which is faster.
.NET supports it nativity
Supports fire and forget
You can peek at the queue, if you have readers that just look. not sure if you can edit a message in the queue.
Cons:
4MB message size limit
2GB Queue size limit
Queue items are held on disk
Not a mainstream MS product, docs are a bit iffy, or were it has been a few years since I used it.
Here is a good blog for MSMQ
Take a look at zeromq. It's one of the fastest message queues around.
I suggest you have a look at TIBCO Enterprise Messaging Service - EMS, which is a high performance messaging product that supports multicasting, routing, supports JMS specification and provides enterprise wide features including your requirements suchas fire-forget and message persistence using file/database using shared state.
As a reference, FEDEX runs on TIBCO EMS
as its messaging infrastructure.
http://www.tibco.com/software/messaging/enterprise_messaging_service/default.jsp
There are lot other references if i provide, you'd really be surprised.
There are so many options in that arena...
Free: MantaRay a peer to peer fully JMS compliant system. The interesting part of Mantaray is that you only need to define where the message goes and MantaRay routes it anyways that will get your message to it's detination - so it is more resistant to failures of individual nodes in your messaging fabric.
Paid: At my day job I administer an IBM WebSphere MQ messaging system with several hundred nodes and have found it to be very good. We also recently purchased Tibco EMS and it seems that it will be pretty nice to use as well.
Related
I've spent hours to find the best way to visualize the results of my Storm system. It seems that there is an infinite combination of technologies and I'm getting completely lost.
I want to avoid the use of a database so from what I have understood my system should have the following features:
a queuing message system (such as Redis, Kafka, ActiveMQ,...) that could be connected to my bolts.
a server that establishes a websocket connection with the browser and stream the messages to it.
a javascript library that updates the front end in real-time.
Could you please correct me if I'm saying something wrong regarding the architecture? And I also would appreciate to know which combination of technologies is the best.
As #Lan said, your question is too wide.
For the minimal use I personnally use Redis and Storm together as redis can be used as a basic queue (beware of persistence problems with redis and clustering if you have to), a shared memory space for storm bolts/spouts (storing configuration, intermediate results...) and a basic message broker (pub/sub support), it also has very good performance in latency and throughput.
You can then use a "classic" backend to plug redis topics to websockets using for instance nodejs with sockjs and redis client, but there are far more solutions for this problematic in many languages.
For the front part, it should be defined by your server choice (for instance sockjs-client or socket.io with nodejs), as fallback strategies are embedded when websockets are not supported in browsers.
To conclude, the best architecture is the one that fits your usage, so it depends.
There are so many ways. I recently built a demo using Apache Storm + Kafka. For visualization I used JQuery --> Node.js (for restful web service) --> Redis. This is just one example. There are so many other combinations that you can consider based on your use case.
Recently started looking into these AMQP (RabbitMQ, ActiveMQ) and ZeroMQ technologies, being interested in distributed systems/computation. Been Googling and StackOverflow'ing around, couldn't find a definite comparison between the two.
The farthest I got is that the two aren't really comparable, but I want to know the differences. It seems to me ZeroMQ is more decentralized (no message broker playing middle-man handling messages/guarenteering delivery) and as such is faster, but is not meant to be a fully fledged system but something to be handled more programmatically, something like Actors.
AMQP on the other hand seems to be a more fully fledged system, with a central message broker ensuring reliable delivery, but slower than ZeroMQ because of this. However, the central broker creates a single point of failure.
Perhaps a metaphor would be client/server vs. P2P?
Are my findings true? Also, what would be the advantages, disadvantages, or use cases of using one over the other? A comparison of the uses of *MQ vs. something like Akka Actors would be nice as well.
EDIT Did a bit more looking around.. ZeroMQ seems to be the new contender to AMQP, seems to be much faster, only issue would be adoption/implementations?
Here's a fairly detailed comparison of AMQP and 0MQ: http://www.zeromq.org/docs:welcome-from-amqp
Note that 0MQ is also a protocol (ZMTP) with several implementations, and a community.
AMQP is a protocol. ZeroMQ is a messaging library.
AMQP offers flow control and reliable delivery. It defines standard but extensible meta-data for messages (e.g. reply-to, time-to-live, plus any application defined headers). ZeroMQ simply provides message delimitation (i.e. breaking a byte stream up into atomic units), and assumes the properties of the underlying protocol (e.g. TCP) are sufficient or that the application will build extra functionality for flow control, reliability or whatever on top of ZeroMQ.
Although earlier versions of AMQP were defined along client/server lines and therefore required a broker, that is no longer true of AMQP 1.0 which at its core is a symmetric, peer-to-peer protocol. Rules for intermediaries (such as brokers) are layered on top of that. The link from Alexis comparing brokered and brokerless gives a good description of the benefits such intermediaries can offer. AMQP defines the rules for interoperability between different components - clients, 'smart clients', brokers, bridges, routers etc -
such that a system can be composed by selecting the parts that are useful.
In ZeroMQ there are NO MESSAGE QUEUES at all, thus the name. It merely provides a way to use messaging semantics over otherwise ordinary sockets.
AMQP is standard protocol for message queueing which is meant to be used with a message-broker handling all message sends and receives. It has a lot of features which are available because it funnels all message traffic through a broker. This may sound slow, but it is actually quite fast when used inside a data centre where host to host latencies are tiny.
I'm not really sure how to respond to your question, which is comparing a lot of different things... but see this which may help you begin to dig into these issues: http://www.rabbitmq.com/blog/2010/09/22/broker-vs-brokerless/
AMQP (Advanced Message Queuing Protocol) is a standard binary wire level protocol that enables conforming client applications to communicate with conforming messaging middleware brokers. AMQP allows cross platform services/systems between different enterprises or within the enterprise to easily exchange messages between each other regardless of the message broker vendor and platform. There are many brokers that have implemented the AMQP protocol like RabbitMQ, Apache QPid, Apache Apollo etc.
ZeroMQ is a high-performance asynchronous messaging library aimed at use in scalable distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ØMQ system can run without a dedicated message broker.
Broker-less is a misnomer as compared to message brokers like ActiveMQ, QPid, Kafka for simple wiring.
It is useful and can be applied to hotspots to reduce network hops and hence latency, as we add reliability, store and forward feature and high availability requirements, you probably need a distributed broker service along with a queue for sharing data to support a loose coupling - decoupled in time - this topology and architecture can be implemented using ZeroMQ, you have to consider your use cases and see, if asynchronous messaging is required and if so, where ZeroMQ would fit, it has a good role in solution it appears and a reasonable knowledge of TCP/IP and socket programming would help you appreciate all others like ZeroMQ, AMQP, etc.
We need to send an XML messages between a point of sale system and a java webservice (outside of our network). the messages contain very sensitive data. The messaging has to be secure and transactional and highly available (24/7) with failover. The solution requires the developement of a broker that does the following:
Poll messages from the POS of system (3 types of messages)
do some transformation to the messages
forward part of the message to the java webservice
store part of the message in a database
notify the POS system of the result
Based on these somewhat simplified requirements, do you believe that Biztalk would be overkill? would MSMQ/WCF do the trick here?
Thank you for your help
Amine
IMO if you have the ability to receive and deliver messages asynchronously, then MSMQ (or other Message Oriented Middleware) would be an obvious choice for reliable, transactional transport, irrespective of the rest of the solution. MSMQ's journalling can also be used for audit and debugging purposes (but you will need a strategy for archiving the journal).
For the Polling, Routing, Mapping / Broker and Auditing requirements you then have the choice of BizTalk, other ESB and EAI products, or a DIY solution.
As you've suggested, it is difficult to justify the cost and learning curve of BizTalk on a single message exchange scenario such as this - you could probably knock up a .NET Windows Service (e.g. using WCF, Workflow Foundation, Transaction Scopes, some XSLT for mapping and a data access layer) in a few days.
However, if this isn't a one-off integration scenario and the need for additional integration arises (more applications to integrate, more services, additional listeners, different communications technologies etc), then it would be advisable for your company to take a long term view on EAI and ESB technologies. IMO the main challenge in integration isn't the initial development work, but is instead the ongoing operational management requirements - e.g. security, auditing, failover, monitoring, handling of bad messages and other exceptions - where products such as BizTalk are really worth the outlay.
Do you want to and have the bandwidth to develop, monitor, and maintain your own custom solution? If you don't mind doing that, then going the route of a custom .net-based, MSMQ/WCF solution might work well.
BizTalk will also cover all of the requirements you have listed. There is a learning curve but it is certainly not insurmountable. The initial ramp-up may be lengthier than would a custom-code solution, but there are considerable benefits, particularly the benefit of having all your requirements reliably met:
secure
transactional
reliable (messages aren't lost)
highly available (24/7)
failover
adapter architecture (includes polling adapters)
transformations
working with external web services
returning correlated responses back to the source system (i.e., orchestrating the end-to-end process)
use a broker (you specifically listed this, and BizTalk is a broker; custom MSMQ and WCF means using no broker)
If BizTalk needs to poll the POS system, then you do not need to worry about using MSMQ. BizTalk can handle transferring messages reliably (they're persisted to SQL Server, while MSMQ persists messages to disk).
Note too that the only way to make MSMQ highly available is to cluster it. So either way you'll need to cluster something.
A BizTalk solution will be easier to maintain over time, particularly if you just want to update your transformations. With versioning you can do so in a way that doesn't require downtime. It'll be tough to update a custom solution without downtime.
Some people have had difficulty in the past with monitoring BizTalk for failed messages, but I have found it to be easier, especially with a tool like SCOM or BizTalk 360, than trying to monitor message queues, which often requires even more custom work to monitor. Just make sure to include monitoring in your cost estimates for the life of your solution.
If you do need auditing, then BizTalk also has you covered. MSMQ Journaling will keep a copy of each message for you, but without significant transaction details and with no out-of-the-box way to search through or archive the data.
Building your own .NET client code to work with a Java web service will likely take a good bit of work regardless of which way you go. With BizTalk that means running a wizard against the endpoint or against the WSDL. With WCF it means doing everything by hand or with the assistance of the svcutil tool.
You should go with MSMQ transporting either way.
If you use MSMQ from .NET you should know its limitation: 4 MB on a message size.
BizTalk on the other hand has MSMQ adapter which overcomes this limitation (if a second BizTalk server listen on the other side of the channel).On top of that BizTalk gives you features like: easy configurable message tracking, visual transformation maps. It can be set up in cluster too (Ent. version only).
But the question is can you (or do you want) afford biztalk licenses and hardware for it servers (it's slower then custom .net solution).
I'm trying to decide if MSMQ is the right tool for communication between our application and a third party web service we are currently communicating with directly. We're looking to uncouple this such that if the service goes down, life could still go on as normal.
I can't find anything outside the usual MS fluff about it being the greatest thing and solving all your problems etc etc. It would be really useful if I could find some information that was somewhere between marketing fluff and API - like an architecture diagram of the components and how they integrate with each other, and more importantly how I integrate with them.
It's probably that I'm just looking for the information in the wrong places, so if someone could point me in the right direction, I'd appreciate it.
A typical MSMQ architecture would be composed of 3 parts...
Message Queue - This would be on one of your servers. You would have to install the MSMQ bits and create your queue.
Client - Your client would insert messages into the queue. I'm assuming you're using .NET. If so, most of what you want is going to be located in the System.Messaging namespace.
Windows Service - This would also run on a server, probably the same server as your queue. Its job would be to watch the queue, process messages as they come in, handle making sure the external service is available, and probably do some logging.
Here's an article that should go into a little more detail and give you some code samples.
MSMQ is a implementation of a message queue as are websphere mq and a bunch of other systems. When looking into the concepts and high level architecture I would suggest reading up on message queue's and how they are applied in disconnected scenario's. I can highly recommend Patterns of Enterprise Application Architecture. For specific examples on msmq check out Pro MSMQ: Microsoft Message Queue Programming it doesn't contain allot of special information but it does group it allot better then most resources available on the internet. This Hello World with MSMQ article would give you a nice overview of what it entails and it's easily executed on a development system.
If you are calling a remote web service from your application, it makes sense to use a queue to decouple your application processing from the remote system. By abstracting the communicating through messaging and having a gateway service that is responsible for communication to the web service, you isolate your application from the latency of the web service and build fault tolerance into the design by reducing the request/response usage inside your application (since messaging is by default asynchronous - you deal with it up front).
There are frameworks for .NET that can make this much easier (such as MassTransit or NServiceBus).
You can also check out SOA Patterns (by Arnon Rotem-Gal-Oz, Manning Press, in MEAP) and Enterprise Integration Patterns (Hohpe,Woolf), the latter of which is an essential read for anyone building a message-based system.
To cut a long story short, I am working on a project where we are rewriting a large web application for all the usual reasons. The main aim of the rewrite is to separate this large single application running on single server into many smaller decoupled applications, which can be run on many servers.
Ok here's what I would like:
I would like HTTP to be the main transport mechanism. When one application for example the CMS has been updated it will contact the broker via http and say "I've changed", then the broker will send back a 200 OK to say "thanks I got the message".
The broker will then look on its list of other applications who wanted to hear about CMS changes and pass the message to the url that the application left when it told the broker it wanted to hear about the message.
The other applications will return 200 OK when they receive the message, if not the broker keeps the message and queues it up for the next time someone tries to contact that application.
The problem is I don't even know where to start or what I need to make it happen. I've been looking at XMPP, ActiveMQ, RabbitMQ, Mule ESB etc. and can see I could spend the next year going around in circles with this stuff.
Could anyone offer any advice from personal experience as I would quite like to avoid learning lessons the hard way.
I've worked with JMS messaging in various software systems since around 2003. I've got a web app where the clients are effectively JMS topic subscribers. By the mere act of publishing a message into a topic, the message gets server-pushed dissemenated to all the subscribing web clients.
The web client is Flex-based. Our middle-tier stack consist of:
Java 6
Tomcat 6
BlazeDS
Spring-Framework
ActiveMQ (JMS message broker)
BlazeDS has ability to be configured as a bridge to JMS. It's a Tomcat servlet that responds to Flex client remoting calls but can also do message push to the clients when new messages appear in the JMS topic that it is configured to.
BlazeDS implements the Comet Pattern for doing server-side message push:
Asynchronous HTTP and Comet architectures
An introduction to asynchronous, non-blocking HTTP programming
Farata Systems has announced that they have modified BlazeDS to work with the Jetty continuations approach to implementing the Comet Pattern. This enables scaling to thousands of Comet connections against a single physical server.
Farata Systems Achieves Performance Breakthrough with Adobe BlazeDS
We are waiting for Adobe to implement support of Servlet 3.0 in BlazeDS themselves as basically we're fairly wedded to using Tomcat and Spring in combo.
The key to the technique of doing massively scalable Comet pattern is to utilize Java NIO HTTP listeners in conjunction to a thread pool (such as the Executor class in Java 5 Concurrency library). The Servlet 3.0 is an async event-driven model for servlets that can be tied together with such a HTTP listener. Thousands (numbers like 10,000 to 20,000) concurrent Comet connections can then be sustained against a single physical server.
Though in our case we are using Adobe Flex technology to turn web clients into event-driven messaging subscribers, the same could be done for any generic AJAX web app. In AJAX circles the technique of doing server-side message push is often referred to as Reverse AJAX. You may have caught that Comet is a play on words, as in the counterpart to Ajax (both household cleaners). The nice thing for us, though, is we just wire together our pieces and away we go. Generic AJAX web coders will have a lot more programming work to do. (Even a generic web app could play with BlazeDS, though - it just wouldn't have any use for the AMF marshaling that BlazeDS is capable of.)
Finally, Adobe and SpringSource are cooperating on establishing a smoother, out-of-the-box integration of BlazeDS in conjunction to the Spring-Framework:
Adobe Collaborates with SpringSource for Enhanced Integration Between Flash and SpringSource Platforms
First of all, don't worry about ESBs. The situation you've described lies well within the bounds of straightforward message-oriented middleware. You only "need" an ESB if you're doing things like mediations, content-based routing, protocol transformations; things where the middleware does stuff to the message, on top of routing it to the right place.
If you have a diverse set of destination applications that need to speak to each other - and it sounds like you do - you're right that messaging over a language agnostic protocol (like XMPP, STOMP or HTTP) is a neat solution. It basically means you don't have to write and run loads of Java daemons to translate messages into your favourite flavour of JMS.
STOMP is increasingly supported by message brokers, especially by the open-source ones, and there's a number of different client libraries. It is a lightweight protocol, specifically designed for messaging so you get a much richer feature set out of the box than you would with HTTP.
For me, XMPP is a bit of a weak option as it's not so well supported on the server side, although it is fun to be able to IM your broker :)
If you are set on HTTP, OpenMQ is very good, and I've personally used its Universal Message Service - basically a webapp wrapper around JMS destinations. It provides a REST-ful interface, with a similar set of verbs as STOMP provides.
As someone has already said, what your describing is basically the Publisher/Subscribe Model. This is very easily achieved using either an ESB or a message queue. I have had some experience with RabbitMQ. Its very good. Nothing gets lost and it deals with the publish subscribe model very well. I have in the past gone down the route on small scale systems of developing my own Message broker with a bespoke protocol over http. I wouldn't advise this, reason being is that as you start to develop it you keep thinking of ways of how to extend it.
RabbitMQ is developed in Erlang but it has java,net,python etc clients that can hook into it very easily. I have used the .net and python clients, it works well. I chose it for Erlangs reputation for creating solid systems that can cope with multiple things going on at the same time, very well. I would call them threads but I think that its smarter than just threads, I think I remember mutterings of the Actor Model and mailboxes, which I recall were pretty neat.
I was in a similar position as yourself but with very bad experiences of other messaging systems (Biztalk et al.) that were too propriety that tied you into a solution. If you can keep the messages separate from the transport and delivery mechanisms, then you can develop your system to your hearts content. I used JSON in the end as the packet sizes are small. You could use anything you like, some opt for SOAP messages, but I feel that these are way too heavy for most stuff, although it does allow you to nicely give XSD schemas to outsiders so that they can/could develop systems that interop with your system in the future.
http://www.rabbitmq.com/tutorials/tutorial-three-java.html, this is a link to the tutorial on the Publish/Subscribe model and how you would achieve it using a message queue system. Its for rabbitMQ, but to be honest it will work with ESB and any other Messaging queue system out there.
ESB (Enterprise Serial Bus) - Consider this when your application have much interaction with two or more external/separate applications where each of these won't communicate in a similar data format. Ex: Some systems may accept objects, XML, JSON, SMTP, TCP/IP, HTTP, HTTPS etc.
ESB has many features like:
Routing,Addressing,Messaging styles,Transport protocols,Service messaging model.
Consider queue system if the producer - consumer applications follows the same type of data format.
Web services (SOAP / REST) is best if one application need the other application to complete the work flow.
Use Queues if the application need asynchronous data transfer.
You're really talking about publish and subscribe with assured delivery. Most MOM software should easily support your use case.
As it was already said earlier, having an ESB for you current case seems to me like to smash a fly with a hammer.
The ESB software itself will be time consuming and will require maintenance. If you go to open source solution, it might be more time consuming than using a licensed solution (IBM, ORACLE, ...).
Of course an ESB would do the job, and it would be really easy to develop a solution, but setting up an ESB would be way more difficult than doing the solution itself.
If your problem is limited to the case described, I would highly suggest you to build a simple architecture over OpenMQ (or similar), and using it through JMS