We are evaluating few options for developing a telecommunication related application platform (and migrating/consolidating some of the standalone apps into the new platform). One of our main concerns is the ability to handle high volume of requests during peak hours.
We feel the TorqueBox seems to an interesting solution worthy of consideration because:
Speed (Next to pure Java performance)
Faster development time over Java
Maintainability
Support for threads/concurrency even though it's Ruby
Faster/Easier front end development with Rails
...
RedHat supported and runs on JBoss (scalability, future development and ability to call Java if necessary)
Has anyone developed/deployed similar application(s) with JRuby/TorqueBox?
Any serious performance bottlenecks ahead? (or why we shouldn't use JRuby and should stick with Java?)
The answer is YES but be aware of memory leaks (gems, threadsafety issues, etc). You have to be familiar with tools like VisualVM, Eclipse MAT and/or NewRelic.
We're successfully using Torquebox on production for some clients on amazon EC2 handling 60k-80k visits per day (new c3 instances are great for Java).
Deployment is also an issue. We're unable to setup any kind of rolling restart because of memory consumption. So every time we deploy using Capistrano a full JBoss restart is needed (no big issue for us).
Bests,
Antonio
yes any mature Java web-server with JRuby is a valuable option. the details of handling high-loads on peak hours will really depend on what kind of app you'll be running, how much "hardware" can you afford to use but in general it's achievable but be aware there might still be some "gotchas" e.g. Ruby libraries (gems) that do not handle thread-safety well. you simply need to understand how to proceed than - which seems you do since you're want to use 'Celluloid.IO' :)
Related
I have a small Akka application that passes many messages between its actors and each actor does some calculations on the data it receives. What I want is to profile this application in order to see which parts of the code take up most time and so on.
I tried VisualVM but I cannot really understand what's going on. I added a picture of the profiler output.
My questions are
What for example is this first line and why does it take up so much time? (scala.concurrent.forkjoin.ForkJoinPool.scan())
Can Akka applications because of their asynchronous behaviour be profiled well at all?
Can I see for instance how long one specific actor(-type) works for one specific message(-type) it receives?
Are there other best-practices for profiling Akka applications?
There are packages not profiled by default and it is their time that is accounted in the profile of scala.concurrent.forkjoin.ForkJoinPool.scan(). If all the hidden packages are allowed to be sampled, the true CPU time consumers will be revealed. For example, the following before/after illustrative profiles uncover that threads are put to sleep most of the time by sun.misc.Unsafe.park waiting to be unparked.
Akka applications can be profiled quite well with proper instrumentation and call tracing. Google's prominent Dapper, a Large-Scale Distributed Systems Tracing Infrastructure paper contains detailed explanation of the technique. Twitter created Zipkin based on that. It is open sourced and has an extension for distributed tracing of Akka. Follow its wiki for a good explanation of how to set up a system that allows to
trace call hierarchies inside an actor system;
debug request processing pipelines (you can log to traces, annotate them with custom key-value pairs);
see dependencies between derived requests and their contribution to resulting response time;
find and analyse slowest requests in your system.
There is also a new kid on the block, Kamon. It is a reactive-friendly toolkit for monitoring applications that run on top of the JVM, which is specially enthusiastic to applications built with the Typesafe Reactive Platform. That definitely means yes for Akka and the integration comes in the form of the kamon-akka and kamon-akka-remote modules that bring bytecode instrumentation to gather metrics and perform automatic trace context propagation on your behalf. Explore the documentation starting from Akka Integration Overview to understand what it can and how to achieve that.
Just a couple of days ago TypeSafe announced that TypeSafe console now is free. I don't know what can be better for profiling Scala/Akka applications. Of cause you can try JProfiler for JVM languages, I've used it with Java projects, but it's not free and for Java.
I was thinking about profiling/metrics in code since I also use Akka/Scala a lot for building production applications, but I also eager to hear alternative ways to make sure that application is healthy.
Metrics (like Dropwizard)
Very good tool for collecting metrics in the code, with good documentation and embedded support for Graphite, Ganglia, Logback, etc.
It has verbose tools for collecting in-app statistics like gauges, counter histograms, timings - information to figure out what is the current state of your app, how many actors were created, etc, if they are alive, what the current state is of majority of actors, etc.
Agree, it's a bit different from profiling but helps a lot to find roots of the problem, especially if integrated with some char building tool.
Profilers like (VisualVM, XRebel)
Since I'm a big fun of doing monitoring, it still answers a slightly different question - what are current insights of my application right now?
But there is quite another matter may disturb us - what of my code is slow (or sloppy)?
For that reason, we have VisualVM and another answers to this question - how to profile Akka actors with VisualVM.
Also, I'd suggest trying XRebel profiler that just adds a bit more firepower to process of figuring out what code makes app slower. It's also paid but on my project it saved a lot of time dealing with sloppy code.
New Relic
I'd suggest it for some playground projects since you can get some monitoring/profiling solutions for free, but on more serious projects I'd go for things I highlighted above.
So I hope, that my overview was helpful.
Note: This question is heavily affected by the main requirement to the web application that I build: high availability and fault tolerance. All the other requirements (like scalability and number of users) is not in question here.
I have got and advice from one of the members of this community to use an Erlang web-server as a back-end for my web application.
The suggestion was that I could use something like Mochiweb as a backend and Django/Ruby on Rails as a front end using JSON and the Service Oriented Model.
The only obvious advantage of this approach that I can understand is that the development of the front-end part is 'as usual' - regular MVC stuff, Ruby on Rails or any other common framework of someone's choice.
But what about other advantages? Do they actually exist?
Sure, Erlang/OTP adds fault-tolerance to the system in question, but doesn't adding a web front-end layer diminish this fault tolerance level to much lower level?
Don't we introduce a 'single point of failure' by coupling Ruby on Rails with Mochiweb? Of course, Mochiweb can cope with faults, but what if something wrong happens on the front-end side?
Technically the Erlang/OTP platform does not do anything about fault-tolerance and high-availability on it's own. It just allows to easily implement concurrent and distributed software. Erlang web server running on a single machine can fail as all other do - just because hardware fails.
So for HA sites it's more important to have proper redundancy and fallback scenarios for cases of hardware and software failures, rather than using any specific software stack or platform. Probably it will just be slightly easier to implement in Erlang compared to other platforms (if you are familiar with it, of course), but absolutely the same results can be achieved using pure Ruby/Python/Java/C or almost any other.
The web industry have tons of experience setting up fault-tolerant frontends. It's just a matter of setting up multiple web machines (often light reverse-proxies) and some sort of HA manager (built into many loadbalancing solutions). The backend is usually the harder part.
I wouldn't use Erlang as a front-end web server if the back end is some other technology.
Many of the benefits of Erlang as a web server come about when the back end is also using Erlang. The biggest of these is lower I/O costs. When your front end and back end are completely separate software stacks, you lose that benefit.
If you're going to build something on Rails, you might as well use something you can get more help with on the front end, such as nginx.
The only place where I found informations on G-WAN web server was the project web site and it looked very much like advertisement.
What I would really know is, for someone who is proficient with C, if it is as easy to use and extend that other architectures. For now I would mostly focus on scripting abilities.
Are C scripts on GWAN easy to write ?
Can you easily update and upload new C scripts to the server (say as easily than some PHP or Java pages on other architectures) ? Do you have to restart the server when doing so ?
Can you easily extend it with third party or existing C libraries ?
Any other feedback welcome.
Well, now G-WAN is available under Linux, I am using it for more than 6 months.
The C scripts are fully-ANSI C compatible so there is no difference for any seasonned C programmer.
To update them on the server, you can edit them directly in the /csp folder (remotely via SSH) or locally on a test machine (and copy them later): G-WAN reloads scripts on-the-fly when they have been changed on disk (no server stop required).
G-WAN C scripts can use any existing library (starting with all those under /usr/lib) without any configuration or interface: you just have to write a '#pragma link' followed by the name of the library at the top of your script.
What I found really useful is the ability to edit C scripts and refresh the view in the Internet browser to see how my code works.
If there is a compilation error, then G-WAN outputs the line in the source code (just like any C compiler).
But where it enters the extraordinary area, is when you have a C script crash: here also it gives you THE LINE NUMBER IN THE SOURCE CODE (with the faulty call and the backtrace).
Kind of black-magic when you are used to Apache modules.
My experience with G-WAN and its C scripts are:
The G-WAN community is very small. Questions you have are mostly answered by its single developer.
I consider the API not mature: it's not as "clean" as Java APIs.
The limitation, but at the same time the power, of C: it's a systems programming language. So writing application logic in it must be done carefully.
You generally need to be a good developer to get good results: if you do something wrong, the server crashes fast and hard (Unix-style).
I've written some scripts now, to try out G-WAN. Overall, it's been very "productive": not much bugs and it works if you follow the guidelines and don't want to do too much funky stuff you expect it to have, like mature web servers. However, I have got the feeling I'm reinventing the wheel a lot of times.
G-WAN also support scripts written in other programming languages (C++, Objective-C, Java, etc.) so you will benefit from whatever native libraries each language implements.
For C scripts, well, the /usr/lib directory lists more than 1,500 libraries that G-WAN can re-use with a simple #pragma link "library".
I found it neat to be able to write a Web application with a part in C, another in C++ and a third one in Java!
Benchmark shown how G-wan fare poorly at handling these tests.
http://joshitech.blogspot.sg/2012/04/performance-nginx-netty-cppcms.html
I have been using G-Wan for about two years. I consider it highly stable and production ready for static files. I have a number of static sites running for over a year with no issues.
I have built some small scale dynamic sites in C with it as demos/test projects. A bittorrent tracker and a real time analytics platform both using the KV Store for data backing.
In my view building large scale dynamic sites in G-Wan is possible but only with a significant investment in development and support. G-Wan is better suited to building robust highly scalable "enterprise grade" applications than tossing something together over a weekend.
I use G-Wan for a CMS http://solicms.com but for now, I use Ruby as primary language.
I have used G-wan for some preliminary testing and it does benchmark well. I have found a few points of concern that make it so that I will not likely use it for any of my projects. I have found that it seems to cache responses for about 0.5secs to speedup the responses/second and I can't have only some of the responses hitting the application code. Also the key/value store is great for cache and temporary data storage but I'm not sure how well it will work as a real back-end storage method.
Can you give examples how you used gSOAP and how well it was integrated in your existing architecture? Have you found development bottlenecks with gSOAP?
We used gSOAP for a bunch of ARM clients to communicate with an AXIS Web Service server. Pros of gSOAP:
very powerful, supports nearly all Web Service constructs
easy to use, its abstraction of WS calls into functions removes all Web Service complexity to the programmer
elegant interfaces in both C and C++
However, we ran into several development bottlenecks:
when using custom datatypes like maps or sets, it takes quite some hacking to get the gSOAP compiler to handle them (marshal/unmarshalling). Especially bad with dynamic data structures.
debugging is hard because of its intrinsic complex network, parsing and memory allocation parts. Do everything possible to stick with static memory allocation.
the mailing list is alive, but developers are not very active on it. Simple questions can get answered quickly, but the toughest problems often go unanswered
forget about optimization. Linking in gSOAP eats about 1MB of memory at runtime (-Os). Runtime performance is fine on our 32MB linux-based ARM board, but there really is little to do about optimization if you need it.
We used gSOAP in a C++-based web server about 4 years back. Overall it worked fine. The only major issue was that the interface was in C and procedural (I understand it is difficult to design a good non-procedural interface). There may be a lot of repeated code while implementing the interface, for which you might have to use macros (we didn't explore the templates option too far then).
We are using gSoap to deploy a web service onto an embedde linux device running an ARM MX processor.
We are using gSOAP to consume a WCF based webservice from an application deployed on a linux device running on ARM processor. The experience is good to a large extent.
We used gSOAP in a web server on ARM ARM9 400MHz device.
gSOAP daemon connected to a database daemon through zeromq library, which is run on the same device.
It supports more than 1000 basic requests wich does not requre connection to database.
Disabling support for multi-referenced SOAP option by the WITH_NOIDREF define helped to decrease serialization time about 4 times faster on big requests with large number of serialization nodes.
What techniques are available to profile services running in WebMethods Integration Server?
That is, to obtain data which would indicate which parts would benefit most from optimisation.
Check the 'Service Profiler for webMethods Integration Server' from http://www.wrightia.com/serviceprofiler.html
Have you thought of trying webMUnit?
You could add time logging around certain method calls to get an idea of how long things are taking to execute. However, the time logging itself will add overhead too, so just keep that in mind.
Or possibly memory profilers such as YourKit Profiler, JProfiler, JProbe etc? Im not sure about these, as I've never used them myself, but they might help by letting you know what's taking up a lot of memory etc
Another tool which will do it is Green Hat tester which is a commercial tool used for testing webMethods. If you use the reporting capabilities of that when running tests you get profiling information generated..
António Abreu's linked tool is a more dedicated profiling tool however and would be the simplest solution.
If you want an "out of the box" solution you could turn on service auditing for the called services and do the legwork yourself, but that's a bit of manual effort.
You can find here a new monitoring tool for IS services: Rich Statistics
I was involved only in development phase so i dont know details but our system was tested with heavy load-testing and hardware was monitored during the process, it actually gave some feedback about bottlenecks. JMeter and sar were used.