I'm trying to gather a bit of knowledge on using an Akka Cluster application for an ETL style application.
I've read through most of the documentation and I'm now looking on moving forward with writing the actual application so I'm here to see if anyone has worked on something before and has any recommendations.
The ETL will need to able to schedule and throttle HTTP requests to a variety of endpoints. I would like to find a way to publish the statistics of the application (response times from endpoints, overall time in the application, and collect errors).
The application should be autoscaled so that when the amount of scheduled requests is increasing over time the Akka Cluster will add nodes to respond to the influx of work.
I was thinking that each scheduled request would produce a UUID to track the work (identifying results and errors that happen). So after the external endpoints respond or don't the results could be placed on a Akka Stream that would post process the requests as part of the T and the L of the application.
If there are good patterns out there or anyone can make a recommendation I would greatly appreciate it.
Cheers!
Take a look at kamon.io -- it's a library to provide metrics and tracing through a variety of technologies, and has an Akka tracing component. From the little I've worked with it, starting a trace requires just one method call.
Related
I am looking for a way to expose an existing event processing system to the external world using a REST interface. I have existing system design where we have RabbitMQ message queues where a publisher could post a message and then wait for the message processed results on a separate queue. Message ID is used to track the output to the original message on the output queue.
Now I want this to be exposed to the external consumers but we don't want to expose our RabbitMQ endpoint for this, so I was wondering if anyone has managed to achieve something similar to this using ExpressJS. Above diagram shows the current thought process
Main challenge I am facing here is that; some of this message processing could take more than couple of minutes, so was not sure how best to develop a API like this. Choices like should I create a polling interface for client here or is there a technology these days that help eliminate the polling on the client API to verify if the message is processed and get the result.
Can someone please help me with a good approach to manage these sort of requirement.
I finally ended up going the webhook way. Now when the REST API service receives a request, the client need to also provide a webhook and this will be registered with the client request and server will call it back when the results are available.
I have an Informatica BDM system (note Big Data Management, not Power Centre) and am having a problem with dropping connections when communicating with a third party web service. This fails the REST web service transformation which in turn kills our batch job.
Rather than fail the entire job, I would like each REST call to potentially be retried several times first.
I looked at the documentation, but I see no option to set a re-try on the REST Web Service Consumer transformation. Did I miss it? Or does one have to construct a re-try loop around it in some other way?
Bad News, But the Informatica BDM is yet to come up with an option to Retry. The need for the feature is already been raised by many users and may be the option will come up in the upcoming releases.
For now, we can only keep track of all the requests and responses manually and hope for the best.
I have a service exposing a REST endpoint that, after a couple of transformations, calls a third-party service also via its REST endpoint.
I would like to implement some sort of throttling on my service to avoid being throttled by this third-party service. Note that my service's endpoint accepts only one request and not a list of them. I'm using Play and we also have Akka Streams as dependency.
My first though was to have my service saving the requests into a database table and then have an Akka Streams Source, leveraging the throttle function, picking tasks, applying the transformations and then calling the external service.
Is this a reasanoble approach or does it have any severe drawbacks?
Thanks!
Why save the requests to the database? Does the queue need to survive restarts and/or do you run a load-balanced setup that needs to somehow synchronize the requests?
If you don't need the above I'd think using only Source.queue to store the task data would work just as well?
And maybe you already thought of this: If you want to make your endpoint more resilient you should allow your API to send a 'sorry, busy' response and drop the request instead of queuing it if your queue grows beyond a certain size.
I have a set of services. Every service contains some components.
Some of them are stateless, some of them are stateful, some are synchronous, some are asynchronous.
I used different approaches to monitoring and alerting.
Log-based alerting and metrics gathering. New Relic based. Own bicycle.
Basically, atm I am looking for a way, how to generalize and aggregate important metrics for all services in single place. One of things, I want is that we monitor more products, than separate services.
As an end result I see it as a single dashboard with small amount of widgets, but looking at those widgets I would be able to say for sure, if services are usable to end-customer.
Probably someone can recommend me some approach/methodology. Or give a reference to some best practices.
I like what you're trying to achieve! A service is not production-ready unless it's thoroughly monitored.
I believe what your're describing goes into the topics of health-checking and metrics.
... I would be able to say for sure, if services are usable to end-customer.
That however will require a little of both ;-) To ensure you're currently fulfilling your SLA, you have to make sure, that your services are all a) running and b) perform as requested. With both problems I suggest to look at the StatsD toolchain. Initially developed by Etsy, it has become the de-facto standard for gathering metrics.
To ensure all your services are running, we're relaying Kubernetes. It takes our description for what should run, be reachable from outside etc. and hosts that on our infrastructure. It also makes sure, that should things die - that they will be restarted. It helps with things like auto-scaling etc. as well! Awesome tooling and kudos to Google!
The way it ensures that is with health-checks. There are multiple ways how you can ensure your service node booted by Kubernetes is alive and kicking (namely HTTP calls and CLI scripts but this should be a modular thing should you need anything else!) If Kubernetes detects unhealthy nodes it will immediately phase them out and start another node instead.
Now, making sure, all your services perform as expected you'll need to gather some metrics. For all of our services (and all individual endpoints), we gather a few metrics via StatsD like:
Requests/sec
number of errors returned (404, etc...)
Response times (Average, Median, Percentiles depending on the services SLA)
Payload size (Average)
sometimes the number of concurrent requests per endpoint, the number of instances currently running
general metrics like the hosts current CPU and memory usage and uptime.
We gather a lot more metrics but that's about the bottom line. Since StatsD has become more of a "protocol specification" than a concrete product there are a myriad of collector, front- and backends to choose from. They help you visualize your systems state and many of them feature alerts of something or some combination of metrics go beyond their thresholds.
Let me know, if this was helpfull!
There's at least 3 types of things you will need to monitor: the host where the service is deployed, the component itself and the SLAs and some of them depend on the software stack you're using as well as the architecture.
With that said, you could for example use Nagios to monitor the hardware where the services are deployed, Splunk for the services metrics/SLAs as well as for any errors that might occur. You can also use SNMP packages in case something goes wrong and you have a more sophisticated support structure, this would be yours triggers. Without knowing how your infrastructure/services are set up it is complicated to go into deeper details.
I'm trying to decide if MSMQ is the right tool for communication between our application and a third party web service we are currently communicating with directly. We're looking to uncouple this such that if the service goes down, life could still go on as normal.
I can't find anything outside the usual MS fluff about it being the greatest thing and solving all your problems etc etc. It would be really useful if I could find some information that was somewhere between marketing fluff and API - like an architecture diagram of the components and how they integrate with each other, and more importantly how I integrate with them.
It's probably that I'm just looking for the information in the wrong places, so if someone could point me in the right direction, I'd appreciate it.
A typical MSMQ architecture would be composed of 3 parts...
Message Queue - This would be on one of your servers. You would have to install the MSMQ bits and create your queue.
Client - Your client would insert messages into the queue. I'm assuming you're using .NET. If so, most of what you want is going to be located in the System.Messaging namespace.
Windows Service - This would also run on a server, probably the same server as your queue. Its job would be to watch the queue, process messages as they come in, handle making sure the external service is available, and probably do some logging.
Here's an article that should go into a little more detail and give you some code samples.
MSMQ is a implementation of a message queue as are websphere mq and a bunch of other systems. When looking into the concepts and high level architecture I would suggest reading up on message queue's and how they are applied in disconnected scenario's. I can highly recommend Patterns of Enterprise Application Architecture. For specific examples on msmq check out Pro MSMQ: Microsoft Message Queue Programming it doesn't contain allot of special information but it does group it allot better then most resources available on the internet. This Hello World with MSMQ article would give you a nice overview of what it entails and it's easily executed on a development system.
If you are calling a remote web service from your application, it makes sense to use a queue to decouple your application processing from the remote system. By abstracting the communicating through messaging and having a gateway service that is responsible for communication to the web service, you isolate your application from the latency of the web service and build fault tolerance into the design by reducing the request/response usage inside your application (since messaging is by default asynchronous - you deal with it up front).
There are frameworks for .NET that can make this much easier (such as MassTransit or NServiceBus).
You can also check out SOA Patterns (by Arnon Rotem-Gal-Oz, Manning Press, in MEAP) and Enterprise Integration Patterns (Hohpe,Woolf), the latter of which is an essential read for anyone building a message-based system.