I need to write automated tests for Enterprise Service Bus (ESB). When ESB is running it performs different scheduled operations including DB queries, SOAP and TCP calls, etc. Also it periodically checks whether some services are active and stops flows in case it is not.
So in my case there will be some problems:
some flows will stop because service is not responding
another flows will continuously run in negative scenariuos producing errors
Is it possible to have mocks continuosly running while executing tests using Citrus Framework?
How else can I overcome this problems with Citrus?
Please note that all server components in Citrus do only live while tests are executed. This is because Citrus automatically starts and stops the server components before/after the test suite. When Citrus is inactive the server components are stopped.
If you want to break these limitations please have a look at the citrus-simulator side project which is a complete standalone simulator for different message transport types (still in beta phase though).
If the limitation mentioned before is Ok for you then you can use a combination of endpoint adapters on the server components in your Citrus project. You could have a static endpoint adapter that always sends a positive response for incoming health check requests.
Please see the documentation on endpoint adapters: http://www.citrusframework.org/reference/html/endpoint-adapter.html
I want to have my application already started on my YARN cluster and allow the users to send additional commands. I am still in the design phase, but I'm confused on the best way about going about this. Is this possible? Could the user send some sort of REST command to the Application Master or Resource Manager that could then be passed to the running YARN Application?
You can if you're willing to build a custom AM and write your own REST API but writing a custom AM is not trivial. As for the RM, you can kill your application or move to another queue via REST API calls but not much else.
https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
The reason I'm asking is I would like to use the out-of-proc mode, but I cannot install a service on each user's workstation, only on a central server. Is the communication between event source and listener service an ETW thing, or is there some kind of RPC I could use?
Yes, the out-of-process mode works by using ETW. All ETW events are system wide so the service just has to listen to ETW events.
ETW only works locally and does not offer a remote solution you could use. Your options are to install a service on each workstation, listen to ETW events (here or here) and forward them to your server with a RPC solution you build yourself. Using MSMQ comes to mind. Or have your application forward the events to your server directly so you don't need the service. Either way, you will have to build it yourself.
To add to Lars' answer, you could also log to SQL. There is a SQL sink you can use but like everything else, to get the most customized fit, you would build your own (or inherit from another class to give you a good starting point). Be careful though. Not all sinks are created the same. They all have their pros and cons. For example, with SQL and Azure sinks, you have to worry about high latency. The XML formatter doesn't write the root starting and ending node so it's not well-formed xml. Whatever reads that file would have to provide them. Good luck!
I want to send a live video stream to a server and I want to perform facial recognition on that video and I would like to get the result back to the client program. Where do I get a server? Can I use Windows Azure here? If yes, can I also make a Python/C++ Server program listen on a particular port?
You haven't talked about the client-side piece. Assuming you're in control of a client app, you could push the video to a Blob, then drop a notification in an Azure queue for a background task to process the uploaded video fragment.
Instead of directly pushing to blobs, you could host a web service that lets you push uploads, and the web service could store the video fragment and then trigger a background processing task.
Running python should be very straightforward - just upload the python exe and any related modules, either with your Windows Azure deployment or in blob storage (then pull them down from blob storage and install them when the VM starts up). As far as port-listening, you can define up to 25 ports that are external-facing. You'd then have your python app listen on the port you defined (either tcp, http, or https).
More info on block and page blobs here. Steve Marx posted this example for installing python in your Web or Worker role.
What are the different approaches for creating scheduled tasks for web applications, with or without a separate web/desktop application?
If we're talking Microsoft platform, then I'd always develop a separate Windows Service to handle such batch tasks.
You can always reference the same assemblies that are being used by your web application to avoid any nasty code duplication.
Jeff discussed this on the Stack Overflow blog -
https://blog.stackoverflow.com/2008/07/easy-background-tasks-in-aspnet/
Basically, Jeff proposed using the CacheItemRemovedCallback as a timer for calling certain tasks.
I personally believe that automated tasks should be handled as a service, a Windows scheduled task, or a job in SQL Server.
Under Linux, checkout cron.
I think Stack Overflow itself is using an ApplicationCache expiration to run background code at intervals.
If you're on a Linux host, you'll almost certainly be using cron.
Under linux you can use cron jobs (http://www.unixgeeks.org/security/newbie/unix/cron-1.html) to schedule tasks.
Use URL fetchers like wget or curl to make HTTP GET requests.
Secure your URLs with authentication so that no one can execute the tasks without knowing the user/password.
I think Windows' built-in Task Scheduler is the suggested tool for this job. That requires an outside application.
This may or may not be what you're looking for, but read this article, "Simulate a Windows Service using ASP.NET to run scheduled jobs". I think StackOverflow may use this method or it was at least talked about using it.
A very simple method that we've used where I work is this:
Set up a webservice/web method that executes the task. This webservice can be secured with username/pass if desired.
Create a console app that calls this web service. If desired, you can have the console app send parameters and/or get back some sort of metrics for output to the console or external logging.
Schedule this executable in the task scheduler of choice.
It's not pretty, but it is simple and reliable. Since the console app is essentially just a heartbeat to tell the app to go do its work, it does not need to share any libraries with the application. Another plus of this methodology is that it's fairly trivial to kick off manually when needed.
Use URL fetchers like wget or curl to make HTTP GET requests.
Secure your URLs with authentication so that no one can execute the tasks without knowing the user/password.
You can also tell cron to run php scripts directly, for example. And you can set the permissions on the PHP file to prevent other people accessing them or better yet, don't have these utility scripts in a web accessible directory...
Java and Spring -- Use quartz. Very nice and reliable -- http://static.springframework.org/spring/docs/1.2.x/reference/scheduling.html
I think there are easier ways than using cron (Linux) or Task Scheduler (Windows). You can build this into your web-app using:
(a) quartz scheduler,
or if you don't want to integrate another 3rd party library into your application:
(b) create a thread on startup which uses the standard Java 'java.util.Timer' class to run your tasks.
I recently worked on a project that does exactly this (obviously it is an external service but I thought I would share).
https://anticipated.io/
You can receive a webhook or an SQS event at a specific scheduled time. Dealing with these schedulers can be a pain so I thought I'd share in such case someone is looking to offload their concerns.