I want to have my application already started on my YARN cluster and allow the users to send additional commands. I am still in the design phase, but I'm confused on the best way about going about this. Is this possible? Could the user send some sort of REST command to the Application Master or Resource Manager that could then be passed to the running YARN Application?
You can if you're willing to build a custom AM and write your own REST API but writing a custom AM is not trivial. As for the RM, you can kill your application or move to another queue via REST API calls but not much else.
https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
I have added Hornetq user using add-uesr.sh script.
I want to write a script to verify hornetq username and password entered by someone is correct or not (before configuring any other components like JMS queues etc).
Does JBOSS CLI provide a way to check the validity of authentication details?
Thank you!
You will not be able to use the JBoss CLI.
I suggest using a Hornteq JMS Client, just extend an example program and pass in username and password via command line paramters or property files. You could also check the correct queues and topics are available to the given user. You can also use the JBoss CLI in Java to check a message is delivered to a queue or topic etc. Make sure your program uses the correct exit code for success or failure.
When using the nekotools server for development, I get all messages in the console. Is it be possible to activate those messages in Apache logs on a server, for debugging purposes?
Answering own question since no answers yet.
In Haxe API I found neko.Web.logMessage(msg:String):Void, it allows sending custom messages to the webserver file logs. Not exactly what I was looking for though.
Does anyone know of a decent way of diagnosing MSMQ access errors. I'm using NServiceBus in a web application. There is a service running that actually creates the private queues, and the web application sends messages to that queue.
The web site uses anonymous access, and the application pool runs using the Network Service account. When my application tries to send a message to the queue (using NServiceBus), I get MSMQ access denied errors. I have tried all combinations of full control with ANONYMOUS LOGON, Everyone, etc, and still no luck. I have tried deleting the queues and letting NServiceBus recreate them, and still no luck.
When I set the IIS App Pool to run using a local administrator account, then it all works fine. How can I work out exactly what permissions I need to apply? Using the accounts/permissions that are added by default clearly isn't working.
You need to set up the queue permissions so that "everyone" can send to any queue.
What are the different approaches for creating scheduled tasks for web applications, with or without a separate web/desktop application?
If we're talking Microsoft platform, then I'd always develop a separate Windows Service to handle such batch tasks.
You can always reference the same assemblies that are being used by your web application to avoid any nasty code duplication.
Jeff discussed this on the Stack Overflow blog -
https://blog.stackoverflow.com/2008/07/easy-background-tasks-in-aspnet/
Basically, Jeff proposed using the CacheItemRemovedCallback as a timer for calling certain tasks.
I personally believe that automated tasks should be handled as a service, a Windows scheduled task, or a job in SQL Server.
Under Linux, checkout cron.
I think Stack Overflow itself is using an ApplicationCache expiration to run background code at intervals.
If you're on a Linux host, you'll almost certainly be using cron.
Under linux you can use cron jobs (http://www.unixgeeks.org/security/newbie/unix/cron-1.html) to schedule tasks.
Use URL fetchers like wget or curl to make HTTP GET requests.
Secure your URLs with authentication so that no one can execute the tasks without knowing the user/password.
I think Windows' built-in Task Scheduler is the suggested tool for this job. That requires an outside application.
This may or may not be what you're looking for, but read this article, "Simulate a Windows Service using ASP.NET to run scheduled jobs". I think StackOverflow may use this method or it was at least talked about using it.
A very simple method that we've used where I work is this:
Set up a webservice/web method that executes the task. This webservice can be secured with username/pass if desired.
Create a console app that calls this web service. If desired, you can have the console app send parameters and/or get back some sort of metrics for output to the console or external logging.
Schedule this executable in the task scheduler of choice.
It's not pretty, but it is simple and reliable. Since the console app is essentially just a heartbeat to tell the app to go do its work, it does not need to share any libraries with the application. Another plus of this methodology is that it's fairly trivial to kick off manually when needed.
Use URL fetchers like wget or curl to make HTTP GET requests.
Secure your URLs with authentication so that no one can execute the tasks without knowing the user/password.
You can also tell cron to run php scripts directly, for example. And you can set the permissions on the PHP file to prevent other people accessing them or better yet, don't have these utility scripts in a web accessible directory...
Java and Spring -- Use quartz. Very nice and reliable -- http://static.springframework.org/spring/docs/1.2.x/reference/scheduling.html
I think there are easier ways than using cron (Linux) or Task Scheduler (Windows). You can build this into your web-app using:
(a) quartz scheduler,
or if you don't want to integrate another 3rd party library into your application:
(b) create a thread on startup which uses the standard Java 'java.util.Timer' class to run your tasks.
I recently worked on a project that does exactly this (obviously it is an external service but I thought I would share).
https://anticipated.io/
You can receive a webhook or an SQS event at a specific scheduled time. Dealing with these schedulers can be a pain so I thought I'd share in such case someone is looking to offload their concerns.