I am using https://host/cli/componentProcessRequest/info/ to get the information about a component process request execution details. But this gives basic information not detailed like logs.
In this process we execute a shell script. I want to get the shell output log through REST API. Is there any way that i can achieve this ?
In the REST documentation for Databricks, you can submit a notebook task as a job to a cluster using the 2.0 API or you can submit a command or python script using the 1.2 API
The 1.2 API allows you to create a context and then all subsequent commands or scripts can be submitted against this context. This allows you to maintain state (dataframes, variables etc) which is much more akin to running notebooks interactively in the browser
What i want is to be able to submit my notebooks into the same context and get the same behaviour as the 1.2 API but this does not seem possible, is there a reason for that? Or am i missing something if it can be done?
My use case is i want to be able to re-run a notebook from the API and have it remember its last state (in the most basic example just knowing its already loaded a dataframe) but more generally having the ability for subsequent jobs to only run what changed since the last run.
As far as I can tell, failing the ability to do this via the 2.0 API, I have 2 options:
Convert my notebook to Python script and have a bootstrap script on client side that invokes an entry point using the 1.2 API within the same context
Create temp tables at checkpoints in my notebook and possibly maintain a special variables dataframe of state variables
Both of these seem unecessarily complex, any other ideas?
Is there a way to check whether all the messages/flow-files in a processor group has cleared or not using rest api in shell script?
Basically, my use case is that I am stopping a nifi processor in my shell script via curl command. Then I need to wait till all the queues are empty and flow files have passed through before proceeding further.
Yes, you can query the status of individual connections or the process group status via the API. The easiest way to do this is perform the action in your browser and use the Developer Tools to monitor the request and copy/paste this to your invoking tool.
We have a handful of Ruby (Rails/Sinatra) apps and are looking for an easy means of managing retention, search and analysis of our logs for these applications.
The initial problem was that every time we'd push a new version of our apps the logs would disappear.
We then started streaming our logs to a file via a terminal using:
cf logs AppName
however the logfiles get very big, very fast and quickly become a problem.
We know that the Bluemix Monitoring and Analytics service provides a lot of the function we need. We're looking that over but want to know if there are other recomended/proven options.
Thanks
We found several 3rd party apps that provide the functions we need.
To use any of these we first had to configure third party logging on Bluemix and used the steps below.
Any 3rd party logger that supports the syslog protocol can be used. The initial setup, registration and configuration of the log manegement service, is well covered at Configuring Selected Third-Party Log Management Services.
What will come out of the configuration step is a syslog URL which will be the destination for your logs.
Once the logging service is configured a user-provided service instance needs to be created to stream the logs to the logging service. We did this using:
cf create-user-provided-service <user-provided-service_name> -l <syslog_URL>
Last step is to bind the service instance to our Ruby apps.
cf bind-service AppName <user-provided-service_name>
For the changes to take effcet, we then had to restage our ruby apps:
cf restage AppName
There was a brief delay between when we'd see the logs generated and when they'd show up in the logger service but overall this is working out ok for us so far.
What are the different approaches for creating scheduled tasks for web applications, with or without a separate web/desktop application?
If we're talking Microsoft platform, then I'd always develop a separate Windows Service to handle such batch tasks.
You can always reference the same assemblies that are being used by your web application to avoid any nasty code duplication.
Jeff discussed this on the Stack Overflow blog -
https://blog.stackoverflow.com/2008/07/easy-background-tasks-in-aspnet/
Basically, Jeff proposed using the CacheItemRemovedCallback as a timer for calling certain tasks.
I personally believe that automated tasks should be handled as a service, a Windows scheduled task, or a job in SQL Server.
Under Linux, checkout cron.
I think Stack Overflow itself is using an ApplicationCache expiration to run background code at intervals.
If you're on a Linux host, you'll almost certainly be using cron.
Under linux you can use cron jobs (http://www.unixgeeks.org/security/newbie/unix/cron-1.html) to schedule tasks.
Use URL fetchers like wget or curl to make HTTP GET requests.
Secure your URLs with authentication so that no one can execute the tasks without knowing the user/password.
I think Windows' built-in Task Scheduler is the suggested tool for this job. That requires an outside application.
This may or may not be what you're looking for, but read this article, "Simulate a Windows Service using ASP.NET to run scheduled jobs". I think StackOverflow may use this method or it was at least talked about using it.
A very simple method that we've used where I work is this:
Set up a webservice/web method that executes the task. This webservice can be secured with username/pass if desired.
Create a console app that calls this web service. If desired, you can have the console app send parameters and/or get back some sort of metrics for output to the console or external logging.
Schedule this executable in the task scheduler of choice.
It's not pretty, but it is simple and reliable. Since the console app is essentially just a heartbeat to tell the app to go do its work, it does not need to share any libraries with the application. Another plus of this methodology is that it's fairly trivial to kick off manually when needed.
Use URL fetchers like wget or curl to make HTTP GET requests.
Secure your URLs with authentication so that no one can execute the tasks without knowing the user/password.
You can also tell cron to run php scripts directly, for example. And you can set the permissions on the PHP file to prevent other people accessing them or better yet, don't have these utility scripts in a web accessible directory...
Java and Spring -- Use quartz. Very nice and reliable -- http://static.springframework.org/spring/docs/1.2.x/reference/scheduling.html
I think there are easier ways than using cron (Linux) or Task Scheduler (Windows). You can build this into your web-app using:
(a) quartz scheduler,
or if you don't want to integrate another 3rd party library into your application:
(b) create a thread on startup which uses the standard Java 'java.util.Timer' class to run your tasks.
I recently worked on a project that does exactly this (obviously it is an external service but I thought I would share).
https://anticipated.io/
You can receive a webhook or an SQS event at a specific scheduled time. Dealing with these schedulers can be a pain so I thought I'd share in such case someone is looking to offload their concerns.