Lately I have found I have to do a lot of long running operations like:
db.whois.find({"basic.last_updated":"1970-01-01 00:00:00"}).forEach(function (doc) {
var parts = doc._id.split('#');
doc.basic.last_updated = parts[1];
db.whois.save(doc);
})
That's fine but since the shell prompt doesn't return until it's complete I am finding I have to keep lots of shells open. Is there any way I can issue a command and send it async so that I can continue using the same shell? Maybe even get a little message when the process is done?
Related
I'm looking for some insight and guidance on using the Foundation.Process type with a PTY (Psuedo Terminal) so that the subprocess can accept input and behave as if it was running via a terminal.
The reason for needing a PTY is that for programs like ssh or in my case (xcodes) which ask for user input including passwords, running these via Foundation.Process does not display the prompts to the user as the output is usually buffered (this works fine in the Xcode debugger console but when running via a real terminal that is buffered the prompts are never displayed in the terminal)
Looking at other threads it seems like correct approach here is create a PTY and use the filehandles to attach to the Process.
While I've got this to work to the point where prompts are now shown, I cant seem to figure out how to pass input back to the process as these are being controlled by the PTY.
Here is my Process setup:
let process = Process()
// Setup the process with path, args, etc...
// Setup the PTY handles
var parentDescriptor: Int32 = 0
var childDescriptor: Int32 = 0
guard Darwin.openpty(&parentDescriptor, &childDescriptor, nil, nil, nil) != -1 else {
fatalError("Failed to spawn PTY")
}
parentHandle = FileHandle(fileDescriptor: parentDescriptor, closeOnDealloc: true)
childHandle = FileHandle(fileDescriptor: childDescriptor, closeOnDealloc: true)
process.standardInput = childHandle
process.standardOutput = childHandle
process.standardError = childHandle
With this setup I then read the parent handle and output any result it gets (such as the input prompts):
parentHandle?.readabilityHandler = { handle in
guard let line = String(data: handle.availableData, encoding: .utf8), !line.isEmpty else {
return
}
logger.notice("\(line)")
}
When process.run() is executed the program runs and I can see it asks for Apple ID: input in my terminal, however, when typing input into the terminal the process does not seem to react to this input.
I've tried forwarding the FileHandle.standardInput:
FileHandle.standardInput.readabilityHandler = { handle in
parentHandle?.write(handle.availableData)
}
But this doesn't seem to work either.
What is the recommended way to setup a PTY with Foundation.Process for executing arbitrary programs and having them behave as if they were being run in a terminal context?
Most of the resources I found online are about other languages and I'd like to stick with Foundation.Process vs. doing anything custom in C/C++ if possible as it just makes it easier to reason about / maintain. The resources for Swift on this topic are very lacking and I've checked out some open source projects that claim to do this but most require manually sending input to the PTY handle vs. accepting them from the user in a terminal.
Any insight / help is very much appreciated!
im trying to execute this tests and I got: 'Failed: Script timeout' and i don't undesrstand where is the error, literally i have do copy paste, in the last test case is working, but in this code i got that error. Thanks in advance.
var name_filter = browser.element(by.xpath(browser.params.constants.GENERALPATHS.CATALOG.NAMEFILTER));
var seriestitle_filter = browser.element(by.xpath(browser.params.constants.GENERALPATHS.CATALOG.SERIESTITLEFILTER));
var catalog_title = browser.element(by.xpath(browser.params.constants.GENERALPATHS.CATALOG.CATALOGTITLE));
await browser.wait(EC.elementToBeClickable(name_filter), waitLongTime);
await browser.wait(EC.elementToBeClickable(seriestitle_filter), waitLongTime);
var catalogues = await element.all(by.className('ui-grid-cell-contents ng-binding ng-scope'));
for(var i=0; i<catalogues.length; i++) {
var catalogue_text = await catalogues[i].getText();
await catalogues[i].click();
await browser.wait(EC.elementToBeClickable(name_filter), waitLongTime);
await browser.wait(EC.elementToBeClickable(seriestitle_filter), waitLongTime);
expect(catalog_title).toEqual(catalogue_text);
}
I've given a recommendation to you some time ago to switch to async/await. I see you're using that, and I hope that made your experience working with protractor better
Unfortunately I can't give you direct answer to your question again, because the script may be causing problems in any line and no way to find out where exactly without the app you're testing. Additionally, there may be a few problems related your config.
However, I tend to help to solve problems with the processes people choose for test development
In this example, I highly recommend you to setup a run configuration in webstorm or vs studio. This will allow you to execute any script line by line, explore variables etc. If you do that, you can easily find the line where you script fails and troubleshoot immediately. If you find the line with the problem but still don't understand why it happens, let me know, I'll try to walk you through the steps I'd do to figure out the issue
Hope it helps, and sorry for a vague answer
I'm having some troubles with celery. Unfortunately the person who set it up isn't working here any more, and until now we've never had problems and thought we understood how it works well enough. Now it has become clear that we don't, and after hours of searching through documentation and other posts on here, I have to admit defeat. Hopefully, someone here can shed some light on what I am missing.
We're using several tasks, all of them are defined in a CELERYBEAT_SCHEDULE like this:
CELERYBEAT_SCHEDULE = {
'runs-every-5-minutes': {
'task': 'tasks.webhook',
'schedule': crontab(minute='*/5'),
'args': (WEBHOOK_BASE + '/task/refillordernumberbuffer', {'refill_count': 1000})
},
'send-sameday-delivery-confirmation': {
'task': 'tasks.webhook',
'schedule': crontab(minute='*/2'),
'args': (WEBHOOK_BASE + '/task/sendsamedaydeliveryconfirmation', {})
},
'send-customer-hotspot-notifications': {
'task': 'tasks.webhook',
'schedule': crontab(hour=9, minute=0),
'args': (WEBHOOK_BASE + '/task/sendcustomerhotspotnotifications', {})
},
}
That's not all of them, but they all work like this. All of those are actually PHP scripts that have no knowledge of the whole celery concept. They are just scripts that execute certain things, and send notifications if necessary. When they are done, they just spit out a JSON response that says success=true.
As far as I know, celery is only used to execute them periodically. We don't have problems with any of them except the last one from my code snippet. That task/script sends out emails, usually 5 to 10, but sometimes a lot more. And that's where the problems start, because (as far as I could examine by watching in celery events, I could honestly not find any confirmation for this in the docs anywhere) when the successful JSOn response from the PHP script doesn't arrive within 3 minutes, celery retries the task, and the script sends a lot of emails again. And again, because just a small amount of emails was saved as "done" form the tasks initial run. This often leads to 4 or 5 retries until finally enough emails were marked as "successfully sent" by the prior retries that finally the last retry finishes under this mystical 3 minute limit.
My questions:
Is there a default time limit? Where is it set? How do I override it? I've read about time_limit and soft_time_limit, but nothing I tried in the config seemed to help. If this is the solution, I would be in need of assistance as to how the settings are properly applied.
Can't I "just" disable the whole retry concept (for one task or for all, doesn't really matter) altogether? It seems to me that we don't need it, as we're running our tasks periodically and missing one due to a temporary error would not matter. I guess that means we shouldn't have used celery in the first place as we're misusing it, but for now I'd just like to understand it better.
Thanks for any help, and sorry if I left anything unclear – happy to answer any follow-up questions and provide more details if necessary.
The rest of the config file goes like this:
## Broker settings.
databases = parse_databases_xml()
settings = parse_custom_settings_xml()
BROKER_URL = 'redis://' + databases['taskqueue']['host'] + '/' + databases['taskqueue']['dbname']
# List of modules to import when celery starts.
CELERY_IMPORTS = ("tasks", )
## Using the database to store task state and results.
CELERY_RESULT_BACKEND = BROKER_URL
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ANNOTATIONS = {
"*": {"rate_limit": "100/m"},
"ping": {"rate_limit": "100/m"},
}
There is no time_limit to be found anywhere, so I don't think we're setting it ourselves. I left out the python imports and the functions that read from our config xml files, as that stuff is all working fine and just concerns some database auth data.
If we have around 30G files (ranged from 50MB to 4GB) need to be uploaded to Google Cloud Storage everyday, according to google docs, gsutil might be the only fitted choice, isn't it?
I want to call gsutil command by Java, now the code below can work. But If I delete that while loop, the program will stop immediately after the runtime.exec(command) but python process was started but doing no uploading and it will soon be killed. I wonder why.
The reason I read from sterr stream is inspired by Pipe gsutil output to file
I decide whether gsutil finish executing by read util the last line of its status output, but is it a reliable way? Is there any better ways to detect whether gsutil execution is end in Java?
String command="python c:/gsutil/gsutil.py cp C:/SFC_Data/gps.txt"
+ " gs://getest/gps.txt";
try {
Process process = Runtime.getRuntime().exec(command);
System.out.println("the output stream is "+process.getErrorStream());
BufferedReader reader=new BufferedReader(new InputStreamReader(process.getErrorStream()));
String s;
while ((s = reader.readLine()) != null){
System.out.println("The inout stream is " + s);
}
} catch (IOException e) {
e.printStackTrace();
}
There are certainly more than one way to upload 30G worth of data per day to GCS. Since you are working in Java, have you considered to use the Cloud Storage API Java client library?
https://developers.google.com/api-client-library/java/apis/storage/v1
As for the specific questions about calling gsutil from Java using Runtime.exec(), I suspect when there is no while loop, the program will exit immediately after creating the sub-process, causing the "process" variable to be GC'ed, which might kill the sub-process.
I think you should wait for the sub-process to complete, which is effectively what the while loop is doing. Or you can just call waitFor() and check the existValue() if you don't care about the output:
http://docs.oracle.com/javase/7/docs/api/java/lang/Process.html
I draw the following pic according to Zhihong Yao's explanation. Hope it can help anyone with the same question as mine.
I have a locked down "kiosk" terminal server.
This terminal server has a perl script as its .Xsession, and launches a Tk interface. When that Tk interface is done, the perl script launches "process2" and lets the user interact with "process2" (which is a graphical application).
If a user tampers with "process2", and make it crash, the user might be able to access the underlying desktop, therefore I would want to check if "process2" is running, and if "process2" is not running on $display, I would want to just execute logout (which would logout the display the perl script is currently running as).
Since the system is running 10 instances of "process2" to 10 different users simultanuosly, I cant just check if "process2" is running on the system with "ps" or someting like that. I need to check if "process2" is running under that specific display $display.
Note that all 10 users log on as the same username in all sessions, so I cannot check all processes run by a specific user, that would return all 10 instances too.
Like:
system("process2 &");
while(1) {
sleep(1);
if (is_it_running("process2", $display) == false) {
system("logout &");
}
}
Its the function "is_it_running" that I need to get to know how it should look.
$display can either contain the raw display number, like this: ":1.0", or it can contain the display number parsed out, like this: "1".
If you use fork and exec instead of system("...&"), you can store the Process IDs of your child processes and more directly check their status. See also perlipc.
Why not just run process2 in the foreground? Then your perl script won't get control back until it's done executing, at which point it can exit:
system("process2");
system("logout");
Of course, if that's the entire script, maybe a bash script would make more sense.
I solved it after many attempts.
Did a piped open
$pidofcall = open(HANDLE, "process2|");
Then I did whatever I did need to do, and I made the server send a signal to me if it loses connection with process2. If I did need to bang out, I simply did a "goto killprocess;" Then I simply had:
killprocess:
kill(9,$pidofcall);
close(HANDLE);
$mw->destroy;
system("logout");