My powershell script runs slowly, is there any way to profile the powershell script?
Posting your script here would really help in giving an accurate answer.
You can use Measure-Command to see how much time each statement in your script is taking. However, you have to wrap each statement in Measure-Command.
Trace-Command can also be used to trace what is happening when the script runs. The output from this cmdlet can be quite verbose.
http://www.jonathanmedd.net/2010/06/powershell-2-0-one-cmdlet-at-a-time-104-trace-command.html
You can do random-pausing in the Powershell debugger. Get the script running, and while it's running, type Ctrl-C. It will halt and then you can display the stack. That will tell you where it is, what it's doing, and why. Do this several times, not just once.
Suppose it is taking twice as long as it could. That means each time you interrupt it the probability you will catch it doing the slow thing is 50%. So if you interrupt it 10 times, you should see that on about 5 samples.
Suppose it is taking 5 times as long as it could. That means 4/5 of the time is being wasted, so you should see it about 8 times out of 10.
Even if as little as 1/5 of the time is being wasted, you should see it about 2 times out of 10. Anything you see on as few as 2 samples, if you can find a faster way to do it, will give you a good speed improvement.
Here's a recent blog about speeding up for loops that shows you how to build a "test harness" for timing loops:
http://www.dougfinke.com/blog/index.php/2011/01/16/make-your-powershell-for-loops-4x-faster/
A quick and simple poor-man's profiler is simply to step through the code in the ISE debugger. You can sometimes feel how slow a part of the code is just by stepping over it or by running to some breakpoint.
Related
I have a script, in which this code fails, with an exit code of -2145124322
$new.ExitCode > $null
$filePath = "wusa.exe"
$argumentList = "`"\\PX_SERVER\Rollouts\Microsoft\VirtualPC\Windows6.1-KB958559-x64-RefreshPkg.msu`" /quiet /norestart"
$exitCode = (Start-Process -FilePath:$filePath -argumentList:$argumentList -wait -errorAction:Stop -PassThru).ExitCode
Write-Host $exitCode
Now, the main script has about 15,000 lines of "other stuff going on", and these lines where not originally exactly like this. The variables are pulled from XML, there is data validation and try/catch blocks, all sorts of stuff. So, I started pulling the pertinent lines out, and put them in a tiny separate script, and hard coded the variables. And there, it works, I get a nice 3010 exit code and off to the races. So, I took my working code, hard coded variables and all, and pasted it back into the original script, and it breaks again.
So, I moved the code out of the function where it belongs, and just put it after I initialize everything and before I start working through the main loop. And there it works! Now, I gotta believe it's the usual "polluted pipeline", but dang if I can figure out what could cause this. My next step I guess is to just start stepping through the code, dropping this nugget in somewhere, run the test, if it works move it farther down, try again. Gack!
So, hopping someone has some insights. Either what it might be, or perhaps an improved test protocol. Or some trick to actually see the whole pipeline and somehow recognize the pollution.
FWIW, I normally work with PoSH v2, but I have tried this with v4 with the exact same results. But perhaps there is some pipeline monitoring feature in a later version that could help with the troubleshooting?
Also, my understanding is that PoSH v2 has issues with negative return codes, so they can't be trusted. But I think newer versions fixed this, correct? So the fact that I get the same code in v4 means it is meaningful to Google? Not that I have found any hint of that exit code anywhere thus far.
Crossed fingers.
EDIT: OK, a little more data. I searched on the exit code without the -, and with DuckDuckGo instead of Google, and found this.
0x8024001E -2145124322 WU_E_SERVICE_STOP Operation did not complete because the service or system was being shut down.
OK, that's some direction. And I have some code that would allow me to kill a service temporarily. But that seems a little draconian. Isn't the whole point of this, like 10th way to install updates from Microsoft, supposed to be to make automation easier? In any case, I can't find any indication there are command line flags for WUSA that would avoid the problem, but I have to believe I am doing something wrong.
Solved! After tracking a number of different errors trying different things, including turning off the firewall and such, it turns out the error isn't that a service won't stop, but that a service won't start. See, some of that 15K lines of code suppresses Windows Update for the duration of my script, because Windows Update causes lots of Autodesk deployments to fail, which is the whole point of my code. Well, of course WUSA needs that service. So, it looks like, rather than suppressing Windows Update for the duration of script execution, I need to be less heavy handed and only suppress for the duration of a deployment task. that will take a few hours to implement and test, but is totally doable. And probably more elegant anyway. Woot!
And yeah, for once it wasn't me pooping in my pipeline unintentionally. ;)
I would appreciate it very much if you helped me with the following most annoying problem:
I'm using PyDev in Eclipse on my Ubuntu 14.04 machine, and every time I run my code in debug mode, it takes around 3-4 minutes to start.
My research yielded, that it takes a very long time to run each "import" statement row (without import statements, the problem vanishes).
Can anyone tell how can I overcome this problem?
Thanks!
I'm attaching:
1) my import statements.
2) my file tree (the file I'm running is in the folder "Gil").
3) and the debug window (during these 3-4 minutes, eclipse adds more and more lines there, that just say "light.py" (this is the file I'm running))
I'm only guessing here, but from your output in PyDev it seems you're executing something with multiprocessing or another thing which creates python subprocesses (which is why I think you're having a new light.py entry every time in the debugger).
Without looking at your code it's a bit hard guessing on what's actually happening, but I can give you some suggestions here:
Make your imports lazier (if you're always executing a new process which has to re-execute all the imports, that can indeed lead to quite more time -- imports in Python are usually slow, even more so with a debugger in place... maybe do a profile in regular mode to actually know what's going on -- if it's open source or you can afford the price, http://www.pyvmmonitor.com/ can probably help you quite a bit here -- if you haven't profiled your code before, you probably have low-hanging fruits which can give you a nice speedup).
Use only programatic breakpoints with the remote debugger (see: http://pydev.org/manual_adv_remote_debugger.html) -- this will make your code run at regular speed until it hits the programmatic breakpoint.
If none of those help, please add more details on your code (are you using stackless, greenlets, threads, multiple processes, etc? -- also 3-4 minutes may be much or not. Without having the original time to get there, it's hard to know...).
I was recently having problems with IDLE stopping responding once it hit a certain point in my code when processing very long strings, as reflected here: What's an efficient way to encode a (very long) string from a dictionary? (Python).
That has since been resolved; the same code that caused IDLE to freeze up ran in a second or two from the command line. Now, out of curiosity, why would this be?
(And, yes, I know I should probably use another IDE. However, at the moment I'm only working on a small project and I like how lean and simple IDLE is.)
IDLE chokes when presenting large results. I entered 'x'*100000, and IDLE took three seconds to display the result. It also froze when I tried to cursor up. The regular Python shell displayed the result practically instantaneously.
I have this Perl script for monitoring a folder in Linux.
To continuously check for any updates to the directory, I have a while loop that sleeps for 5 minutes in-between successive loops :
while(1) {
...
sleep 300;
}
Nobody on my other question suggested using cron for scheduling instead of a for loop.
This while construct, without any break looks ugly to me as compared to submitting a cronjob using crontab :
0 */5 * * * ./myscript > /dev/null 2>&1
Is cron the right choice? Are there any advantages of using the while loop construct?
Are there any better ways of doing this except the loop and cron?
Also, I'm using a 2.6.9 kernel build.
The only reasons I have ever used the while solution is if either I needed my code to be run more than once a minute or if it needed to respond immediately to an external event, neither of which appear to be the case here.
My thinking is usually along the lines of: cron has been tested by millions and millions of people over decades so it's at least as reliable as the code I've just strung together.
Even in situations where I've used while, I've still had a cron job to restart my script in case of failure.
My advice would be to simply use cron. That's what it's designed for. And, as an aside, I rarely redirect the output to /dev/null, that makes it too hard to debug. Usually I simply redirect to a file in the /tmp file system so that I can see what's going on.
You can append as long as you have an automated clean-up procedure and you can even write to a more private location if you're worried about anyone seeing stuff in the output.
The bottom line, though, is that a rare failure can't be analysed if you're throwing away the output. If you consider your job to be bug-free then, by all means, throw the output away but I rarely consider my scripts bug-free, just in case.
Why don't you make the build process that puts the build into the directory do the notification? (See SO 3691739 for where that comes from!)
Having cron run the program is perfectly acceptable - and simpler than a permanent loop with a sleep, though not by much.
Against a cron solution, since the process is a simple one-shot, you can't tell what has changed since the last time it was run - there is no state. (Or, more accurately, if you provide state - via a file, probably - you are making life much more complex than running a single script that keeps its state internally.)
Also, stopping the notification service is less obvious. If there's a single process hanging around, you kill it and the notifications stop. If the notifications are run by cron, then you have to know that they're run out of a crontab, know whose crontab it is, and edit that entry in order to stop it.
You should also consider persuading your company to upgrade to a version of Linux where the inotify mechanism is available.
If you go for the loop instead of cron and want your job run at regular intervals, sleep(300) tends to drift. (consider the execution time of the rest of your script)
I suggest using a construct like this:
use constant DELAY => 300;
my $next=time();
while (1){
$next+=DELAY;
...;
sleep ($next-time());
};
Yet another alternative is the 'anacron' utility.
if you don't want to use cron.
this http://upstart.ubuntu.com/ can be used to babysit processes.
or you can use watch whichever is easier.
I'm running a long simulation in MATLAB that I've realized I need to stop and rerun. However, MATLAB is really into this calculation, and it's stopped responding. How can I interrupt this run without killing MATLAB?
(I realize this is a problem with many Windows programs, but it's really acute with MATLAB.)
Go to the command window, and hit Ctrl-C a lot. From my experience, on a single-core machine you do not have a chance, unless you do lots of output. On a multi-core or multi-processor machine, you'll probably stop it eventually, but it takes time.
See also http://www.mathworks.com/support/solutions/en/data/1-188VX/index.html
Added: it is a good practice to (1) save a snapshot of your workspace before running anything really long and (2) within a very long calculation, write some of the variables to a file from time to time, so that you can resume the calculation if it was interrupted (by power failure, e.g.).
How well MATLAB responds to CTRL-C rather depends on what it's doing. If it's in the middle of a BLAS or LAPACK call for example, it will not respond until that call returns. If you're in a block of code where lots of lines of MATLAB are being executed, you can expect CTRL-C to be more responsive.
I have got a very simple trick to pause (or stop) a non-responsive execution.
If my simulation is running a long loop I always do the following:
for ii = 1:N
do_stuff();
clear empty_script;
empty_script;
end
And then create a file empty_script.m containing the following:
%keyboard
Whenever I want to pause execution I open an external text editor and uncomment the line saying keyboard in empty_script.m. That leaves me in debugging mode where I can watch variables, modify stuff or even stop the program.
Another strategy for dealing with this problem is to introduce a very short pause somewhere in the calculation (especially in a FOR or WHILE loop), as in:
for ii = 1:N
do_stuff();
pause(0.1);
end
This increases the chances that your maniacal Ctrl-C'ing will actually stop it.
you can find the MATLAB process in the windows task manager and set the priority as high or low and let other program to have lower or higher priority. In my experience, it is an efficient way.
if you wont to stop and rerun then killing is not bad choise
Go to windows task manager-> Processes then fined MATLAB.exe and push the End Process button