I would appreciate it very much if you helped me with the following most annoying problem:
I'm using PyDev in Eclipse on my Ubuntu 14.04 machine, and every time I run my code in debug mode, it takes around 3-4 minutes to start.
My research yielded, that it takes a very long time to run each "import" statement row (without import statements, the problem vanishes).
Can anyone tell how can I overcome this problem?
Thanks!
I'm attaching:
1) my import statements.
2) my file tree (the file I'm running is in the folder "Gil").
3) and the debug window (during these 3-4 minutes, eclipse adds more and more lines there, that just say "light.py" (this is the file I'm running))
I'm only guessing here, but from your output in PyDev it seems you're executing something with multiprocessing or another thing which creates python subprocesses (which is why I think you're having a new light.py entry every time in the debugger).
Without looking at your code it's a bit hard guessing on what's actually happening, but I can give you some suggestions here:
Make your imports lazier (if you're always executing a new process which has to re-execute all the imports, that can indeed lead to quite more time -- imports in Python are usually slow, even more so with a debugger in place... maybe do a profile in regular mode to actually know what's going on -- if it's open source or you can afford the price, http://www.pyvmmonitor.com/ can probably help you quite a bit here -- if you haven't profiled your code before, you probably have low-hanging fruits which can give you a nice speedup).
Use only programatic breakpoints with the remote debugger (see: http://pydev.org/manual_adv_remote_debugger.html) -- this will make your code run at regular speed until it hits the programmatic breakpoint.
If none of those help, please add more details on your code (are you using stackless, greenlets, threads, multiple processes, etc? -- also 3-4 minutes may be much or not. Without having the original time to get there, it's hard to know...).
Related
I was recently having problems with IDLE stopping responding once it hit a certain point in my code when processing very long strings, as reflected here: What's an efficient way to encode a (very long) string from a dictionary? (Python).
That has since been resolved; the same code that caused IDLE to freeze up ran in a second or two from the command line. Now, out of curiosity, why would this be?
(And, yes, I know I should probably use another IDE. However, at the moment I'm only working on a small project and I like how lean and simple IDLE is.)
IDLE chokes when presenting large results. I entered 'x'*100000, and IDLE took three seconds to display the result. It also froze when I tried to cursor up. The regular Python shell displayed the result practically instantaneously.
Whether or not I compile a Racket program seems to make no difference to the runtime performance.
Is it just the loading of the file initially that is improved by compilation? In other words, does running racket src.rkt do a jit compilation on the fly, which is why I see no difference in compiling vs interactive?
Even for tight loops of integer arithmetic, where I thought some difference would occur, the profile times are equivalent whether or not I previously did a raco make.
Am I missing something simple?
PS, I notice that I can run racket against the source file (.rkt) or .zo file. Does racket automatically use the .zo if one is found that corresponds to the .rkt file, or does the .zo file need to be used explicitly? Either way, it makes no difference to the performance numbers I'm seeing.
Yes, you're right.
Racket compiles code in two stages: first, the code is compiled into bytecode form, and then when it runs it gets jitted into machine code. When you compile a file, you're basically creating the bytecode which saves on re-compiling it later. Since that's usually not something that takes a lot of time for small pieces of code, you won't see any noticeable difference in runtimes. For an extreme example, you can delete all *.zo files in the collection tree and start DrRacket -- it will take a lot of time to start since there's a ton of code, but once it does start, it would run almost as usual. (It would also be slow to click "run" since that will reload and recompile some files.) Another concern for bigger pieces of code is that the compilation process can make memory consumption higher, but that's also not an issue with smaller pieces of code.
See also the Performace chapter in the guide for hints on how to improve performance.
Racket will always compile your code, regardless of whether it is run interactively at the REPL or run from the command-line. Here is the section in the guide that explains it. In interactive mode, the compiler turns every expression/definition into bytecode in memory and executes that. Otherwise, the compilers outputs the bytecode to zo files.
Note: Eli replied at the same time I did. See his response for more details.
for those who don't know, imenu is a thing in emacs that lets a mode insert one or more menu items into the menu bar. The most common usage is to make a "table of contents" accessible from a drop-down menu, so the user can quickly jump to declarations of functions or classes or sections in a document, etc.
imenu has a couple different ways of working - in the first and more commonly used way, a major mode provides regexps to imenu, and imenu uses those regexps to perform the scan of the buffer and build the index. A major mode sets this up by putting the list of regexps into imenu-generic-expression. The second way is for the major mode to perform its own scan. It can do this by instead setting the variable imenu-create-index-function to the name of a function defined by themode, which returns a list containing the table of contents.
I'm doing the latter - imenu-create-index-function - but sometimes the fn takes a looong time to run, say 3 or 4 seconds or more, which freezes the UI. If I make the operation asynchronous, that would solve that problem.
I know about asynch processes. The scan logic is implemented in elisp. Is it possible to run elisp in an asynch process? If so, how?
Or, is there a way to run regular elisp asynchronously in emacs, without resorting to an asynch process?
I think the way font-lock does it is, it fontifies on idle. It keeps state and fontifies a little at a time, always remembering where it left off, what else needs to be fontified, what has changed since the last fontification run, etc. Is my understanding correct? Maybe I could use this incremental approach .
Recommendations?
To run elisp asynchronously you can use either run-with-idle-timer or run-with-timer. I imagine you'll want the idle version. Check the documentation links for more details.
Note: If the code takes 3 or 4 seconds to run, it'll still take that long (and freeze your Emacs while it runs), so if you can break the work up into small enough chunks that it only takes .5 seconds or so at a time, that might work well.
One package that I use all the time, pabbrev.el, uses idle timers really well - I never notice it running. That might be a good package to examine to see how it breaks up the work (it is scanning all open buffers and building up a word frequency list).
The answers posted by TreyJackson and jeremiahd were valid back in year 2011. Now, in 2018, here is a link to the emacs documentation for asynchronous processes.
You can run elisp in an asynch process by spawning emacs in batch mode as the process, see http://www.emacswiki.org/emacs/BatchMode . Other than that, there's basically nothing as far as I know.
It looks like http://nschum.de/src/emacs/async-eval/ basically wraps the boilerplate necessary to do this. No clue if it's actively maintained or anything though.
I'm about to rewrite a large portion of a project that I have developed over the last 10years while learning perl. There is alot of optimisation that can be gained.
A key part of the code is a large if/elsif block that require xxx.cgi files depending on a POST value. Eg:
if($FORM{'action'} eq "1"){require "1.cgi";}
elsif($FORM{'action'} eq "2"){require "2.cgi";}
elsif($FORM{'action'} eq "3"){require "3.cgi";}
elsif($FORM{'action'} eq "4"){require "4.cgi";}
It has many more irritations but just how expensive is using "require" in perl?
require itself has a relatively low cost in any case and, if you require the same file more than once within a single run of your program, it will detect that the file has already been loaded and not attempt to load it a second time. However, if you have a long and highly-populated search path (#INC) and you require (or use) a lot of files, it's possible that all of the directory searches could add up; this isn't common (and doesn't sound likely in your case), but it can be improved by reorganizing your module directories so that the things you're loading show up earlier in #INC.
The potentially-major performance hit referred to by earlier answers is the cost of compiling the code in the files you require. Getting rid of the require by moving the code into your main program will not help with this, as the code will still need to be compiled. In your case, it would probably make things worse, as it would cause the code for all options to be compiled on every one rather than only compiling the code used by the one action selected by the user.
As has been said, it really depends on the actual code in those files. Your best bet would be to do tests using Devel::NYTProf and/or Benchmark to see where the most time is being spent in your code if you are unhappy with its performance.
You can also read Profiling Perl on perl.com, but it is a bit outdated as it uses Devel::DProf.
Not answer to your primary question, but still a good idea for code refactor i read recently in Ovid blog.
The first time, possibly expensive; Perl has to search a path to find the file and load it up. Subsequent times, it's cheap -- a table is consulted and the file isn't actually loaded a second time. If this is in a CGI that is run once per request and then exited, then this is not too good.
It's really going to depend on the size of the files you're calling to. If you have massive CGI files, then it might detriment the performance of your software. If we're talking 6 or 7 lines of code each, then no issue. Try benchmarking your program's performance with and without, and make your own judgement.
SOLVED see Edit 2
Hello,
I've been writing a Perl program to handle automatic upgrading of local (proprietary) programs (for the company I work for).
Basically, it runs via cron, and unfortunately has a memory leak (or something similar). The problem is that the leak only happens when I'm not looking (aka when run via cron, not via command line).
My code does not contain any circular (or other) references, so the commonly cited tools will not help me (Devel::Cycle, Devel::Peek).
How would I go about figuring out what is using so much memory that the kernel kills it?
Basically, the code SFTPs into a server (using ```sftp...`` `), calls OpenSSL to verify the file, and then SFTPs more if more files are needed, and installs them (untars them).
I have seen delays (~15 sec) before the first SFTP session, but it has never used so much memory as to be killed (in my presence).
If I can't sort this out, I'll need to re-write in a different language, and that will take precious time.
Edit: The following message is printed out by the kernel which led me to believe it was a memory leak:
[100023.123] Out of memory: kill process 9568 (update.pl) score 325406 or a child
[100023.123] Killed Process 9568 (update.pl)
I don't believe it is an issue with cron because of the stalling (for ~15 sec, sometimes) when running it via the command-line. Also, there are no environmental variables used (at least by what I've written, maybe underlying things do?)
Edit 2: I found the issue myself, with help from the below comment by mobrule (in response to this question). It turns out that the script was called from a crontab of a user (non-root) just once a day and that (non-root privs) caused a special infinite loop situation.
Sorry guys, I feel kinda stupid for not finding this before, but thanks.
mobrule, if you submit your comment as an answer, I will accept it as it lead to me finding the problem.
End Edits
Thanks,
Brian
P.S. I may be able to post small snippets of code, but not the whole thing due to company policy.
You could try using Devel::Size to profile some of your objects. e.g. in the main:: scope (the .pl file itself), do something like this:
use Devel::Size qw(total_size);
foreach my $varname (qw(varname1 varname2 ))
{
print "size used for variable $varname: " . total_size($$varname) . "\n";
}
Compare the actual size used to what you think is a reasonable value for each object. Something suspicious might pop out immediately (e.g. a cache that is massively bloated beyond anything that sounds reasonable).
Other things to try:
Eliminate bits of functionality one at a time to see if suddenly things get a lot better; I'd start with the use of any external libraries
Is the bad behaviour localized to just one particular machine, or one particular operating system? Move the program to other systems to see how its behaviour changes.
(In a separate installation) try upgrading to the latest Perl (5.10.1), and also upgrade all your CPAN modules
How do you know that it's a memory leak? I can think of many other reasons why the OS would kill a program.
The first question I would ask is "Does this program always work correctly from the command line?". If the answer is "No" then I'd fix these issues first.
On the other hand if the answer is "Yes", I would investigate all the differences between having the program executed under cron and from the command line to find out why it is misbehaving.
If it is run by cron, that shouldn't it die after iteration? If that is the case, hard for me to see how a memory leak would be a big deal...
Are you sure it is the script itself, and not the child processes that are using the memory? Perhaps it ends up creating a real lot of ssh sessions , instead of doing a bunch of stuff in one session?