I have a Perl script, fetch url like http://1.1.1.1/1.jpg from MySQL using DBI, and download this jpg file using LWP::Simple. It's a infinite loop.
while (1) {
my $url=&fetch_url_from_mysql;
if ($url){
&download_jpg($url);
} else {
sleep 1;
}
}
Plain simple. I suppose the memory usage would be stay in certain amount. But after one month of continuous running of this script. The memory usage is 7.5G!
How can I profile it?
For profiling, set an explitict exit. Create a counter, and exit from your program if your iteration is equal or bigger than this.
For profiling, use NYTprof:
perl -d:NYTProf script.pl
nytprofhtml
But you are dealing with a memory leak here.
Read this to find a memory leak: How can I find memory leaks in long-running Perl program?
Most probably you have a variable that will never be freed. Perl frees memory if a variable goes out of scope, but one of your variables never goes out of scope.
Use $variable=undef to free up the memory.
If you port your whole script maybe we could find a leak in it.
regards,
Related
This question is related to an answer to a former question about memory handling by Perl. I've learned that one can free memory in Perl by explicitly using the undef function on an available scalar and using Devel::Peek or Devel::Size or such one can see how many memory is allocated for a scalar. In all those cases the scalars debugged are used within their scope.
But is it possible to debug things like allocated memory outside the scope of variables, just on the level of a Perl interpreter? Something like searching for all allocated memory for all "things" that are a scalar in the current interpreter and print their associated data, like current value or such?
And if that's the case, if one does already have that information, is one even able to free the known memory? Just like calling undef on a scalar, but without the scalar, something more low level, like on those "things" output of Devel::Peek.
What I'm thinking about is having a mod_perl cleanup handler executed after a request, scanning the current mod_perl interpreter for large chunks of data and freeing them manually. Simply because I decide that large blocks of allocated data are of no use anymore, even if Perl thinks otherwise:
Finally and perhaps the biggest win is memory re-use: as calls are made into Perl subroutines, memory allocations are made for variables when they are used for the first time. Subsequent use of variables may allocate more memory, e.g. if a scalar variable needs to hold a longer string than it did before, or an array has new elements added. As an optimization, Perl hangs onto these allocations, even though their values "go out of scope".
https://perl.apache.org/docs/2.0/user/intro/overview.html#Threads_Support
I could find a lot of monitoring and debugging packages around low level memory access, but no hint yet how one could call something like the undef function on some low level Perl struct in Perl. Might simply not be possible without any XS or such...
is it possible to debug things like allocated memory outside the scope of variables
There really isn't any such memory. Any memory allocated outside of variables is surely needed. As you yourself point out, it's the memory allocated for variables that make up most "wasted" space.
but no hint yet how one could call something like the undef function on some low level Perl struct in Perl.
It's because there are no such structs.
Just like calling undef on a scalar, but without the scalar, something more low level, like on those "things" output of Devel::Peek.
Devel::Peek's only function, Dump, outputs things in variables. Like you've said, undef is what you'd want to clear these.
From the above, it's obvious you want to know how to free the memory associated with the variables in subs.
You also overlooked the fact that many operators have an associated variable (called "target") in which they return their result.
Approach 1
A simple way to clear all those variables would be to selectively clear the symbol table (%::). This would effectively "unload" every module. Be sure not clear core components (perl -E'say for sort keys %::'). And don't forget to clear %INC so the modules can be reloaded.
If clearing the symbol table is the approach you want to take, it might be less risky and time-consuming to take a snapshot of %:: early on, and restore that snapshot when it's time to clear the symbol.
Approach 2
If you didn't want to reload the modules, you could take attempt to locate every sub, and undef their vars, then undef the vars of their ops.
A sub's vars exists within its pads. Conveniently, so do opcode targets. There's a pad for each level of recursion the sub has experienced.
Given a reference to a sub, you can find the variables in a sub's pads. You can refer to PadWalker for an example of how to do this. You can't actually use PadWalker since it only returns one variable per variable name, even if there are more than one (due to more than one variable being declared with the same name, or due to recursion).
Captured variables and our variables should be left untouched. It's possible to detect whether a pad entry is one of these. (Again, refer to PadWalker.)
(Obviously, you could also looking into freeing the sub's extra pads!)
How do you find all the subs? Well, navigating the symbol table will give you most of them. Finding anon ones will be trickier.
Approach 3
The most efficient approach is to simply terminate the mod_perl thread/process. A new clean one will automatically be spawned. It's also the simplest to implement, as it's simply a configuration change (setting MaxRequestsPerChild to 1).
Another form of wasted memory is a memory leak. That's another large question, so I'm not touching it.
I think you are looking for this answer to a similar question.
Everything you really need to know you can find in the internals to the Devel::MAT::* submodules. Namely the Devel::MAT::Dumper.xs, which has the structure of the heap for perl interpreter. The module is designed to dump the heap at signal and analyze it later, but I think you can turn it into a runtime check. If you need help on reading the XS, look here.
To debug memory allocation you should recompile perl with -Accflags=-DPERL_MEM_LOG DOC
(see related question about how to recompile perl)
You maybe will be interested in MEMORY DEBUGGERS
To free perl scalar, just like when it leave her scope:
{ # Scope enter
my $x; # Memory allocation
} # << HERE $x is freed
You should just reduce its variable REFCNT to zero by SvREFCNT_dec macro DOC
To free an SV that you've created, call SvREFCNT_dec(SV*) . Normally this call is not necessary (see Reference Counts and Mortality).
Here is pseudo code:
{
$x;
call_xs_sub( $x ); # << HERE $x is freed
}
XS pseudo code:
call_xs_sub( SV *sv ) {
...
SvREFCNT_dec( sv ); # <<HERE scalar is freed
...
}
To spy every memory allocation you should walk perl arenas.
At compile time you may view every place where variable is declared and accessed with help of B::Xref module
Or run perl with -Dm option (The perl should be compiled with corresponding options. See this topic):
perl -Dm script.pl
I have a C program with perl running as a thread. I would like to restrict the perl interpreter to use memory from a chunk that I pre-allocated (about 2GB). Wonder if it's possible and how to do it.
Thanks.
I'm reasonably certain there is no way to do that in a normal Perl binary, but all perl's memory allocation code is nicely packaged in the malloc.c file in the source code. That file also has lots of comments on how Perl's memory allocation works under the hood. It's shouldn't be too hard to create a locally modified perl that does what you want, I think.
I know that Perl uses reference count based garbage collection.
When a variable goes out of scope, the reference count is decremented and if REFcount goes to 0, the memory is de-allocated.
But when I trace a small example which is shown below, I couldn't able to find the de-allocation happening.
print "start..";
sub func
{
my $length = 8*1024*1024;
my $array = [1..$length];
}
func();
print "done..";
In the example, when the program starts, Perl.exe occupies ~ 3 MB physical memory.
After allocation during the func() call, Perl.exe occupies ~ 370 MB memory.
But after the func() call , the allocated memory should be garbage collected. why is it not done?
Looking forward for your replies.
According to the question "How can I free an array or hash so my program shrinks?" in perlfaq3:
You usually can't. Memory allocated to lexicals (i.e. my() variables)
cannot be reclaimed or reused even if they go out of scope. It is
reserved in case the variables come back into scope. Memory allocated
to global variables can be reused (within your program) by using
undef() and/or delete().
On most operating systems, memory allocated to a program can never be
returned to the system. That's why long-running programs sometimes re-
exec themselves. Some operating systems (notably, systems that use
mmap(2) for allocating large chunks of memory) can reclaim memory that
is no longer used, but on such systems, perl must be configured and
compiled to use the OS's malloc, not perl's.
In general, memory allocation and de-allocation isn't something you
can or should be worrying about much in Perl.
See also How can I make my Perl program take less memory?
Perl may have marked the memory as freed, but it doesn't necessarily mean that it has been freed back to the OS. Your Perl program may reuse that memory. Try running func again. You shouldn't see an increase in the amount of memory used.
You may want to set the environment variable PERL_DESTRUCT_LEVEL and see if that makes any difference, but I doubt it.
Garbage collection is not one of Perl's greatest strengths.
While investigating a long running perl program for memory leaks I tried to use Test::LeakTrace.
Looking at one of the leaks it reports I can narrow down the leaking code to just:
/$?/
So running: perl -MTest::LeakTrace::Script -e'/$?/' prints:
leaked SCALAR(0x10d3d48) from -e line 1.
Why is this, do I need to worry about it ?
Update: Also tried Devel::LeakTrace::Fast, it's not complaining about the same code.
Assuming you got a leak. Then this:
perl -e'/$?/ for 1..1E9'
should make your process grow in memory
ps -o rss,vsz <PID>
In my case it stays stable all the way. You should check it for your setup. It could be that leak your module detects is some late destruction. You could write a note to the module authors to help you figure out its output, you can help them to improve it...
BTW another thing confirming "no leak" for me is that on
perl -MTest::LeakTrace::Script -e'/$?/ for 1..1000'
I don't see multiple leaked scalars, just one.
I'm trying to enable the garbage collector of my script to do a better job. There's a ton of memory that it should be able to reclaim, but something is stopping it.
I've used Devel::Cycle a bit and that's allowed me to get closer but I'm not quite there.
How do I find out the current reference count for a Perl hash (the storage for my objects)?
Is there a way to track who is holding a reference to an object? Perhaps a sort of Tie that says, whenever someone points are this object, remember who that someone is.
You are looking for Devel::Refcount.
If you are worried about returning unused memory to the OS, you should know that is not possible in general. The memory footprint of your Perl program will be proportional to the largest allocation during the lifetime of your program.
See How can I make my Perl program take less memory? in the Perl FAQ list as well as Mini-Tutorial: Perl's Memory Management (as pointed out by #Evan Carroll in the comments).