I have reopened this question because, unfortunately, I was unable to solve the problem. My script runs in the background continuously, but after minutes to hours it uses a lot more memory than after the start. The script is 3000 lines long, uses different runspaces and native .net functions. I have tried to find the memory leak by commenting out the functions, without success. It is probably a combination of the different functions. I think I really need a way to analyze the memory at the runtime. What can I do?
Related
I am executing one java program via eclipse and I was executing the exact same program yesterday and my program execution was only taking 10 min yesterday, today the same program is taking more than an hour and I did not change any single thing in my code. could you plwase give me a solution to revert back to the old duration of my program execution that I had yesterday
If you did not change anything in your sourcecode, I see the following possible reasons for this:
Side-effects on the machine you are running the program on, like other (maybe hidden) processes soak up cpu time and slow down your program.
This could also be the machine itself being slower (slowdown from to much heat, etc.)
Your code is doing some "random" things that require longer runs sometimes (sounds unlikely tho)
Somehow eclipse is causing an issue (try to run your program without it)
Your java runtime might cause a problem (sounds unlikely aswell, but maybe updating it to the newest version can help)
I have one exe which collect some information and once information collected saved in local machine. I have managed loop such that it will do same task for infinite time.
But exe stops execution after couple of hours (approx 5-6 hours), it neither crashed nor gives exception.
I tried to find reason in windbg but I haven't got any exception in it.
Now, Could anyone help me to detect problem?
should I go for sysinternal tool or any other, which debugger tool should I use?
A few things jump to mind that could be killing your program:
Out of memory condition
Stack overflow
Integer wrap in loop counter
Programs that run forever are notoriously difficult to write correctly, because your memory management must be perfect. Without more information though, it's impossible to answer this question.
If the executable is not yours and is Naitive C/C++ code, you may want to capture first chance exception dumps or monitor the exe using Windows debug tools (such as DebugDiag or ADPlus).
Alternatively, if you have access to the developer of the executable, they may add more tracing to the exe (ETW or otherwise) to understand the possible failure points in the code.
I am new to powershell but has written up a few scripts running on a windows2003 server. It's definitely more powerful than cmd scripting (maybe due to me having a programming background). However, when I delve further, I noticed that:
Each script launched will run under 1 powershell process, ie.
you see a new powershell process for each script.
the scripts I tested for memory are really simple, say, build a
string or query an environment variable, then Start-Sleep for 60
sec, So nothing needy (as to memory usage). But each process takes
around >30MB. Call me stingy, but as there are memory-intensive
applications scheduled to run everyday, and if I need to schedule a
few powershell scripts to run regularly and maybe some script
running continuously as a service, I'd certainly try to keep memory
consumption as low as possible. <-- This is because we recently
experienced a large application failure due to lack of memory.
I have not touched on C# yet, but would anyone reckon that it sometimes may be better to write the task in C#?
Meanwhile, I've seen posts regarding memory leak in powershell. Am I right to think that the memory created by the script will be withing the process space of powershell, so that when the script terminates hence powershell terminates, the memory created get cleared?
My PowerShell.exe 2.0 by itself (not running a script) is ~30MB on XP. This shouldn't worry you much with the average memory per machine these days. Regarding memory leaks, there have been cases where people use 3rd party libraries that have memory leaks when objects arn't properly disposed of. To address those you have to manually invoke the garbage collectorusing [gc]::Collect(), but this is rare. Other times i've seen people use Get-Content to read a very large file and assign it to a variable before using it. This will take alot of memory as well. In that case you can use the pipeline to read the file portions at a time to reduce your memory footprint.
1 - Yes, a new process is created. The same is true when running a cmd script, vb script, or C# compiled executable.
2 - Loading the powershell host and runtime will take some non-trivial amount of memory, which will vary from system to system and version to version. It will generally be a heavier-weight process than a cmd shell or a dedicated C# exe. For those MB, you are getting the rich runtime and library support that makes Powershell so powerful.
General comments:
The OS allocates memory per-process. Once a process terminates, all of its memory is reclaimed. This is the general design of any modern OS, and is not specific to Powershell or even Windows.
If your team is running business-critical applications on hardware such that a handful of 30MB processes can cause a catastrophic failure, you have bigger problems. Opening a browser and going to Facebook will eat more memory than that.
In the time it takes you to figure out some arcane batch script solution, you could probably create a better solution in Powershell, and your company could afford new dedicated hardware with the savings in billable hours :-)
You should use the tool which is most appropriate for the job. Powershell is often the right tool, but not always. It's great for automating administrative tasks in a Windows environment (file processing, working with AD, scheduled tasks, setting permissions, etc, etc). It's less great for high-performance, heavily algorithmic tasks, or for complex coding against raw .NET APIs. For these tasks, C# would make more sense.
Powershell has huge backing/support from Microsoft (and a big user community!), and it's been made very clear that it is the preferred scripting environment for Windows going forward. All new server-side tech for Windows has powershell support. If you are working in admin/IT, it would be a wise investment to build up some skills in Powershell. I would never discourage someone from learning C#, but if your role is more IT than dev then Powershell will be the right tool much more often, and your colleagues are more likely to also understand it.
Powershell requires (much) more resources (RAM) than cmd so if all you need is something quick and simple, it makes more sense to use cmd.
CMD uses native Win32 calls and Powershell uses the .Net framework. Powershell takes longer to load, and can consume a lot more RAM than CMD.
"I monitored a Powershell session executing Get-ChildItem. It grew to
2.5GB (all of it private memory) after a few minutes and was no way nearly finished. CMD “dir /o-d” with a small scrollback buffer
finished in about 2 minutes, and never took more than 300MB of
memory."
https://qr.ae/pGmwoe
I work on an average (~ 20k lines of code, Objective-C mixed with C++), and I am figthing to hunt down an EXC_BAD_ACCESS error.
I have tried all the common techniques (like enabling NSZombie, guard edges,etc.) So far, I have ruled out the possibility to access a released object, and the double-free error.
It seems that something writes on a memory space where it shouldn't. But, as many memory errors, it's not happening all the time, and it's not crashing always in the same place.
(Sometimes I receive the "object was modified after being freed" message).
Sometimes, the overwritten memory belongs to the allocator, and it crashes on malloc, or on free().
And, of course, some changes in the app may influence the bug's behaviour - if I try to comment out parts of the code, the error appears less often, so it's more difficult to find it.
Finally, I have been looking into using valgrind, but it seems that all those who used it worked on the simulator. but my code must run on the actual device (some code is ARM-specific)
Are there any general tips on how to debug such errors?
Note: The app involves video processing, so the amount of memory used is fairly large.
There are some special tools available on the XCode. You could try to use them in order to analyse your code.
http://developer.apple.com/library/mac/#featuredarticles/StaticAnalysis/index.html
It will produce some warning in case of invalid objects usage so it could help you to find a problem.
If you feel that the C++ code is causing the issue you could copy the C++ out of your iPhone project and create a Mac project. With this you could set up various stress tests. And, you should be able to use valgrind as well.
I am currently programming a iPhone-app for my maturity research. But there is an behavior I don't understand: Sometimes when i compile my project there is:
Thread 1: Program received signal : "EXC_BAD_ACCESS".
But when I compile the same code a second, or a third time the code just runs fine and i can't get why. I use some MonteCarloSimulation but when it fails it fails executing one of the first 100 simulations. But when every thing runs fine it executes 1000000 simulations without an error.. Really strange isn't it?
Do you have any idea? Can this be an issue of Xcode or arc?
All other things just work perfect.
Do you have to get any further information? I can also send you my code as an email.
This usually means you're trying to access an object that has already been deallocated.
In order to debug these things, Objective C uses something called "NSZombie" that will keep those objects around so you can at least see what it is that's trying to be called. See this question for some details on how to use it.
This is typically caused by accessing some memory that's been corrupted, chances are you have a reference to an object which has been deleted. A lot of the time you may find that the memory where the object was located has not yet been overwritten, so when you attempt to access that memory your data is still intact and there is no problem, hence it working some of the time.
Another scenario would be that you've got some code writing into memory using a bad reference, so you're writing into an area you shouldn't be. Depending on the memory layout when the program starts, this could have no effect some of the time but cause something catastrophic at other times.