Powershell memory usage - expensive? - powershell

I am new to powershell but has written up a few scripts running on a windows2003 server. It's definitely more powerful than cmd scripting (maybe due to me having a programming background). However, when I delve further, I noticed that:
Each script launched will run under 1 powershell process, ie.
you see a new powershell process for each script.
the scripts I tested for memory are really simple, say, build a
string or query an environment variable, then Start-Sleep for 60
sec, So nothing needy (as to memory usage). But each process takes
around >30MB. Call me stingy, but as there are memory-intensive
applications scheduled to run everyday, and if I need to schedule a
few powershell scripts to run regularly and maybe some script
running continuously as a service, I'd certainly try to keep memory
consumption as low as possible. <-- This is because we recently
experienced a large application failure due to lack of memory.
I have not touched on C# yet, but would anyone reckon that it sometimes may be better to write the task in C#?
Meanwhile, I've seen posts regarding memory leak in powershell. Am I right to think that the memory created by the script will be withing the process space of powershell, so that when the script terminates hence powershell terminates, the memory created get cleared?

My PowerShell.exe 2.0 by itself (not running a script) is ~30MB on XP. This shouldn't worry you much with the average memory per machine these days. Regarding memory leaks, there have been cases where people use 3rd party libraries that have memory leaks when objects arn't properly disposed of. To address those you have to manually invoke the garbage collectorusing [gc]::Collect(), but this is rare. Other times i've seen people use Get-Content to read a very large file and assign it to a variable before using it. This will take alot of memory as well. In that case you can use the pipeline to read the file portions at a time to reduce your memory footprint.

1 - Yes, a new process is created. The same is true when running a cmd script, vb script, or C# compiled executable.
2 - Loading the powershell host and runtime will take some non-trivial amount of memory, which will vary from system to system and version to version. It will generally be a heavier-weight process than a cmd shell or a dedicated C# exe. For those MB, you are getting the rich runtime and library support that makes Powershell so powerful.
General comments:
The OS allocates memory per-process. Once a process terminates, all of its memory is reclaimed. This is the general design of any modern OS, and is not specific to Powershell or even Windows.
If your team is running business-critical applications on hardware such that a handful of 30MB processes can cause a catastrophic failure, you have bigger problems. Opening a browser and going to Facebook will eat more memory than that.
In the time it takes you to figure out some arcane batch script solution, you could probably create a better solution in Powershell, and your company could afford new dedicated hardware with the savings in billable hours :-)
You should use the tool which is most appropriate for the job. Powershell is often the right tool, but not always. It's great for automating administrative tasks in a Windows environment (file processing, working with AD, scheduled tasks, setting permissions, etc, etc). It's less great for high-performance, heavily algorithmic tasks, or for complex coding against raw .NET APIs. For these tasks, C# would make more sense.
Powershell has huge backing/support from Microsoft (and a big user community!), and it's been made very clear that it is the preferred scripting environment for Windows going forward. All new server-side tech for Windows has powershell support. If you are working in admin/IT, it would be a wise investment to build up some skills in Powershell. I would never discourage someone from learning C#, but if your role is more IT than dev then Powershell will be the right tool much more often, and your colleagues are more likely to also understand it.

Powershell requires (much) more resources (RAM) than cmd so if all you need is something quick and simple, it makes more sense to use cmd.
CMD uses native Win32 calls and Powershell uses the .Net framework. Powershell takes longer to load, and can consume a lot more RAM than CMD.
"I monitored a Powershell session executing Get-ChildItem. It grew to
2.5GB (all of it private memory) after a few minutes and was no way nearly finished. CMD “dir /o-d” with a small scrollback buffer
finished in about 2 minutes, and never took more than 300MB of
memory."
https://qr.ae/pGmwoe

Related

PowerShell Memory Leak

I have reopened this question because, unfortunately, I was unable to solve the problem. My script runs in the background continuously, but after minutes to hours it uses a lot more memory than after the start. The script is 3000 lines long, uses different runspaces and native .net functions. I have tried to find the memory leak by commenting out the functions, without success. It is probably a combination of the different functions. I think I really need a way to analyze the memory at the runtime. What can I do?

exe stops execution after couple of hours

I have one exe which collect some information and once information collected saved in local machine. I have managed loop such that it will do same task for infinite time.
But exe stops execution after couple of hours (approx 5-6 hours), it neither crashed nor gives exception.
I tried to find reason in windbg but I haven't got any exception in it.
Now, Could anyone help me to detect problem?
should I go for sysinternal tool or any other, which debugger tool should I use?
A few things jump to mind that could be killing your program:
Out of memory condition
Stack overflow
Integer wrap in loop counter
Programs that run forever are notoriously difficult to write correctly, because your memory management must be perfect. Without more information though, it's impossible to answer this question.
If the executable is not yours and is Naitive C/C++ code, you may want to capture first chance exception dumps or monitor the exe using Windows debug tools (such as DebugDiag or ADPlus).
Alternatively, if you have access to the developer of the executable, they may add more tracing to the exe (ETW or otherwise) to understand the possible failure points in the code.

Perl scripts, to use forks or threads?

I am writing a couple fo scripts that go and collect data from a number of servers, the number will grow and im trynig to future proof my scripts, but im a little stuck.
so to start off with I have a script that looks up an IP in a mysql database and then connects to each server grabs some information and then puts it into the database again.
What i have been thinknig is there is a limited amount of time to do this and if i have 100 servers it will take a little bit of time to go out to each server get the information and then push it to a db. So I have thought about either using forks or threads in perl?
Which would be the prefered option in my situation? And hs anyone got any examples?
Thanks!
Edit: Ok so a bit more inforamtion needed: Im running on Linux, and what I thought was i could get the master script to collect the db information, then send off each sub process / task to connect and gather information then push teh information back to the db.
Which is best depends a lot on your needs; but for what it's worth here's my experience:
Last time I used perl's threads, I found it was actually slower and more problematic for me than forking, because:
Threads copied all data anyway, as a thread would, but did it all upfront
Threads didn't always clean up complex resources on exit; causing a slow memory leak that wasn't acceptable in what was intended to be a server
Several modules didn't handle threads cleanly, including the database module I was using which got seriously confused.
One trap to watch for is the "forks" library, which emulates "threads" but uses real forking. The problem I faced here was many of the behaviours it emulated were exactly what I was trying to get away from. I ended up using a classic old-school "fork" and using sockets to communicate where needed.
Issues with forks (the library, not the fork command):
Still confused the database system
Shared variables still very limited
Overrode the 'fork' command, resulting in unexpected behaviour elsewhere in the software
Forking is more "resource safe" (think database modules and so on) than threading, so you might want to end up on that road.
Depending on your platform of choice, on the other hand, you might want to avoid fork()-ing in Perl. Quote from perlfork(1):
Perl provides a fork() keyword that
corresponds to the Unix system call of
the same name. On most Unix-like
platforms where the fork() system call
is available, Perl's fork() simply
calls it.
On some platforms such as Windows
where the fork() system call is not
available, Perl can be built to
emulate fork() at the interpreter
level. While the emulation is
designed to be as compatible as
possible with the real fork() at the
level of the Perl program, there are
certain important differences that
stem from the fact that all the pseudo
child "processes" created this way
live in the same real process as far
as the operating system is concerned.

Reliably detect 16 bit process

Due to the need to run a 15+ year old application I wish to create a watchdog program to ensure a 16 bit application is running on a 32 bit version of Windows XP Pro and start it if necessary. Normally I'd use EnumWindows() to look for the application's window. Unfortunately, this doesn't work, or at least not reliably, for 16 bit apps.
Given that I have absolutely no control over the code in the application in question, how can I reliably detect whether or not it's running? I'm coding this in C (not C++ or C#).
You'll probably need to enumerate processes (e.g., EnumProcesses with GetModuleFilenameEx or CreateToolhelp32Snapshot with Process32First and Process32Next). If you don't find an instance of ntvdm.exe, then no 16-bit process is running, and you can stop there. If you do find an instance of ntvdm.exe, you can use VDMEnumTaskWOWEx to enumerate the 16-bit processes running in it.
Back when it was still under the original owners, I posted an article on CodeGuru demonstrating how to do all of this. It'd take a bit of work to make it compile under a modern compiler (e.g., it's old enough that it uses iostream.h instead of iostream, but the process enumeration part should still be pretty much right (though, looking at things, you'll also need to enable the SeDebugPrivilege for it to work on Windows 7).

How are Operating Systems "Made"?

Creating an OS seems like a massive project. How would anyone even get started?
For example, when I pop Ubuntu into my drive, how can my computer just run it?
(This, I guess, is what I'd really like to know.)
Or, looking at it from another angle, what is the least amount of bytes that could be on a disk and still be "run" as an OS?
(I'm sorry if this is vague. I just have no idea about this subject, so I can't be very specific. I pretend to know a fair amount about how computers work, but I'm utterly clueless about this subject.)
Well, the answer lives in books: Modern Operating Systems - Andrew S. Tanenbaum is a very good one. The cover illustration below.
The simplest yet complete operating system kernel, suitable for learning or just curiosity, is Minix.
Here you can browse the source code.
(source: cs.vu.nl)
Operating Systems is a huge topic, the best thing I can recommend you if you want to go really in depth on how a operating systems are designed and construced it's a good book:
Operating System Concepts
If you are truly curious I would direct you to Linux from Scratch as a good place to learn the complete ins and outs of an operating system and how all the pieces fit together. If that is more information than you are looking for then this Wikipedia article on operating systems might be a good place to start.
A PC knows to look at a specific sector of the disk for the startup instructions. These instructions will then tell the processor that on given processor interrupts, specific code is to be called. For example, on a periodic tick, call the scheduler code. When I get something from a device, call the device driver code.
Now how does an OS set up everything with the system? Well hardware's have API's also. They are written with the Systems programmer in mind.
I've seen a lot of bare-bones OS's and this is really the absolute core. There are many embedded home-grown OS's that that's all they do and nothing else.
Additional features, such as requiring applications to ask the operating system for memory, or requiring special privileges for certain actions, or even processes and threads themselves are really optional though implemented on most PC architectures.
The operating system is, simply, what empowers your software to manage the hardware. Clearly some OSes are more sophisticated than others.
At its very core, a computer starts executing at a fixed address, meaning that when the computer starts up, it sets the program counter to a pre-defined address, and just starts executing machine code.
In most computers, this "bootstrapping" process immediately initializes known peripherals (like, say, a disk drive). Once initialized, the bootstrap process will use some predefined sequence to leverage those peripherals. Using the disk driver again, the process might read code from the first sector of the hard drive, place it in a know space within RAM, and then jump to that address.
These predefined sequence (the start of the CPU, the loading of the disk) allows the programmers to star adding more and more code at the early parts of the CPU startup, which over time can, eventually, start up very sophisticated programs.
In the modern world, with sophisticated peripherals, advanced CPU architectures, and vast, vast resources (GBs or RAM, TB of Disk, and very fast CPUs), the operating system can support quite powerful abstractions for the developer (multiple processes, virtual memory, loadable drivers, etc.).
But for a simple system, with constrained resourced, you don't really need a whole lot for an "OS".
As a simple example, many small controller computers have very small "OS"es, and some may simply be considered a "monitor", offering little more than easy access to a serial port (or a terminal, or LCD display). Certainly, there's not a lot of needs for a large OS in these conditions.
But also consider something like a classic Forth system. Here, you have a system with an "OS", that gives you disk I/O, console I/O, memory management, plus the actual programming language as well as an assembler, and this fits in less than 8K of memory on an 8-Bit machine.
or the old days of CP/M with its BIOS and BDOS.
CP/M is a good example of where a simple OS works well as a abstraction layer to allow portable programs to run on a vast array of hardware, but even then the system took less than 8K of RAM to start up and run.
A far cry from the MBs of memory used by modern OSes. But, to be fair, we HAVE MBs of memory, and our lives are MUCH MUCH simpler (mostly), and far more functional, because of it.
Writing an OS is fun because it's interesting to make the HARDWARE print "Hello World" shoving data 1 byte at a time out some obscure I/O port, or stuffing it in to some magic memory address.
Get a x86 emulator and party down getting a boot sector to say your name. It's a giggly treat.
Basically... your computer can just run the disk because:
The BIOS includes that disk device in the boot order.
At boot, the BIOS scans all bootable devices in order, like the floppy drive, the harddrive, and the CD ROM. Each device accesses its media and checks a hard-coded location (typically a sector, on a disk or cd device) for a fingerprint that identifies the media, and lists the location to jump to on the disk (or media) where instructions start. The BIOS tells the device to move its head (or whatever) to the specified location on the media, and read a big chunk of instructions. The BIOS hands those instructions off to the CPU.
The CPU executes these instructions. In your case, these instructions are going to start up the Ubuntu OS. They could just as well be instructions to halt, or to add 10+20, etc.
Typically, an OS will start off by taking a large chunk of memory (again, directly from the CPU, since library commands like 'GlobalAlloc' etc aren't available as they're provided by the yet-to-be-loaded-OS) and starts creating structures for the OS itself.
An OS provides a bunch of 'features' for applications: memory management, file system, input/output, task scheduling, networking, graphics management, access to printers, and so on. That's what it's doing before you 'get control' : creating/starting all the services so later applications can run together, not stomp on each other's memory, and have a nice API to the OS provided services.
Each 'feature' provide by the OS is a large topic. An OS provides them all so applications just have to worry about calling the right OS library, and the OS manages situations like if two programs try to print at the same time.
For instance, without the OS, each application would have to deal with a situation where another program is trying to print, and 'do something' like print anyway, or cancel the other job, etc. Instead, only the OS has to deal with it, applications just say to the OS 'print this stuff' and the OS ensure one app prints, and all other apps just have to wait until the first one finishes or the user cancels it.
The least amount of bytes to be an OS doesn't really make sense, as an "OS" could imply many, or very few, features. If all you wanted was to execute a program from a CD, that would be very very few bytes. However, that's not an OS. An OS's job is to provide services (I've been calling them features) to allow lots of other programs to run, and to manage access to those services for the programs. That's hard, and the more shared resources you add (networks, and wifi, and CD burners, and joysticks, and iSight video, and dual monitors, etc, etc) the harder it gets.
http://en.wikipedia.org/wiki/Linux_startup_process you are probably looking for this.
http://en.wikipedia.org/wiki/Windows_NT_startup_process or this.
One of the most recent operating system projects I've seen that has a serious backing has been a MS Research project called Singularity, which is written entirely in C#.NET from scratch.
To get an idea how much work it takes, there are 2 core devs but they have up to a dozen interns at any given time, and it still took them two years before they could even get the OS to a point where it would bootup and display BMP images (it's how they use to do their presentations). It took much more work before they could even get to a point where there was a command line (like about 4yrs).
Basically, there are many arguments about what an OS actually is. If you got everyone agreed on what an OS specifically is (is it just the kernel? everything that runs in kernel mode? is the shell part of OS? is X part of OS? is Web browser a part of OS?), your question is answered! Otherwise, there's no specific answer to your question.
Oh, this is a fun one. I've done the whole thing at one point or another, and been there through a large part of the evolution.
In general, you start writing a new OS by starting small. The simplest thing is a bootstrap loader, which is a small chunk of code that pulls a chunk of code in and runs it. Once upon a time, with the Nova or PDP computers, you could enter the bootstrap loader through the front panel: you entered the instructions hex number by hex number. The boot loader than reads some medium into memory, and set the program counter to the start address of that code.
That chunk of code usualy loads something else, but it doesn't have to: you can write a program that's meant to run on the bare metal. That sort of program does something useful on its own.
A real operating system is bigger, and has more pieces. you need to load programs, put them in memory, and run them; you need to provide code to run the IO devices; as it gets bigger, you need to manage memory.
If you want to really learn how it works, find Doug Comer's Xinu books, and Andy Tannenbaum's newest operating system book on Minix.
You might want to get the book The Design and Implementation of the FreeBSD Operating system for a very detailed answer. You can get it from Amazon or this link to FreeBSD.org's site looks like the book as I remember it: link text
Try How Computers Boot Up, The Kernel Boot Process and other related articles from the same blog for a short overview of what a computer does when it boots.
What a computer does when its start is heavily dependent (maybe obvious?) on the CPU design and other "low-level stuff"; therefore it's kind of difficult to anticipate what your computer does when it boots.
I can't believe this hasn't been mentioned... but a classic book for an overview of operating system design is Operating Systems - Design and Implementation written by Andrew S Tanenbaum, the creator of MINIX. A lot of the examples in the book are geared directly towards MINIX as well.
If you would like to learn a bit more, OS Dev is a great place to start. Especially the wiki. This site is full of information as well as developers who write personal operating systems for a small project/hobby. It's a great learning resource too, as there are many people in the same boat as you on OSDev who want to learn what goes into an OS. You might end up trying it yourself eventually, too!
the operating system (OS) is the layer of software that controls the hardware. The simpler the hardware, the simpler the OS, and vice-versa ;-)
if the early days of microcomputers, you could fit the OS into a 16K ROM and hard-wire the motherboard to start executing machine code instructions at the start of the ROM address space. This 'bootstrap' process would then load the code for the drivers for the other devices like the keyboard, monitor, floppy drive, etc., and within a few seconds your machine would be booted and ready for use.
Nowadays... same principle, but a lot more and more complex hardware ;-)
Well you have something linking the startup of the chip to a "bios", then to a OS, that is usually a very complicated task done by a lot of services of code.
If you REALY want to know more about this i would recomend reading a book... about microcontrllers, especially one where you create a small OS in c for a 8051 or the like.. or learn some x86 assembly and create a very small "bootloader OS".
You might want to check out this question.
An OS is a program, just like any other application you write. The main purpose of this program is that it allows you to run other programs. Modern OSes take advantage of modern hardware to ensure that programs do not clash with one another.
If you are interested in writing your own OS, check out my own question here:
How to get started in operating system development
You ask how few bytes could you put on disk and still run as an OS? The answer depends on what you expect of your OS, but the smallest useful OS that I know of fits in 1.7 Megabytes. It is Tom's Root Boot disk and it is a very nice if small OS with "rescue" applications that fits on one floppy disk. Back in the days when every machine had a floppy drive and not every machine had a CD-ROM drive, I used to use it frequently.
My take on it is that it is like your own life. AT first, you know very little - just enough to get along. This is similar to what the BIOS provides - it knows enough to look for a disk drive and read information off of it. Then you learn a little bit more when you go to elementary school. This is like the boot sector being read into memory and being given control. Then you go to high school, which is like the OS kernel loading. Then you go to college (drivers and other applications.) Of course, this is the point at which you are liable to CRASH. HE HE.
Bottom line is that layers of more and more capability are slowly loaded on. There's nothing magic about an OS.
Reading through here will give you an idea of what it took to create Linux
https://netfiles.uiuc.edu/rhasan/linux/
Another really small operating system that fits on one disk is QNX (when I last looked at it a long time ago, the whole OS, with GUI interface, web browser, disk access and a built in web server, fit on one floppy drive).
I haven't heard too much about it since then, but it is a real time OS so it is designed to be very fast.
Actually, some people visit a 4-year college to get a rough idea on this..
At its core, OS is extremely simple. Here's the beginner's guide to WHAT successful OS are made to do:
1. Manage CPU using scheduler which decides which process (program's running instance) to be scheduled.
2. Manage memory to decide which all processes use it for storing instruction(code) and data(variables).
3. Manage I/O interfaces such as disk drives, alarms, keyboard, mouse.
Now, above 3 requirements give rise to need for processes to communicate(and not fight!), to interact with outside world, help applications to do what they want to do.
To dig deeper into HOW it does that, read Dinosaur book :)
So, you can make OS as small as you want to as long as you manage to handle all hardware resources.
When you bootup, BIOS tells CPU to start reading bootloader(which loads first function of OS which resides at fixed address in memory--something like main() of small C program). Then this creates functions and processes and threads and starts the big bang!
Firstly, reading reading and reading about, what is OS; then what are the uses/ Types/ nature / objective/ needs/ of the different OS's.
Some of the links are as follows; newbie will enjoy these links:
Modern OS - this gives Idea about general OS.
Start of OS - this gives basics of what it really takes to MAKE OS, how we can make it and how one can modify a present open source code of OS by himself.
Wiki OS - Gives idea about the different Os's used in different fields and uses of it(Objects / features of OS.)
Let's see in general what OS contains (Not the sophisticatedLinux or Windows)
OS need a CPU and to dump a code in it you need a bootloader.
OS must be have the objectives to fullfill and those objectives mustbe defined in a wrapper which is called Kernel
Inside you could have scheduling time and ISR's (Depends on the objective and OS you need to make)
OS development is complicated. There are some websites like osdev or lowlevel.eu (german) dedicated to the topic. There are also some books, that others have mentioned already.
I can't help but also reference the "Write your own operating system" video series on youtube, as I'm the one who made it :-)
See https://www.youtube.com/playlist?list=PLHh55M_Kq4OApWScZyPl5HhgsTJS9MZ6M