Consider the following fork bomb in Python (source):
import os
while 1:
os.fork()
I'm too afraid to test it out myself, but I'm somewhat skeptical that if I just took this program and ran it my computer would just freeze up and die. Assuming this is true, my question is -- what mechanisms or policies is my operating system using to fight it off?
My question can be viewed as sort of an "application" problem to what one might learn in an OS class.
As expected, when I tried it out on my machine, the computer froze and I had to hard reboot. So definitely don't do this on a regular basis.
The last error that I was able to capture from the program was:
BlockingIOError: [Errno 11] Resource temporarily unavailable
File "fork_bomb.py", line 3, in <module>
os.fork()
So at some point, the OS couldn't handle the OS fork calls and returned an error. The only other useful message I can see from /var/log/syslog is
cgroup: fork rejected by pids controller in /user.slice/user-1000.slice/session-2.scope
Cgroups are a way to restrict resources from processes within a particular cgroup. So presumably, the python processes were in a cgroup that had reached its pid/task limit. So that's one way the OS tries to deal with fork bombs, is limiting tasks using cgroups. Of course, the infinite loop of forks, even if the forks were failing, still required overhead from requesting resources from the OS, hence the system freeze.
Theoretically, another way the OS can try to limit fork bombs is through memory limits. Ignoring copy-on-write, if all the forked processes required extra memory, the Linux OOM (out of memory) killer will be called. This kernel process will be awakened when memory is tight and then its job is to start killing processes that it thinks will help free up sufficient memory to keep the system running. Memory limits can be set using cgroups or by setting the minimum free memory using /proc/sys/vm/min_free_kbytes.
Related
The scheduler I have been working on for my OS class has been getting a "*** stack smashing detected ***" error on the VM I'm using (I'm using Vagrant with virtualbox). This error occurs roughly 50% of the time I run the program.
When switching to the VM cluster provided by our professor (connected using SSH on the aforementioned VM), the error never showed up.
My first instinct was that my local VM didn't have enough memory allocated to it and that somehow the code I was running was going out of bounds of where my VM could access. (the test involved performing 128 matrix multiplications of varying sizes each in its own thread)
Can anyone confirm if this is a feasible explanation? My fear is the error is just being ignored on the other VM (I use the same makefile for both that compiles with flags -g and -lm).
Thanks!
Stack smashing detected is caused when your program overwrites "canary" memory that is above the area where its local variables are located. It's usually due to writing more elements of a local array than were allocated for it. A bug-free program should never do this on any machine, no matter how much or how little memory is available. So your program is buggy and needs to be fixed.
In particular, this error is not caused by simply running out of stack space.
Most likely the other VM has its compiler configured to disable this check by default. You may be able to re-enable it with -fstack-protector. But either way, you should investigate and fix this bug on whichever machine lets you reproduce it.
This is a common error message and there are many general answers that have not worked for me.
I think I have isolated this particular problem to the PostgreSQL data directory being symlinked to an external hard drive.
FATAL: could not create shared memory segment: No space left on device
DETAIL: Failed system call was shmget(key=5432001, size=56, 03600).
HINT: This error does *not* mean that you have run out of disk space. It occurs either if all available shared memory IDs have been taken, in which case you need to raise the SHMMNI parameter in your kernel, or because the system's overall limit for shared memory has been reached.
$ sysctl -a | grep sysv
kern.sysv.shmmax: 412316860416
kern.sysv.shmmin: 8
kern.sysv.shmmni: 64
kern.sysv.shmseg: 128
kern.sysv.shmall: 100663296
$ sudo cat /etc/sysctl.conf
kern.sysv.shmmax=412316860416
kern.sysv.shmmin=8
kern.sysv.shmmni=64
kern.sysv.shmseg=128
kern.sysv.shmall=100663296
PostgreSQL version 9.4.15. From my PostgreSQL config
shared_buffers = 128MB
Don't know what other settings would be relevant.
Other environment details:
The external hard drive with the data directory is at only 50% capacity. My RAM usage when this happens is ~60% capacity.
I have not been able to determine an exact set of steps that reproduces the bug. I have an external hard drive with a PostgreSQL data directory and a local folder with another data directory. In my project, I'll symlink to one or the other depending on which copy of data I want to use. As far as I have noticed, the problem only appears when I've been working off the symlinked hard drive and when I unplug it without stopping the server and then plug it back in. But it doesn't happen every time when I perform those steps.
I don't expect anyone to be able to point to the specific problem given the above description.
But how can I get more useful information next time I'm in a bugged state? Are there any system commands that would help identify the exact problem?
...It occurs either if all available shared memory IDs have been taken, in which case you need to raise the SHMMNI parameter in your kernel, or because the system's overall limit for shared memory has been reached.
How can I check if if all available shared memory IDs have been taken or if the system's overall limit for shared memory has been reached and what do I do with the answer?
From the bind9 man page, I understand that the named process starts one worker thread per CPU if it was able to determine the number of CPUs. If its unable to determine, a single worker thread is started.
My question is how does it calculate the number of CPUs? I presume by CPU, it means cores. The Linux machine I work is customized and has kernel 2.6.34 and does not support lscpu or nproc utilities. named is starting a single thread even if i give -n 4 option. Is there any other way to force named to start multiple threads?
Thanks in advance.
In FreeBSD kernel, how can I first stop all the cores, and then run my code (can be a kernel module) on all the cores? Also, when finished, I can let them restore contexts and continue executing.
Linux has APIs like this, I believe FreeBSD also has a set of APIs to do this.
edit:
Most likely I did not clarify what I want to do. First, the machine is x86_64 SMP.
I set a timer, when the time is over; to stop all the threads (including kernel threads) on all cores; save context; run my code on one core to do some kernel stuff; when finished, restore the context and let them continue running; periodically. The other kernel threads and processes are not affected (without changing their relative priority).
I assume that your "code" (the kernel module) actually takes advantage of SMP inherently already.
So, one approach you can do is:
Set the affinity of all your processes/threads to your desired cpus (sched_setaffinity)
Set each of your threads to use Real-Time (RT) scheduling.
If it is a kernel module, you can do this manually in your module (I believe), by changing the scheduling policy for your task_struct to SCHED_RR (or SCHED_FIFO) after pinning each process to a core.
In userspace, you can use the FreeBSD rtprio command (http://www.freebsd.org/cgi/man.cgi?query=rtprio&sektion=1):
rtprio, idprio -- execute, examine or modify a utility's or process's
realtime or idletime scheduling priority
The effect will be: Your code will run first before any other non-essential process in the system, until your code finishes.
I am hosting IIS based web service applications on Windows 2008 64-bit system running on a Quad core 8G machine. Ran into couple of instances when W3WP was running at 7.6G of memory usage. Nothing else was responding on the system including RDP. Right click on the process from the task manager and creating the dumps, froze the system and all its threads for a long time (close to 30minutes). When the freeze up occurred during off hours, we let the dump run for a while (ran close to 1 hour) but still dump didn't complete. In the interest of getting the system up, we had to kill IIS
Tried other tools like procexp, debug diag etc to create full memory dump and all have the same results
So, what tool does the community use to grab dump files quickly? Or without freezing all the threads? I realize latter might be a rhetorical question. But what are the options for generating such a large dump file without locking up the system for a long time?
IMO you shouldn't have to wait until the process memory grows to 8 GB. I am sure with something like 3 - 4 GB you should be able to detect the memory leak.
Procdump has an option based on memory threshold
-m Memory commit threshold in MB at which to create a dump of the process.
I would you this option to dump the memory of the process.
And also SSD would help in writing faster.
WPA a.k.a xperf (http://msdn.microsoft.com/en-us/performance/cc825801.aspx) is a powerfull tool, to diagnose the applications. You will get call stack of the culprit allocation. You dont have to collect the dump and it is no-invasive and does not load much in production systems
Complete step by step information is available here. http://msdn.microsoft.com/en-us/library/ff190906(v=VS.85).aspx.