In some operating system,for any process there is a stack and a heap.Both grows towards each other.There must be a guard band between them to check for overlapping.Can anyone give me some illustration about it.I want to write my own function for checking stack overflow error.
In a system like that, you would normally have a guard word or something similar at the top of the heap, something like 0xa55a or 0xdeadbeef.
Then, periodically, that guard word is checked to see if it's been corrupted. If so something has overwritten the memory.
Now this may not necessarily be a stack overflow, it may be a rogue memory write. But, in both those cases, something is seriously wrong so you may as well treat them the same.
Of course, more modern operating systems may take the approach of using the assistance of the hardware such as in the Intel chips. In those, you can set up a stack segment to a specific size and, if you try to write outside of there (using the stack selector), you'll get a trap raised.
The heap in that case would be using a different selector so as to be kept separate.
Many operating systems place a guard page (or similar techniques) between stack and heap to protect against such attack vectors. I haven't seen canaries (the method mentioned by paxdiablo) there yet, they're mostly used to guard against stack-internal overflows (aka to guard the return address).
Guard pages on Windows: http://msdn.microsoft.com/en-us/library/aa366549(VS.85).aspx
Linux had an interesting exploit based on this problem some time ago though: http://www.h-online.com/open/news/item/Root-privileges-through-Linux-kernel-bug-Update-1061563.html
Related
(Title: Can calling an operating system function cause a stack overflow?)
Or is the function always guaranteed to return an error code of some kind to indicate the failure, without crashing the program in the process?
If it can indeed cause a stack overflow, how can you make sure your program never has a chance to crash?
I'm interested in the behavior under Windows, though answers about how other operating systems manage this (e.g. Linux syscalls) are informative as well.
[Edit: For clarification, I'm talking about the behavior of programs as machine code, run by the processor. No high level languages (other than, perhaps, C); I mean actual OS functionality].
I am running a recursive function in MATLAB and gets this error message:
"Maximum recursion limit of 500 reached. Use set(0,'RecursionLimit',N)
to change the limit. Be aware that exceeding your available stack space can
crash MATLAB and/or your computer."
I wonder, can it really happen that MATLAB and/or my computer can crash if I change the recursion limit and exceed my stack space? It seems strage to me. Why doesn't MATLAB just halt and quit automatically if the available stack space is exceeded?
And how do I know how big recursion limit I can choose without danger?
The computer isn't going to crash. Matlab might, that depends on their handling of the stack. The warning is there most likely because Matlab is multi-platform, and it can run on many different OSes and architectures, some of which might not be as safe as Windows or most POSIXes.
In practice, when working with a thread stack on a Windows machine, the "stack overflow" error is actually "access violation" - you try to access memory that doesn't belong to you not actually an access violation (instead, it's a special kind of page fault, basically), but the idea is similar - the OS notifies you first that you're reaching the ceiling of the stack, and if you exceed the last few "safety" pages, it will give you the actual stack overflow. This is very handy, since it's "free" - you don't have to check if you still have stack space every time you push, and it gives you some extra safety.
Depending on how Matlab handles that exception, it might gracefuly report an error and continue, or it might crash.
Of course, handling stack overflows can be quite tricky, but not for the OS. Windows can handle those errors just fine, it doesn't really care.
The recursion limit depends a lot on how much data Matlab actually has to store in each step of the recursion. The typical stack size of a Windows thread is about 256-1024 kiB (and it's configurable for threads you start on your own), which is actually quite a lot, unless you're passing a lot of big arguments. Consider that with a method that takes two integers and doesn't have any variables, you'd need about 20 000 call deep recursion to exceed even the 256 kiB stack space (on 32-bit).
However, stack overflows are usually a problem in code. You most often run into them by choosing the wrong recursion exit condition. That's part of the reason why the stack overflow is handled by an "exception", rather than allocating more memory for the stack - every trivial recursion error would start crashing all your applications and possibly even the OS. So, first make sure that you actually need a recursion that deep :)
Actually here is a very similar question presented by Mathworks.
Here is how they say you can produce a recursion problem, I think it had 32 bit in mind so you may need to increase the 5000:
create the following file:
function retVal = myrecursivefun(inVal, recursions)
recursions = recursions - 1;
inVal = inVal + 1;
if recursions > 0
retVal = myrecursivefun(inVal, recursions);
else
retVal = inVal;
end
Then run it as follows to crash MATLAB:
set(0,'RecursionLimit', 5000);
myrecursivefun(1, 5000);
Personal note: I think the default recursion limit of 500 makes sense. Matlab programmers won't often want to go beyond this and most of the time this limit is hit because of a mistake.
Also, I think matlab has a little more overhead for function calls than low level languages like C++, therefore you will usually want to avoid deep recursion in the first place.
I'm looking at a crash dump. Some variables seem perfectly viewable in windbg, while others just say "memory access error". What causes this? Why do some variables have sensical values while others simply list ?
It appears that all the problems are associated with following pointers. I'm certain that while many of these pointers are uninitialized the vast majority of them should be pointing somewhere valid. Based on the nature of this crash (a simple null ptr dereference) I'm fairly certain the whole process hasn't gone out to lunch.
Mini-dumps are fairly useless, they don't contain a snapshot of all in use memory. Instead, all they contain are some critical structures/lists (e.g. the loaded module list) and the contents of the crashing stack.
So, any pointer that you try to follow in the dump will just give you question marks. Grab a full memory dump instead and you'll be able to see what these buffers point to.
-scott
If they are local pointer variables, what is most likely happening is that the pointers are not initialized, or that stack location has been reused to contain another variable, that may not be a pointer. In both cases, the pointer value may point to a random, unreadable portion of memory.
I have a pretty weird problem in my iPhone app which is, I think, related to memory getting corrupted:
At one point, I need to sort an array, which I do with -[sortArrayUsingFunction].
The result is not correct unless I either allocate some memory with something like void *test = malloc(2 * sizeof( int )) before the method call or have, e.g., a call to NSLog() (which is never invoked) in the sorting function.
In other words: the sorting only works if I slightly increase the memory that was used before calling the sorting function. I think this is because at some point, memory gets corrupted.
How do you debug something like this?
It sounds like some of your code is using already released objects. A lot of help with debugging this kind of errors is provided in Apple’s great Mac OS X Debugging Magic tech note, especially the foundation part.
For your case I'd disable autorelease pools (setting the environment variable NSEnableAutoreleasePool=NO) or use the zombie feature (NSZombieEnabled=YES) to find places where you send messages to released objects.
Try running your program in the simulator under Valgrind:
http://valgrind.org/
And how to use it under the simulator:
http://landonf.bikemonkey.org/code/iphone/iPhone_Simulator_Valgrind.20081224.html
You may have to change the VALGRIND path in the code example depending on where it gets installed.
Such things can be a challenge to debug. There are some tools for detecting out-of-bounds accesses and such on other platforms, so I presume there would be something for the iPhone, however I don't know of any.
Perhaps you should store two copies of the array, and compare them for differences. Print out the differences. The nature of the "junk" that was introduced to one of the arrays might give a hint as to where it came from.
Also just go through the code that runs before this point, and re-read it (or better yet, get someone else to read it). You might spot a bug.
I'm developing an operating system and rather than programming the kernel, I'm designing the kernel. This operating system is targeted at the x86 architecture and my target is for modern computers. The estimated number of required RAM is 256Mb or more.
What is a good size to make the stack for each thread run on the system? Should I try to design the system in such a way that the stack can be extended automatically if the maximum length is reached?
I think if I remember correctly that a page in RAM is 4k or 4096 bytes and that just doesn't seem like a lot to me. I can definitely see times, especially when using lots of recursion, that I would want to have more than 1000 integars in RAM at once. Now, the real solution would be to have the program doing this by using malloc and manage its own memory resources, but really I would like to know the user opinion on this.
Is 4k big enough for a stack with modern computer programs? Should the stack be bigger than that? Should the stack be auto-expanding to accommodate any types of sizes? I'm interested in this both from a practical developer's standpoint and a security standpoint.
Is 4k too big for a stack? Considering normal program execution, especially from the point of view of classes in C++, I notice that good source code tends to malloc/new the data it needs when classes are created, to minimize the data being thrown around in a function call.
What I haven't even gotten into is the size of the processor's cache memory. Ideally, I think the stack would reside in the cache to speed things up and I'm not sure if I need to achieve this, or if the processor can handle it for me. I was just planning on using regular boring old RAM for testing purposes. I can't decide. What are the options?
Stack size depends on what your threads are doing. My advice:
make the stack size a parameter at thread creation time (different threads will do different things, and hence will need different stack sizes)
provide a reasonable default for those who don't want to be bothered with specifying a stack size (4K appeals to the control freak in me, as it will cause the stack-profligate to, er, get the signal pretty quickly)
consider how you will detect and deal with stack overflow. Detection can be tricky. You can put guard pages--empty--at the ends of your stack, and that will generally work. But you are relying on the behavior of the Bad Thread not to leap over that moat and start polluting what lays beyond. Generally that won't happen...but then, that's what makes the really tough bugs tough. An airtight mechanism involves hacking your compiler to generate stack checking code. As for dealing with a stack overflow, you will need a dedicated stack somewhere else on which the offending thread (or its guardian angel, whoever you decide that is--you're the OS designer, after all) will run.
I would strongly recommend marking the ends of your stack with a distinctive pattern, so that when your threads run over the ends (and they always do), you can at least go in post-mortem and see that something did in fact run off its stack. A page of 0xDEADBEEF or something like that is handy.
By the way, x86 page sizes are generally 4k, but they do not have to be. You can go with a 64k size or even larger. The usual reason for larger pages is to avoid TLB misses. Again, I would make it a kernel configuration or run-time parameter.
Search for KERNEL_STACK_SIZE in linux kernel source code and you will find that it is very much architecture dependent - PAGE_SIZE, or 2*PAGE_SIZE etc (below is just some results - many intermediate output are deleted).
./arch/cris/include/asm/processor.h:
#define KERNEL_STACK_SIZE PAGE_SIZE
./arch/ia64/include/asm/ptrace.h:
# define KERNEL_STACK_SIZE_ORDER 3
# define KERNEL_STACK_SIZE_ORDER 2
# define KERNEL_STACK_SIZE_ORDER 1
# define KERNEL_STACK_SIZE_ORDER 0
#define IA64_STK_OFFSET ((1 << KERNEL_STACK_SIZE_ORDER)*PAGE_SIZE)
#define KERNEL_STACK_SIZE IA64_STK_OFFSET
./arch/ia64/include/asm/mca.h:
u64 mca_stack[KERNEL_STACK_SIZE/8];
u64 init_stack[KERNEL_STACK_SIZE/8];
./arch/ia64/include/asm/thread_info.h:
#define THREAD_SIZE KERNEL_STACK_SIZE
./arch/ia64/include/asm/mca_asm.h:
#define MCA_PT_REGS_OFFSET ALIGN16(KERNEL_STACK_SIZE-IA64_PT_REGS_SIZE)
./arch/parisc/include/asm/processor.h:
#define KERNEL_STACK_SIZE (4*PAGE_SIZE)
./arch/xtensa/include/asm/ptrace.h:
#define KERNEL_STACK_SIZE (2 * PAGE_SIZE)
./arch/microblaze/include/asm/processor.h:
# define KERNEL_STACK_SIZE 0x2000
I'll throw my two cents in to get the ball rolling:
I'm not sure what a "typical" stack size would be. I would guess maybe 8 KB per thread, and if a thread exceeds this amount, just throw an exception. However, according to this, Windows has a default reserved stack size of 1MB per thread, but it isn't committed all at once (pages are committed as they are needed). Additionally, you can request a different stack size for a given EXE at compile-time with a compiler directive. Not sure what Linux does, but I've seen references to 4 KB stacks (although I think this can be changed when you compile the kernel and I'm not sure what the default stack size is...)
This ties in with the first point. You probably want a fixed limit on how much stack each thread can get. Thus, you probably don't want to automatically allocate more stack space every time a thread exceeds its current stack space, because a buggy program that gets stuck in an infinite recursion is going to eat up all available memory.
If you are using virtual memory, you do want to make the stack growable. Forcing static allocation of stack sized, like is common in user-level threading like Qthreads and Windows Fibers is a mess. Hard to use, easy to crash. All modern OSes do grow the stack dynamically, I think usually by having a write-protected guard page or two below the current stack pointer. Writes there then tell the OS that the stack has stepped below its allocated space, and you allocate a new guard page below that and make the page that got hit writable. As long as no single function allocates more than a page of data, this works fine. Or you can use two or four guard pages to allow larger stack frames.
If you want a way to control stack size and your goal is a really controlled and efficient environment, but do not care about programming in the same style as Linux etc., go for a single-shot execution model where a task is started each time a relevant event is detected, runs to completion, and then stores any persistent data in its task data structure. In this way, all threads can share a single stack. Used in many slim real-time operating systems for automotive control and similar.
Why not make the stack size a configurable item, either stored with the program or specified when a process creates another process?
There are any number of ways you can make this configurable.
There's a guideline that states "0, 1 or n", meaning you should allow zero, one or any number (limited by other constraints such as memory) of an object - this applies to sizes of objects as well.