I am currently reading OS and read about internal and external memory fragmentation.
Internal fragmentation is based on fixed size partitioning. For example = paging is based on fixed size partitioning and hence, paging suffers from internal fragmentation.
On the other hand, External fragmentation is based on variable size partitioning.
For example = segmentation is based on dynamic variable size partitioning and hence, segmentation suffers from external fragmentation.
So, my doubt is there is internal fragmentation in paging, so it has 0 external fragmentation or there is something very small, so we can neglect that
and
Similarly, for segmentation, does it also has 0 internal fragmentation or very small, that can be neglected?
Is my understanding right ?
Internal fragmentation is sujected to "Fixed size partitioning scheme" and external fragmentation to "variable size partitioning ".
No, there can never be external fragementation in fixed size partitioning because the leftover space cannot be used to allocate to any other process. External fragmentation occurs only when "there is space available which can be allocated to the process but due to non availability of enough contiguous space,the available space cannot be allocated".
On the other hand,in case of variable size partitioning, there can never be internal fragmentation because the lefover space can be allocated to the process of same or less than the available space(though the probability of allotment could be very less).
We can remove internal fragmentation and external fragmentation, if we can use a method "non-contiguous allocation" in "variable size partitioning".
Related
Is Memory address translation only useful when the total size of virtual memory
(summed over all processes) needs to be larger than physical memory?
Basically, the size of virtual memory depends on what you call "virtual memory". If you call virtual memory the virtual memory of one process then virtual memory has the same size than physical memory. If you call virtual memory the whole virtual memory of all processes than virtual memory can (technically) have an infinite size. This is because every process can have a whole address space. The virtual address space of one process cannot be bigger than physical memory because the processor has limited bits to address this memory. In modern long mode the processor has only 48 bits to address RAM at the byte level. This gives a very big amount of RAM but most systems will have 8GB to 32GB.
Technically on a 8GB RAM computer, every process could have 8GB allocated. I say technically because eventually, the computer will constantly be removing page frames from RAM and that will put too much overhead on the OS and on the computer which will make your system freeze. In the end, the size of the sum of the virtual memory of every process is limited by the capacity of your system (and OS) to have an efficient page swapping algorithm (and on your willingness to have a slow system).
Now to answer your question, paging (virtual memory) is used also to avoid fragmentation and for securing the system. With the old segmentation model, fragmentation was an issue because you had to run a complex algorithm to determine which part of memory a process gets. With paging, the smallest granularity of memory is 4KB. This makes everything much easier because a small process just gets a 4KB page and the process can work in that page the way it wants. While a bigger process will get several pages and can allocate more pages by doing a system call. There is still the issue of external fragmentation but it is mostly due to latency of accessing high memory vs low memory. Basically, paging solves the issue of external fragmentation because a process can get a page anywhere (where it's available) and it will not make a difference (except for high vs low memory). There is still the issue of internal fragmentation with paging.
Paging also secures the system. With segmentation you had several levels of ring protection. With paging you have only user or supervisor. With segmentation, the memory is not well protected because one process can access the memory of another process in the same segment. With paging, there are 2 different protections. The first protection is the ring itself (user vs supervisor) the second are the page tables. The page tables isolate one process from another because the memory accesses are translated to other positions in RAM. It is the job of the OS to fill the page tables properly so that one process doesn't have access to the physical memory allocated to another process. The user vs supervisor bit in the page tables, prevent one process from accessing the kernel except via a system call interface (the instruction syscall in assembly for x86).
Is it possible to say that internal fragmentation occurs only in physical memory and external fragmentation occurs only in virtual memory ?
If we can't say that, could you explain where internal and external fragmentation can happen ?
I disagree that internal fragmentation occurs only in physical memory. The unused memory is also marked as used in the free list, and it's when using this free list that the OS allocates more than what is needed. I would argue that the problem of internal fragmentation doesn't occur in any one type of memory, but is an issue of the allocation algorithm the OS is using. It's an issue in both.
I'm confused by this.
Are pages only memory units that exist in secondary memory or do they also exist in RAM too?
A memory page is the smallest unit of memory used by a virtual memory manager. A page can be backed by physical RAM, or by swap space or a page file on a hard drive. Pages backed by RAM have much faster IO, but as RAM gets full the OS may have to swap out pages to the hard drive.
Pages do not exist [physically] at all. A page is simply a redirection mechanism.
The operating system sets up of linear, logical address space for each process. The logical address space is organized into pages that in turn may map to:
A physical page frame of memory
No where
Somewhere on disk and managed by the operating system.
Paging is a memory management scheme by which a computer stores and retrieves data from secondary storage for use in main memory. Pages are used in RAM too, as a solution of external fragmentation.External fragmentation is a situation when total free space is enough to hold another process but space available is not contiguous. Compaction is one of the solution but for processes which are run-time loaded only. So, Paging is the true solution for external fragmentation where we implement page table which gives illusion that process has been given contiguous memory. Every address from CPU is broken down to page number and offset.
I have a program that routinely uses massive arrays, where the memory is allocated using mmap
Does anyone know the typical overheads of allocating address space in large amounts before the memory is committed, either if allocating with MAP_NORESERVE or backing the space with a sparse file? It5 strikes me mmap can't be free since it must make page table entries for the allocated space. I want to have some idea of this overhead before implementing an algorithm I'm considering.
Obviously the answer is going to be platform dependent, im most interested in x64 linux, sparc solaris and sparc linux. I'm thinking that the availability of 1mb pages makes the overhead rather less on a sparc than x64.
The overhead of mmap depends on the way you use it. And it is typically negligible when you use it in an appropriate way.
In linux kernel, mmap operation can be divided into two parts:
look for a free address range that can hold the mapping
Create/enlarge vma struct in address space (mm_struct)
So allocate large amount of memory use mmap do not introduce more
overhead than small ones.
So you should allocate memory as larger as possible in each time. (avoid mutiple times of small mmap)
And you may provide the start address explicitly (if possible). This could save some time in kernel in looking for an large enough free space.
If your application is an multi-threaded program. You should avoid concurrent calls to mmap. That is because the address space is protected by a reader-writer lock and the mmap always takes the writer lock. mmap latency will be orders of magnitude greater in this case.
Moreover, mmap only create the mapping but not the page table. Pages are allocated in the page fault handler when being touched. Page fault handler would take the reader lock that protects address space and can also affects mmap performance.
In this case, you should always try to reuse your large array instead of munmap it and mmap again. (Avoid pagefaults)
As processes are loaded and removed from memory , the free memory space is broken into little pieces ,causing fragmentation ... but how does this happen ?
And what is the best solution to external fragmentation ?
External fragmentation exists when there is enough total memory to satisfy a request (from a process usually), but the total required memory is not available at a contiguous location i.e, its fragmented.
Solution to external fragmentation :
1) Compaction : shuffling the fragmented memory into one contiguous location.
2) Virtual memory addressing by using paging and segmentation.
External Fragmentation
External fragmentation happens when a dynamic memory allocation algorithm allocates some memory and a small piece is left over that cannot be effectively used. If too much external fragmentation occurs, the amount of usable memory is drastically reduced. Total memory space exists to satisfy a request, but it is not contiguous.
see following example
0x0000 0x1000 0x2000
A B C //Allocated three blocks A, B, and C, of size 0x1000.
A C //Freed block B
Now Notice that the memory that B used cannot be included for an allocation larger than B's size
External fragmentation can be reduced by compaction or shuffle memory contents to place all free memory together in one large block. To make compaction feasible, relocation should be dynamic.External fragmentation is also avoided by using paging technique.
The best solution to avoid external fragmentation is Paging.
Paging is a memory management technique usually used by virtual memory operating systems to help ensure that the data you need is available as quickly as possible.
for more see this : What's the difference between operating system "swap" and "page"?
In case of Paging there is no external fragmentation but it doesn't avoid internal fragmentation.