I am following a walk through of the GDT. I can get a grasp of how the data structure is implemented and it's purpose.
However, what I don't understand is how the GDT is used. What is a scenario where the CPU needs to load a segment and how does the GDT do that? If the GDT is involved in a keyboard driven interrupt, that would be a great example to explain.
GDT or Global Descriptor Table contains information about segments of the memory. The address of GDT is stored in one of the special registers called GDTR. Each segment maps to a region of memory. The purpose of the segment is to provide hardware memory protection. The CPU does not really load the entire segment as segment usually refer to a memory region. Rather, when you use one of the segment registers (CS, DS, SS, etc) with an offset to address a memory region, CPU will perform check with info stored in GDT. For example, if you have set one of the segment to be read-only then later you try to write to it, the cpu will prevent the access. For a system do use segment, GDT is involved every time someone access memory.
Today's OS rarely uses segment. Most of them set up "Flat Memory Model" which each segment spans entire memory space. Paging and virtual memory are used for memory protection. Segment exists mainly because of backward-compatibility.
You can read more about GDT on osdev
For a modern OS that doesn't use segmentation (much); you'd probably find that the GDT contains:
2 or 3 descriptors for "CPL=0" and "CPL=3" code (if a 64-bit OS supports older 32-bit processes then it will probably have "32-bit CPL=3 code" and "64-bit CPL=3 code", in addition to "64-bit CPL=0 code")
a descriptor for "CPL=0" stack
a descriptor for "CPL=3" stack and data
a descriptor per CPU for the CPU's TSS (Task State Segment)
for 32-bit (not so much for 64-bit where you can use swapgs) one or two descriptors per CPU that are used to find "CPU local" data and/or "thread local" data
(optionally, less likely); a descriptor that defines a call gate for the kernel's API
(optionally, for special purposes); one or more descriptors that describe an LDT (sometimes used for emulation)
For other operating systems the GDT could contain anything.
Related
We usually learn Virtual memory and Paging at the same time in Operating System and they seem dependent. However, I wonder if they exist independently of each other?
The answer to your question depends on how you define "Virtual Memory". If you define it just as "the addresses that the application sees", then yes Virtual Memory can exist without paging.
Prior to paging, systems used segmentation to isolate user processes. To put it in simple words every process has it's own segment. All the addresses it "sees" are just offsets inside the segment. The hardware implicitly adds the segment base to the address requested by the application to get the Physical addresses. Just like the page table, the segment bases can be modified only by the kernel and it can effectively isolate memory for processes at the same time allowing scope for sharing some parts of memory between processes too.
Segments also have limits which are checked before every access to ensure that the user doesn't use a very big offset and spill into other process.
Segmentation support has been removed from Intel X86_64 architectures where the segment registers do exist but are always set to 0. Only the two segment registers %fs and %gs continue to exist. But the limit checks on them is not performed by the hardware. These segments are now used by the OS for thread local storage.
In the following link;
https://www.openhub.net/p/f9-kernel
F9 Microkernel runs on Cortex M, but Cortex M series doesn't have MMU. My knowledge on MMU and Virtual Memory are limited hence the following quesitons.
How the visibility of entire physical memory is prevented for each process without MMU?
Is it possible to achieve isolation with some static memory settings without MMU. (with enough on chip RAM to run my application and kernel then, just different hard coded memory regions for my limited processes). But still I don't will this prevent the access?
ARM Cortex-M processors lack of MMU, and there is optional memory protection unit (MPU) in some implementations such as STMicroelectronics' STM32F series.
Unlike other L4 kernels, F9 microkernel is designed for MPU-only environments, optimized for Cortex M3/M4, where ARMv7 Protected Memory System Architecture (PMSAv7) model is supported. The system address space of a PMSAv7 compliant system is protected by a MPU. Also, the available RAM is typically small (about 256 Kbytes), but a larger Physical address space (up to 32-bit) can be used with the aid of bit-banding.
MPU-protected memory is divided up into a set of regions, with the number of regions supported IMPLEMENTATION DEFINED. For example, STM32F429, provides 8 separate memory regions. In PMSAv7, the minimum protect region size is 32 bytes, and maximum is up to 4 GB. MPU provides full access over:
Protection region
Overlapping protection region
Access permissions
Exporting memory attributes to the system
MPU mismatches and permission violations invoke the programmable priority MemManage fault handler.
Memory management in F9 microkernel, can split into three conceptions:
memory pool, which represents the area of PAS with specific attributes (hardcoded in mem map table).
address space - sorted list of fpages bound to particular thread(s).
flexible page - unlike traditional pages in L4, fpage represent in MPU region instead.
Yes, but ....
There is no requirement for an MMU at all, things just get less convenient and flexible. Practically, anything that provides some form of isolation (e.g. MPU) might be good enough to make a system work - assuming you do need isolation at all. If you don't need it for some reason and just want the kernel to do scheduling, then a kernel can do this without an MMU or MPU also.
While understanding the concept of Paging in Memory Management, I came through the terms "logical memory" and "physical memory". Can anyone please tell me the diff. between the two ???
Does physical memory = Hard Disk
and logical memory = RAM
There are three related concepts here:
Physical -- An actual device
Logical -- A translation to a physical device
Virtual -- A simulation of a physical device
The term "logical memory" is rarely used because we normally use the term "virtual memory" to cover both the virtual and logical translations of memory.
In an address translation, we have a page index and a byte index into that page.
The page index to the Nth path in the process could be called a logical memory. The operating system redirects the ordinal page number into some arbitrary physical address.
The reason this is rarely called logical memory is that the page made be simulated using paging, becoming a virtual address.
Address transition is a combination of logical and virtual. The normal usage is to just call the whole thing "virtual memory."
We can imagine that in the future, as memory grows, that paging will go away entirely. Instead of having virtual memory systems we will have logical memory systems.
Not a lot of clarity here thus far, here goes:
Physical Memory is what the CPU addresses on its address bus. It's the lowest level software can get to. Physical memory is organized as a sequence of 8-bit bytes, each with a physical address.
Every application having to manage its memory at a physical level is obviously not feasible. So, since the early days, CPUs introduced abstractions of memory known collectively as "Memory Management." These are all optional, but ubiquitous, CPU features managed by your kernel:
Linear Memory is what user-level programs address in their code. It's seen as a contiguous addresses space, but behind the scenes each linear address maps to a physical address. This allows user-level programs to address memory in a common way and leaves the management of physical memory to the kernel.
However, it's not so simple. User-level programs address linear memory using different memory models. One you may have heard of is the segmented memory model. Under this model, programs address memory using logical addresses. Each logical address refers to a table entry which maps to a linear address space. In this way, the o/s can break up an application into different parts of memory as a security feature (details out of scope for here)
In Intel 64-bit (IA-32e, 64-bit submode), segmented memory is never used, and instead every program can address all 2^64 bytes of linear address space using a flat memory model. As the name implies, all of linear memory is available at a byte-accessible level. This is the most straightforward.
Finally we get to Virtual Memory. This is a feature of the CPU facilitated by the MMU, totally unseen to user-level programs, and managed by the kernel. It allows physical addresses to be mapped to virtual addresses, organized as tables of pages ("page tables"). When virtual memory ("paging") is enabled, tables can be loaded into the CPU, causing memory addresses referenced by a program to be translated to physical addresses transparently. Page tables are swapped in and out on the fly by the kernel when different programs are run. This allows for optimization and security in process/memory management (details out of scope for here)
Keep in mind, Linear and Virtual memory are independent features which can work in conjunction. If paging is disabled, linear addresses map one-to-one with physical addresses. When enabled, linear addresses are mapped to virtual memory.
Notes:
This is all linux/x86 specific but the same concepts apply almost everywhere.
There are a ton of details I glossed over
If you want to know more, read The Intel® 64 and IA-32 Architectures Software Developer Manual, from where I plagiarized most of this
I'd like to add a simple answer here.
Physical Memory : This is the memory that is actually present and every process needs space here to execute their code.
Logical Memory:
To a user program the memory seems contiguous,Suppose a program needs 100 MB of space in memory,To this program a virtual address space / Logical address space starts from 0 and continues to some finite number.This address is generated by CPU and then The MMU then maps this virtual address to real physical address through some page table or any other way the mapping is implemented.
Please correct me or add some more content here. Thanks !
Physical memory is RAM; Actually belongs to main memory. Logical address is the address generated by CPU. In paging,logical address is mapped into physical address with the help of page tables. Logical address contains page number and an offset address.
An address generated by the CPU is commonly referred to as a logical address, whereas an address seen by the memory unit—that is, the one loaded into the memory-address register of the memory—is commonly referred to as a physical address
The physical address is the actual address of the frame where each page will be placed, whereas the logical address is the address generated by the CPU for each page.
What exactly is a frame?
Processes are retrieved from secondary memory and stored in main memory using the paging storing technique.
Processes are kept in secondary memory as non-contiguous pages, which implies they are stored in random locations.
Those non-contiguous pages are retrieved into main Memory as a frame by the paging operating system.
The operating system divides the memory frame size equally in main memory, and all processes retrieved from secondary memory are stored concurrently.
An operating system/computer architecture question here. I was reading about caches, about how virtually indexing the cache is an option to reduce address translation time. I came across the following:
"Virtual cache difficulties include:
Aliasing
Two different virtual addresses may have the same physical address."
I can't think of a scenario when this can occur. It's been a while since my O/S days and I'm drawing a blank.
Could someone provide an example? Thanks
Two processes might have a shared mapping. E.g., in Unix, executable code is typically mapped into a region shared between all processes that execute the same program. (In fact, a single process might have several mappings of the same underlying memory, e.g. when it mmap's the same file twice.)
I believe that the executable sections of programs can possibly be shared between processes--thus being mapped twice.
For example: if you load two instances of vim, there will be two processes. Both process will likely map to the same executable code in physical memory.
shmat() is a typical example of same physical address being mapped as two different virtual address in two different processes.
If you do pmap -x pid_A .
you will you see the virtual mem map for process A similarly for Process B.
Actual Phy mem is not exposed to the user-space program.
Now SayProcess A and B share a shared memory segment and shared memory pointer be sh_mem_ptr_A and Sh_mem_ptr_B.
If you print these pointers their address(virtual) will be different.
Because Sh_mem_ptr_A is a part of memory map of Process A, Similarly sh_mem_ptr_B for Process B.
Kernel maintains the maaping of Virtual-to- phy addr. By page table and offset.
Higher bits map to the page table and offset maps to offset in the page table. So If you notice the Lower order bits of sh_mem_ptr_A and sh_mem_ptr_B they will be same(but may not be true always).
Also each process is allocated 4GB of virtual space (in 32 bit system), out of which 1 GB (depends upon Os to Os) is mapped for OS. Since OS is common for all processes, so the lower 1GB of virtual addresses are common for all the process, which are mapped to same OS physical pages.
Can any one please make me clear what is the difference between virtual memory and swap space?
And why do we say that for a 32-bit machine the maximum virtual memory accessible is 4 GB only?
There's an excellent explantation of virtual memory over on superuser.
Simply put, virtual memory is a combination of RAM and disk space that running processes can use.
Swap space is the portion of virtual memory that is on the hard disk, used when RAM is full.
As for why 32bit CPU is limited to 4gb virtual memory, it's addressed well here:
By definition, a 32-bit processor uses
32 bits to refer to the location of
each byte of memory. 2^32 = 4.2
billion, which means a memory address
that's 32 bits long can only refer to
4.2 billion unique locations (i.e. 4 GB).
There is some confusion regarding the term Virtual Memory, and it actually refers to the following two very different concepts
Using disk pages to extend the conceptual amount of physical memory a computer has - The correct term for this is actually Paging
An abstraction used by various OS/CPUs to create the illusion of each process running in a separate contiguous address space.
Swap space, OTOH, is the name of the portion of disk used to store additional RAM pages when not in use.
An important realization to make is that the former is transparently possible due to the hardware and OS support of the latter.
In order to make better sense of all this, you should consider how the "Virtual Memory" (as in definition 2) is supported by the CPU and OS.
Suppose you have a 32 bit pointer (64 bit points are similar, but use slightly different mechanisms). Once "Virtual Memory" has been enabled, the processor considers this pointer to be made as three parts.
The highest 10 bits are a Page Directory Entry
The following 10 bits are a Page Table Entry
The last 12 bits make up the Page Offset
Now, when the CPU tries to access the contents of a pointer, it first consults the Page Directory table - a table consisting of 1024 entries (in the X86 architecture the location of which is pointed to by the CR3 register). The 10 bits Page Directory Entry is an index in this table, which points to the physical location of the Page Table. This, in turn, is another table of 1024 entries each of which is a pointer in physical memory, and several important control bits. (We'll get back to these later). Once a page has been found, the last 12 bits are used to find an address within that page.
There are many more details (TLBs, Large Pages, PAE, Selectors, Page Protection) but the short explanation above captures the gist of things.
Using this translation mechanism, an OS can use a different set of physical pages for each process, thus giving each process the illusion of having all the memory for itself (as each process gets its own Page Directory)
On top of this Virtual Memory the OS may also add the concept of Paging. One of the control bits discussed earlier allows to specify whether an entry is "Present". If it isn't present, an attempt to access that entry would result in a Page Fault exception. The OS can capture this exception and act accordingly. OSs supporting swapping/paging can thus decide to load a page from the Swap Space, fix the translation tables, and then issue the memory access again.
This is where the two terms combine, an OS supporting Virtual Memory and Paging can give processes the illusion of having more memory than actually present by paging (swapping) pages in and out of the swap area.
As to your last question (Why is it said 32 bit CPU is limited to 4GB Virtual Memory). This refers to the "Virtual Memory" of definition 2, and is an immediate result of the pointer size. If the CPU can only use 32 bit pointers, you have only 32 bit to express different addresses, this gives you 2^32 = 4GB of addressable memory.
Hope this makes things a bit clearer.
IMHO it is terribly misleading to use the concept of swap space as equivalent to virtual memory. VM is a concept much more general than swap space. Among other things, VM allows processes to reference virtual addresses during execution, which are translated into physical addresses with the support of hardware and page tables. Thus processes do not concern about how much physical memory the system has, or where the instruction or data is actually resident in the physical memory hierarchy. VM allows this mapping. The referenced item (instruction or data) may be resident in L1, or L2, or RAM, or finally on disk, in which case it is loaded into main memory.
Swap space it is just a place on secondary memory where pages are stored when they are inactive. If there is no sufficient RAM, the OS may decide to swap-out pages of a process, to make room for other process pages. The processor never ever executes instruction or read/write data directly from swap space.
Notice that it would be possible to have swap space in a system with no VM. That is, processes that directly access physical addresses, still could have portions of it on
disk.
Though the thread is quite old and has already been answered. Still would like to share this link as this is the simplest explanation I have found so far. Below link has got diagrams for better visualization.
Key Difference: Virtual memory is an abstraction of the main memory. It extends the available memory of the computer by storing the inactive parts of the content RAM on a disk. Whenever the content is required, it fetches it back to the RAM. Swap memory or swap space is a part of the hard disk drive that is used for virtual memory. Thus, both are also used interchangeably.
Virtual memory is quiet different from the physical memory. Programmers get direct access to the virtual memory rather than physical memory. Virtual memory is an abstraction of the main memory. It is used to hide the information of the real physical memory of the system. It extends the available memory of the computer by storing the inactive parts of the RAM's content on a disk. When the content is required, it fetches it back to the RAM. Virtual memory creates an illusion of a whole address space with addresses beginning with zero. It is mainly preferred for its optimization feature by which it reduces the space requirements. It is composed of the available RAM and disk space.
Swap memory is generally called as swap space. Swap space refers to the portion of the virtual memory which is reserved as a temporary storage location. Swap space is utilized when available RAM is not able to meet the requirement of the system’s memory. For example, in Linux memory system, the kernel locates each page in the physical memory or in the swap space. The kernel also maintains a table in which the information regarding the swapped out pages and pages in physical memory is kept.
The pages that have not been accessed since a long time are sent to the swap space area. The process is referred to as swapping out. In case the same page is required, it is swapped in physical memory by swapping out a different page. Thus, one can conclude that swap memory and virtual memory are interconnected as swap memory is used for the technique of virtual memory.
difference-between-virtual-memory-and-swap-memory
"Virtual memory" is a generic term. In Windows, it is called as Paging or pagination. In Linux, it is called as Swap.