Where is the PCI configuration address space located at? Is it located in the main system memory? If yes, is this memory address space predefined (hardwire) or change after each reboot?
PCI configuration space is present in PCI card and it is mapped to CPU addressible space. In fact, every PCI function has its own configuration space.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
Trying to have an understanding of data flow on a modern consumer desktop computers.
Looking first at a SATA port. If you want to load some bytes into RAM does the CPU send that request to the memory controller which handles the request to the SATA device and is that data than moved onto RAM by the memory controller or does the CPU cache or registers get involved with the data at all?
I assume the OS typically blocks the thread until an I/O request is completed. Does the memory controller send an interrupt to let the OS know it can schedule that thread into the queue again?
Ethernet: So assuming the above steps are complete where some bytes of a file have been loaded onto RAM does the memory get moved to ethernet controller by the memory controller or does the CPU get involved holding any of this data?
What if you use socket with localhost? Do we just do a round about with the memory controller or do we involve the ethernet controller at all?
SATA to SATA storage transfer buffered anywhere?
I know that is a lot of questions, if you can comment on any I would appreciate it! I am really trying to understand the fundamentals here. I have a hard time moving on the higher levels of abstraction without these details...
The memory controller doesn't create requests itself (it has no DMA or bus mastering capabilities).
The memory controller is mostly about routing requests to the right places. For example, if the CPU (or a device) asks to read 4 bytes from physical address 0x12345678, then the memory controller uses the physical address to figure out whether to route that request to a PCI bus, or a different NUMA node/different memory controller (e.g. using quickpath or hyper-transport or omni-path links to other chips/memory controllers), or to its locally attached RAM chips. If the memory controller forwards a request to its locally attached RAM chips; then memory controller also handles the "which memory channel" and timing part; and may also handle encryption and ECC (both checking/correcting errors, and reporting them to OS).
Most devices support bus mastering themselves.
Because most operating systems use paging "contiguous in virtual memory" often doesn't imply "contiguous in physical memory". Because devices only deal with physical addresses and most transfers are not contiguous in physical memory; most devices support the use of "lists of extents". For example, if a disk controller driver wants to read 8 KiB from a disk, then the driver may tell the disk controller "get the first 2 KiB from physical address 0x11111800, then the next 4 KiB from physical address 0x22222000, then the last 2 KiB from physical address 0x33333000"; and the disk controller will follow this list to transfer the pieces of an 8 KiB transfer to the desired addresses.
Because devices use physical addresses and almost all software (including kernel, drivers) typically primarily use virtual addresses, something (kernel) will have to convert the virtual addresses (e.g. from a "read()" function call) into the "list of extents" that the device driver/s (probably) need. When an IOMMU is being used (for security or virtualization) this conversion may include configuring the IOMMU to suit the transfer; and in that case its better to think of them as "device addresses" that the IOMMU may convert/translate into physical addresses (where the device uses "device addresses" and not actual physical addresses).
For some (relatively rare, mostly high-end server) cases; the motherboard/chipset may also include some kind of DMA engine (e.g. "Intel QuickData technology"). Depending on the DMA engine; this may be able to inject data directly into CPU caches (rather than only being able to transfer to/from RAM like most devices), and may be able to handle direct "from one device to another device" transfers (rather than having to use RAM as a buffer). However; in general (because device drivers need to work when there's no "relatively rare" DMA engine) it's likely that any DMA engine provided by the motherboard/chipset won't be supported by the OS or drivers well (and likely that the DMA engine won't be supported or used at all).
Looking first at a SATA port. If you want to load some bytes into RAM does the CPU send that request to the memory controller which handles the request to the SATA device and is that data than moved onto RAM by the memory controller or does the CPU cache or registers get involved with the data at all?
The CPU is not involved during the operation. It is a AHCI which makes the transfer which is a PCI DMA device.
The AHCI from Intel (https://www.intel.ca/content/www/ca/en/io/serial-ata/serial-ata-ahci-spec-rev1-3-1.html) is used for SATA disks. Meanwhile, for more modern NVME disks an NVME PCI controller is used (https://wiki.osdev.org/NVMe).
The OS will basically write in RAM to write to the registers of the AHCI. This way, it will be able to tell the AHCI to do stuff and tell it to write to RAM at a certain positions (probably in a buffer provided/allocated by the user mode process which asks for the data on disk). The operation is DMA so the CPU is not really involved.
From C++, you ask for data probably using fstream or direct API calls made to the OS in the library provided by the OS. For example, on Windows, you can use the WriteFile() function (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-writefile) to write to some files.
Underneath, the library is a thin wrapper which makes system calls (it is the same for the standard C++ library). You can browse my answer at Who sets the RIP register when you call the clone syscall? for more information on system calls and how they work on modern systems.
The memory controller is not really involved. It is involved to write to the registers of the AHCI but Brendan's answer is probably more useful for that matter because I'm not really aware.
I assume the OS typically blocks the thread until an I/O request is completed. Does the memory controller send an interrupt to let the OS know it can schedule that thread into the queue again?
Yes the OS blocks the thread and yes the AHCI triggers an MSI interrupt on command completion.
The OS will put the process on the queue of processes which are waiting for IO. The AHCI does trigger interrupts using the MSI capability of PCI devices. The MSI capability is a special capability which allows to bypass the IO APIC of modern x86 processors and to directly send an inter-processor interrupt to the local APIC. The local APIC then looks in the IDT as you would expect for the vector to trigger and makes the processor jump to the handler associated.
It is the handler number which differentiates between the devices that triggered the interrupt which makes it easy for the OS to just place a proper handler for the device on that interrupt number (a driver).
The OS has a driver model which takes for account the different types of devices that will be used on modern computers. The driver model will often be in the form of a virtual filesystem. The virtual filesystem basically presents every devices to the upper layer of the OS as a file. The upper layer basically makes open, read, write and ioctl calls on the file. Underneath, the driver will do complex things like triggering read/write cycles by writing the registers of the AHCI and put the process on other queues to wait for IO.
From user mode (like when calling fstream's open method), it is really a syscall which is made. The syscall handler will check all permissions and make sure that everything is okay with the request before returning a handle to the file.
Ethernet: So assuming the above steps are complete where some bytes of a file have been loaded onto RAM does the memory get moved to ethernet controller by the memory controller or does the CPU get involved holding any of this data?
The Ethernet controller is also a PCI DMA device. It reads and writes in RAM directly.
The Ethernet controller is a PCI DMA device. I never wrote an Ethernet driver but I can tell it just reads and writes in RAM directly. Now for network communications you will have sockets which act similarly to files but are not part of the virtual filesystem. The network cards (including Ethernet controllers) are not presented to the upper layer as files. Instead, you use sockets to communicate with the controller. Sockets not implemented in the C++ standard library but are present on all widespread platforms as OS provided libraries that must be used from C++ or C. Sockets are also system calls on their own.
What if you use socket with localhost? Do we just do a round about with the memory controller or do we involve the ethernet controller at all?
The ethernet controller is probably not involved.
If you use sockets with localhost, the OS will simply send the data to the loopback interface. The Wikipedia article is quite direct here (https://en.wikipedia.org/wiki/Localhost):
In computer networking, localhost is a hostname that refers to the current computer used to access it. It is used to access the network services that are running on the host via the loopback network interface. Using the loopback interface bypasses any local network interface hardware.
SATA to SATA storage transfer buffered anywhere?
It is transferred to RAM from the first device then transferred to the second device afterwards.
On the link I provided earlier for the AHCI it is stated that:
This specification defines the functional behavior and software interface of the Advanced Host Controller Interface (ACHI), which is a hardware mechanism that allows software to communicate with Serial ATA devices. AHCI is a PCI class device that acts as a data movement engine between system memory and Serial ATA devices.
The AHCI isn't for moving from SATA to SATA. It is for moving from SATA to RAM or from RAM to SATA. The SATA to SATA operation thus involves bringing the data in RAM then to move the data from RAM to the other SATA device.
The question is conceptual/theoretical, nothing about anything I am working practically on.
I understand in the virtual memory layout you have the heap, stack, data, memory mapped, etc sections
I was wondering how would it work if for example on macosx the driver wants to access one of the registers in the memory mapped IO(MMIO) region ?
I would assume that it would need to some how know the physical page number to where the MMIO region is present, but how would it get this? Any function call ?
Also, what if after a while the page was swapped out and in to a different page number ? Would it need to check/fetch the page number to the MMIO registers every time ?
How MMIO pages are discovered depends on the kind of device and platform we're talking about. On something as sophisticated and complex as a Mac, schemes like ACPI are used for assigning and enumerating MMIO addresses. Some addresses are most likely also hardcoded in Firmware and accessible to the OS via the Firmware's API. On simple embedded platforms, you'll often just find MMIO ranges hard-coded to some specific physical address.
PCI devices advertise their MMIO ranges in the configuration space, and system software (Firmware and/or OS) can decide where in physical address space the device's MMIO ranges should be located.
If the (x86/-64) CPU is running in paged/long mode, which for a Mac is true even in (EFI) firmware, the MMIO pages need to be mapped into virtual memory address space using the page table, in order to be software-accessible.
Also, what if after a while the page was swapped out and in to a
different page number ?
A MMIO-backed virtual page will not be swapped out - the only reason a virtual page would be swapped out would be because the system is low on unused physical system memory, and the OS needs to reclaim some. Removing an MMIO-backed page from the page table doesn't free any system memory.
In order to be executed by the cpu, a program must be loaded into RAM. A program is just a sequence of machine instructions (like the x86 instruction set) that a processor can understand (because it physically implements their semantic through logic gates).
I can more or less understand how a local instruction (an instruction executed inside the cpu chipset) such as 'ADD R1, R2, R3' works. Even how the cpu interfaces with the ram through the northbridge chipset using the data bus and the address bus is clear enough to me.
What I am struggling with is the big picture.
For example how can a file be saved into an hard disk?
Let's say that the motherboard uses a SATA interface to communicate with the HDD.
Does this mean that this SATA interface has an instruction set which can be used by the cpu by preparing SATA instructions written in the correct format?
Does the same apply with the PCI interface, the AGP interface and so on?
Is all the hardware communication basically accomplished through determining a stardard interface for some task and implementing it (by the companies that create hardware chipsets) with an instruction set that any other hardware components can query?
Is my high level understanding of hardware and software interaction correct?
Nearly. It's actually more general than an instruction.
Alot of these details are architecture specific, so I will stick to a high level general overview of how this can be done.
The CPU can read and write to RAM without a problem correct? You can issue instructions that read and write to any memory address. So rather than try to extend the CPU to understand every possible hardware interface out there, hardware manufacturers simply map sections of the address space (where RAM normally would be) to hardware.
Say for example you want to save a file to a hard drive. This is a possible sequence of command that would occur:
The command register of the hard drive controller is address 0xF00, an address that is outside of RAM but accessible to the CPU
Write the instruction to the command register that indicates we want to write to the hard drive.
There could be conceivably an address register at 0xF01 that tells the hard drive controller where to save the data
Tell the hard drive controller that the data I want to write is at some address in RAM, and initiate the write sequence.
There are a myriad of other ways this can be conceivably be done, but the important thing to note is that it is simply using the instructions that the CPU already has for using RAM.
All of this can be done by the CPU without any special instructions on the CPU side, just read and write to an address. You can imagine this being extended, there is a special place in the address space for the USB controller that contains a list of USB devices, there is a special place for the PCI device list, each PCI devices has several registers that can be read and written to instruct them to do things.
Essentially the role of a device driver is to know how these special registers are to be read and written, what kind of commands devices can accept, etc. Often, as is the case with many graphics cards, what these registers do is not documented to the public and so we rely on their drivers to run the cards correctly.
Address of video memory (0xB8000), who is mapping video memory to this address?
The routine which is copying the data from the address and putting to the screen, Is it a processor inbuilt function (Is this driver comes with the processor)?
What happens when you write to the address:
That area of the address space is not mapped to RAM, instead it gets sent across the system bus to your VGA card. The BIOS set this up with your VGA card at boot time (Lots of address ranges are memory mapped to various devices). No code is being executed on the CPU to plot the pixels when you write to this area of the address space. The VGA card receives this information instead of your RAM and does this itself.
If you wanted you could look BIOS functions calls and have it reconfigure the hardware so you could plot pixels instead of place characters at the video address instead. You could even probe it to see if it supports VESA and switch to a nice 1280*768 32bpp resolution. The BIOS would then map an area of the address space of your choosing to the VGA card for you.
More about the BIOS:
The BIOS is a program that comes with your motherboard that your CPU executes when it first powers up. It sets up all the hardware, maps all the memory mapped devices, creates various useful tables, assigns IO ports, hooks interrupts up to a bunch of routines it leaves in memory. It then loads your bootsector from a device and jumps to your OS code.
The left behind routines and data structures enable you to get your OS off the ground. You can load sectors off a disk, write text to the screen, get information about the system (Memory maps, ACPI tables, MP Tables .etc). Without these routines and data structures, it would be a lot harder if not impossible to make an acceptable bootsector and have all the information about the system to build a functional kernel.
However the routines are dated, slow and have very restrictive limitations. For one the routines left in memory are 16bit real mode code, so as soon as you switch to 32bit protected mode you have to switch back constantly or use VM86 mode to access them (Completely unaccessible in 64bit mode, Apparently emulating the instructions with a modified linux x86emu library is an option though). So the routines are generally very slow. So you will need to write your own drivers from scratch if you move away from real mode programming.
In most cases, the PC monitor is a VGA-compatible device, which by standard includes a mode for setting text buffer (32 KB sized) through MMIO starting from the address of 0xB8000.
The picture above summarizes how MMIO works.
First of all, is virtual memory a hardware feature of the system, or is it implemented solely by OS?
During link-time relocation, the linker assigns run-time addresses to each section and each symbol, in the generated executable
Do those run-time addresses correspond to virtual addresses?
What if the system for which the executable is generated, does not use virtual memory?
Next, if virtual memory is not used, then the application's address space is limited to the physical address space allocated for it by OS after load-time relocation
Does page fault occur if no virtual memory is used?
I think, it does: in case if the physical page containing the requested physical address has not yet been stored in RAM, then page fault should occur, which is serviced by OS page fault handler
Finally, is paging possible without virtual memory?
I'm asking because paging is always mentioned together with virtual memory, but it seems that the presence of virtual memory is not required to do paging
Thanks
Wow, a lot of questions.
Where is virtual memory implemented? The underlying hardware needs to support virtual memory. Remember, when you access a memory address in your program, the CPU needs some way to obtain the data belonging to this address. If you only have physical access, then the operation is directly sent to the memory controller. In systems with virtual memory you have an MMU (memory management unit), which translates a virtual address into a physical one. (Note, that some microcontrollers provide a stripped-down version, called a Memory-Protection Unit (MPU), which does not provide this translation step, but at least allows access rights checking.)
Do link-time addresses correspond to virtual addresses at runtime? In general, link-time addresses correspond to runtime virtual addresses. However, there is a mode where this is not the case: position-independent code. Here, the virtual addresses are determined at load time by a dynamic linker. This approach is usually used to load dynamically linked libraries (DLL / .so) to an application. For more details on that topic, you migth want to check out "Linkers and Loaders".
What if my target system does not have virtual memory? If your system does not support virtual memory, from the compiler's/loader's perspective nothing really changes: you still need to generate code to access memory. The only difference is that your CPU does no additional translation from a virtual to a physical address anymore.
Are there page faults if there is no virtual memory? There are no page faults if you don't have virtual memory. However, in case of an MPU you might still see access violations reported by the hardware, if your application tries to access an address it is not supposed to read/write. Note, that physical addresses (better: data pointed to by physical addresses) don't need to be loaded into RAM. They are just pointers into the RAM which is already there.
Is paging possible without virtual memory? 'Paging' and 'Virtual Memory' are often used to denote the same thing. However, paging may also refer to the concept of splitting memory into chunks of the same size - pages. The answer to your question depends on what you mean by paging. ;)