I am right now trying to understand the booting procedure and how does the processors initializes. I have read the Multiprocessor Specification section B.4.2 but it is not clear to me what INIT IPI and STARTUP IPI does to the BSP and AP.
In general; INIT IPI is like a soft reset for the (logical) CPU, that puts it into a "wait for SIPI state". The Intel manuals have a table showing the default values of various registers after power on, after reset, and after INIT IPI.
The Startup IPI is a way to tell the CPU to start executing at a certain address (an address derived from the "vector field" of the Startup IPI) before a usable IDT can be set up. This also bumps the CPU out of the "wait for SIPI state". Some (most) CPUs will respond to a Startup IPI when they aren't in the "wait for SIPI state", but without a previous INIT IPI you can't expect the CPU to be in a known/safe state at the time.
All CPUs respond to INIT IPI and Startup IPI the same. The main difference between BSP and AP is during power on (BSP executes firmware while AP CPUs wait).
Related
In a single processor system, when powered on the processor starts executing the boot rom code and the multiple stages of the boot. However how does this work in a multi process system? Does one processor act as the master? Who decides which processor is the master and the others helpers?
How and where is it configured?
Are the page tables shared between the processors? The processor caches are obviously different, at least the L1 caches is.
Multiprocessor Booting
1 One processor designated as ‘Boot Processor’ (BSP)
– Designation done either by Hardware or BIOS
– All other processors are designated AP (Application Processors)
2- BIOS boots the BSP
3- BSP learns system configuration
4- BSP triggers boot of other AP
– Done by sending an Startup IPI (inter processor interrupt) signal to
the AP
look here
and here for more details
In FreeBSD kernel, how can I first stop all the cores, and then run my code (can be a kernel module) on all the cores? Also, when finished, I can let them restore contexts and continue executing.
Linux has APIs like this, I believe FreeBSD also has a set of APIs to do this.
edit:
Most likely I did not clarify what I want to do. First, the machine is x86_64 SMP.
I set a timer, when the time is over; to stop all the threads (including kernel threads) on all cores; save context; run my code on one core to do some kernel stuff; when finished, restore the context and let them continue running; periodically. The other kernel threads and processes are not affected (without changing their relative priority).
I assume that your "code" (the kernel module) actually takes advantage of SMP inherently already.
So, one approach you can do is:
Set the affinity of all your processes/threads to your desired cpus (sched_setaffinity)
Set each of your threads to use Real-Time (RT) scheduling.
If it is a kernel module, you can do this manually in your module (I believe), by changing the scheduling policy for your task_struct to SCHED_RR (or SCHED_FIFO) after pinning each process to a core.
In userspace, you can use the FreeBSD rtprio command (http://www.freebsd.org/cgi/man.cgi?query=rtprio&sektion=1):
rtprio, idprio -- execute, examine or modify a utility's or process's
realtime or idletime scheduling priority
The effect will be: Your code will run first before any other non-essential process in the system, until your code finishes.
If we look at a Haswell architectural diagram today we can see that there are PCIe lanes directly connected to the CPU (for graphics) as well as some of them routed to the the platform controller hub (southbridge replacement):
If we look Intel 8 series data-sheet (the specification of the C222) we will find that the Intel C222 contains the I/O APIC used to route legacy INTx interrupts (Chapter 5.10). My question is what happens if a legacy INTx interupt requests arrives directly at the CPU (over the PCIe 3.0 lanes). Does that have to be forwarded to the C222 first or is there another I/O APIC in the system agent that I will have to program in that case? Also, with Intel Virtualization Technology for Directed I/O there is now an additional indirection, the interrupt remapping table. Is that table in the system agent (former northbridge) on the CPU or on the C222 and does that mean all interrupts from the PCIe 3.0 lanes need to be routed to the C222 first in case the remapping is enabled?
Legacy INTx interupt requests arriving at a root port in the CPU are forwarded to the I/O APIC in the PCH.
There is a separate VT-d instance in the CPU (perhaps even a separate instance per root port), so message-signaled interrupts arriving at a root port do not go through the PCH.
I am wondering if it is, in theory, possible to enable hyperthreads after they have been disabled in the BIOS and vice-versa. As it turns out, if hyper-threads are disabled they do still show up in the MADT tables of ACPI as disabled cores. This is a sample output from the MADT with a processor having 4 cores and 2 threads per core and hyper-threading disabled.
CPU 0: APIC_ID=0 ACPI_PROCESSOR_ID=0 ENABLED=1
CPU 1: APIC_ID=2 ACPI_PROCESSOR_ID=1 ENABLED=1
CPU 2: APIC_ID=4 ACPI_PROCESSOR_ID=2 ENABLED=1
CPU 3: APIC_ID=6 ACPI_PROCESSOR_ID=3 ENABLED=1
CPU 4: APIC_ID=255 ACPI_PROCESSOR_ID=4 ENABLED=0
CPU 5: APIC_ID=255 ACPI_PROCESSOR_ID=5 ENABLED=0
CPU 6: APIC_ID=255 ACPI_PROCESSOR_ID=6 ENABLED=0
CPU 7: APIC_ID=255 ACPI_PROCESSOR_ID=7 ENABLED=0
I'm wondering if (a) there is an option to enable these cores at runtime (without rebooting and going through the BIOS). And (b) what (well defined or not) state is a hyperthread/processor in, if it is not enabled (i.e., is it executing hlt or mwait instructions with local APIC disabled for example?).
What I read in the ACPI specification (5.2.12.2 Processor Local APIC Structure) is the following for the enabled flag:
If zero, this processor is unusable, and the operating system
support will not attempt to use it.
However, if someone knows, I'm interested in knowing about what is the actual state a disabled hyperthread is in. For example, did the MP Initialization Protocol Algorithm as described in Intels Software Developers Manual Volume 3 (Section 8.4.3) execute on the disabled hyper-threads during initialization?
(a) Sorry to say but you cannot with 99.99% certainty unless you have access to the processor initialization code or your BIOS vendor happened to comment out a few different lines of code. The number of cores & threads are locked in at the end of the cold boot process.
(b) I'm pretty sure when the HT disabled bit is set. The logical Processor 1 and secondary APIC are disabled.
I've started reading about VMM and wondered to myself how does the hypervisor knows a privileged instruction (for ex, cpuid) happened inside a VM and not real OS ?
let's say I've executed cpuid, a trap will occur and a VMEXIT would happen, how does the hypevisor
would know that the instruction happened inside my regular OS or inside a VM ?
First off, you are using the wrong terminology. When an OS runs on top of a hypervisor, the OS becomes the VM (virtual-machine) itself and the hypervisor is the VMM (=virtual machine monitor). A VM can also be called "guest". Thus: OS on top of hypervisor = VM = guest (these expressions mean the same thing).
Secondly, you tell the CPU that it's executing inside the VM from the moment you've executed VMLAUNCH or VMRESUME, assuming you're reading about Intel VMX. When for some reason the VM causes a hypervisor trap, we say that "a VM-exit occured" and the CPU knows it's no longer executing inside the VM. Thus:
Between VMLAUNCH/VMRESUME executions and VM-exits we are in the VM and CPUID will trap (causing a VM-exit)
Between VM-exits and VMLAUNCH/VMRESUME executions we are in the VMM (=hypervisor) and CPUID will NOT TRAP, since we already are in the hypervisor
Instructions that are privileged generate exceptions when executed in user mode. The exception is usually an undefined instruction exception. The hypervisor hooks this exception, inspects the executing instruction and then returns control to the VM. When the host OS calls the same instruction, it is in a supervisor or elevated privilege and usually no exception is generated when it executes the instruction. So generally, these issues are handled by the CPU.
However, if an instruction is not available on the processor (say floating point emulation), then the hypervisor may emulate for the VM and chain to the OS handler if not. Possibly it may even allow the OS to handle the emulation for both VMs and user tasks in the OS.
So generally, this question is unanswerable for a generic CPU. It depends on how the instruction is emulated in the VM. However, the best case is that the hypervisor does not emulate any OS instructions. Emulations will not only slow down the VMs, but the entire CPU, including user processes in the host OS.