I'm trying to trace memory accesses made by a single core QEMU KVM enabled VM using the Intel Pin instrumentation tool.
As QEMU creates 2 child threads from a parent, which one should be used for tracing the VMs memory accesses in host?
qemu-system-x86_64 -enable-kvm -m 1024 -hda image.qcow2
Gives rise to:
qemu-system-x86(6068)-+
|-{qemu-system-x86}(6069)
|-{qemu-system-x86}(6072)
When attaching pin tool to the parent process (pid=6068) the system hangs, on attaching to the child (pid=6069) the VM hangs up, on attaching to the child (pid=6072) the pin tool creates output w/o hangup. How can I confirm if I'm attached to the right thread so that I'm tracing the VM's memory accesses?
Related
I saw the following discussion and had some questions:
live resize of a NVMe drive
If the physical capacity of the nvme device changes (e.g., from 10GB to 20GB), how the operating system detect it without rebooting?
In the above link, re-scanning pci bus is solution.
When the re-scanning be executed, does the operating system ask the nvme device to update its meta-info (e.g., capacity, etc.) ?
How does OS interact with disk specifically? (How to read changed device parameters from the disk, not the old device parameters in memory?)
This is an AWS virtual-machine probably so the disk is actually a virtual-disk. You can't resize a physical disk like you upgrade its capacity physically (you'd need to change the disk).
With that said, this machine probably runs on top of a type 1 hypervisor. What I understand about these is that the virtual machines (VMs) run as processes on a different ring on top of a minimal operating-system (hypervisor). When the VMs execute privileged instructions, it will trigger a protection fault and the hypervisor can thus inspect who actually triggered the fault (was it the guest kernel or a user mode process within the guest kernel?). If it was the guest kernel than it can execute that instruction on behalf of the guest. Otherwise, it will probably do what a real kernel would do (trigger an exception). It can tell the difference because the guest kernel runs in a different ring than the ring 3 (user mode).
With that said, the NVME device isn't PCI it is NVME. The host controller of the NVME drive is PCI. To rescan the NVME drives, you will read/write to some registers that are memory mapped in RAM and ask the NVME PCI host controller what is the size of the different disks that are found. PCI is known to be hot pluggable (similarly to USB) in some cases but mostly not on consumer motherboards. I don't think you'll get any interrupt when a PCI device is hot plugged so you are left doing a rescan of the devices.
For NVME, it depends on the host controller if you'll get an interrupt when a disk is swapped/changes size. As to virtual-disks, it probably depends on a lot of different things. You could definitely be left doing a PCI rescan here. I guess it depends on the hypervisor, on the OS and on the host controller configurations.
First consider the situation when there is only one operating system installed. Now I run some executable. Processor reads instructions from the executable file and preforms these instructions. Even though I can put whatever instructions I want into the file, my program can't read arbitrary areas of HDD (and do many other potentially "bad" things).
It looks like magic, but I understand how this magic works. Operating system starts my program and puts the processor into some "unprivileged" state. "Unsafe" processor instructions are not allowed in this state and the only way to put the processor back to "privileged" state is give the control back to kernel. Kernel code can use all the processor's instructions, so it can do potentially unsafe things my program "asked" for if it decides it is allowed.
Now suppose we have VMWare or VirtualBox on Windows host. Guest operating system is Linux. I run a program in guest, it transfers control to guest Linux kernel. The guest Linux kernel's code is supposed to be run in processor's "privileged" mode (it must contain the "unsafe" processor instructions!). But I strongly doubt that it has an unlimited access to all the computer's resources.
I do not need too much technical details, I only want to understand how this part of magic works.
This is a great question and it really hits on some cool details regarding security and virtualization. I'll give a high-level overview of how things work on an Intel processor.
How are normal processes managed by the operating system?
An Intel processor has 4 different "protection rings" that it can be in at any time. The ring that code is currently running in determines the specific assembly instructions that may run. Ring 0 can run all of the privileged instructions whereas ring 3 cannot run any privileged instructions.
The operating system kernel always runs in ring 0. This ring allows the kernel to execute the privileged instructions it needs in order to control memory, start programs, write to the HDD, etc.
User applications run in ring 3. This ring does not permit privileged instructions (e.g. those for writing to the HDD) to run. If an application attempts to run a privileged instruction, the processor will take control from the process and raise an exception that the kernel will handle in ring 0; the kernel will likely just terminate the process.
Rings 1 and 2 tend not to be used, though they have a use.
Further reading
How does virtualization work?
Before there was hardware support for virtualization, a virtual machine monitor (such as VMWare) would need to do something called binary translation (see this paper). At a high level, this consists of the VMM inspecting the binary of the guest operating system and emulating the behavior of the privileged instructions in a safe manner.
Now there is hardware support for virtualization in Intel processors (look up Intel VT-x). In addition to the four rings mentioned above, the processor has two states, each of which contains four rings: VMX root mode and VMX non-root mode.
The host operating system and its applications, along with the VMM (such as VMWare), run in VMX root mode. The guest operating system and its applications run in VMX non-root mode. Again, both of these modes each have their own four rings, so the host OS runs in ring 0 of root mode, the host OS applications run in ring 3 of root mode, the guest OS runs in ring 0 of non-root mode, and the guest OS applications run in ring 3 of non-root mode.
When code that is running in ring 0 of non-root mode attempts to execute a privileged instruction, the processor will hand control back to the host operating system running in root mode so that the host OS can emulate the effects and prevent the guest from having direct access to privileged resources (or in some cases, the processor hardware can just emulate the effect itself without getting the host involved). Thus, the guest OS can "execute" privileged instructions without having unsafe access to hardware resources - the instructions are just intercepted and emulated. The guest cannot just do whatever it wants - only what the host and the hardware allow.
Just to clarify, code running in ring 3 of non-root mode will cause an exception to be sent to the guest OS if it attempts to execute a privileged instruction, just as an exception will be sent to the host OS if code running in ring 3 of root mode attempts to execute a privileged instruction.
I've started reading about VMM and wondered to myself how does the hypervisor knows a privileged instruction (for ex, cpuid) happened inside a VM and not real OS ?
let's say I've executed cpuid, a trap will occur and a VMEXIT would happen, how does the hypevisor
would know that the instruction happened inside my regular OS or inside a VM ?
First off, you are using the wrong terminology. When an OS runs on top of a hypervisor, the OS becomes the VM (virtual-machine) itself and the hypervisor is the VMM (=virtual machine monitor). A VM can also be called "guest". Thus: OS on top of hypervisor = VM = guest (these expressions mean the same thing).
Secondly, you tell the CPU that it's executing inside the VM from the moment you've executed VMLAUNCH or VMRESUME, assuming you're reading about Intel VMX. When for some reason the VM causes a hypervisor trap, we say that "a VM-exit occured" and the CPU knows it's no longer executing inside the VM. Thus:
Between VMLAUNCH/VMRESUME executions and VM-exits we are in the VM and CPUID will trap (causing a VM-exit)
Between VM-exits and VMLAUNCH/VMRESUME executions we are in the VMM (=hypervisor) and CPUID will NOT TRAP, since we already are in the hypervisor
Instructions that are privileged generate exceptions when executed in user mode. The exception is usually an undefined instruction exception. The hypervisor hooks this exception, inspects the executing instruction and then returns control to the VM. When the host OS calls the same instruction, it is in a supervisor or elevated privilege and usually no exception is generated when it executes the instruction. So generally, these issues are handled by the CPU.
However, if an instruction is not available on the processor (say floating point emulation), then the hypervisor may emulate for the VM and chain to the OS handler if not. Possibly it may even allow the OS to handle the emulation for both VMs and user tasks in the OS.
So generally, this question is unanswerable for a generic CPU. It depends on how the instruction is emulated in the VM. However, the best case is that the hypervisor does not emulate any OS instructions. Emulations will not only slow down the VMs, but the entire CPU, including user processes in the host OS.
I am trying to understand hardware assisted virtualization for a project with ARM CortexA8 and using the ARM Trustzone feature. I am new to this topic therefore I started with Wiki entries to understand more.
Wikipedia explains hardware assisted virtialization and adds a line in the definitionas:
Full virtualization is used to simulate a complete hardware
environment, or virtual machine, in which an unmodified guest
operating system (using the same instruction set as the host machine)
executes in complete isolation.
The text in bold is a bit confusing. How is the same instruction set of the processor used to provide two isolated environment? Can someone explain it? ArmTrustzone manual also talk of a "virtual processor core" to provide security. Please throw some light.
thanks
The phrase "using the same instruction set as the host machine" means that the guest OS is not aware of the virtualization layer and behaves as if it is executed on a real machine (with the same instruction set). This is in contrast to the para-virtualization paradigm in which the guest OS is aware of virtualization and calls some specific VMM functions, i.e. hypercalls.
No, CPU has not additional instructions. Virtual machine instruction set is translated by a hypervisor component called VMM (virtual machine manager) to be executed on the physical CPU.
Physical CPU with assisted Virtualization introduced only a new ring 0 mode called VMX that allow the virtual machine to execute some instructions in ring 0.
Recently i`ve been studying something about hardware-supported virtualization.
I read about 3 states of host cpu ,thus the most common userspace,kernelspace and A New Guest State.And as i can see from the ps command,there is a process for the vm i started,and some 'sub'-threads for each cpu owned by the virtual machine.Also i noticed when the vm runs some io related program,some more threads will be created on the host,which i guess might be the responses of qemu for hardware emulation.
So here comes my question:For any certain time(time in guest state,not the other two),does a vcpu thread represent a guestOS process running(i mean 'occupy' and 'exclusively')?just the same as a physical cpu,for any given time in userspace,a user process is running on it.
This may sound a little stupid,I just want to figure it out for further research.
To make this question simple:
is the vcpu thread which runs on host machine associated with some guestOS process at any given time?
To further simplify it:
is it right when i say the guestOS processes are actually running on the host CPU directly and scheduled as ordinary host-processes?the difference between the two kinds of process being what we called virtualization?
Maybe i need another threads to solve some questions about guestOS process switching,but before that,hope you guys can help me with this one.
sincerely
MeNok
I posted this question on LQ and got the answer.
http://www.linuxquestions.org/questions/linux-virtualization-90/a-guestos-process-occupies-vcpu-at-any-given-time-4175419271/
VCPU is not a thread in host. KVM allows guest to run directly on a physical CPU with a less privilege guest mode. A timer interrupt will cause CPU back from guest mode to host mode and return to KVM. Since KVM is scheduled in kernel mode, a guest should be also scheduled in the host as well.