Regarding Intel VMX, I found that there is IA-32e mode guest bit field in vm-entry controls.
Based on the Intel manual, when this bit is set, the VM entries start from IA-32e mode.
What is the benefit of starting VCPU from ia-32e mode?
Is it just one of VM optimization to fast-forward some initial boot steps of VCPU?
Also, if this bit has not been set, what is the default mode of the VCPU when it initially enters to the VM?
Related
First consider the situation when there is only one operating system installed. Now I run some executable. Processor reads instructions from the executable file and preforms these instructions. Even though I can put whatever instructions I want into the file, my program can't read arbitrary areas of HDD (and do many other potentially "bad" things).
It looks like magic, but I understand how this magic works. Operating system starts my program and puts the processor into some "unprivileged" state. "Unsafe" processor instructions are not allowed in this state and the only way to put the processor back to "privileged" state is give the control back to kernel. Kernel code can use all the processor's instructions, so it can do potentially unsafe things my program "asked" for if it decides it is allowed.
Now suppose we have VMWare or VirtualBox on Windows host. Guest operating system is Linux. I run a program in guest, it transfers control to guest Linux kernel. The guest Linux kernel's code is supposed to be run in processor's "privileged" mode (it must contain the "unsafe" processor instructions!). But I strongly doubt that it has an unlimited access to all the computer's resources.
I do not need too much technical details, I only want to understand how this part of magic works.
This is a great question and it really hits on some cool details regarding security and virtualization. I'll give a high-level overview of how things work on an Intel processor.
How are normal processes managed by the operating system?
An Intel processor has 4 different "protection rings" that it can be in at any time. The ring that code is currently running in determines the specific assembly instructions that may run. Ring 0 can run all of the privileged instructions whereas ring 3 cannot run any privileged instructions.
The operating system kernel always runs in ring 0. This ring allows the kernel to execute the privileged instructions it needs in order to control memory, start programs, write to the HDD, etc.
User applications run in ring 3. This ring does not permit privileged instructions (e.g. those for writing to the HDD) to run. If an application attempts to run a privileged instruction, the processor will take control from the process and raise an exception that the kernel will handle in ring 0; the kernel will likely just terminate the process.
Rings 1 and 2 tend not to be used, though they have a use.
Further reading
How does virtualization work?
Before there was hardware support for virtualization, a virtual machine monitor (such as VMWare) would need to do something called binary translation (see this paper). At a high level, this consists of the VMM inspecting the binary of the guest operating system and emulating the behavior of the privileged instructions in a safe manner.
Now there is hardware support for virtualization in Intel processors (look up Intel VT-x). In addition to the four rings mentioned above, the processor has two states, each of which contains four rings: VMX root mode and VMX non-root mode.
The host operating system and its applications, along with the VMM (such as VMWare), run in VMX root mode. The guest operating system and its applications run in VMX non-root mode. Again, both of these modes each have their own four rings, so the host OS runs in ring 0 of root mode, the host OS applications run in ring 3 of root mode, the guest OS runs in ring 0 of non-root mode, and the guest OS applications run in ring 3 of non-root mode.
When code that is running in ring 0 of non-root mode attempts to execute a privileged instruction, the processor will hand control back to the host operating system running in root mode so that the host OS can emulate the effects and prevent the guest from having direct access to privileged resources (or in some cases, the processor hardware can just emulate the effect itself without getting the host involved). Thus, the guest OS can "execute" privileged instructions without having unsafe access to hardware resources - the instructions are just intercepted and emulated. The guest cannot just do whatever it wants - only what the host and the hardware allow.
Just to clarify, code running in ring 3 of non-root mode will cause an exception to be sent to the guest OS if it attempts to execute a privileged instruction, just as an exception will be sent to the host OS if code running in ring 3 of root mode attempts to execute a privileged instruction.
In virtualization, the hypervisor doesn't need to use system call?
is it always running in Kernel mode or it can run also in user mode?
On x86, a hypervisor runs in VMX root mode. Within this mode, transitions between kernel and user mode work just as they do in VMX non-root mode (i.e., in a guest) or when VMX is off.
However, in my experience, most hypervisors always run in kernel mode.
I am trying to understand hardware assisted virtualization for a project with ARM CortexA8 and using the ARM Trustzone feature. I am new to this topic therefore I started with Wiki entries to understand more.
Wikipedia explains hardware assisted virtialization and adds a line in the definitionas:
Full virtualization is used to simulate a complete hardware
environment, or virtual machine, in which an unmodified guest
operating system (using the same instruction set as the host machine)
executes in complete isolation.
The text in bold is a bit confusing. How is the same instruction set of the processor used to provide two isolated environment? Can someone explain it? ArmTrustzone manual also talk of a "virtual processor core" to provide security. Please throw some light.
thanks
The phrase "using the same instruction set as the host machine" means that the guest OS is not aware of the virtualization layer and behaves as if it is executed on a real machine (with the same instruction set). This is in contrast to the para-virtualization paradigm in which the guest OS is aware of virtualization and calls some specific VMM functions, i.e. hypercalls.
No, CPU has not additional instructions. Virtual machine instruction set is translated by a hypervisor component called VMM (virtual machine manager) to be executed on the physical CPU.
Physical CPU with assisted Virtualization introduced only a new ring 0 mode called VMX that allow the virtual machine to execute some instructions in ring 0.
Recently i`ve been studying something about hardware-supported virtualization.
I read about 3 states of host cpu ,thus the most common userspace,kernelspace and A New Guest State.And as i can see from the ps command,there is a process for the vm i started,and some 'sub'-threads for each cpu owned by the virtual machine.Also i noticed when the vm runs some io related program,some more threads will be created on the host,which i guess might be the responses of qemu for hardware emulation.
So here comes my question:For any certain time(time in guest state,not the other two),does a vcpu thread represent a guestOS process running(i mean 'occupy' and 'exclusively')?just the same as a physical cpu,for any given time in userspace,a user process is running on it.
This may sound a little stupid,I just want to figure it out for further research.
To make this question simple:
is the vcpu thread which runs on host machine associated with some guestOS process at any given time?
To further simplify it:
is it right when i say the guestOS processes are actually running on the host CPU directly and scheduled as ordinary host-processes?the difference between the two kinds of process being what we called virtualization?
Maybe i need another threads to solve some questions about guestOS process switching,but before that,hope you guys can help me with this one.
sincerely
MeNok
I posted this question on LQ and got the answer.
http://www.linuxquestions.org/questions/linux-virtualization-90/a-guestos-process-occupies-vcpu-at-any-given-time-4175419271/
VCPU is not a thread in host. KVM allows guest to run directly on a physical CPU with a less privilege guest mode. A timer interrupt will cause CPU back from guest mode to host mode and return to KVM. Since KVM is scheduled in kernel mode, a guest should be also scheduled in the host as well.
According to a marked answer on stackoverflow.com here and another reference here, I understand that :
Hypervisor virtualization = below the OS and a hardware virtualization where the hardware is designed to support virtualization
Non-Hypervisor virtualization = on top of the OS (like an application software), that is purely software virtualization
But we do also have Type1 and Type2 classifications for hypervisors and it seems to me that Type2 is purely Software Virtualization ... so does this mean that Non-Hypervisor Virtualization is equivalent to Type 2 Hypervisor or are there some subtle differences??
Or is it the case that these terms all are loosely defined??
Thanks in advance.
it seems to me that Type2 is purely Software Virtualization
Don't conflate "Type 1 vs Type 2" and "Hardware vs Software" Virtualization. In fact, there is actually a middle ground between hardware and software: There is Full hardware (HVM), "partial" hardware (PVM), and Pure Software (SW).
I'll try to clarify by expanding all 6 combinations:
Type 1 + Full Hardware (HVM) - This allows a hypervisor like Xen HVM to boot an unmodified guest OS. This is actually slow because the hypervisor must decode "telegraph messages" that the guest OS is trying to send to the hardware. (i.e. writing to the disk drive involves repeatedly storing bytes in location 0xblahblah.)
Type 1 + Paravirtualization (PVM) - This is when you modify the guest OS a little to call the Hypervisor directly for some tasks, like disk I/O. This is faster because the guest just says "here, write this page of bytes" and doesn't have to do (virtualized) I/O on each byte. You know you're doing PVM when you install special drivers. Of course, sometimes the OS has virtual drivers built in already. For example, any modern Linux kernel will switch to PVM mode at boot automatically when it detects it's running under Xen, KVM, UML, etc.
Type 1 + Pure Software (SW) - I'm not sure if this exists, but it wouldn't be that hard to build. Since software emulation is slow, the overhead of booting a real OS and running Type 2 isn't a big deal.
Type 2 + Full Hardware (HVM) - This allows you to boot an un-modified Windows under VirtualBox or KVM. You know it's type 2 when you can reboot all your Guests and still play MP3s in the background :)
Type 2 + Paravirtualization (PVM) - This happens any time you install guest drivers, or boot a modern Linux kernel under VirtualBox/KVM.
Type 2 + Pure Software (SW) - early versions of Bochs and Qemu. (Latter versions actually have hardware assisted modes too.) You can tell they are "pure software" because they allow you to run software that you normally can't run without it. (i.e. I've run Windows '95 under Bochs on an ARM processor, and I've booted an ARM distro on an x86 under Qemu.)
There is also another subject that is unlike the above:
Container technology. Containers like Docker/Rkt/LXD don't fit in the above table. Applications in Containers are ordinary programs calling the kernel in ordinary ways, no Hypervisor involved.
It's just that containers use the Kernel features of cgroups and namespaces to make an app "feel" like it's in a VM. Each container gets a 'partitioned' view of the system: It's own filesystem, it's own user IDs, it's own process IDs, it's own hostname + IP address, etc. But from the outside, you can see all processes in all containers with 'ps'.
In my mind, Non-Hypervisor virtualization means a virtualization layer that runs something OTHER than an OS on top of it -- most commonly virtualizing the user-level environment of some other operatoring system. For example, the WINE project is non-hypervisor virtualization -- it allows running win32 programs on a linux (or other) host. There's no attempt to run an actual Windows OS or emulate 'bare' hardware for a virtualized OS. Instead the virtual layer provides the user-level abstractions and system calls for windows directly.
Contrast this with a hypervisor which may be either type 1 (running on bare metal) or type2 (running on an OS) and which provides hardware-level abstractions and which you run an entire OS on top of.
A Hypervisor, by definition, emulates hardware. (That may or may not physically exist) - it may virtualize some as well.
Virtualization intercepts a call and redirects it elsewhere.
They are two different but interrelated topics.
Type 1 Hypervisors run on "bare metal" and sit between the hardware and your virtual operating systems (the hypervisor itself is the operating system). For example, VMWare ESX, Citrix XenServer or Microsoft Hyper-V
Type 2 Hypervisors run on top of your existing operating system and may support either hardware or software virtualization. For example both QEmu and Bochs) emulate an entire CPU, optionally even a different CPU architecture. Both are Type 2 Hypervisors but have significant overhead on performance due to the emulation required.
VMware Workstation/Server/Player/Fusion, Parallels, Virtualbox are all examples of Type 2 hypervisors that support Hardware-assisted Virtualization - this means rather than emulating the CPU instructions, the CPU instructions can pass through directly with no emulation or translation -- effectively running with no loss of performance if the processor supports it.
Next up, non-hypervisor virtualization which is (effectively) application virtualization. The hardware itself is not being emulated in any way at all, the virtualization layer is just intercepting certain system calls and virtualizing those. Examples in this category include VMWare Thinapp, Microsoft App-V and many more. Windows Vista itself virtualizes certain registry and disk writes to areas where the user doesn't have permission to write. This virtualization in Vista is critical for backwards compatibility with many legacy applications.
Finally we have pure emulators - no virtualization is happening here. In this category we have WINE and to some extent Cygwin. Also Bochs fits in this category as well as a Type 2 Hypervisor since there is no virtualization, just hardware emulation. DOSEMU is another one that fits in here.
I'm sure I've missed plenty of examples, but
(I'll post my comment to #answer-16868851 here since I miss few points to fulfill "You must have 50 reputation to comment" requirement)
BraveNewCurrency writes:
Type 1 + Pure Software (SW) - I'm not sure if this exists, but it wouldn't be that hard to build. Since software emulation is slow, the overhead of booting a real OS and running Type 2 isn't a big deal.
So far I've found only one Type 1 hypervisor capable of doing this -- it's VMware ESXi.
vSphere 5 Documentation Center | ESXi Hardware Requirements say:
■ To support 64-bit virtual machines, support for hardware virtualization (Intel VT-x or AMD RVI) must be enabled on x64 CPUs.
Hence, 32-bit guests work without VT-x in it.
As I see zero competition for it camed (either proprietary or opensource), I guess trapping sensitive CPU instructions without VT-x support (that is in Pure Software) is serious challenge in practice.
While following doesn't relate to the original question already, v5.0 (and v4.x) requires 64-bit support from CPU though:
■ ESXi 5.0 will install and run only on servers with 64-bit x86 CPUs.
■ ESXi 5.0 requires a host machine with at least two cores.
Those interested in running Type 1 + SW hypervisor on 32-bit machines (like me) may use it's earlier versions. Minimum system requirements for installing ESXi/ESX (1003661) says:
ESX 3.5.x
The hardware requirements for ESX 3.5.x are the same as those listed in the ESX 3.0.x section, with the following additions.
[...]
ESX 3.0.x
You need these hardware and system resources to install and use ESX 3:
At least two processors:
1500 MHz Intel Xeon and above, or AMD Opteron (32bit mode) for ESX
1500 MHz Intel Xeon and above, or AMD Opteron (32bit mode) for Virtual SMP
1500 MHz Intel Viiv or AMD A64 x2 dual-core processors
+ ESX 3.5 Installation Guide repeats this in following section / subsection:
ESX Server 3 Requirements
This section discusses the minimum and maximum hardware configurations supported by ESX Server 3 version 3.5.
Minimum Server Hardware Requirements
...
Hence, Pure (and 32-bit only) Software :)