Virtual Machines & VMware, Part I
December 21, 2001
By: Jay Munro
Ever want to try out a new operating system on your pc without trashing your existing Operating System? Got a legacy application that won’t run on your current OS? Are you a developer who needs to test your code on a number of platforms? Would you like to test distributed applications on a network without requiring a server farm? Find out what companies like Symantec and Merrill Lynch know--you can do these tasks with a Virtual Machine (VM) on a single PC.
A Virtual Machine is a software environment that encapsulates one or more Operating Systems and
applications that actually run inside or "under" the VM. The OS can’t tell the difference between operating in a VM or in a "real" machine. Within a Virtual Machine, you can do almost anything you can do with a real PC, with complete safety. Furthermore, the type of virtual machine we’ll soon discuss is encapsulated in a file that can be moved from one PC to another, without worrying about hardware compatibility. Virtual machines are isolated entities, and the security of a host is not threatened by an errant application. A virtual machine can let you use a given platform’s operating system software under another OS on the same platform, such as running a copy of Linux in a VM on your Windows 2000 PC. Virtual machine files can be stored on a server, and used more than one person, so a team of developers can use a common configuration to test.
In this article we will look at where Virtual Machines came from, how they work, how they differ from software emulators, and finally a look at the new version of VMware Workstation 3.0 for Windows. VMware is an industrial strength virtual machine company with three levels of VM products: VMware Workstation, VMware GSX server, and VMware ESX server. We’ll be concentrating on the Workstation for Windows product, but we’ll also discuss a few of the differences between the products.
The idea of a virtual machine is not new--its roots actually go back almost to the beginning of computing itself. Initially, the concept of a virtual machine came about in the 1960’s on mainframes as a way to create less complex multi user time share environments. As described by Melinda Varian’s canonical paper "VM and the VM Community: Past, Present, and Future", a time sharing system was developed by a group of MIT programmers, working with equipment donated by IBM. The system, "Compatible Time Share System" or CTSS, was initially developed in 1961 and evolved over the years as the example of how to do time sharing. CTSS was designed much like current multitasking systems, doling out processing time in scheduled slices. The system provided a supervisory program that controlled resources, and scheduled time shares for foreground and background tasks. The key to its operation was the supervisor program’s control of Trap Interrupts. By trapping interrupts, the control program was able to isolate the users or processes from each other.
As the systems developed, changes in the hardware were made to support relocation of memory, a key process to facilitating a virtual memory system. Without the ability to relocate (page) memory, entire programs would need to be swapped in and out to active memory address space. With virtual memory, a big performance boost could be realized.
In the 1960’s, the concept of a "virtualizable machine" was developed, and virtual machine technology became a very popular subject of study, and the key focus of user organizations and conferences in the late 60’s and 70’s. For some deeper background, in late 1964, a project at MIT called CP-40 (Control Program for IBM System/360 Model 40 mainframes) really turned the corner on virtual machines. The idea was to create an operating system that would let each mainframe user have their own IBM System/360 virtual machine (which was originally called a pseudo machine). The subsequent release of a single user virtual environment running atop CP-40 called CMS (Cambridge Monitor System) was the beginning of a long line of IBM VM
operating system products. Later CMS would be called Conversational Monitor System, and it worked in conjunction with CP on IBM System/370 systems. Though IBM was reluctant to invest in VM technology in the 60’s and 70’s, it became fairly successful in the 80’s, and they still sell VM systems today.
According to Mendel Rosenblum PhD, VMware’s Chief scientist and co-founder, standard college textbooks on computer architecture do not have a discussion of the virtualization of processors. Virtual Machines were also not in Intel’s mind as they designed the 64bit IA64 chip architecture, as the processors (like Itanium and the upcoming McKinley) are not completely virtualizable. Rosenblum offers a tongue in cheek theory that maybe the engineers had not been exposed to virtual machine technology in college, so they never considered implementing such capability.
During the 1990’s, at Stanford University, work was being done on designing and building scalable multi-processor machines-- ones that could scale to 1000 or more processors. Rosenblum and a group of graduate students were tackling the problem from the operating system point of view, and came on the idea of building a virtual machine monitor (VMM) with an existing operating system, rather than designing one from scratch. In talking with various software and operating system development companies, the feedback Rosenblum received was favorable on the idea, but he had trouble securing development assistance from those companies. Since Rosenblum’s team was going to use Microsoft operating systems, the idea of having Microsoft work on the project in conjunction with their operating systems seemed like the best idea.
However, rather than asking Microsoft to help integrate a virtual machine directly into the OS, Mendel came up with the idea of running a VMM under the Microsoft operating systems, without requiring core OS modifications. As an extension to that concept, Mendel and his graduate students started experimenting with creating a virtual machine monitor that can also run other commodity operating systems within a VM for use on single or multiprocessor systems.
Since many of the graduate students had friends in entrepreneurial enterprises in nearby Silicon Valley, they started kicking around the idea of starting a company. The initial research and development at Stanford was targeted at building VMM’s on a server. However, the added complexity of the logistics of partnering with server companies soon gave the team the realization that they should start smaller, so they scaled back to a workstation platform to work out the design. VMware Workstation was born, and a company as well.
Before we get too far, we should take a second to define some terms. A Virtual Machine (VM) is defined by Popek and Goldberg (in their paper "Formal requirements for virtualizable third generation architectures," Communications of the ACM, Vol 17, July 1974) as an "efficient, isolated duplicate of a real machine". A real machine has a number of systems that it provides to an operating system and applications for use. Starting at the core, the CPU and motherboard chipset provides a set of instructions and other foundational elements for processing data, memory allocation, and I/O handling. On top of that are hardware devices and resources such as memory, video, audio, disk drives, CDROMs, and ports (USB, Parallel, serial). In a "real machine", the BIOS provides a low level interface that an operating system can use to access various motherboard and I/O resources. With a real machine, when an operating system accesses a hardware device, it typically communicates through a low-level device driver that interfaces directly to physical hardware device memory or I/O ports.
The effective opposite of a real machine is an emulator. An emulator will reproduce everything from the CPU instructions to the I/O devices in software. An emulator can provide cross-CPU operation, such as running Windows software on a Mac. Unfortunately, an emulator takes a performance hit since it must translate every instruction, function call, or data transfer. In addition, an emulator is quite complex, as it needs to emulate most if not all of the CPU instructions.
The functionality and abstraction level of a Virtual Machine lies between a real machine and an emulator. A virtual machine is an environment created by a Virtual Machine Monitor (VMM). The VMM can create one or more VM’s on a single machine. While an emulator provides a complete layer between the operating system or application and the hardware, a VMM manages one or more VMs, with each VM providing facilities for an application or "guest OS" to believe it’s running in a normal environment with access to physical hardware. Instead, when applications or guest OSs execute low-level instructions that inspect or modify hardware state, they appear to the app or OS to be directly executing on the hardware, but are instead virtualized by the VM and passed to the VMM. For traps or interrupts occurring at the application level, they can pass directly to the VMM, which in turn interacts with the hardware.
In the case of VMWare, we’ll see that its VMM isn’t in direct control of the hardware, but actually runs atop a primary or "host" OS for low-level hardware I/O control, while it also plants a low-level driver in the host OS to handle various low-level VM management chores and certain hardware interactions. And VMWare can spawn multiple VMs, each capable of hosting its own guest OS. Confusing? Maybe a little, but before we get into VMWare directly, let’s explore the basic theory of VMMs and VMs a bit more.
As mainframe hardware and software evolved, a concept of virtualization of the hardware developed. A protected processor has two basic modes (and associated instructions)--Privileged (aka System, Supervisor, or Kernel mode) and Non-privileged (aka User mode). Popek and Goldberg’s rules of virtualization state that a virtualizable processor architecture allows any instruction that inspects or modifies machine state to be trapped when executed in any but the most privileged mode. When a trap occurs, rather than generating an exception and crashing, the instruction is sent to the VMM. This allows the VMM to take complete control of the machine. The VMM then either executes the instruction on the processor, or emulates the results and returns them to the VM or app. Some instruction results may be cached by the VMM, so the instruction doesn’t need to be reissued. For example, an instruction that gets the state of a port can be cached if the port hasn’t changed and the VMM knows it.
In a typical mainframe configuration, the VMM runs on top of the hardware in privileged mode, and the VMs run in user mode. When the VM issues non-privileged instructions to the processor, the VMM passes them through to be executed directly. For example, an instruction such as Add or Sub, doesn’t affect the state of the hardware so the VMM executes it directly on the hardware. Most instructions fall into this category, so the performance of many applications running within a VM vs. the Real system is better than running under emulation.
Privileged instructions can only be executed by a process running in a privileged mode of the same or higher level. Privileged instructions issued by processes running in user mode cause a trap. This provides the basis for the isolation of a VM from the rest of the machine. Processors include instructions that can affect the state of the machine, such as I/O instructions, or instructions to modify or manipulate segment registers, processor control registers, flags, etc. VMware calls these "sensitive" instructions. Such sensitive instructions can affect the VMM and other processes if executed by a given process without regard for other running processes. In a virtual machine, sensitive instructions cannot be executed on the CPU directly by applications or guest OSs for that reason. A primary job of a VMM is to trap and handle all such machine-specific sensitive
instructions.
For a processor to be fully virtualized, all sensitive instructions are privileged and are able to be trapped. On traditional virtualized mainframes, when privileged instructions are issued from the VM, they are trapped and passed to the VMM for handling. A specific example of a sensitive instruction would be PushF (push flags) and PopF (pop flags) instructions that save and retrieve the current state flags of the CPU.
As mentioned earlier, one or more processes running in a "real machine" would rely on the OS to manage sensitive instructions. With multiple programs running and issuing sensitive instructions, the OS multiplexes among them and issues one protected instruction at a time to the CPU.
instructions and get the CPU’s attention. This is why the hardware needs to be virtualizable to allow a VMM to keep each VM isolated. On a mainframe, the hardware, and more specifically, the processor, is designed to be virtualizable.
VMM’s are classified as two basic types: the standalone VMM, and the hosted VMM. The stand alone Virtual Machine Monitor is basically an operating system that can create Virtual Machines. The standalone system is the classic main frame type architecture. Such VMMs are typically limited in their hardware support, and similar to an operating system, they require device drivers for each hardware device.
In contrast, a hosted VMM runs as an application on an existing host operating system. The hosted VMM can take advantage of the host OS for memory management, processor scheduling, hardware drivers, and
resource management.
Unfortunately x86 processors, along with most other processors, are not virtualizable, so running a traditional standalone mainframe-style VMM is impossible. However, VMware has combined some patent-pending technology with a hosted VMM architecture to overcome deficiencies of the x86 PC platform, and take
advantage of running on a full-function operating system.
The x86 Difference
With a PC system, there are a couple of things that get in the way of running a VMM directly on the hardware in the same manner as on a mainframe system. First, an x86 CPU is not fully virtualizable as mentioned earlier, which means that the VMM cannot trap all privileged instructions. According to Darryl Ramm, Director of Advanced Technology at VMware, the company’s proprietary technology provides a mechanism for handling non-trapping instructions within a VMM architecture. He emphasized that the core operation of the VMM in VMware products shares much in common with traditional mainframe style VMMs. This includes software running in the VM executing directly on the underlying processor, and strong isolation between virtual machines based on CPU hardware privilege levels. VMware would not talk directly about the technology until their patents have been awarded, but did give us a look at much of the operation of their hosted VMM. Second, the PC platform has a huge variety of devices and hardware that it supports. For a VMM to work directly on the hardware, it would need drivers for every IO device, requiring an almost OS-like selection. Because of this impediment, a VMM runs best on the PC platform on a hosted system, that is, coexisting with a host operating system. In such a configuration, the host OS provides the driver and resource support across a wide range of hardware. In addition, VMware’s underlying technology provides workarounds for the lack of full processor virtualizability. VMware Workstation is a hosted VMM.
VMware offers three virtual machine systems--Workstation, GSX Server, and ESX Server. Workstation is hosted on Windows NT, Windows 2000, or Windows XP Professional. A Linux-hosted version is also available. GSX Server is similar to Workstation in looks, and is hosted on Windows NT or Windows 2000 Server, or Linux. ESX Server is a stand alone system, running directly on the hardware itself, without the need for a host operating system. The products are similar enough in operation across the line, so most things we discuss about Workstation will apply to all.
To understand how Workstation installs and runs, it helps to look at the way the Intel Architecture (IA32) platform works. On the IA32 architecture, the protection mechanism provides four privilege levels, 0 through 3. The levels are most commonly interpreted as rings (see figure below). Intel processors also provide Protected and Real modes of operation along with Virtual-86 mode. The privilege levels exist only in Protected Mode. Exactly how VMware’s VMM is able to handle Real Mode software is a part of the technology they won’t discuss publicly. You are probably aware that DOS was a Real Mode OS and
Windows 9x/Me was comprised of 16- and 32-bit code that ran mostly in Protected Mode, but also had some Real Mode functionality. Windows 2000 and XP are 32-bit OSs that pretty much run in Protected Mode, and Unix and Linux are Protected Mode OSs. Virtual-86 mode is typically used to run a DOS shell from within Windows. Virtual-86 mode can only support running Real Mode environments and is not related to how VMware VMMs operate. Real, Protected, and Virtual-86 mode software can all run transparently within a VMware Workstation VMM.
Theoretically, according to Intel, the host operating system runs its kernel at Ring 0, device drivers at Rings 1 & 2, and applications on Ring 3. In practice, most operating systems run completely at Ring 0, and
applications in Ring 3.
Processor instructions such as those used for interrupt handling are only accessible from Ring 0, or if a Protected Mode application tries to directly access memory through the CPU, it will cause a General Protection exception.
When Workstation installs, it creates three main components-- the VMX driver, Virtual Machine Monitor (VMM), and the VMware application (VMapp). The VMX driver and VMM operate at Ring 0. The Workstation application runs in Ring 3 and looks like any other application to the host operating system.
During installation, the VMX driver is installed within the operating system to gain the high privilege levels that drivers are permitted and that the VMM requires. When executed, the application (through the VMX driver), loads the VMM into Kernel memory, giving it Ring 0 privileges. At this point, the host operating system only knows about the VMX driver and the Workstation application, and is completely ignorant of the VMM. Also at this point, your machine contains two "worlds"-- the host world and the VMM world. The VMM world can communicate directly with the processor hardware or through the VMX driver to the host world. When a "world switch" is done between the VMM and host worlds, all user and system CPU state needs to be saved, which takes a toll in terms of performance as compared with a real system, or a mainframe VMM.
The Workstation application gives you a wizard and configuration editor that lets you create and configure VM’s. The VM looks like a raw machine in which you can install an operating system and applications. We’ll discuss setting up a VM with Workstation in more detail in Part II of this story, but essentially you define drives, I/O, and other peripherals from the configuration editor.
When guest OSs or applications are running purely computational programs, they will run in a standalone mode directly through the VMM to the CPU. Workstation differs from traditional mainframe VMMs in I/O operations as it uses the host OS to perform actual I/O on behalf of a virtual machine. I/O instructions are privileged instructions and the VMM intercepts each request, and switches to the host world. Once in the host world, the Workstation VMapp (VM application portion) performs the I/O request (such as disk read, or network access) on behalf of the VMM through standard host OS system calls. Requests may be translated from low level calls to a higher level call to get equivalent data. For example, a request from a guest OS to read sectors from its disk is translated to a read() by the Workstation VMapp to get the corresponding data. The data in then returned to the guest OS through the VMX driver and the VMM.
By using a hosted environment, some proprietary technology, and the tag team process, VMware can create PC based virtual machine monitors, despite the limited virtualization by the Intel platform. In addition, the VM can take advantage of the operating system’s abstraction of a large number of devices, without having to provide its own drivers. Unfortunately, for I/O intensive operations, performance degradation as compared with a real system is evident. (We present test results in Part II of this story). In addition, the VMM, while it has full system and hardware privileges, must be a good citizen and cooperate by letting the host OS schedule CPU time. Since VMware and the created guest VMs also run as applications under the host OS, they run the risk of being swapped out of memory at inconvenient times. The bottom line is that the performance of the VMM and ultimately any guest OS or application running within a VM is dependent on the scheduling and allocation decisions that the host OS makes. Host OS scheduling of priority levels for any or all VMs can be set in the Workstation preferences panel.
While VMware can output to a wide variety of devices through the host operating system, it is actually limited to a narrow range of generic devices in within the actual VM’s. When you install an operating system on a real PC, the hardware provides specific devices (such as keyboards, printer ports, and hard disks) that the Operating system interfaces to with drivers. Any added cards or peripherals also require driver support. One of the tasks of a VM is to provide a similar environment to the guest operating system. Since the operating system sees the VM environment as a physical PC, so drivers need to be installed to interface with the VM’s hardware emulation.
When you create a guest VM, by default it supports a standard PS/2 keyboard, PS/2 mouse, floppy drive, IDE controllers with ATA disks, and ATAPI CDROMs. You can add SoundBlaster16 sound support, serial, and parallel ports. The VM also lets you install a virtual BusLogic SCSI controller, a virtual AMD Lance PCNet Ethernet card, and VMware’s own virtual SVGA video controller. Most of the hardware devices that VMware emulates are standard branded devices, so guest operating systems already have the drivers built in. However, with the VMware virtual display card, it is not emulating a specific name brand model, such as a Matrox G450. The virtual display card will work with the guest OS’s generic VGA driver, just as any hardware display card does. However, like its hardware counterpart, the VMware virtual display gives better performance when you install the manufacturers driver in your guest OS.
As we described above, any time a VM accesses an I/O device, it gets a performance hit from a world switch. For a keyboard or mouse, this is not a major problem since they are very short duration and low processing overhead events. However, for high throughput devices such as a network controller, the overhead can
become a problem. To overcome some of the overhead problem with world switches, the device emulation in VMware’s VMM understands the context in which I/O requests are made, and uses that knowledge to reduce world switches. In the case of the virtual network card, the VM will make lots of I/O requests, but only a percentage of them will actually require moving packets. The rest are either getting or setting the state of the I/O port. Since the VMapp translates raw I/O requests into higher level network requests, the VM only really needs to pass packet requests with a world switch. A large part of a network packet transaction consists of In/Out requests that are just getting the state of the ports. For these requests, the VMM emulates the data that the VM is looking for. This technique is uses on other I/O devices as well, minimizing the performance penalty of going to the host OS. The following diagram shows the steps that a packet send and receive goes through in VMware Workstation.
VMware’s network architecture (courtesy of VMware)
This concludes the first half of our story. In the second half we’ll delve deeply into VMware’s features and operations, and how you can take great advantage of its unique functionality.
