I'm currently doing a course about Operating Systems.
I understand that a kernel is a core part of an operating system that acts as a bridge between user applications and the data processing elements of a computer such as the CPU.
Why do we need Drivers then (e.g Touchpad drivers), doesn't the kernel control all computer hardware?
Because there are literally thousands (if not hundreds of thousands) of companies that produce hardware devices. The Operating System company cannot write software to handle all of them, so they provide a generic programming model using which those hardware companies can write software that could talk to their hardware.
Important to note is that although drivers are not actually part of the kernel as such, they do have some low-level privileges (direct access to hardware) because their code runs in Executive mode, unlike normal applications which run in User mode and generally do not access hardware directly. The whole point is that once the OS provides a way to write hardware-controlling software (called drivers), any vendor or person is free to write a software that could take advantage of the specialized features of their/his hardware device.
Also note that some hardware devices follow well-known standard (such as keyboards, mice, many video drivers, monitors etc.) and most OS come with built-in support for those devices. On the other hand, some devices do not have or follow standards, while still other can have both kinds of features, i.e. a subset of features that could be accessed by the default driver, plus a subset of features that doesn't follow any standards and thus is not supported by default driver. In all such cases, hardware manufacturers supply their own driver that knows the ins and outs of their hardware and therefore can take advantage of all features efficiently.
Certainly kernel does control all the hardware. However, there are way too many different hardware devices out there.
To deal with this vast variety of different devices, developers write specialized modules - and these are called drivers.
Related
Preface: I'm a student about to take a course in Operating Systems. I thought I'd do some prep by watching a series on YouTube first.
Throughout the course of watching about 10 of the videos in this series, I have learned that roughly the operating system's purpose is to serve as an interface for System Programs/Applications/I/O devices/etc. to communicate with the system's hardware.
This got me thinking about how Apple's hardware is not modularly customizable. How Apple users can't swap out hardware components as easily as users on a system running Windows OS. I began to think that most likely this is because the OS Apple implements is built very specifically with the original hardware their products come with to run as efficiently as possible.
Is there any truth to this logic? I'm basically just trying to apply what I've been learning to a "real-life" example.
This got me thinking about how Apple's hardware is not modularly customizable.
It is quite customizable but not with their own hardware. The "hardware" that Apple is shipping is mostly x86-64 CPUs with a recent chipset like an xHCI, an AHCI and a modern PCI network card, etc. This is unless you have an M1 computer which is their most recent product based on an ARM architecture. They licensed the architecture from ARM ltd. and are manufacturing their own CPU. I think this is a very good and open decision from Apple unlike several bad ideas they had about their phones like removing the 3.5mm jack or using a lightning plug instead of USB-C.
If you do have an x86-64 CPU, the OS Apple built called MacOS can run on the computer. It is simply forbidden by Apple's license of use. The fact that their hardware is less customizable has mostly to do with screws and the way that the case is made than with the OS itself.
I'm curious about how the drivers work out of the box in x86 motherboards, like display, USB controllers etc.
For example :
Booting a toy custom kernel in x86 can display to screen without doing any extra work on the drivers space, however for an Android phone which is an embedded system, it seems almost impossible to display to screen with my own toy custom kernel (as there is information out there available about the memory map of the device and how the display is interfaced with the device).
Why is that I/O works out of the box in x86 motherboards and doesn't on embedded computers?
x86 PC firmware has standard software interfaces (a lot like system calls), either modern UEFI or legacy BIOS int 0x10 and other interrupts.
The key point is that it's not just bare-metal x86, it's IBM PC-compatible which means software and even emulated legacy hardware like a PS/2 port, VGA, and even legacy interrupt controller.
If you didn't have all this help from firmware (for the benefit of bootloaders and toy OSes), you'd have a much harder job, e.g. needing at least a basic USB-hid and USB host-controller driver to get keyboard input. The lowest level function to handle a user input
Why is that I/O works out of the box in x86 motherboards and doesn't on embedded computers?
That's not your real question. Embedded machines have working I/O hardware, they just don't come with portable software APIs/ABIs wrapped around drivers as part of the firmware.
I think most vendor's SDK comes with functions to access the I/O hardware (after maybe doing some fiddling to get it into a usable state). i.e. to write your own driver for it.
Embedded doesn't need this in firmware because it's expected that the kernel will be customized for the hardware.
Wouldn't it be better to have a BIOS or UEFI for maximum portability? Does it have any drawbacks to include one?
Yes: code size in the boot ROM, and someone has to write + debug that code. This costs time and developer salary.
No point in booting up what's nearly an OS (a UEFI environment) just to load a kernel which is going to take over the HW anyway.
Also a downside in boot time: any code that runs other than loading the kernel where it wants to be is wasted CPU time that slows down the boot. Having a very lightweight interface that just lets you get your kernel loaded, and leaving all the I/O to it, makes sense for this.
Unlike x86 PCs, there's no expectation that you can use this hardware with an OS install disc / image you downloaded that isn't specifically customized for this hardware.
It's not intended to be easy for hobbyists to play with using training-wheels APIs. Real OSes on this hardware won't use such APIs so why provide them in the first place?
Does the OS get this information from the BIOS or does it scan the buses on its own to detect what hardware is installed on the system. Having looked around online different sources say different things. Some saying the BIOS detects the hardware and then stores it in memory which the OS then reads, others saying the OS scans buses (e.g pci) to learn of the hardware.
I would have thought with modern OSs it would ignore the BIOS and do it itself.
Any help would be appreciated.
Thanks.
Generally speaking, most modern OSes (Windows and Linux) will re-scan the detected hardware as part of the boot sequence. Trusting the BIOS to detect everything and have it setup properly has proven to be unreliable.
In a typical x86 PC, there are a combination of techniques used to detect attached hardware.
PCI and PCI Express busses has a standard mechanism called Configuration Space that you can scan to get a list of attached devices. This includes devices installed in a PCI/PCIe slot, and also the controller(s) in the chipset (Video Controller, SATA, etc).
If an IDE or SATA controller is detected, the OS/BIOS must talk to the controller to get a list of attached drives.
If a USB controller is detected, the OS/BIOS loaded a USB protocol stack, and then enumerates the attached hubs and devices.
For "legacy" ISA devices, things are a little more complicated. Even if your motherboard does not have an ISA slot on it, you typically still have a number of "ISA" devices in the system (Serial Ports, Parallel Ports, etc). These devices typically lack a truly standardized auto-detection method. To detect these devices, there are 2 options:
Probe known addresses - Serial Ports are usually at 0x3F8, 0x2F8, 0x3E8, 0x2E8, so read from those addresses and see if there is something there that looks like a serial port UART. This is far from perfect. You may have a serial port at a non-standard address that are not scanned. You may also have a non-serial port device at one of those addresses that does not respond well to being probed. Remember how Windows 95 and 98 used to lock up a lot when detecting hardware during installation?
ISA Plug-n-Play - This standard was popular for a hot minute as ISA was phased out in favor of PCI. You probably will not encounter many devices that support this. I believe ISA PnP is disabled by default in Windows Vista and later, but I am struggling to find a source for that right now.
ACPI Enumeration - The OS can rely on the BIOS to describe these devices in ASL code. (See below.)
Additionally, there may be a number of non-PnP devices in the system at semi-fixed addresses, such as a TPM chip, HPET, or those "special" buttons on laptop keyboards. For these devices to be explained to the OS, the standard method is to use ACPI.
The BIOS ACPI tables should provide a list of on-motherboard devices to the OS. These tables are written in a language called ASL (or AML for the compiled form). At boot time, the OS reads in the ACPI tables and enumerates any described devices. Note that for this to work, the motherboard manufacturer must have written their ASL code correctly. This is not always the case.
And of course, if all of the auto-detection methods fail you, you may be forced to manually install a driver. You do this through the Add New Hardware Wizard in Windows. (The exact procedure varies depending on the Windows version you have installed.)
I see a lot of info about system hardware, except for memory ,one of the main important part besides the cpu, which funnily isn't really mentioned as well.
This is fair because perhaps there's so many things to enumerate, you kind of lose sight of the forest through the trees.
For memory on x86/64 platforms you will want to query either BIOS or EFI for a memory map. for BIOS this is int 0x15 handle 0xe820. EFI has it's own mechanism which provides similar information.
This will show you which memory ranges are reserved by hardware etc. in order for your OS to know to leave them alone. (ok you have to built that part too of course ;D)
For other platforms, often the OS will be configured for a fixed memory size, like in embedded platforms. There is no BIOS for you, and performing a sort of bruteforce on memory is unreliable at best. (as far as i know! - not much experience outside of x86/64!!!)
For the CPU you will definitely want to look into MSRs, control registers and CPUID functions to enumerate the CPU and see what it's capable of. you can query if for example 64 bit mode is supported, and some other features which might not be present on all cpus.
For other hardware like pci etc, i would recommend like myron-semack said to look into PCI specification, pci-express, and importantly ACPI as implementing that will make you handle hardware and powermanagement a . bit more generically / according to newer standards.
Just a thought, if we have to make our application cross-platform, then is it possible to create a cross-application OS?
No.
Lets say you do go and invest - a monumental amount of - effort in building you're Uber-OS (that will run Mac apps, Linux apps, Unix apps, Android apps, i-phone apps, Nokia apps, Symbian apps, SAP apps, Windows Apps etc).
Then there's nothing stopping someone writing a new OS that you don't support.
P.S. And there are hundreds (if not thousands) of different hand held devices out there for scanning products, weights and mesures etc many of which have their own flavour of OS.
Technically yes as long as you limit the scope of all to all applications that run on major OSes.
It is theoretically possible to create an OS that could handle applications run on the 4-5 most common OSes but the amount of work involved would be monumental.
Every time a new feature was added to any of the OSes, you'd need to add it to your OS too - So as well as being almost impossible to build, you'd need a large enough dev team to stay ahead of 4-5 of the largest dev teams/groups in the world.
No but with virtualization you could have a single computer that can run any application.
First there is the practical impossibility of successfully following the evolution of an indefinite number of operating systems. Do we take embedded OS into account? How about one-shot OS for specific applications? How about proprietary OS with no access to documentation?
Then there is also the - very difficult, if not impossible - problem of merging the various paradigms used in the wild. Ideally you would want OS services like the clipboard, or networking or ... or ... to work in a uniform way and allow applications to cooperate as if targeted to the same OS.
(Let's not even think about the various hardware-dependent applications.)
After all this, you should also consider what the application development for your own OS would be like...
I wonder if this is a good case for Gödel's incompleteness theorems :-)
PS: That said, there are quite a few projects attempting to bridge the various OS gaps:
http://en.wikipedia.org/wiki/List_of_computer_system_emulators
http://en.wikipedia.org/wiki/List_of_emulators#Operating_System_emulators
What you can do is use virtual machines, such as VMWare's software, and emulate several operating systems on the same physical machine.
What do you define by an operating system that can run all applications?
Applications are mostly written in a higher level language and then translated into binary code that differs between machine architectures (like Intel and PowerPC) and operating systems (like Windows or Unix-based systems).
Java for example is only cross-platform because not the language itself is cross-platform (any high level language is), but because there exist Java virtual machines for different architectures and operating systems that abstract the heterogeneity of the underlying system.
It is definitely not theoretically impossible (nothing is except for some mathematical problems), but can you imagine what one would have to do in order to make such a thing work? You can basically run Linux programs in Windows with CygWin, you can also run Windows programs in Linux with Wine. All of those try to create a small operating system (e.g. the Windows core) into your other OS (e.g. Linux). This is probably not what you want.
To summarize, I can't imagine anyone really trying to do that. With all the money in the world, seriously. Better invest in writing native apps for the operating systems you want to support.
Someone told me that for most operating systems, the drivers become a part of the kernel. How does this happen? Does the kernel decompile itself, add the driver, and recompile itself? Or are the drivers plug ins for the kernel? Are drivers even their own separate programs?
I'm going to answer this even though it was asked 7 years ago for those who stumble on it all these years later.
The Kernel is the heart of the Operating System and thus includes support for a wide-variety of functionality which will be relied on system-wide.
The System Service Routines do not have to be present under a single kernel image (e.g. on Windows, there's FltMgr.sys to handle the Filesystem, and this is communicated with through device control routines).
A kernel-mode device driver is in layman's terms, an extension to the kernel. You'll be executing with a Current Privilege Level of 0 (which is for Ring 0) and thus the restrictions on memory access and usage of specific instructions will be revoked for you. Your trust level will also be better.
A kernel-mode device driver will essentially be a "module" to the kernel. You have a Dynamic Link Library (DLL) in user-mode on Windows, or a Dylib on OS X... Think of a kernel-mode device driver as the kernel-mode equivalent, except it doesn't have to be about extending the actual kernel, it can be for functionality which must be implemented at a kernel-level for an third-party application.
Furthermore, the idea behind it is that the main kernel is able to provide access to a set of APIs for the third-party kernel-mode software to rely on. Otherwise, the third-party developer would have to implement literally everything themselves, and that would basically be "OS development".
Bullet-points:
Kernel-Mode device drivers have the same privileges as the actual OS kernel.
Kernel-Mode device drivers are supposed to have the ability to use some of the OS kernel's APIs to assist with implementation of functionality which works with how the OS was designed (e.g. Filesystem, Process, Registry (or equivalent), memory management, etc.).
Kernel-Mode device drivers are basically an "extension" to the OS kernel because you are not replacing the actual kernel but you're also at the same privilege level (mentioned as #1) and could theoretically extend system-wide functionality if you felt like it (e.g. add support for user-mode software to rely on the device drivers functionality through communication methods like call gates or Inter-Process Communication).
I hope this somewhat helped any future stumblers of this extremely old thread; the question was really good.
A driver is compiled into a library that exposes a known interface. The kernel then scans for drivers on startup and loads them into kernel memory. Some operating systems, such as linux, also support kernel modules that can be loaded / unloaded while the OS is running...