I want to clarify before the question that I am not an established professional programmer in any position at any firm. This is solely to satisfy curiosity, and will not pertain to any task or project at this time.
As I understand it, firmware is software placed on hardware to grant it autonomous functionality from instructions, which is given through some form of input; As long as the input stream is readable, which is made possible through drivers. Drivers are software packages with pre-written reference libraries that recognize a specific set of instructions for each possible function in the attached device.
NOTE: not quoted, so I'm aware that this could be inaccurate.
What I want to know is how firmware or drivers are placed on devices without installation through an OS or storage medium; such as a DVD or USB? Specifically firmware installed by manufacturers, like bios and keyboard drivers that are present on all computers. I'm assuming these are less or not reliant on compilation in order to function properly, which is the sole reason I'm asking this question.
Can firmware be developed without compilation?
References
Demystifying Firmware
C++ Kernel Development
Starting Firmware Development
These just explain that an OS is a type of firmware, and that firmware is primarily developed in C with Assembly and C++ as plausible alternatives; pertaining to kernel development as well.
Yes, especially in the larger components. An example involving lua is http://nodelua.org/doc/index/
However, firmware development is typically an extremely memory (and frequently CPU) constrained environment.
C (or traditionally, assembler) is often preferred because it can produce extremely small executables, and is very efficient in stack usage. This matters when you're counting memory in bytes, or kilobytes.
Using a non-compiled language means you need to include a tiny interpreter, and you might not be able to set aside enough memory for this.
You've made an edit, wherein you suggest that an "OS is a type of firmware".
This can be true, in a manner of speaking.
Often firmware itself can consist of an operating system, with components. As an example, the firmware in some home internet routers will contain an OS (which might very well be linux!), however it is still regarded as firmware. There is a bit of a grey area between a computer that is an "embedded device with firmware", vs that of a 'regular computer with regular software', but generally firmware is a computer system running in a very constrained environment, often with very specific uses.
NetBSD includes Lua in it's kernel. Many systems have been developed that do not use Assembly (except for a small part of it), C, or C++, but instead use some other language - though it is typically still compiled for size and performance reasons.
As for the actual transfer of firmware (whatever the form it may be in), this depends on the device in question.
Some devices require that the firmware be burned into the components. (In ROM, though there are various types of ROM and some can be rewritten).
Other devices require that the firmware be transferred when the device is turned on.
And yet others have SDCards or battery-backed RAM or whatever that allow storing the firmware across reboots.
Related
Some background:
I am a Computer Engineering major attending school right now, and I just completed a project that created a microprocessor with a super simple instruction set that ran on an FPGA. I chose to implement a simple file storage scheme, a VGA text only display output, and a PS/2 keyboard input. I wrote two main programs, a firmware that was in ROM in the processor and a kernel that provided a bunch of library type functions, and was capable of loading and executing files from the filesystem. This project was challenging and overall a lot of fun.
My Question:
I want to do some super low level programming on a modern computer, but I can't seem to find any resources or documentation that help me get started. To be clear, I want to find the proper documentation that would help me to write a program in C, x86, or x86-64 assembly that I could compile, and format into some form of bootable data. I know this is a daunting task, and typically not something a hobbyist would take on, but I know that it's possible (Terry Davis's TempleOS).
Are there any websites or books that I can read that will contain the specifics needed to make something like this?
Look out, you might just catch the bug. OS Development, though having a very small demographic, is still quite a thriving hobby. Once you start, you may not ever give it up.
Since your subject line states 64-bit and you use the term modern hardware, be advised that modern hardware no longer has the older style BIOS, where the developer wrote the boot process which included the video out, file system in, and other standard routines. Modern hardware now uses an EFI firmware which does all of the booting for you, including the reading from the file system(s). For modern hardware, OS development really starts with the OS Loader, the part that loads the OS, and this is done in a high level language such as C/C++. Very little if any assembly at all, in fact that is its point/purpose.
Don't get too discouraged, currently a lot of computers still allow the old style boot. However, the old style boot starts in 16-bit mode, moves to 32-bit mode, then if desired, moves to long mode (64-bit). There also are emulators that you can use so you don't have to have a separate system, just to test your development. I prefer Bochs myself, but I am a little bias since I wrote some of the code for it, namely most of the (original) USB emulation.
If you wish to dip your toes into this hobby, there are numerous places to start. I personally wrote a few books on the subject. They show you how to start from when the time the POST gives up control to your boot code, up to the point of a minimal Round Robin style task/thread switching OS, with all the necessary hardware and software basics to do so. There is a forum to OS Development, along with its wiki.
Again, a project like this is not for the faint at heart, though it is an enjoyable hobby most have found to be a very good learning experience.
Do interpreted languages such as Java and Python need an operating system to work?
For example, on a bare-metal ARM microcontroller, can an interpreter be installed such that we can have both compiled code such as C, and interpreted code such as Python working together, Or is an OS needed to support this?
Of course you can write an interpreter that runs on bare-metal, it is just that if the platform does not have an OS any run-time support the language needs must be part of the interpreter. To the extent in some cases that such an interpreter might essentially be an OS. That is if it provides the services to operate a system, it could be called an operating system.
It is not perhaps as simple as interpreted vs compiled. Java for example runs on a virtual machine and is "compiled" to bytecode. The bytecode is interpreted (or just-in-time compiled in some cases), rather then the Java source directly. In an embedded system, it is possible that you would deploy cross-compiled bytecode on the target rather then the source. Certainly however JVMs exist for bare-metal. Some support multi-threading through a third party RTOS, others either have that support built-in or do not support threading at all.
There are interpreters for cut-down subsets of JavaScript and Python that run on bare-metal microcontrollers. I am not sure about full implementations, but it is technically possible given sufficient run-time support even if not explicitly implemented. To fully support some of these languages along with all the standard and third-party libraries and frameworks a developer might expect, may require so much run-time support and resource that it is simpler to deploy and OS, so implementations for resource constrained systems are often subsets or have restricted libraries.
Java needs a VM - virtual machine. It isn't interpreted, but executes byte code. Interpreted would mean grabbing the source in run-time as it goes, like BASIC.
When Java was new and exciting around year 2000, everyone thought it would be the next big general-purpose language, replacing C++. The syntax was so clean, it was "pure OO" and not some "filthy hybrid".
It was the major buzz word of the time. Schools stopped teaching C and C++. MCU manufacturers started to make chips with Java VM in hardware. Microsoft made their own Java "standard". Everyone was high on the Java hype.
Then as the Internet hype as whole collapsed in 2002, it took the Java hype with it. In the sober hang-over afterwards, people started to realize that things like byte code, VMs and garbage collection probably don't belong on bare metal systems.
They went back to using compiled C for hardware-related programming. Or in fact they never stopped, since Java never quite made it there, save for some oddball exotic architectures.
Java remained used only in the areas were it was suitable, namely web, desktop and mobile development. And so it got a second golden age when the smart phone hype struck around 2010.
No. See for example picoJava, which is one of several solutions for running Java natively. You can't get closer to bare metal than running bytecode on the CPU.
No. Some 8-bit computers had interpreted languages in ROM despite not having anything reasonably resembling a modern operating system. The Apple 2 is one example. You could boot the system without any disks or tapes, and it would go straight to a BASIC prompt, where you could write basic (no pun intended) programs.
Note that an operating system is somewhat of a vague term when speaking about these days - these 8-bit computers did have some level of firmware, and this firmware did provide some OS-type functionality like access to basic peripherals. In these days, what we now know as an OS was more commonly called a "DOS" - a Disk Operating System. MS-DOS is one of them, as well as Apple's ProDOS. These DOS's evolved into our modern-day operating systems (e.g. Windows 95 was based on top of MS-DOS, while modern Windows versions derive from a separate branch that was largely re-implemented with more modern techniques), so one could claim that their ancestors are the closest they had to what we now call an OS.
But what is an interpreter but a piece of software?
In a more theoretical sense, an interpreter is simply software - a program that takes input and produces output. Suppose you were to implement a custom solid-state Turing Machine. In this case, your "input" would be the program to be interpreted, and the "output" would be the program's behavior. If "software" can run without an operating system, then an interpreter can.
Is this model a little simplified? Of course. The difference is a matter of degree, not nature. Add very basic user input and output capabilities (e.g. a TTY) and you have the foundation to implement all, or nearly all, of the basic functionality of a language such as Java byte code, Python, or BASIC. The main things you would be missing are libraries and whatnot that depend on things like screen manipulation, multiprocessing, and networking, but you could handle them with time too.
I need to know Does a operating system design for specific processors category?
and also can any operating system run on any microprocessor?
Generally speaking, an operating system is not designed for a specific processor; though some do make assumptions about the hardware and computer system over all that might not be available in all systems. That said, for an operating system to run on a partical architecture, there is usually code that performs some specific, critical functions that is implemented for a specific architecture, frequently being written in assembly (I know of no OS that doesn't do this). To enable a new architecture, this code needs to be rewritten for the new machine, so that means new assembly most of the time. As mentioned in the comments, there are operating systems that only run on a single architecture like Windows, while others have these specific components for a number of architectures and thus can run on a number of processors like Linux. Note however the same exactly binary will not run across architectures, the operating system needs to be rebuilt for each architecture and possibly even for the same architecture if the system itself is different enough (as can be the case with some small MCUs).
So to answer your two questions directly: no, an OS is not uaually designed for a specific processor, and no, any OS cannot run on any processor.
Historically, operating systems have been designed for specific hardware. In some cases, such as eunuchs, the system was reworked so that it could be ported to multiple systems.
M$ ported Windoze to the Alpha processor in order to placate Digital and avoid lawsuits.
[C]an any operating system run on any microprocessor?
No.
I am making my own operating system for fun. I'm reading and learning a lot from the Intel's 64bit and IA-32 architecture manual which is helpful for developing an operating system.
But one thing that I just can't find anywhere is how to make device drivers. The only posts I can find about device drivers on the internet are about making Linux or Windows drivers. But it's different from making drivers for my own kernel.
Can someone experienced please point me into the right direction, enlighten me where I can learn making my own device drivers for my own operating system?
For a start, simple keyboard or mouse drivers are enough, but I will appreciate anything, be it simple keyboard drivers, or complex network drivers
TL;DR: I need a solid source explaining how to make device drivers for my own kernel. I can't find any and hope that someone have stumbled upon one
(I'm not asking "how to make drivers?", but rather asking "can I have links to books/posts about making drivers?")
All a kernel-mode device driver is, is code executing under the kernel, which is not treated as the "main" kernel image itself - consider it as an "extension" to the main kernel image. On Windows for example, kernel-mode device drivers (which tend to use the *.sys file extension, which is also belonging to the Portable Executable (PE) Format) are treated as modules to ntoskrnl.exe (which is the main kernel image on Windows).
The idea is that your kernel-mode device drivers will have access to routines exported by your main kernel image (e.g. if you want to implement mechanisms like kernel-mode callbacks). You could start by making your own simple format for your kernel to understand, and then you can start implementing export support appropriate for your own file format (for the main kernel image) and import support (for the other kernel-mode software to make use of the APIs which it should have access to, provided by the main kernel image).
You'll have a lot of work to do if you want a good design implementation.
When reading the documentation of BLE chip from a semiconductor manufacturer that there is a small real time kernel on board, not OS. This is probably just a stupid question, but what would be the difference between a full fledged RTOS vs a real time kernel in such a scenario.
There are a variety of opinions on that question. One, held by systems programmers and the like, is that an OS kernel is a minimal set of services to abstract the underlying computer hardware, making it easier to write an OS or even applications on the machine. OS kernels typically operate in their own space, separate from user space where the rest of the OS (or even applications) reside. There is also the concept of a microkernel (and nanokernel), which is intended specifically to serve higher level (e.g., user space) OS services--although different microkernels partition OS services differently in kernel vs. user space. Check Wikipedia as a start (although some of its related entries have some factual errors).
Sometimes the embedded computing community refers to a kernel as a minimal OS, expecting applications, not more OS functionality, to use the kernel.
This may seem like an insignificant distinction, but it does effect the design and implementation of the kernel.