I'm implementing a PCIe driver, and I'd like to understand at what level the interrupts can be or should be enabled/disabled. I intentionally do not specify OS, as I'm assuming it should be relevant for any platform. By levels I mean the following:
OS specific interrupts handling framework
Interrupts can be disabled or enabled in the PCI/PCIe configuration space registers, e.g. COMMAND register
Interrupts also can be masked at device level, for instance we can
configure device not trigger certain interrupts to the host
I understand that whatever interrupt type is being used on PCIe (INTx emulation, MSI or MSI-X), it has to be delivered to the host OS.
So my question is really -- do we actually have to enable or disable interrupts on every layer, or it's sufficient only at the closest to hardware, e.g. in relevant PCI registers?
Disabling interrupts at the various levels usually has completely different purposes.
Disabling interrupts:
In the OS (really, this means in the CPU) - This is generally about avoiding race conditions. In particular, if state/memory corruption could occur during a particular section of code if the CPU happened to be interrupted, then that section of code will need to disable interrupt handling. Interrupt handlers must not acquire normal locks (by definition they can't be suspended), and they must not attempt to acquire a spin-lock that is held by the thread currently scheduled on the same CPU (because that thread is blocked from progressing by the very same interrupt handler!) so ensuring data safety with interrupt handlers can be tricky. Handling interrupts promptly is generally a good thing, so you want to absolutely minimise such sections in any code you write. Do as much of your interrupt handling in secondary interrupt handlers as possible to avoid such situations. Secondary interrupt handlers are really just callbacks on a regular OS thread which doesn't have any of the restrictions of a primary interrupt handler.
PCI/PCIe configuration - It's my understanding this is mainly about routing interrupts, and is something you normally do once when your driver loads (or is activated by a client) and again when your driver unloads (or is deactivated). This may also be affected by power management events. In some OSes, the PCI(e) level is actually handled for you when you activate PCI device interrupts via higher-level APIs.
On-device - This is usually an optimisation to avoid interrupting the CPU when it doesn't need to be interrupted. The most common scenario is that an event happens on the device, so an interrupt is generated. The driver's primary interrupt handler checks the device registers if the driver needs to do any processing. If so, it disables interrupts on the device, and schedules the driver's secondary interrupt handler to run. The OS eventually runs the secondary handler, which processes whatever information the device has provided, until it runs out of things to do. Then it enables interrupts again, checks once more if there's any work pending from the device and if there are none, it terminates. (If there are items to process in this last check, it re-disables interrupts and starts over from the beginning.) The idea is that until the secondary interrupt handler has finished processing, there really is no point triggering the primary interrupt handler, and a waste of resources, if additional events arrive, because the driver is already busy processing the event queue. The final check after re-enabling interrupts is to avoid a race condition between an event arriving and re-enabling interrupts.
I hope that answers your question.
Related
As far as I know that external events such as hardware interrupts or system calls can trigger control transfer from user space to kernel space. But what if there is no external event or system calls, will the running process keep running forever?
It could, but typically the kernel programs a timer device to cause an interrupt at some point in the future. When that interrupt arrives, the kernel has the opportunity to decide what to do next.
I'm currently taking a class in Operating Systems, and everything has been smooth until I encountered Concurrency and Mutual Exclusion.
Up until this chapter in the text I am currently reading, I was under the impression that the OS handled calls to certain I/O operations such as printers through queues and interrupts, and the OS also handled the scheduling of processes.
But in this section "Mutual exclusion: Hardware support", it states for a process to guarantee mutual exclusion it is sufficient to block all interrupts, and this can be done through interrupt disabling, however the cost is high since the processor is limited in its ability to interleave(Stallings, p. 211).
If this is a capability, whats stopping a programmer from placing his entire program within a critical section by disabling interrupts? And why can't the OS handle calls to critical resources, in the way that was previously stated(I/O queues & interrupts), but we must rely on programmers to identify their critical sections?
I understand the need for to identify critical sections with shared variables and memory space, but I am baffled as to why a program needs to identify its critical section with regard to I/O devices such as a printers and why the OS can't.
This is not [entirely] correct:
But in this section "Mutual exclusion: Hardware support", it states for a process to guarantee mutual exclusion it is sufficient to block all interrupts, and this can be done through interrupt disabling, however the cost is high since the processor is limited in its ability to interleave.
Processors generally support multiple means of synchronization. The simplest is uninterruptible instructions. These will be generally be short instructions such as set a bit or branch if the bit was set already. Such instructions allow synchronization within a single processor.
As you mention, disabling interrupts is another method. Generally, interrupts have priorities. Usually you can disable all interrupts that has a priority lower than specified. That allows disabling all or some interrupts.
Disabling interrupts only works when locking resources that are not shared by multiple processors.
That is why the quote you have in the context you have it is not [entirely] correct. Disabling interrupts on a processor does not synchronize when there are multiple processors. However, in theory, an operating system could disable all interrupts on all processors but such system would be seriously brain damaged because that would hamper the performance of a multi-processor system. But that might work in, say, a quick-and-dirty student project operating system.
If this is a capability, whats stopping a programmer from placing his entire program within a critical section by disabling interrupts?
Disabling interrupts is only possible in kernel mode.
Another method of hardware synchronization is interlocked instructions. These are instructions that lock the memory of the operands and prevent other processors from accessing that memory while the instruction is executing. Sometimes are are simple add integer interlocked and bit set (or clear) and branch interlocked instructions.
In an operating system, what is the difference between a system call and an interrupt? Are all system calls interrupts? Are all interrupts system calls?
Short Answer:
They are different things.
A system call is call by software running on the OS to services
provided by the OS.
An interrupt is usually external hardware component notifying the CPU/Microprocessor about an event that needs handling in software (usually a driver).
I say usually external, because some interrupts can be raised by software (soft interrupt)
Are all system calls interrupts? Depends
Are all interrupts system calls? No
Long answer:
The OS manages CPU time and other hardware connected to the CPU (Memory (RAM), HDD, keyboard, to name a few). It exposes services that allow user programs to access the underlying hardware and these are system calls. Usually these deal with allocating memory, reading/writing files, printing a document and so on.
When the OS interacts with other hardware it usually does so through a driver layer which sets-up the task for the hardware to perform and interrupt once the job is done, so the printer may interrupt once the document is printed or it runs out of pages. It is therefore often the case that a system call leads to generation of interrupts.
Are all system calls interrupts - Depends as they may be implemented as soft interrupts. So when a user program makes a system call, it causes a soft interrupt that results in the OS suspending the calling process, and handle the request itself, then resume the process. But, and I quote from Wikipedia,
"For many RISC processors this (interrupt) is the only technique provided, but
CISC architectures such as x86 support additional techniques. One
example is SYSCALL/SYSRET, SYSENTER/SYSEXIT (the two mechanisms were
independently created by AMD and Intel, respectively, but in essence
do the same thing). These are "fast" control transfer instructions
that are designed to quickly transfer control to the OS for a system
call without the overhead of an interrupt"
The answer to your question depends upon the underlying hardware (and sometimes operating system implementation). I will return to that in a bit.
In an operating system, what is the difference between a system call and an interrupt?
The purpose of an interrupt handler and a system call (and a fault handler) is largely the same: to switch the processor into kernel mode while providing protection from inadvertent or malicious access to kernel structures.
An interrupt is triggered by an asynchronous external event.
A system call (or fault or trap) is triggered synchronously by executing code.
Are all system calls interrupts? Are all interrupts system calls?
System calls are not interrupts because they are not triggered asynchronously by the hardware. A process continues to execute its code stream in a system call, but not in an interrupt.
That being said, Intel's documentation often conflates interrupt, system calls, traps, and faults, as "interrupt."
Some processors treat system calls, traps, faults and interrupts largely the same way. Others (notably Intel) provide different methods for implementing system calls.
In processors that handle all of the above in the same way, each type of interrupt, trap, and fault has a unique number. The processor expects the operating system to set up a vector (array) of pointers to handlers. In addition, there are one or more handlers available for an operating system to implement system calls
Depending upon the number of available handlers, the OS may have a separate handler for each system call or use a register value to determine what specific system function to execute.
In such a system, one can execute an interrupt handler synchronously the same way one invokes a system call.
For example, on the VAX the CHMK #4 instruction, invokes the 4th kernel mode handler. In intel land there is an INT instruction that does roughly the same.
Intel processors have supported the SYSCALL mechanism that provides a different way to implement system calls.
I'm reading about interrupt handling in mondern CPUs and operating systems, but I can't figure out one point:
As soon as some hardware device changes the state (current/voltage?) on an interrupt pin of the CPU, the CPU stops after processing the prevailing instruction and jumps to execute the interrupt handler code. Now imagine the interrupt handler code has to change some kind of state in scheduler's data structures, however before the OS was interrupted it was also fumbling around in the same structures. That would lead to messed up data, so there must be a solution.
I would guess the OS and the interrupt handler both use a semaphore, implemented through some atomic compare/set memory operation to protect the shared data structures. However, if the OS gets interrupted while holding such a semaphore, the interrupt handler could not do anything and the interrupt would just vanish, because busy waiting for that semaphore would never return control to the OS, hence the lock is never released.
How is this problem solved? There must be some trick that I'm missing...
Maybe an hardware detail you are missing can explain your confusion.
Whenever an hardware interrupt occurs, something along these lines happen:
1 - The CPU goes to a privileged mode, further hardware interrupts are disabled (normally a bit in the processor flags register), and execution jumps to the interrupt handler.
2 - Once the OS interrupt handling is done, it re-enables hardware interrupts, so further interrupts can happen.
So, in short, the OS/interrupt handler can control when hardware interrupts are allowed to interrupt the normal flow.
An easy solution to your problem would be just have the OS disable hardware interrupts while messing with those data structures.
In practice, things get more complex to minimize interrupt latency.
Things can change from one architecture to another, but the basic principle is still that further hardware interrupts are disabled when one happens, and they can be enabled/disabled (provided the CPU is running in the required privileged modes).
Check the end part of this: http://en.wikibooks.org/wiki/X86_Assembly/Advanced_Interrupts
I am now reading materials about preemptive multitasking - and one thing escapes me.
All of the materials imply, that operating system somehow interrupts the running processes on the CPU from the "outside", therefore causing context switching and the like.
However, I can't imagine how would that work, when operating system's kernel is just another process on the CPU. When another process is already occuping the CPU, how can the OS cause the switch from the "outside"?
The OS is not just another process. The OS controls the behavior of the system when an interrupt occurs.
Before the scheduler starts a process, it arranges for a timer interrupt to be sent when the timeslice ends. Assuming nothing else happens before then, the timer will fire, and the kernel will take over control. If it elects to schedule a different process, it will switch things out to allow the other process to run and then return from the interrupt.
Hardware can signal the processor - this is called an "interrupt" - and when it occurs, control is transferred to the kernel (regardless of which process was executing at the time). This function is built in to the processor. Specifically, control is transferred to an "interrupt handler" which is a function/method within the kernel.The kernel can schedule a timer interrupt, for instance, so that this happens periodically. Once an interrupt occurs and control is transferred to the kernel, the kernel can pass control back to the originally executing process, or another process that is scheduled.