Question about Message Signaled Interrupts (MSI) on x86 LAPIC system - operating-system

Hi I'm writing a kernel and plan to use MSI interrupt for PCI devices.
However, I'm also quite confused by the documentations.
My understanding about MSI are as follow:
From PCI device point of view:
Documentations indicate that I
need to find Capabillty ID = 0x05 to locate 3 registers: Message control (MCR), Message Address (MAR) and Message Data (MDR) registers
MCR provide control functionality for MSI interrupt,
MAR provide the physical address the PCI device
will write once interrupt occurs
MDR forms out the actual data it will write into the physical address
From CPU point of view:
Documentation shows that Message Address register contains fixed top of 0xFEE, and following by destination ID (LAPIC ID) and other controlling bits as follow:
The Message Data register will contain the following information, including the interrupt vector:
After reading all of these, I am thinking if the APIC_ID is 0x0h would the Message Address conflict with the Local APIC memory mapping? Although the address of FEE00000~FEE00010 are reserved.
In addition, is it true that the vector number in MDR is corresponding to the IDT vector number. In other words, if I put MAR = 0xFEE0000C (Destination ID = 0, Using logical APIC ID) and MDR = 0x0032 (edge trigger, Vector = 50) and enable the MSI interrupt, then once the device issues an interrupt CPU would correspondingly run the function pointed by IDT[50]? After that I write 0h to EOI register to end it?
Finally, according to the documentation, the upper 32 bit of MAR is not used? Can anyone help on this?
Thanks a lot!

Your understanding of how to detect and program MSI in a PCI (or PCIe) device is correct.*
The message address controls the destination (which CPU the interrupt is sent to), while the message data contains the vector number. For normal interrupts, all bits of the message data should be 0 except for the low 8 bits, which contain the vector.
The vector is an index into the IDT, so if the message data is 0x0032, the interrupt is delivered through entry 50 of the IDT.**
If the Destination ID in an interrupt message is 0, the Message Address of the MSI does match the default address of the local APIC, but they do not conflict, because the APIC can only be written by the CPU and MSIs can only be written by devices.
On x86 platforms, the upper 32 bits of the message address must be 0. This can be done by setting the upper part of the message address to 0 or by programming the device to use a 32-bit message address (in which case the upper message address register is not used). The PCI spec was designed to work with systems where 64-bit MSI addresses are used, but x86 systems never use the upper 32 bits of the message address.
Reprogramming the APIC base address by writing to the APIC_BASE MSR does not affect the address range used for MSI; it is always 0xFEExxxxx.
* You should also look at the MSI-X capability, because some devices support MSI-X but not MSI. MSI-X is a bit more flexible, which inevitably makes it a bit more complicated.
** When using the MSI capability, the message data isn't exactly the value in the Message Data Register (MDR). The MSI capability allows the device to use several contiguous vectors. When the device sends an interrupt message, it replaces the low bits of the MDR with a different value depending on the interrupt cause within the device.

Related

Does modern operating-systems use the MCFG ACPI table to find the registers of the xHCI?

I'm working on a minimal hobby operating-system. I want to write a driver for xHCI in order to support USB keyboards and mouses. I'm using QEMU for virtualization. I'm passing the -device qemu-xhci parameter to qemu so that it emulates the xHCI. I was wondering if the modern operating-systems use the MCFG ACPI table to find the configuration space of PCI (and the registers of the xHCI)? I was also wondering how to enable PCI-Express in QEMU so that I can find a MCFG table in RAM so that I can support the latest technology in my hobby OS.
To find all xHCI controllers you search PCI configuration space for devices ("functions") with matching "class/subclass/progID" values (see note 2); which means that you have to find a way to access PCI configuration space first.
On 80x86; there are 3 possible ways to access PCI configuration space - 2 that use IO ports ("mechanism #1" and the deprecated "mechanism #2"), and one that maps PCI configuration space into the physical address space (called "Enhanced Configuration Access Mechanism").
If the Enhanced Configuration Access Mechanism is supported; the MCFG ACPI table describes how PCI configuration space is mapped into the physical address space. Primarily; PCI buses are described as "groups of buses", where each group (defined by a "starting bus number" and "total buses in this group" pair) has a base physical address, and the correct physical address for a PCI function is determined by finding information for the relevant group of buses for the requested bus number, then doing a calculation like:
physical_address = base_physical_address_for_group +
(bus_number - starting_bus_number_for_group) << 20 +
device_number << 15 +
function_number << 12 +
offset;
Note 1: because most operating systems use virtual memory it's possible for an OS to create a nice "virtually linear" mapping of the ("possibly physically disjoint") physical memory areas described by MCFG ACPI table (while using the same page full of zeros mapped as read-only to fill any gaps in the "virtually linear mapping"); so that the OS can use a simplified approach (with no need to find information for the relevant group of buses) like:
virtual_address = PCI_config_space_base_virtual_address +
bus_number << 20 +
device_number << 15 +
function_number << 12 +
offset;
Note 2: An OS doesn't/shouldn't literally search PCI configuration space each time it wants to start a device driver for one specific type of device. Instead an OS typically enumerates PCI buses once during boot (and possibly after boot in response to a notification if "hot-plug PCI" is supported) and starts device drivers based on the results of that enumeration. In other words, it's more like "I found an xHCI controller and need to start the appropriate driver" and not like "I want to start an xHCI driver and need to find the appropriate device/s".

Can't receive from USB bulk endpoint despite Windows enumerates and libusb reads descriptor of STM32 custom device class

For a fast ADC sampling USB device, I am using the USB 2.0 High Speed capable STM32F733 with the embedded USB-HS PHY. In USBView, I can see that the device is enumerated, the libusb code opens the device and claims interface, but when I try to receive data with libusb_bulk_transfer, the operation times out (return code -12). Things I have tried: I have confirmed than when I request data with libusb_bulk_transfer, the device is interrupted. Note: I have DMA enabled in my class configuration C file and it is not clear to me how that is triggered. I have verified that the transfersize and packet count registers are being set correctly by the LL library function, and that when I request data from
Any tips on debugging such problems will be much appreciated - this board is my undergrad thesis due in under two months!
Desktop sequence:
libusb_get_device_list, libusb_get_device_descriptor, libusb_open, libusb_get_string_descriptor_ascii, libusb_free_device_list, libusb_bulk_transfer(devh, fat_EPIN_ADDR, inframe, fat_EPIN_SIZE, &gotBytes, 100). Where gotBytes is integer, and inframe is a large array.
Device firmware:
MX_USB_DEVICE_Init();
uint8_t txBuffer[10*fat_EPIN_SIZE];
while (1)
{
USBD_LL_Transmit(&hUsbDeviceHS, Custom_fat_EPIN_ADDR, txBuffer, Custom_fat_EPIN_SIZE);
HAL_Delay(1);
}
Custom_fat_EPIN_SIZE is 0x200 and the endpoint address is 0x81 (EP IN 1)
Installed driver for device is WinUSB (verified in Device Manger to be winusb.sys), and I am linking libusb-1.0 into my desktop program. You can find my source code at https://gitlab.com/tywonemi-school-stuff/silicon-radar-fun, the firmware is My SW/v1 and the desktop software is a Qt Creator project in My SW/Viewer, of note is usb.cpp. You can also compare with testing project/HIDTest, which is code that I tested with STM32F303 nucleo dev board where I was able to read an array through IN bulk endpoint with the Viewer application. However, F3 has the USB peripheral, while F7 has OTG_USB, and I am now attempting USB 2.0 compliant HS so there may be more protocol-based pitfalls. You can also find the output of the device descriptor etc from USBView in my SW/USBView_broken.txt
EDIT 1: I have found finally some concrete error in the STM32 behavior. The DMAADDR is set for EPIN 0x81, and never increments, despite the DMA being enabled. I have went through literally every occurrence of the word "DMA" in the USB_OTG periphery.
I thought it might be that my linker script makes my array be stored in DTCM or similar, and the OTG DMA can't access it, but the address of txBuffer is 0x2003EBEC which is in SRAM2. The AHB matrix in the reference manual clearly shows, that the USB OTG HS DMA is master for a bus that SRAM2 is a slave of. And DTCM is connected too. I will look for application notes for USB OTG HS DMA - it just seems to be refusing to copy data!
I have fixed my issue by disabling the DMA setting. I have re-read the relevant portions of the reference manual and still don't know how exactly the values propagate into the Tx FIFOs. It is possible that DMA-less operation will be a major bottleneck in my project, I might return to this later.

Question on PCI Express(PCIe) configuration space access on VirtualBox

Hi I'm trying to access the PCIe configuration space with MMIO method on a kernel base.
Before I drop my question, my platform is Windows 10, VirtualBox 6.0.10.
My virtual machine set as default except the following:
chipset choosed ICH9
Core number set to 4
Memory set to 1GB
Added IDE controller(no HD connected)
After boot, the printing shows that valid memory address are:0x0~0x9FC00 and 0x100000~0x3FEF0000 as displayed in following screen shot.
While type 1 is RAM, 2 is ROM or Reserved, 3 is ACPI Reclaim Memory and 4 is ACPI NVS Memory.
Furthermore I retrieved base address of the PCIe configuration memory map base address from MCFG as showed in the following screen shot.
It can be seen that:
The configuration space base is 0x3F000000 which overlap with the valid memory space.
The first 8 byte of 0x3F000000~0x3F000008 is all 0, which should be first 8 byte of bus:0, device:0, function:0.
So whether I should not use VirtualBox, or I should do some other operations to enable PCIe MMIO accessibility of configuration space?
Thanks so much!!
You've probably parsed the "MFCG ACPI table" incorrectly, or used the wrong (virtual?) address for the "MFCG ACPI table" and forgot to check signature and checksum.
The "Base_addr:" doesn't make sense, and the "Start_PCI_bus: 0, End_PCI_bus: 0" doesn't make sense either.

Device tree address and reg and property

I'm struggling to understand where to get the address of a device on a device tree? As an example how do I know that I should set <0x00900000 0x20000> in here.
Is memory mapped IO done in the hardware (the processor itself) or in software and do I just have to pass the right address in the device tree?
Is the address hardcoded on the processor or can I just set an arbitrary address? I cannot find anything in my reference manual about setting a certain address in the device tree
These kind of addresses can be found in the Reference Manual of the processor.
You can find the link here.
Take a look at the chapter 48 (OCRAM On-chip RAM Memory Controller) and more specifically at the section 48.2.1 (page 4118):
The total on-chip RAM size for the chip is 128 Kbytes, organized as 16K x 64 bits,mapped from 0x00900000 to 0x0091FFFF
This is where come from the values <0x00900000 0x20000> from the dtsi file, corresponding to the base address and the offset.
These values are in dts/dtsi file provided by the chip maker.

entry() get into the different address from the entry point I set in the Elf

Recently I'm learning about the OS. And I want to write a simple bootloader, which change the real mode to protect mode and then load the simple kernel.
But I can't figure out the entry address problem.
At first I put the bootloader in the first sector of the OS.img(qemu), and then the kernel begin at the second sector.
Here's readelf result of my kernel:
The entry point address is 0x800c.
And the LMA and VMA are below:
A part of the bootloader which read elf-type kernel and then get into the entry(),which is the entry point address.
However, when I disassemble the bootloader, the entry() is below:
Call *0x8018, not *0x800c.
I don't know why this happen.
Could you please help me?
call *0x8018 performs a call to an address that is stored at 0x8018, that's correct since ELFHDR is 0x8000 and offset of e_entry in the header is 0x18.
The real problem is in the way you load segments into memory. Each segment should be loaded at address p_vaddr from file offset p_offset. Notice that in your case p_vaddr is 0x8000, that the same place in memory you loaded elf header to and that's why ELFHDR->e_entry gets overwritten. The easiest solution would be to load elf header at different address.
Source: http://www.skyfree.org/linux/references/ELF_Format.pdf