How to setup SR-IOV with Samsung PM1733/1735 NVMe SSD - virtualization

Question

The Samsung PM1733/1735 NVMe SSDs declare the SR-IOV supports. And I hope to set up a NVMe SR-IOV environment with a PM1735 SSD. The dev description of my SSD is /dev/nvme1.
However, I tried with commands like
echo 4 > /sys/class/nvme/nvme1/device/sriov_numvfs.
The dmesg information contains that
[ 1239.732278] nvme nvme2: pci function 0000:86:00.1
[ 1239.732359] nvme 0000:86:00.1: enabling device (0000 -> 0002)
[ 1239.732365] pci 0000:86:00.2: [144d:a824] type 00 class 0x010802
[ 1239.733053] iommu: Adding device 0000:86:00.2 to group 164
[ 1239.733402] nvme nvme3: pci function 0000:86:00.2
[ 1239.733473] pci 0000:86:00.3: [144d:a824] type 00 class 0x010802
[ 1239.733477] nvme 0000:86:00.2: enabling device (0000 -> 0002)
[ 1239.734117] iommu: Adding device 0000:86:00.3 to group 165
[ 1239.734485] nvme nvme4: pci function 0000:86:00.3
[ 1239.734557] pci 0000:86:00.4: [144d:a824] type 00 class 0x010802
[ 1239.734782] nvme 0000:86:00.3: enabling device (0000 -> 0002)
[ 1239.735066] iommu: Adding device 0000:86:00.4 to group 166
[ 1239.735352] nvme nvme5: pci function 0000:86:00.4
[ 1239.735631] nvme 0000:86:00.4: enabling device (0000 -> 0002)
[ 1270.289730] nvme nvme2: Device not ready; aborting initialisation
[ 1270.289794] nvme nvme2: Removing after probe failure status: -19
[ 1270.293724] nvme nvme4: Device not ready; aborting initialisation
[ 1270.293727] nvme nvme5: Device not ready; aborting initialisation
[ 1270.293732] nvme nvme5: Removing after probe failure status: -19
[ 1270.293783] nvme nvme4: Removing after probe failure status: -19
[ 1270.297647] nvme nvme3: Device not ready; aborting initialisation
[ 1270.297703] nvme nvme3: Removing after probe failure status: -19
The lspci | grep Non can find that
86:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a824
86:00.1 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a824
86:00.2 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a824
86:00.3 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a824
86:00.4 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a824
Are the VFs correctly created?
And another question is how can I bind the namespaces with the VFs. I used nvme-cli to create namespaces from the /dev/nvme, but how can I bind these namespaces with VFs?

Answer 1

Allocate VQ and VI using nvme cli command nvme virt-mgmt and enable that controller and export that vf to vm.
example commands
nvme list-secondary /dev/nvme1n1 --> output of list of all secondary controllers
nvme virt-mgmt /dev/nvme1n1 -c 1 -r 0 -n 2 --> allocate 2 VQs for secondary controller 1
nvme virt-mgmt /dev/nvme1n1 -c 1 -r 1 -n 2 --> allocate 2 VIs for secondary controller 1
nvme virt-mgmt /dev/nvme1n1 -c -a 9 --> enable secondary controller 1
nvme list-secondary /dev/nvme1n1 --> check first entry is updated and now shows as online.
Now export this vf to vm.

Answer 2

You might need to check nvme spec 1.4 (page 320, section 8.5 Virtualization Enhancements). I guess you also need to configure VQ and VI).

Answer 3

vitaliy-kachmar
You will still see in host as "Initialization Aborted" for VFs.
After VI and VQ and online variables are set, now you can export the VF to a Guest OS/hypervisor. In Guest OS, you can check that nvme block device is generated, as soon as OS startup completed.

Answer 4

First create nvme namespaces using nvme create command and then attach using nvme attach-ns to the controller, create single or multi-namespaces, then create VF and try export.
To export VF to VM.
Install virt-manager and install Guest OS (centos or unbuntu etc..) in virt-manager, i used Ubuntu. you can also refer - https://help.ubuntu.com/community/KVM/VirtManager
Now, you have VFs and VMs.
To export, select any Guest OS
select open and select add-hardware option in new window and select PCI and add one of VF 0000:86:00.1/2/3/4 device
Run guest OS.
In lsblk you will find one char device /dev/nvme0 and block devices /dev/nvme0n1 and now you can run fio on the block device