kernel/linux.git/include/linux/vfio_pci_core.h, branch v6.18.42

vfio/pci: Fix racy bitfields and tighten struct layout

2026-07-18T14:53:11+00:00

commit e73638e55f861758d49f14d7bb5dba3035981cd7 upstream. Bitfield operations are not atomic, they use a read-modify-write pattern, therefore we should be careful not to pack bitfields that can be concurrently updated into the same storage unit. This split takes a binary approach: flags that are only modified pre/post open/close remain bitfields, flags modified from user action, including actions that reach across to another device (ex. reset) use dedicated storage units. Note that the virq_disabled and bardirty flags are relocated to fill an existing hole in the structure. Bitfield justifications: has_dyn_msix: written only in vfio_pci_core_enable() pci_2_3: written only in vfio_pci_core_enable() reset_works: written only in vfio_pci_core_enable() extended_caps: written only in vfio_cap_len() under vfio_config_init() has_vga: written only in vfio_pci_core_enable() nointx: written only in vfio_pci_core_enable() needs_pm_restore: written only in vfio_pci_probe_power_state() disable_idle_d3: written only at .init in vfio_pci_core_init_dev() Dedicated storage units: virq_disabled: written by guest INTx command writes in vfio_basic_config_write() while the device is open bardirty: written by guest BAR writes in vfio_basic_config_write() while the device is open pm_intx_masked: written in the runtime-PM suspend path. pm_runtime_engaged: written by low-power feature entry/exit paths needs_reset: set in vfio_pci_core_disable() and cleared for devices in the set by vfio_pci_dev_set_try_reset() sriov_active: written by vfio_pci_core_sriov_configure() via sysfs sriov_numvfs while bound. Fixes: 9cd0f6d5cbb6 ("vfio/pci: Use bitfield for struct vfio_pci_core_device flags") Cc: stable@vger.kernel.org Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Alex Williamson Reviewed-by: Kevin Tian Link: https://lore.kernel.org/r/20260615191241.688297-4-alex.williamson@nvidia.com Signed-off-by: Alex Williamson Signed-off-by: Greg Kroah-Hartman

vfio/pci: Latch disable_idle_d3 per device

2026-07-18T14:53:11+00:00

commit 4575e9aac5336d1365138c0284773bf8da4b1fa3 upstream. When disable_idle_d3 was introduced in vfio-pci, it directly manipulated the device power state with pci_set_power_state(). There were no refcounts to maintain or balanced operations, we could unconditionally bring the device to D0 and conditionally move it to D3hot. Therefore the module parameter was made writable. Later, in commit c61302aa48f7 ("vfio/pci: Move module parameters to vfio_pci.c"), as part of the vfio-pci-core split, the writable aspect of the module parameter was nullified. The parameter value could still be changed through sysfs, but the vfio-pci driver latched the values into vfio-pci-core globals at module init. Loading the vfio-pci module, or unloading and reloading, with non-default or different values could change the globals relative to existing devices bound to vfio-pci variant drivers. Runtime PM was introduced in commit 7ab5e10eda02 ("vfio/pci: Move the unused device into low power state with runtime PM"), which marks the point where power states became refcounted. PM get and put operations need to be balanced, but the same module operations noted above can change the global variables relative to those devices already bound to vfio-pci variant drivers. This introduces a window where PM operations can now become unbalanced. To resolve this with a narrow footprint for stable backports, the disable_idle_d3 flag is latched into the vfio_pci_core_device at the time of initialization, such that the device always operates with a consistent value. NB. vfio_pci_dev_set_try_reset() now unconditionally raises the runtime PM usage count around bus reset to account for disable_idle_d3 becoming a per-device rather than global flag. When this flag is set, the additional get/put pair is harmless and allows continued use of the shared vfio_pci_dev_set_pm_runtime_get() helper. Fixes: 7ab5e10eda02 ("vfio/pci: Move the unused device into low power state with runtime PM") Cc: stable@vger.kernel.org Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Alex Williamson Reviewed-by: Kevin Tian Link: https://lore.kernel.org/r/20260615191241.688297-2-alex.williamson@nvidia.com Signed-off-by: Alex Williamson Signed-off-by: Greg Kroah-Hartman

vfio/pci: Use a private flag to prevent power state change with VFs

2026-07-18T14:53:11+00:00

commit 40ef3edf151e184d021917a5c4c771cc0870844a upstream. The current implementation uses pci_num_vf() while holding the memory_lock to prevent changing the power state of a PF when VFs are enabled. This creates a lockdep circular dependency warning because memory_lock is held during device probing. [ 286.997167] ====================================================== [ 287.003363] WARNING: possible circular locking dependency detected [ 287.009562] 7.0.0-dbg-DEV #3 Tainted: G S [ 287.015074] ------------------------------------------------------ [ 287.021270] vfio_pci_sriov_/18636 is trying to acquire lock: [ 287.026942] ff45bea2294d4968 (&vdev->memory_lock){+.+.}-{4:4}, at: vfio_pci_core_runtime_resume+0x1f/0xa0 [ 287.036530] [ 287.036530] but task is already holding lock: [ 287.042383] ff45bea3a96b8230 (&new_dev_set->lock){+.+.}-{4:4}, at: vfio_group_fops_unl_ioctl+0x44d/0x7b0 [ 287.051879] [ 287.051879] which lock already depends on the new lock. [ 287.051879] [ 287.060070] [ 287.060070] the existing dependency chain (in reverse order) is: [ 287.067568] [ 287.067568] -> #2 (&new_dev_set->lock){+.+.}-{4:4}: [ 287.073941] __mutex_lock+0x92/0xb80 [ 287.078058] vfio_assign_device_set+0x66/0x1b0 [ 287.083042] vfio_pci_core_register_device+0xd1/0x2a0 [ 287.088638] vfio_pci_probe+0xd2/0x100 [ 287.092933] local_pci_probe_callback+0x4d/0xa0 [ 287.098001] process_scheduled_works+0x2ca/0x680 [ 287.103158] worker_thread+0x1e8/0x2f0 [ 287.107452] kthread+0x10c/0x140 [ 287.111230] ret_from_fork+0x18e/0x360 [ 287.115519] ret_from_fork_asm+0x1a/0x30 [ 287.119983] [ 287.119983] -> #1 ((work_completion)(&arg.work)){+.+.}-{0:0}: [ 287.127219] __flush_work+0x345/0x490 [ 287.131429] pci_device_probe+0x2e3/0x490 [ 287.135979] really_probe+0x1f9/0x4e0 [ 287.140180] __driver_probe_device+0x77/0x100 [ 287.145079] driver_probe_device+0x1e/0x110 [ 287.149803] __device_attach_driver+0xe3/0x170 [ 287.154789] bus_for_each_drv+0x125/0x150 [ 287.159346] __device_attach+0xca/0x1a0 [ 287.163720] device_initial_probe+0x34/0x50 [ 287.168445] pci_bus_add_device+0x6e/0x90 [ 287.172995] pci_iov_add_virtfn+0x3c9/0x3e0 [ 287.177719] sriov_add_vfs+0x2c/0x60 [ 287.181838] sriov_enable+0x306/0x4a0 [ 287.186038] vfio_pci_core_sriov_configure+0x184/0x220 [ 287.191715] sriov_numvfs_store+0xd9/0x1c0 [ 287.196351] kernfs_fop_write_iter+0x13f/0x1d0 [ 287.201338] vfs_write+0x2be/0x3b0 [ 287.205286] ksys_write+0x73/0x100 [ 287.209233] do_syscall_64+0x14d/0x750 [ 287.213529] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 287.219120] [ 287.219120] -> #0 (&vdev->memory_lock){+.+.}-{4:4}: [ 287.225491] __lock_acquire+0x14c6/0x2800 [ 287.230048] lock_acquire+0xd3/0x2f0 [ 287.234168] down_write+0x3a/0xc0 [ 287.238019] vfio_pci_core_runtime_resume+0x1f/0xa0 [ 287.243436] __rpm_callback+0x8c/0x310 [ 287.247730] rpm_resume+0x529/0x6f0 [ 287.251765] __pm_runtime_resume+0x68/0x90 [ 287.256402] vfio_pci_core_enable+0x44/0x310 [ 287.261216] vfio_pci_open_device+0x1c/0x80 [ 287.265947] vfio_df_open+0x10f/0x150 [ 287.270148] vfio_group_fops_unl_ioctl+0x4a4/0x7b0 [ 287.275476] __se_sys_ioctl+0x71/0xc0 [ 287.279679] do_syscall_64+0x14d/0x750 [ 287.283975] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 287.289559] [ 287.289559] other info that might help us debug this: [ 287.289559] [ 287.297582] Chain exists of: [ 287.297582] &vdev->memory_lock --> (work_completion)(&arg.work) --> &new_dev_set->lock [ 287.297582] [ 287.310023] Possible unsafe locking scenario: [ 287.310023] [ 287.315961] CPU0 CPU1 [ 287.320510] ---- ---- [ 287.325059] lock(&new_dev_set->lock); [ 287.328917] lock((work_completion)(&arg.work)); [ 287.336153] lock(&new_dev_set->lock); [ 287.342523] lock(&vdev->memory_lock); [ 287.346382] [ 287.346382] *** DEADLOCK *** [ 287.346382] [ 287.352315] 2 locks held by vfio_pci_sriov_/18636: [ 287.357125] #0: ff45bea208ed3e18 (&group->group_lock){+.+.}-{4:4}, at: vfio_group_fops_unl_ioctl+0x3e3/0x7b0 [ 287.367048] #1: ff45bea3a96b8230 (&new_dev_set->lock){+.+.}-{4:4}, at: vfio_group_fops_unl_ioctl+0x44d/0x7b0 [ 287.376976] [ 287.376976] stack backtrace: [ 287.381353] CPU: 191 UID: 0 PID: 18636 Comm: vfio_pci_sriov_ Tainted: G S 7.0.0-dbg-DEV #3 PREEMPTLAZY [ 287.381355] Tainted: [S]=CPU_OUT_OF_SPEC [ 287.381356] Call Trace: [ 287.381357] [ 287.381358] dump_stack_lvl+0x54/0x70 [ 287.381361] print_circular_bug+0x2e1/0x300 [ 287.381363] check_noncircular+0xf9/0x120 [ 287.381364] ? __lock_acquire+0x5b4/0x2800 [ 287.381366] __lock_acquire+0x14c6/0x2800 [ 287.381368] ? pci_mmcfg_read+0x4f/0x220 [ 287.381370] ? pci_mmcfg_write+0x57/0x220 [ 287.381371] ? lock_acquire+0xd3/0x2f0 [ 287.381373] ? pci_mmcfg_write+0x57/0x220 [ 287.381374] ? lock_release+0xef/0x360 [ 287.381376] ? vfio_pci_core_runtime_resume+0x1f/0xa0 [ 287.381377] lock_acquire+0xd3/0x2f0 [ 287.381378] ? vfio_pci_core_runtime_resume+0x1f/0xa0 [ 287.381379] ? lock_is_held_type+0x76/0x100 [ 287.381382] down_write+0x3a/0xc0 [ 287.381382] ? vfio_pci_core_runtime_resume+0x1f/0xa0 [ 287.381383] vfio_pci_core_runtime_resume+0x1f/0xa0 [ 287.381384] ? __pfx_pci_pm_runtime_resume+0x10/0x10 [ 287.381385] __rpm_callback+0x8c/0x310 [ 287.381386] ? ktime_get_mono_fast_ns+0x3d/0xb0 [ 287.381389] ? __pfx_pci_pm_runtime_resume+0x10/0x10 [ 287.381390] rpm_resume+0x529/0x6f0 [ 287.381392] ? lock_is_held_type+0x76/0x100 [ 287.381394] __pm_runtime_resume+0x68/0x90 [ 287.381396] vfio_pci_core_enable+0x44/0x310 [ 287.381398] vfio_pci_open_device+0x1c/0x80 [ 287.381399] vfio_df_open+0x10f/0x150 [ 287.381401] vfio_group_fops_unl_ioctl+0x4a4/0x7b0 [ 287.381402] __se_sys_ioctl+0x71/0xc0 [ 287.381404] do_syscall_64+0x14d/0x750 [ 287.381405] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 287.381406] ? trace_irq_disable+0x25/0xd0 [ 287.381409] entry_SYSCALL_64_after_hwframe+0x77/0x7f Introduce a private flag 'sriov_active' in the vfio_pci_core_device struct. This allows the driver to track the SR-IOV power state requirement without relying on pci_num_vf() while holding the memory_lock. The lock is now only held to set the flag and ensure the device is in D0, after which pci_enable_sriov() can be called without the lock. Fixes: f4162eb1e2fc ("vfio/pci: Change the PF power state to D0 before enabling VFs") Cc: stable@vger.kernel.org Suggested-by: Jason Gunthorpe Suggested-by: Alex Williamson Signed-off-by: Raghavendra Rao Ananta Link: https://lore.kernel.org/r/20260514173449.3282188-1-rananta@google.com [promote bitfield to plain bool to avoid storage-unit races] Signed-off-by: Alex Williamson Signed-off-by: Greg Kroah-Hartman

vfio: refactor vfio_pci_mmap_huge_fault function

2026-05-23T11:06:29+00:00

[ Upstream commit 9b92bc7554b543dc00a0a0b62904a9ef2ad5c4b0 ] Refactor vfio_pci_mmap_huge_fault to take out the implementation to map the VMA to the PTE/PMD/PUD as a separate function. Export the new function to be used by nvgrace-gpu module. Move the alignment check code to verify that pfn and VMA VA is aligned to the page order to the header file and make it inline. No functional change is intended. Cc: Shameer Kolothum Cc: Alex Williamson Cc: Jason Gunthorpe Reviewed-by: Shameer Kolothum Signed-off-by: Ankit Agrawal Link: https://lore.kernel.org/r/20251127170632.3477-2-ankita@nvidia.com Signed-off-by: Alex Williamson Stable-dep-of: 948b71aa81cd ("drivers/vfio_pci_core: Change PXD_ORDER check from switch case to if/else block") Signed-off-by: Sasha Levin

vfio/pci: Disable qword access to the PCI ROM bar

2026-01-08T09:17:21+00:00

[ Upstream commit dc85a46928c41423ad89869baf05a589e2975575 ] Commit 2b938e3db335 ("vfio/pci: Enable iowrite64 and ioread64 for vfio pci") enables qword access to the PCI bar resources. However certain devices (e.g. Intel X710) are observed with problem upon qword accesses to the rom bar, e.g. triggering PCI aer errors. This is triggered by Qemu which caches the rom content by simply does a pread() of the remaining size until it gets the full contents. The other bars would only perform operations at the same access width as their guest drivers. Instead of trying to identify all broken devices, universally disable qword access to the rom bar i.e. going back to the old way which worked reliably for years. Reported-by: Farrah Chen Closes: https://bugzilla.kernel.org/show_bug.cgi?id=220740 Fixes: 2b938e3db335 ("vfio/pci: Enable iowrite64 and ioread64 for vfio pci") Cc: stable@vger.kernel.org Signed-off-by: Kevin Tian Tested-by: Farrah Chen Link: https://lore.kernel.org/r/20251218081650.555015-2-kevin.tian@intel.com Signed-off-by: Alex Williamson Signed-off-by: Greg Kroah-Hartman

vfio/pci: Use RCU for error/request triggers to avoid circular locking

2025-12-18T13:03:27+00:00

[ Upstream commit 98693e0897f754e3f51ce6626ed5f785f625ba2b ] Thanks to a device generating an ACS violation during bus reset, lockdep reported the following circular locking issue: CPU0: SET_IRQS (MSI/X): holds igate, acquires memory_lock CPU1: HOT_RESET: holds memory_lock, acquires pci_bus_sem CPU2: AER: holds pci_bus_sem, acquires igate This results in a potential 3-way deadlock. Remove the pci_bus_sem->igate leg of the triangle by using RCU to peek at the eventfd rather than locking it with igate. Fixes: 3be3a074cf5b ("vfio-pci: Don't use device_lock around AER interrupt setup") Signed-off-by: Alex Williamson Reviewed-by: Jason Gunthorpe Link: https://lore.kernel.org/r/20251124223623.2770706-1-alex@shazbot.org Signed-off-by: Alex Williamson Signed-off-by: Sasha Levin

vfio/pci: Do vf_token checks for VFIO_DEVICE_BIND_IOMMUFD

2025-08-05T21:41:14+00:00

This was missed during the initial implementation. The VFIO PCI encodes the vf_token inside the device name when opening the device from the group FD, something like: "0000:04:10.0 vf_token=bd8d9d2b-5a5f-4f5a-a211-f591514ba1f3" This is used to control access to a VF unless there is co-ordination with the owner of the PF. Since we no longer have a device name in the cdev path, pass the token directly through VFIO_DEVICE_BIND_IOMMUFD using an optional field indicated by VFIO_DEVICE_BIND_FLAG_TOKEN. Fixes: 5fcc26969a16 ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD") Tested-by: Shameer Kolothum Reviewed-by: Yi Liu Signed-off-by: Jason Gunthorpe Reviewed-by: Kevin Tian Link: https://lore.kernel.org/r/0-v3-bdd8716e85fe+3978a-vfio_token_jgg@nvidia.com Signed-off-by: Alex Williamson

vfio/pci: Fix typo in macro to declare accessors

2024-06-21T18:47:01+00:00

Correct spelling of DECLA[RA]TION Suggested-by: Ramesh Thomas Signed-off-by: Gerd Bayer Link: https://lore.kernel.org/r/20240619115847.1344875-4-gbayer@linux.ibm.com Signed-off-by: Alex Williamson

vfio/pci: Support 8-byte PCI loads and stores

2024-06-21T18:47:01+00:00

Many PCI adapters can benefit or even require full 64bit read and write access to their registers. In order to enable work on user-space drivers for these devices add two new variations vfio_pci_core_io{read|write}64 of the existing access methods when the architecture supports 64-bit ioreads and iowrites. Signed-off-by: Ben Segal Co-developed-by: Gerd Bayer Signed-off-by: Gerd Bayer Link: https://lore.kernel.org/r/20240619115847.1344875-3-gbayer@linux.ibm.com Signed-off-by: Alex Williamson

vfio/pci: Use unmap_mapping_range()

2024-05-31T21:15:51+00:00

With the vfio device fd tied to the address space of the pseudo fs inode, we can use the mm to track all vmas that might be mmap'ing device BARs, which removes our vma_list and all the complicated lock ordering necessary to manually zap each related vma. Note that we can no longer store the pfn in vm_pgoff if we want to use unmap_mapping_range() to zap a selective portion of the device fd corresponding to BAR mappings. This also converts our mmap fault handler to use vmf_insert_pfn() because we no longer have a vma_list to avoid the concurrency problem with io_remap_pfn_range(). The goal is to eventually use the vm_ops huge_fault handler to avoid the additional faulting overhead, but vmf_insert_pfn_{pmd,pud}() need to learn about pfnmaps first. Also, Jason notes that a race exists between unmap_mapping_range() and the fops mmap callback if we were to call io_remap_pfn_range() to populate the vma on mmap. Specifically, mmap_region() does call_mmap() before it does vma_link_file() which gives a window where the vma is populated but invisible to unmap_mapping_range(). Suggested-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian Link: https://lore.kernel.org/r/20240530045236.1005864-3-alex.williamson@redhat.com Signed-off-by: Alex Williamson