kernel/linux.git/include/linux/swiotlb.h, branch v7.3-rc1

Merge tags 'dma-mapping-7.3-2026-08-24' and 'dma-mapping-7.3-2026-08-24-2' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux

2026-08-24T18:35:46+00:00

Pull dma-mapping updates from Marek Szyprowski: - swiotlb: - new configuration option for the default pool size (Jagadeesh Pagadala) - reduce overhead for high watermark tracking (chenhuguanshen) - minor code cleanups and improvements (Vova Sharaienko, Honglei Huang and Marek Szyprowski) - add proper tracking of the shared DMA state through direct, pool and swiotlb paths (Aneesh Kumar K.V) This is important for confidential-computing * tag 'dma-mapping-7.3-2026-08-24' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux: dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS MAINTAINERS: update tree for DMA MAPPING HELPERS dma/swiotlb: introduce Kconfig option for compile-time default pool size dma-direct: Improve readability of the dma_direct_map_sg() for P2PDMA case iommu/dma: simplify dma_iova_destroy() and drop the free_iova helper dma-coherent: use KiB in DMA allocation logs dma-coherent: fix spacing coding style issue * tag 'dma-mapping-7.3-2026-08-24-2' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux: (23 commits) swiotlb: remove unused SWIOTLB_FORCE flag dma: swiotlb: handle set_memory_decrypted() failures dma: swiotlb: free dynamic pools from process context dma-direct: rename ret to cpu_addr in alloc helpers dma-direct: select DMA address encoding from __DMA_ATTR_ALLOC_CC_SHARED dma-direct: set decrypted flag for remapped DMA allocations dma-direct: make dma_direct_map_phys() honor DMA_ATTR_CC_SHARED dma-direct: Move dma_direct_map_phys() to dma/direct.c dma-direct: pass attrs to dma_capable() for DMA_ATTR_CC_SHARED checks dma-mapping: make dma_pgprot() honor __DMA_ATTR_ALLOC_CC_SHARED dma: swiotlb: track pool encryption state and honor DMA_ATTR_CC_SHARED dma: swiotlb: pass mapping attributes by reference dma-pool: track decrypted atomic pools and select them via attrs dma-direct: use __DMA_ATTR_ALLOC_CC_SHARED in alloc/free paths dma-mapping: Add internal shared allocation attribute coco: arm64: s390: powerpc: Mark secure guests with CC_ATTR_GUEST_MEM_ENCRYPT dma-direct: swiotlb: handle swiotlb alloc/free outside __dma_direct_alloc_pages s390: Expose protected virtualization through cc_platform_has() swiotlb: Preserve allocation virtual address for dynamic pools dma: free atomic pool pages by physical address ...

dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS

2026-08-13T06:02:35+00:00

Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale. Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=track_hiwater" parameter on demand for debugging. Note that when CONFIG_DEBUG_FS is enabled but hiwater tracking is disabled, the "io_tlb_used" metric reports an approximate value rather than an instantaneously exact one. Suggested-by: Fan Du Signed-off-by: Jun Miao Co-developed-by: Fan Du Signed-off-by: Fan Du Tested-by: chenhuguanshen Signed-off-by: chenhuguanshen Reviewed-by: Michael Kelley Tested-by: Michael Kelley Link: https://lore.kernel.org/r/20260812070459.637077-1-frankchen158@126.com Signed-off-by: Marek Szyprowski

swiotlb: remove unused SWIOTLB_FORCE flag

2026-07-31T06:44:52+00:00

SWIOTLB_FORCE has no remaining in-tree users. Forced bouncing is now controlled through the swiotlb=force command line option via swiotlb_force_bounce. Remove the unused flag and simplify the force_bounce initialization. Reviewed-by: Jason Gunthorpe Signed-off-by: Aneesh Kumar K.V (Arm) Link: https://lore.kernel.org/r/20260717180442.110954-24-aneesh.kumar@kernel.org Signed-off-by: Marek Szyprowski

dma: swiotlb: free dynamic pools from process context

2026-07-31T06:44:52+00:00

swiotlb_dyn_free() is used after removing a dynamic swiotlb pool from RCU-protected lists. It can call swiotlb_free_tlb(), which may need to restore the encryption state of an unencrypted pool with set_memory_encrypted() before freeing the pages. RCU callbacks run in atomic context, but set_memory_encrypted() is not guaranteed to be atomic-safe on all architectures. For example, page attribute updates may allocate page tables or take sleeping locks. Use queue_rcu_work() for dynamic pool freeing instead. This keeps the RCU grace period before freeing a published pool, while running the actual pool teardown from workqueue context. Use the same helper for the transient-pool error path, since that path may also be reached from atomic DMA mapping context. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) Link: https://lore.kernel.org/r/20260717180442.110954-22-aneesh.kumar@kernel.org Signed-off-by: Marek Szyprowski

dma: swiotlb: track pool encryption state and honor DMA_ATTR_CC_SHARED

2026-07-31T06:40:59+00:00

Teach swiotlb to distinguish between encrypted and decrypted bounce buffer pools, and make allocation and mapping paths select a pool whose state matches the requested DMA attributes. Add a cc_shared flag to io_tlb_mem, initialize it for the default and restricted pools, and propagate __DMA_ATTR_ALLOC_CC_SHARED into swiotlb pool allocation. Reject swiotlb alloc/map requests when the selected pool does not match the required encrypted/decrypted state. Also return DMA addresses with the matching phys_to_dma_{encrypted, unencrypted} helper so the DMA address encoding stays consistent with the chosen pool. Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) Link: https://lore.kernel.org/r/20260717180442.110954-14-aneesh.kumar@kernel.org Signed-off-by: Marek Szyprowski

dma: swiotlb: pass mapping attributes by reference

2026-07-31T06:40:58+00:00

Change swiotlb_tbl_map_single() to take the DMA mapping attributes by reference and update the direct callers accordingly. This is a preparatory change for a follow-up patch which updates the attributes based on the selected swiotlb pool. Keeping the signature change separate makes the follow-up patch easier to review. No functional change in this patch. Reviewed-by: Jason Gunthorpe Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) Link: https://lore.kernel.org/r/20260717180442.110954-13-aneesh.kumar@kernel.org Signed-off-by: Marek Szyprowski

dma-direct: swiotlb: handle swiotlb alloc/free outside __dma_direct_alloc_pages

2026-07-31T06:40:58+00:00

Move swiotlb allocation out of __dma_direct_alloc_pages() and handle it in dma_direct_alloc() / dma_direct_alloc_pages(). This is needed for follow-up changes that simplify the handling of memory encryption/decryption based on the DMA attribute flags. swiotlb backing pages are already mapped decrypted by swiotlb_update_mem_attributes() and rmem_swiotlb_device_init(), so dma-direct should not call dma_set_decrypted() on allocation nor dma_set_encrypted() on free for swiotlb-backed memory. Update alloc/free paths to detect swiotlb-backed pages and skip encrypt/decrypt transitions for those paths. Keep the existing highmem rejection in dma_direct_alloc_pages() for swiotlb allocations. Only for "restricted-dma-pool", we currently set `for_alloc = true`, while rmem_swiotlb_device_init() decrypts the whole pool up front. This pool is typically used together with "shared-dma-pool", where the shared region is accessed after remap/ioremap and the returned address is suitable for decrypted memory access. So existing code paths remain valid. Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) Reviewed-by: Mostafa Saleh Link: https://lore.kernel.org/r/20260717180442.110954-8-aneesh.kumar@kernel.org Signed-off-by: Marek Szyprowski

dma/swiotlb: introduce Kconfig option for compile-time default pool size

2026-07-17T07:15:38+00:00

The SWIOTLB bounce buffer pool size is hardcoded at 64 MB via IO_TLB_DEFAULT_SIZE with no compile-time knob to adjust it. On memory-constrained embedded or mobile platforms equipped with a hardware IOMMU (e.g., ARM SMMU) covering most DMA-capable devices, reserving 64 MB at boot is unnecessarily wasteful — the SWIOTLB is only exercised for devices that bypass the IOMMU or have restricted DMA address ranges. Introduce CONFIG_SWIOTLB_DEFAULT_SIZE_MB, an integer Kconfig option (range 1–64 MB, default 64) that allows platforms to set a smaller compile-time default. IO_TLB_DEFAULT_SIZE is updated to derive from this value when CONFIG_SWIOTLB is enabled, preserving the existing 64 MB default when the option is not configured. The runtime "swiotlb=" kernel parameter override remains fully supported and takes precedence over the compile-time default. Signed-off-by: Jagadeesh Pagadala Signed-off-by: Bibek Kumar Patro Reviewed-by: Michael Kelley Link: https://lore.kernel.org/r/20260702-swiotlb-v2-1-9205f3ba5408@oss.qualcomm.com Signed-off-by: Marek Szyprowski

swiotlb: reduce swiotlb pool lookups

2024-07-10T05:59:03+00:00

With CONFIG_SWIOTLB_DYNAMIC enabled, each round-trip map/unmap pair in the swiotlb results in 6 calls to swiotlb_find_pool(). In multiple places, the pool is found and used in one function, and then must be found again in the next function that is called because only the tlb_addr is passed as an argument. These are the six call sites: dma_direct_map_page: 1. swiotlb_map -> swiotlb_tbl_map_single -> swiotlb_bounce dma_direct_unmap_page: 2. dma_direct_sync_single_for_cpu -> is_swiotlb_buffer 3. dma_direct_sync_single_for_cpu -> swiotlb_sync_single_for_cpu -> swiotlb_bounce 4. is_swiotlb_buffer 5. swiotlb_tbl_unmap_single -> swiotlb_del_transient 6. swiotlb_tbl_unmap_single -> swiotlb_release_slots Reduce the number of calls by finding the pool at a higher level, and passing it as an argument instead of searching again. A key change is for is_swiotlb_buffer() to return a pool pointer instead of a boolean, and then pass this pool pointer to subsequent swiotlb functions. There are 9 occurrences of is_swiotlb_buffer() used to test if a buffer is a swiotlb buffer before calling a swiotlb function. To reduce code duplication in getting the pool pointer and passing it as an argument, introduce inline wrappers for this pattern. The generated code is essentially unchanged. Since is_swiotlb_buffer() no longer returns a boolean, rename some functions to reflect the change: * swiotlb_find_pool() becomes __swiotlb_find_pool() * is_swiotlb_buffer() becomes swiotlb_find_pool() * is_xen_swiotlb_buffer() becomes xen_swiotlb_find_pool() With these changes, a round-trip map/unmap pair requires only 2 pool lookups (listed using the new names and wrappers): dma_direct_unmap_page: 1. dma_direct_sync_single_for_cpu -> swiotlb_find_pool 2. swiotlb_tbl_unmap_single -> swiotlb_find_pool These changes come from noticing the inefficiencies in a code review, not from performance measurements. With CONFIG_SWIOTLB_DYNAMIC, __swiotlb_find_pool() is not trivial, and it uses an RCU read lock, so avoiding the redundant calls helps performance in a hot path. When CONFIG_SWIOTLB_DYNAMIC is *not* set, the code size reduction is minimal and the perf benefits are likely negligible, but no harm is done. No functional change is intended. Signed-off-by: Michael Kelley Reviewed-by: Petr Tesarik Signed-off-by: Christoph Hellwig

swiotlb: remove alloc_size argument to swiotlb_tbl_map_single()

2024-05-07T11:29:28+00:00

Currently swiotlb_tbl_map_single() takes alloc_align_mask and alloc_size arguments to specify an swiotlb allocation that is larger than mapping_size. This larger allocation is used solely by iommu_dma_map_single() to handle untrusted devices that should not have DMA visibility to memory pages that are partially used for unrelated kernel data. Having two arguments to specify the allocation is redundant. While alloc_align_mask naturally specifies the alignment of the starting address of the allocation, it can also implicitly specify the size by rounding up the mapping_size to that alignment. Additionally, the current approach has an edge case bug. iommu_dma_map_page() already does the rounding up to compute the alloc_size argument. But swiotlb_tbl_map_single() then calculates the alignment offset based on the DMA min_align_mask, and adds that offset to alloc_size. If the offset is non-zero, the addition may result in a value that is larger than the max the swiotlb can allocate. If the rounding up is done _after_ the alignment offset is added to the mapping_size (and the original mapping_size conforms to the value returned by swiotlb_max_mapping_size), then the max that the swiotlb can allocate will not be exceeded. In view of these issues, simplify the swiotlb_tbl_map_single() interface by removing the alloc_size argument. Most call sites pass the same value for mapping_size and alloc_size, and they pass alloc_align_mask as zero. Just remove the redundant argument from these callers, as they will see no functional change. For iommu_dma_map_page() also remove the alloc_size argument, and have swiotlb_tbl_map_single() compute the alloc_size by rounding up mapping_size after adding the offset based on min_align_mask. This has the side effect of fixing the edge case bug but with no other functional change. Also add a sanity test on the alloc_align_mask. While IOMMU code currently ensures the granule is not larger than PAGE_SIZE, if that guarantee were to be removed in the future, the downstream effect on the swiotlb might go unnoticed until strange allocation failures occurred. Tested on an ARM64 system with 16K page size and some kernel test-only hackery to allow modifying the DMA min_align_mask and the granule size that becomes the alloc_align_mask. Tested these combinations with a variety of original memory addresses and sizes, including those that reproduce the edge case bug: * 4K granule and 0 min_align_mask * 4K granule and 0xFFF min_align_mask (4K - 1) * 16K granule and 0xFFF min_align_mask * 64K granule and 0xFFF min_align_mask * 64K granule and 0x3FFF min_align_mask (16K - 1) With the changes, all combinations pass. Signed-off-by: Michael Kelley Reviewed-by: Petr Tesarik Signed-off-by: Christoph Hellwig