summaryrefslogtreecommitdiff
path: root/drivers/iommu/amd_iommu.c
AgeCommit message (Collapse)AuthorFilesLines
2016-07-26iommu/amd: Initialize dma-ops domains with 3-level page-tableJoerg Roedel1-1/+1
A two-level page-table can map up to 1GB of address space. With the IOVA allocator now in use, the allocated addresses are often more closely to 4G, which requires the address space to be increased much more often. Avoid that by using a three-level page-table by default. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-26iommu/amd: Update Alias-DTE in update_device_table()Joerg Roedel1-1/+8
Not doing so might cause IO-Page-Faults when a device uses an alias request-id and the alias-dte is left in a lower page-mode which does not cover the address allocated from the iova-allocator. Fixes: 492667dacc0a ('x86/amd-iommu: Remove amd_iommu_pd_table') Cc: stable@vger.kernel.org Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-14iommu/amd: Use container_of to get dma_ops_domainJoerg Roedel1-10/+26
This is better than storing an extra pointer in struct protection_domain, because this pointer can now be removed from the struct. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-14iommu/amd: Flush iova queue before releasing dma_ops_domainJoerg Roedel1-3/+14
Before a dma_ops_domain can be freed, we need to make sure it is not longer referenced by the flush queue. So empty the queue before a dma_ops_domain can be freed. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-14iommu/amd: Handle IOMMU_DOMAIN_DMA in ops->domain_free call-backJoerg Roedel1-8/+17
This domain type is not yet handled in the iommu_ops->domain_free() call-back. Fix that. Fixes: 0bb6e243d7fb ('iommu/amd: Support IOMMU_DOMAIN_DMA type allocation') Cc: stable@vger.kernel.org # v4.2+ Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-14iommu/amd: Use dev_data->domain in get_domain()Joerg Roedel1-6/+1
Using the cached value is much more efficient than calling into the IOMMU core code. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-14iommu/amd: Optimize map_sg and unmap_sgJoerg Roedel1-26/+85
Optimize these functions so that they need only one call into the address alloctor. This also saves a couple of io-tlb flushes in the unmap_sg path. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Introduce dir2prot() helperJoerg Roedel1-7/+13
This function converts dma_data_direction to iommu-protection flags. This will be needed on multiple places in the code, so this will save some code. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Implement timeout to flush unmap queuesJoerg Roedel1-0/+28
In case the queue doesn't fill up, we flush the TLB at least 10ms after the unmap happened to make sure that the TLB is cleaned up. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Implement flush queueJoerg Roedel1-4/+67
With the flush queue the IOMMU TLBs will not be flushed at every dma-ops unmap operation. The unmapped ranges will be queued and flushed at once, when the queue is full. This makes unmapping operations a lot faster (on average) and restores the performance of the old address allocator. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Allow NULL pointer parameter for domain_flush_complete()Joerg Roedel1-1/+1
If domain == NULL is passed to the function, it will queue a completion-wait command on all IOMMUs in the system. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Set up data structures for flush queueJoerg Roedel1-1/+39
The flush queue is the equivalent to defered-flushing in the Intel VT-d driver. This patch sets up the data structures needed for this. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Remove align-parameter from __map_single()Joerg Roedel1-10/+3
This parameter is not required anymore because the iova-allocations are always aligned to its size. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Remove other remains of old address allocatorJoerg Roedel1-311/+5
There are other remains in the code from the old allocatore. Remove them all. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Make use of the generic IOVA allocatorJoerg Roedel1-157/+26
Remove the old address allocation code and make use of the generic IOVA allocator that is also used by other dma-ops implementations. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Remove special mapping code for dma_ops pathJoerg Roedel1-93/+14
Use the iommu-api map/unmap functions instead. This will be required anyway when IOVA code is used for address allocation. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Pass gfp-flags to iommu_map_page()Joerg Roedel1-3/+4
Make this function ready to be used in the DMA-API path. Reorder parameters a bit while at it. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Implement apply_dm_region call-backJoerg Roedel1-0/+15
It is used to reserve the dm-regions in the iova-tree. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Create a list of reserved iova addressesJoerg Roedel1-0/+69
Put the MSI-range, the HT-range and the MMIO ranges of PCI devices into that range, so that these addresses are not allocated for DMA. Copy this address list into every created dma_ops_domain. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Allocate iova_domain for dma_ops_domainJoerg Roedel1-1/+19
Use it later for allocating the IO virtual addresses. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-07-13iommu/amd: Init unity mappings only for dma_ops domainsJoerg Roedel1-2/+4
The default domain for a device might also be identity-mapped. In this case the kernel would crash when unity mappings are defined for the device. Fix that by making sure the domain is a dma_ops domain. Fixes: 0bb6e243d7fb ('iommu/amd: Support IOMMU_DOMAIN_DMA type allocation') Cc: stable@vger.kernel.org # v4.2+ Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-06-15iommu/amd: Set AMD iommu callbacks for platform bus driverWan Zongshun1-0/+4
AMD has more drivers will use ACPI to platform bus driver later, all those devices need iommu support, for example: eMMC driver. For latest AMD eMMC controller, it will utilize sdhci-acpi.c driver, which will rely on platform bus to match device and driver, where we will set 'dev' of struct platform_device as map_sg parameter passing to iommu driver for DMA request, so the iommu-ops are needed on the platform bus. Signed-off-by: Wan Zongshun <Vincent.Wan@amd.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-05-20Merge tag 'iommu-updates-v4.7' of ↵Linus Torvalds1-118/+149
git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu Pull IOMMU updates from Joerg Roedel: "The updates include: - rate limiting for the VT-d fault handler - remove statistics code from the AMD IOMMU driver. It is unused and should be replaced by something more generic if needed - per-domain pagesize-bitmaps in IOMMU core code to support systems with different types of IOMMUs - support for ACPI devices in the AMD IOMMU driver - 4GB mode support for Mediatek IOMMU driver - ARM-SMMU updates from Will Deacon: - support for 64k pages with SMMUv1 implementations (e.g MMU-401) - remove open-coded 64-bit MMIO accessors - initial support for 16-bit VMIDs, as supported by some ThunderX SMMU implementations - a couple of errata workarounds for silicon in the field - various fixes here and there" * tag 'iommu-updates-v4.7' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu: (44 commits) iommu/arm-smmu: Use per-domain page sizes. iommu/amd: Remove statistics code iommu/dma: Finish optimising higher-order allocations iommu: Allow selecting page sizes per domain iommu: of: enforce const-ness of struct iommu_ops iommu: remove unused priv field from struct iommu_ops iommu/dma: Implement scatterlist segment merging iommu/arm-smmu: Clear cache lock bit of ACR iommu/arm-smmu: Support SMMUv1 64KB supplement iommu/arm-smmu: Decouple context format from kernel config iommu/arm-smmu: Tidy up 64-bit/atomic I/O accesses io-64-nonatomic: Add relaxed accessor variants iommu/arm-smmu: Work around MMU-500 prefetch errata iommu/arm-smmu: Convert ThunderX workaround to new method iommu/arm-smmu: Differentiate specific implementations iommu/arm-smmu: Workaround for ThunderX erratum #27704 iommu/arm-smmu: Add support for 16 bit VMID iommu/amd: Move get_device_id() and friends to beginning of file iommu/amd: Don't use IS_ERR_VALUE to check integer values iommu/amd: Signedness bug in acpihid_device_group() ...
2016-05-19Merge tag 'pci-v4.7-changes' of ↵Linus Torvalds1-2/+1
git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci Pull PCI updates from Bjorn Helgaas: "Enumeration: - Refine PCI support check in pcibios_init() (Adrian-Ken Rueegsegger) - Provide common functions for ECAM mapping (Jayachandran C) - Allow all PCIe services on non-ACPI host bridges (Jon Derrick) - Remove return values from pcie_port_platform_notify() and relatives (Jon Derrick) - Widen portdrv service type from 4 bits to 8 bits (Keith Busch) - Add Downstream Port Containment portdrv service type (Keith Busch) - Add Downstream Port Containment driver (Keith Busch) Resource management: - Identify Enhanced Allocation (EA) BAR Equivalent resources in sysfs (Alex Williamson) - Supply CPU physical address (not bus address) to iomem_is_exclusive() (Bjorn Helgaas) - alpha: Call iomem_is_exclusive() for IORESOURCE_MEM, but not IORESOURCE_IO (Bjorn Helgaas) - Mark Broadwell-EP Home Agent 1 as having non-compliant BARs (Prarit Bhargava) - Disable all BAR sizing for devices with non-compliant BARs (Prarit Bhargava) - Move PCI I/O space management from OF to PCI core code (Tomasz Nowicki) PCI device hotplug: - acpiphp_ibm: Avoid uninitialized variable reference (Dan Carpenter) - Use cached copy of PCI_EXP_SLTCAP_HPC bit (Lukas Wunner) Virtualization: - Mark Intel i40e NIC INTx masking as broken (Alex Williamson) - Reverse standard ACS vs device-specific ACS enabling (Alex Williamson) - Work around Intel Sunrise Point PCH incorrect ACS capability (Alex Williamson) IOMMU: - Add pci_add_dma_alias() to abstract implementation (Bjorn Helgaas) - Move informational printk to pci_add_dma_alias() (Bjorn Helgaas) - Add support for multiple DMA aliases (Jacek Lawrynowicz) - Add DMA alias quirk for mic_x200_dma (Jacek Lawrynowicz) Thunderbolt: - Fix double free of drom buffer (Andreas Noever) - Add Intel Thunderbolt device IDs (Lukas Wunner) - Fix typos and magic number (Lukas Wunner) - Support 1st gen Light Ridge controller (Lukas Wunner) Generic host bridge driver: - Use generic ECAM API (Jayachandran C) Cavium ThunderX host bridge driver: - Don't clobber read-only bits in bridge config registers (David Daney) - Use generic ECAM API (Jayachandran C) Freescale i.MX6 host bridge driver: - Use enum instead of bool for variant indicator (Andrey Smirnov) - Implement reset sequence for i.MX6+ (Andrey Smirnov) - Factor out ref clock enable (Bjorn Helgaas) - Add initial imx6sx support (Christoph Fritz) - Add reset-gpio-active-high boolean property to DT (Petr Štetiar) - Add DT property for link gen, default to Gen1 (Tim Harvey) - dts: Specify imx6qp version of PCIe core (Andrey Smirnov) - dts: Fix PCIe reset GPIO polarity on Toradex Apalis Ixora (Petr Štetiar) Marvell Armada host bridge driver: - add DT binding for Marvell Armada 7K/8K PCIe controller (Thomas Petazzoni) - Add driver for Marvell Armada 7K/8K PCIe controller (Thomas Petazzoni) Marvell MVEBU host bridge driver: - Constify mvebu_pcie_pm_ops structure (Jisheng Zhang) - Use SET_NOIRQ_SYSTEM_SLEEP_PM_OPS for mvebu_pcie_pm_ops (Jisheng Zhang) Microsoft Hyper-V host bridge driver: - Report resources release after stopping the bus (Vitaly Kuznetsov) - Add explicit barriers to config space access (Vitaly Kuznetsov) Renesas R-Car host bridge driver: - Select PCI_MSI_IRQ_DOMAIN (Arnd Bergmann) Synopsys DesignWare host bridge driver: - Remove incorrect RC memory base/limit configuration (Gabriele Paoloni) - Move Root Complex setup code to dw_pcie_setup_rc() (Jisheng Zhang) TI Keystone host bridge driver: - Add error IRQ handler (Murali Karicheri) - Remove unnecessary goto statement (Murali Karicheri) Miscellaneous: - Fix spelling errors (Colin Ian King)" * tag 'pci-v4.7-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci: (48 commits) PCI: Disable all BAR sizing for devices with non-compliant BARs x86/PCI: Mark Broadwell-EP Home Agent 1 as having non-compliant BARs PCI: Identify Enhanced Allocation (EA) BAR Equivalent resources in sysfs PCI, of: Move PCI I/O space management to PCI core code PCI: generic, thunder: Use generic ECAM API PCI: Provide common functions for ECAM mapping PCI: hv: Add explicit barriers to config space access PCI: Use cached copy of PCI_EXP_SLTCAP_HPC bit PCI: Add Downstream Port Containment driver PCI: Add Downstream Port Containment portdrv service type PCI: Widen portdrv service type from 4 bits to 8 bits PCI: designware: Remove incorrect RC memory base/limit configuration PCI: hv: Report resources release after stopping the bus ARM: dts: imx6qp: Specify imx6qp version of PCIe core PCI: imx6: Implement reset sequence for i.MX6+ PCI: imx6: Use enum instead of bool for variant indicator PCI: thunder: Don't clobber read-only bits in bridge config registers thunderbolt: Fix double free of drom buffer PCI: rcar: Select PCI_MSI_IRQ_DOMAIN PCI: armada: Add driver for Marvell Armada 7K/8K PCIe controller ...
2016-05-09Merge branches 'arm/io-pgtable', 'arm/rockchip', 'arm/omap', 'x86/vt-d', ↵Joerg Roedel1-125/+221
'ppc/pamu', 'core' and 'x86/amd' into next
2016-05-09iommu/amd: Remove statistics codeJoerg Roedel1-95/+0
The statistics are not really used for anything and should be replaced by generic and per-device statistic counters. Remove the code for now. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-21iommu/amd: Move get_device_id() and friends to beginning of fileJoerg Roedel1-54/+54
They will be needed there later. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-21iommu/amd: Don't use IS_ERR_VALUE to check integer valuesJoerg Roedel1-10/+10
Use the better 'var < 0' check. Fixes: 7aba6cb9ee9d ('iommu/amd: Make call-sites of get_device_id aware of its return value') Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-15iommu/amd: Signedness bug in acpihid_device_group()Dan Carpenter1-1/+1
"devid" needs to be signed for the error handling to work. Fixes: b097d11a0fa3f ('iommu/amd: Manage iommu_group for ACPI HID devices') Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-11iommu/amd: Fix checking of pci dma aliasesJoerg Roedel1-11/+76
Commit 61289cb ('iommu/amd: Remove old alias handling code') removed the old alias handling code from the AMD IOMMU driver because this is now handled by the IOMMU core code. But this also removed the handling of PCI aliases, which is not handled by the core code. This caused issues with PCI devices that have hidden PCIe-to-PCI bridges that rewrite the request-id. Fix this bug by re-introducing some of the removed functions from commit 61289cbaf6c8 and add a alias field 'struct iommu_dev_data'. This field carrys the return value of the get_alias() function and uses that instead of the amd_iommu_alias_table[] array in the code. Fixes: 61289cbaf6c8 ('iommu/amd: Remove old alias handling code') Cc: stable@vger.kernel.org # v4.4+ Tested-by: Tomasz Golinski <tomaszg@math.uwb.edu.pl> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-07iommu/amd: Set AMD iommu callbacks for amba busWan Zongshun1-1/+12
AMD Uart DMA belongs to ACPI HID type device, and its driver is basing on AMBA Bus, need also IOMMU support. This patch is just to set the AMD iommu callbacks for amba bus. Signed-off-by: Wan Zongshun <Vincent.Wan@amd.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-07iommu/amd: Manage iommu_group for ACPI HID devicesWan Zongshun1-1/+32
This patch creates a new function for finding or creating an IOMMU group for acpihid(ACPI Hardware ID) device. The acpihid devices with the same devid will be put into same group and there will have the same domain id and share the same page table. Signed-off-by: Wan Zongshun <Vincent.Wan@amd.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-07iommu/amd: Add iommu support for ACPI HID devicesWan Zongshun1-9/+60
Current IOMMU driver make assumption that the downstream devices are PCI. With the newly added ACPI-HID IVHD device entry support, this is no longer true. This patch is to add dev type check and to distinguish the pci and acpihid device code path. Signed-off-by: Wan Zongshun <Vincent.Wan@amd.com> Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-07iommu/amd: Make call-sites of get_device_id aware of its return valueWan Zongshun1-10/+41
This patch is to make the call-sites of get_device_id aware of its return value. Signed-off-by: Wan Zongshun <Vincent.Wan@amd.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-04-07iommu/amd: Add new map for storing IVHD dev entry type HIDWan Zongshun1-0/+1
This patch introduces acpihid_map, which is used to store the new IVHD device entry extracted from BIOS IVRS table. It also provides a utility function add_acpi_hid_device(), to add this types of devices to the map. Signed-off-by: Wan Zongshun <Vincent.Wan@amd.com> Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-02-29iommu/amd: Detach device from domain before removalJoerg Roedel1-0/+4
Detach the device that is about to be removed from its domain (if it has one) to clear any related state like DTE entry and device's ATS state. Reported-by: Kelly Zytaruk <Kelly.Zytaruk@amd.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-01-29iommu/amd: Correct the wrong setting of alias DTE in do_attachBaoquan He1-1/+1
In below commit alias DTE is set when its peripheral is setting DTE. However there's a code bug here to wrongly set the alias DTE, correct it in this patch. commit e25bfb56ea7f046b71414e02f80f620deb5c6362 Author: Joerg Roedel <jroedel@suse.de> Date: Tue Oct 20 17:33:38 2015 +0200 iommu/amd: Set alias DTE in do_attach/do_detach Signed-off-by: Baoquan He <bhe@redhat.com> Tested-by: Mark Hounschell <markh@compro.net> Cc: stable@vger.kernel.org # v4.4 Signed-off-by: Joerg Roedel <jroedel@suse.de>
2016-01-07iommu/amd: Remove an unneeded conditionDan Carpenter1-5/+3
get_device_id() returns an unsigned short device id. It never fails and it never returns a negative so we can remove this condition. Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Preallocate dma_ops apertures based on dma_maskJoerg Roedel1-7/+53
Preallocate between 4 and 8 apertures when a device gets it dma_mask. With more apertures we reduce the lock contention of the domain lock significantly. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Use trylock to aquire bitmap_lockJoerg Roedel1-3/+17
First search for a non-contended aperture with trylock before spinning. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Make dma_ops_domain->next_index percpuJoerg Roedel1-10/+29
Make this pointer percpu so that we start searching for new addresses in the range we last stopped and which is has a higher probability of being still in the cache. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Relax locking in dma_ops pathJoerg Roedel1-59/+11
Remove the long holding times of the domain->lock and rely on the bitmap_lock instead. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Initialize new aperture range before making it visibleJoerg Roedel1-13/+20
Make sure the aperture range is fully initialized before it is visible to the address allocator. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Build io page-tables with cmpxchg64Joerg Roedel1-3/+13
This allows to build up the page-tables without holding any locks. As a consequence it removes the need to pre-populate dma_ops page-tables. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Allocate new aperture ranges in dma_ops_alloc_addressesJoerg Roedel1-19/+10
It really belongs there and not in __map_single. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Optimize dma_ops_free_addressesJoerg Roedel1-2/+3
Don't flush the iommu tlb when we free something behind the current next_bit pointer. Update the next_bit pointer instead and let the flush happen on the next wraparound in the allocation path. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Remove need_flush from struct dma_ops_domainJoerg Roedel1-24/+6
The flushing of iommu tlbs is now done on a per-range basis. So there is no need anymore for domain-wide flush tracking. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Iterate over all aperture ranges in dma_ops_area_allocJoerg Roedel1-17/+11
This way we don't need to care about the next_index wrapping around in dma_ops_alloc_addresses. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Flush iommu tlb in dma_ops_free_addressesJoerg Roedel1-2/+4
Instead of setting need_flush, do the flush directly in dma_ops_free_addresses. Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-12-28iommu/amd: Rename dma_ops_domain->next_address to next_indexJoerg Roedel1-13/+13
It points to the next aperture index to allocate from. We don't need the full address anymore because this is now tracked in struct aperture_range. Signed-off-by: Joerg Roedel <jroedel@suse.de>
generated by cgit v1.2.3 (git 2.39.0) at 2025-02-02 20:49:15 +0300