summaryrefslogtreecommitdiff
path: root/drivers/crypto/intel
AgeCommit message (Collapse)AuthorFilesLines
2025-07-31Merge tag 'v6.17-p1' of ↵Linus Torvalds41-447/+1261
git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 Pull crypto update from Herbert Xu: "API: - Allow hash drivers without fallbacks (e.g., hardware key) Algorithms: - Add hmac hardware key support (phmac) on s390 - Re-enable sha384 in FIPS mode - Disable sha1 in FIPS mode - Convert zstd to acomp Drivers: - Lower priority of qat skcipher and aead - Convert aspeed to partial block API - Add iMX8QXP support in caam - Add rate limiting support for GEN6 devices in qat - Enable telemetry for GEN6 devices in qat - Implement full backlog mode for hisilicon/sec2" * tag 'v6.17-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (116 commits) crypto: keembay - Use min() to simplify ocs_create_linked_list_from_sg() crypto: hisilicon/hpre - fix dma unmap sequence crypto: qat - make adf_dev_autoreset() static crypto: ccp - reduce stack usage in ccp_run_aes_gcm_cmd crypto: qat - refactor ring-related debug functions crypto: qat - fix seq_file position update in adf_ring_next() crypto: qat - fix DMA direction for compression on GEN2 devices crypto: jitter - replace ARRAY_SIZE definition with header include crypto: engine - remove {prepare,unprepare}_crypt_hardware callbacks crypto: engine - remove request batching support crypto: qat - flush misc workqueue during device shutdown crypto: qat - enable rate limiting feature for GEN6 devices crypto: qat - add compression slice count for rate limiting crypto: qat - add get_svc_slice_cnt() in device data structure crypto: qat - add adf_rl_get_num_svc_aes() in rate limiting crypto: qat - relocate service related functions crypto: qat - consolidate service enums crypto: qat - add decompression service for rate limiting crypto: qat - validate service in rate limiting sysfs api crypto: hisilicon/sec2 - implement full backlog mode for sec ...
2025-07-27crypto: keembay - Use min() to simplify ocs_create_linked_list_from_sg()Thorsten Blum1-2/+2
Use min() to simplify ocs_create_linked_list_from_sg() and improve its readability. Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-27crypto: qat - make adf_dev_autoreset() staticGiovanni Cabiddu2-2/+1
The function adf_dev_autoreset() is only used within adf_aer.c and does not need to be exposed outside the compilation unit. Make it static and remove it from the header adf_common_drv.h. This does not introduce any functional change. Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-27crypto: qat - refactor ring-related debug functionsGiovanni Cabiddu1-9/+14
Refactor the functions `adf_ring_start()` and `adf_ring_next()` to improve readability. This does not introduce any functional change. Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-27crypto: qat - fix seq_file position update in adf_ring_next()Giovanni Cabiddu1-1/+3
The `adf_ring_next()` function in the QAT debug transport interface fails to correctly update the position index when reaching the end of the ring elements. This triggers the following kernel warning when reading ring files, such as /sys/kernel/debug/qat_c6xx_<D:B:D:F>/transport/bank_00/ring_00: [27725.022965] seq_file: buggy .next function adf_ring_next [intel_qat] did not update position index Ensure that the `*pos` index is incremented before returning NULL when after the last element in the ring is found, satisfying the seq_file API requirements and preventing the warning. Fixes: a672a9dc872e ("crypto: qat - Intel(R) QAT transport code") Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-27crypto: qat - fix DMA direction for compression on GEN2 devicesGiovanni Cabiddu2-5/+5
QAT devices perform an additional integrity check during compression by decompressing the output. Starting from QAT GEN4, this verification is done in-line by the hardware. However, on GEN2 devices, the hardware reads back the compressed output from the destination buffer and performs a decompression operation using it as the source. In the current QAT driver, destination buffers are always marked as write-only. This is incorrect for QAT GEN2 compression, where the buffer is also read during verification. Since commit 6f5dc7658094 ("iommu/vt-d: Restore WO permissions on second-level paging entries"), merged in v6.16-rc1, write-only permissions are strictly enforced, leading to DMAR errors when using QAT GEN2 devices for compression, if VT-d is enabled. Mark the destination buffers as DMA_BIDIRECTIONAL. This ensures compatibility with GEN2 devices, even though it is not required for QAT GEN4 and later. Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Fixes: cf5bb835b7c8 ("crypto: qat - fix DMA transfer direction") Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - flush misc workqueue during device shutdownGiovanni Cabiddu3-0/+7
Repeated loading and unloading of a device specific QAT driver, for example qat_4xxx, in a tight loop can lead to a crash due to a use-after-free scenario. This occurs when a power management (PM) interrupt triggers just before the device-specific driver (e.g., qat_4xxx.ko) is unloaded, while the core driver (intel_qat.ko) remains loaded. Since the driver uses a shared workqueue (`qat_misc_wq`) across all devices and owned by intel_qat.ko, a deferred routine from the device-specific driver may still be pending in the queue. If this routine executes after the driver is unloaded, it can dereference freed memory, resulting in a page fault and kernel crash like the following: BUG: unable to handle page fault for address: ffa000002e50a01c #PF: supervisor read access in kernel mode RIP: 0010:pm_bh_handler+0x1d2/0x250 [intel_qat] Call Trace: pm_bh_handler+0x1d2/0x250 [intel_qat] process_one_work+0x171/0x340 worker_thread+0x277/0x3a0 kthread+0xf0/0x120 ret_from_fork+0x2d/0x50 To prevent this, flush the misc workqueue during device shutdown to ensure that all pending work items are completed before the driver is unloaded. Note: This approach may slightly increase shutdown latency if the workqueue contains jobs from other devices, but it ensures correctness and stability. Fixes: e5745f34113b ("crypto: qat - enable power management for QAT GEN4") Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Cc: stable@vger.kernel.org Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - enable rate limiting feature for GEN6 devicesSuman Kumar Chakraborty2-0/+91
Add support for enabling rate limiting(RL) feature for QAT GEN6 by initializing the rl_data member in adf_hw_device_data structure. Implement init_num_svc_aes() for GEN6 which will populate the number of AEs associated with the RL service type. Implement adf_gen6_get_svc_slice_cnt() for GEN6 which will return the slice count that can support the RL service type. Co-developed-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - add compression slice count for rate limitingSuman Kumar Chakraborty2-0/+2
In QAT GEN4 devices, the compression slice count was tracked using the dcpr_cnt field. Introduce a new cpr_cnt field in the rate limiting (RL) infrastructure to track the compression (CPR) slice count independently. The cpr_cnt value is populated via the RL_INIT admin message. The existing dcpr_cnt field will now be used exclusively to cache the decompression slice count, ensuring a clear separation between compression and decompression tracking. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - add get_svc_slice_cnt() in device data structureSuman Kumar Chakraborty6-14/+26
Enhance the adf_hw_device_data structure by introducing a new callback function get_svc_slice_cnt(), which provides a mechanism to query the total number of accelerator available on the device for a specific service. Implement adf_gen4_get_svc_slice_cnt() for QAT GEN4 devices to support this new interface. This function returns the total accelerator count for a specific service. Co-developed-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - add adf_rl_get_num_svc_aes() in rate limitingSuman Kumar Chakraborty6-1/+40
Enhance the rate limiting (RL) infrastructure by adding adf_rl_get_num_svc_aes() which can be used to fetch the number of engines associated with the service type. Expand the structure adf_rl_hw_data with an array that contains the number of AEs per service. Implement adf_gen4_init_num_svc_aes() for QAT GEN4 devices to calculate the total number of acceleration engines dedicated to a specific service. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - relocate service related functionsSuman Kumar Chakraborty5-36/+38
Rename (1) is_service_enabled() to adf_is_service_enabled(), and (2) srv_to_cfg_svc_type() to adf_srv_to_cfg_svc_type(), and move them to adf_cfg_services.c which is the appropriate place for configuration-related service logic. This improves code organization and modularity by grouping related service configuration logic in a single location. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - consolidate service enumsSuman Kumar Chakraborty9-50/+47
The enums `adf_base_services` (used in rate limiting) and `adf_services` define the same values, resulting in code duplication. To improve consistency across the QAT driver: (1) rename `adf_services` to `adf_base_services` in adf_cfg_services.c to better reflect its role in defining core services (those with dedicated accelerators), (2) introduce a new `adf_extended_services` enum starting from `SVC_BASE_COUNT`, and move `SVC_DCC` into it, as it represents an extended service (DC with chaining), and (3) remove the redundant `adf_base_services` enum from the rate limiting implementation. This does not introduce any functional change. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - add decompression service for rate limitingSuman Kumar Chakraborty3-0/+4
Add a new base service type ADF_SVC_DECOMP to the QAT rate limiting (RL) infrastructure. This enables RL support for the decompression (DECOMP) service type, allowing service-level agreements (SLAs) to be enforced when decompression is configured. The new service is exposed in the sysfs RL service list for visibility. Note that this support is applicable only to devices that provide the decompression service, such as QAT GEN6 devices. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - validate service in rate limiting sysfs apiSuman Kumar Chakraborty3-2/+10
The sysfs interface 'qat_rl/srv' currently allows all valid services, even if a service is not configured for the device. This leads to a failure when attempting to add the SLA using 'qat_rl/sla_op'. Add a check using is_service_enabled() to ensure the requested service is enabled. If not, return -EINVAL to prevent invalid configurations. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - enable telemetry for GEN6 devicesVijay Sundar Selvamani4-0/+348
Enable telemetry for QAT GEN6 devices by defining the firmware data structures layouts, implementing the counters parsing logic and setting the required properties on the adf_tl_hw_data data structure. As for QAT GEN4, telemetry counters are exposed via debugfs using the interface described in Documentation/ABI/testing/debugfs-driver-qat_telemetry. Co-developed-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: Vijay Sundar Selvamani <vijay.sundar.selvamani@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - add decompression service to telemetryVijay Sundar Selvamani1-0/+3
QAT GEN6 devices offer decompression as an additional service. Update the telemetry ring pair service interface to support monitoring decompression operations. Co-developed-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: Vijay Sundar Selvamani <vijay.sundar.selvamani@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - enable power management debugfs for GEN6 devicesGeorge Abraham P4-1/+159
The QAT driver includes infrastructure to report power management (PM) information via debugfs. Extend this support to QAT GEN6 devices by exposing PM debug data through the `pm_status` file. This implementation reports the current PM state, power management hardware control and status registers (CSR), and per-domain power status specific to the QAT GEN6 architecture. The debug functionality is implemented in adf_gen6_pm_dbgfs.c and initialized as part of the enable_pm() function. Co-developed-by: Vijay Sundar Selvamani <vijay.sundar.selvamani@intel.com> Signed-off-by: Vijay Sundar Selvamani <vijay.sundar.selvamani@intel.com> Signed-off-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - relocate power management debugfs helper APIsGeorge Abraham P4-86/+108
Relocate the power management debugfs helper APIs in a common file adf_pm_dbgfs_utils.h and adf_pm_dbgfs_utils.c so that it can be shared between device generations. When moving logic from adf_gen4_pm_debugfs.c to adf_pm_dbgfs_utils.c, the include kernel.h has been replaced with the required include. This does not introduce any functional change. Signed-off-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - fix virtual channel configuration for GEN6 devicesSuman Kumar Chakraborty2-6/+6
The TCVCMAP (Traffic Class to Virtual Channel Mapping) field in the PVC0CTL and PVC1CTL register controls how traffic classes are mapped to virtual channels in QAT GEN6 hardware. The driver previously wrote a default TCVCMAP value to this register, but this configuration was incorrect. Modify the TCVCMAP configuration to explicitly enable both VC0 and VC1, and map Traffic Classes 0 to 7 → VC0 and Traffic Class 8 → VC1. Replace FIELD_PREP() with FIELD_MODIFY() to ensure that only the intended TCVCMAP field is updated, preserving other bits in the register. This prevents unintended overwrites of unrelated configuration fields when modifying TC to VC mappings. Fixes: 17fd7514ae68 ("crypto: qat - add qat_6xxx driver") Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - add live migration enablers for GEN6 devicesMałgorzata Mielnik3-0/+13
The current implementation of the QAT live migration enablers is exclusive to QAT GEN4 devices and resides within QAT GEN4 specific files. However, the underlying mechanisms, such as the relevant CSRs and offsets, can be shared between QAT GEN4 and QAT GEN6 devices. Add the necessary enablers required to implement live migration for QAT GEN6 devices to the abstraction layer to allow leveraging the existing QAT GEN4 implementation. Signed-off-by: Małgorzata Mielnik <malgorzata.mielnik@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - relocate and rename bank state structure definitionSuman Kumar Chakraborty4-45/+47
The `bank_state` structure represents the state of a bank of rings. As part of recent refactoring, the functions that interact with this structure have been moved to a new unit, adf_bank_state.c. To align with this reorganization, rename `struct bank_state` to `struct adf_bank_state` and move its definition to adf_bank_state.h. Also relocate the associated `struct ring_config` to the same header to consolidate related definitions. Update all references to use the new structure name. This does not introduce any functional change. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - relocate bank state helper functionsMałgorzata Mielnik6-209/+258
The existing implementation of bank state management functions, including saving and restoring state, is located within 4xxx device files. However, these functions do not contain GEN4-specific code and are applicable to other QAT generations. Relocate the bank state management functions to a new file, adf_bank_state.c, and rename them removing the `gen4` prefix. This change enables the reuse of such functions across different QAT generations. Add documentation to bank state related functions that were moved from QAT 4xxx specific files to common files. This does not introduce any functional change. Signed-off-by: Małgorzata Mielnik <malgorzata.mielnik@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - replace CHECK_STAT macro with static inline functionSuman Kumar Chakraborty1-14/+19
The macro CHECK_STAT is used to check that all ring statuses match the saved state during restoring the state of bank. Replace the CHECK_STAT macro with the static inline function `check_stat()` to improve type safety, readability, and debuggability. This does not introduce any functional change. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - use pr_fmt() in adf_gen4_hw_data.cSuman Kumar Chakraborty1-4/+6
Add pr_fmt() to adf_gen4_hw_data.c logging and update the debug and error messages to utilize it accordingly. This does not introduce any functional changes. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: qat - disable ZUC-256 capability for QAT GEN5Bairavi Alagappan1-8/+1
The ZUC-256 EEA (encryption) and EIA (integrity) algorithms are not supported on QAT GEN5 devices, as their current implementation does not align with the NIST specification. Earlier versions of the ZUC-256 specification used a different initialization scheme, which has since been revised to comply with the 5G specification. Due to this misalignment with the updated specification, remove support for ZUC-256 EEA and EIA for QAT GEN5 by masking out the ZUC-256 capability. Fixes: fcf60f4bcf549 ("crypto: qat - add support for 420xx devices") Signed-off-by: Bairavi Alagappan <bairavix.alagappan@intel.com> Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-07-18crypto: keembay - Fix dma_unmap_sg() nents valueThomas Fourier1-1/+7
The dma_unmap_sg() functions should be called with the same nents as the dma_map_sg(), not the value the map function returned. Fixes: 472b04444cd3 ("crypto: keembay - Add Keem Bay OCS HCU driver") Signed-off-by: Thomas Fourier <fourier.thomas@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-06-26crypto: qat - Use crypto_shash_export_coreHerbert Xu1-7/+7
Use crypto_shash_export_core to export the core hash state without the partial blocks. Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-06-23crypto: qat - restore ASYM service support for GEN6 devicesSuman Kumar Chakraborty1-1/+9
Support for asymmetric crypto services was not included in the qat_6xxx by explicitly setting the asymmetric capabilities to 0 to allow for additional testing. Enable asymmetric crypto services on QAT GEN6 devices by setting the appropriate capability flags. Fixes: 17fd7514ae68 ("crypto: qat - add qat_6xxx driver") Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-06-23crypto: qat - remove duplicate masking for GEN6 devicesSuman Kumar Chakraborty1-1/+0
The ICP_ACCEL_CAPABILITIES_CIPHER capability is masked out redundantly for QAT GEN6 devices. Remove it to avoid code duplication. This does not introduce any functional change. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-06-23crypto: qat - lower priority for skcipher and aead algorithmsGiovanni Cabiddu1-6/+6
Most kernel applications utilizing the crypto API operate synchronously and on small buffer sizes, therefore do not benefit from QAT acceleration. Reduce the priority of QAT implementations for both skcipher and aead algorithms, allowing more suitable alternatives to be selected by default. Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Link: https://lore.kernel.org/all/20250613012357.GA3603104@google.com/ Cc: stable@vger.kernel.org Acked-by: Eric Biggers <ebiggers@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-06-13crypto: qat - add support for decompression service to GEN6 devicesSuman Kumar Chakraborty7-1/+25
Add support to configure decompression as a separate service for QAT GEN6 devices. A new arbiter configuration has been added to map the hardware decompression threads to all ring pairs. The decompression service is enabled via sysfs by writing "decomp" to "/sys/bus/pci/devices/<BDF>/qat/cfg_services". The decompression service is not supported on QAT GEN2 and GEN4 devices, and attempting it results in an invalid write error. The existing compression service for QAT GEN2 and GEN4 devices remains unchanged and supports both compression and decompression operations on the same ring pair. Co-developed-by: Karthikeyan Gopal <karthikeyan.gopal@intel.com> Signed-off-by: Karthikeyan Gopal <karthikeyan.gopal@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-06-13crypto: qat - fix state restore for banks with exceptionsSvyatoslav Pankratov1-7/+22
Change the logic in the restore function to properly handle bank exceptions. The check for exceptions in the saved state should be performed before conducting any other ringstat register checks. If a bank was saved with an exception, the ringstat will have the appropriate rp_halt/rp_exception bits set, causing the driver to exit the restore process with an error. Instead, the restore routine should first check the ringexpstat register, and if any exception was raised, it should stop further checks and return without any error. In other words, if a ring pair is in an exception state at the source, it should be restored the same way at the destination but without raising an error. Even though this approach might lead to losing the exception state during migration, the driver will log the exception from the saved state during the restore process. Signed-off-by: Svyatoslav Pankratov <svyatoslav.pankratov@intel.com> Fixes: bbfdde7d195f ("crypto: qat - add bank save and restore flows") Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-06-13crypto: qat - allow enabling VFs in the absence of IOMMUAhsan Atta1-1/+0
The commit ca88a2bdd4dd ("crypto: qat - allow disabling SR-IOV VFs") introduced an unnecessary change that prevented enabling SR-IOV when IOMMU is disabled. In certain scenarios, it is desirable to enable SR-IOV even in the absence of IOMMU. Thus, restoring the previous functionality to allow VFs to be enumerated in the absence of IOMMU. Fixes: ca88a2bdd4dd ("crypto: qat - allow disabling SR-IOV VFs") Signed-off-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Michal Witwicki <michal.witwicki@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-06-13crypto: qat - use unmanaged allocation for dc_dataSuman Kumar Chakraborty1-2/+2
The dc_data structure holds data required for handling compression operations, such as overflow buffers. In this context, the use of managed memory allocation APIs (devm_kzalloc() and devm_kfree()) is not necessary, as these data structures are freed and re-allocated when a device is restarted in adf_dev_down() and adf_dev_up(). Additionally, managed APIs automatically handle memory cleanup when the device is detached, which can lead to conflicts with manual cleanup processes. Specifically, if a device driver invokes the adf_dev_down() function as part of the cleanup registered with devm_add_action_or_reset(), it may attempt to free memory that is also managed by the device's resource management system, potentially leading to a double-free. This might result in a warning similar to the following when unloading the device specific driver, for example qat_6xxx.ko: qat_free_dc_data+0x4f/0x60 [intel_qat] qat_compression_event_handler+0x3d/0x1d0 [intel_qat] adf_dev_shutdown+0x6d/0x1a0 [intel_qat] adf_dev_down+0x32/0x50 [intel_qat] devres_release_all+0xb8/0x110 device_unbind_cleanup+0xe/0x70 device_release_driver_internal+0x1c1/0x200 driver_detach+0x48/0x90 bus_remove_driver+0x74/0xf0 pci_unregister_driver+0x2e/0xb0 Use unmanaged memory allocation APIs (kzalloc_node() and kfree()) for the dc_data structure. This ensures that memory is explicitly allocated and freed under the control of the driver code, preventing manual deallocation from interfering with automatic cleanup. Fixes: 1198ae56c9a5 ("crypto: qat - expose deflate through acomp api for QAT GEN2") Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-23crypto: qat - add missing header inclusionArnd Bergmann2-0/+2
Without this header, the build of the new qat_6xxx driver fails when CONFIG_PCI_IOV is not set: In file included from drivers/crypto/intel/qat/qat_common/adf_gen6_shared.c:7: drivers/crypto/intel/qat/qat_common/adf_gen4_pfvf.h: In function 'adf_gen4_init_pf_pfvf_ops': drivers/crypto/intel/qat/qat_common/adf_gen4_pfvf.h:13:34: error: 'adf_pfvf_comms_disabled' undeclared (first use in this function) 13 | pfvf_ops->enable_comms = adf_pfvf_comms_disabled; | ^~~~~~~~~~~~~~~~~~~~~~~ Fixes: 17fd7514ae68 ("crypto: qat - add qat_6xxx driver") Signed-off-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-19crypto: qat - enable reporting of error counters for GEN6 devicesSuman Kumar Chakraborty1-0/+2
Enable the reporting of error counters through sysfs for QAT GEN6 devices and update the ABI documentation. This enables the reporting of the following: - errors_correctable - hardware correctable errors that allow the system to recover without data loss. - errors_nonfatal: errors that can be isolated to specific in-flight requests. - errors_fatal: errors that cannot be contained to a request, requiring a Function Level Reset (FLR) upon occurrence. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-19crypto: qat - enable RAS support for GEN6 devicesSuman Kumar Chakraborty4-0/+1325
Enable the reporting and handling of errors for QAT GEN6 devices. Errors are categorized as correctable, non-fatal, or fatal. Error handling involves reading the error source registers (ERRSOU0 to ERRSOU3) to determine the source of the error and then decoding the actual source reading specific registers. The action taken depends on the error type: - Correctable and Non-Fatal errors. These error are logged, cleared and the corresponding counter is incremented. - Fatal errors. These errors are logged, cleared and a Function Level Reset (FLR) is scheduled. This reports and handles the following errors: - Accelerator engine (AE) correctable errors - Accelerator engine (AE) uncorrectable errors - Chassis push-pull (CPP) errors - Host interface (HI) parity errors - Internal memory parity errors - Receive interface (RI) errors - Transmit interface (TI) errors - Interface for system-on-chip (SoC) fabric (IOSF) primary command parity errors - Shared RAM and slice module (SSM) errors Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-14crypto: iaa - Optimize rebalance_wq_table()Yury Norov1-21/+14
The function opencodes for_each_cpu() by using a plain for-loop. The loop calls cpumask_weight() inside the conditional section. Because cpumask_weight() is O(1), the overall complexity of the function is O(node * node_cpus^2). Also, cpumask_nth() internally calls hweight(), which, if not hardware accelerated, is slower than cpumask_next() in for_each_cpu(). If switched to the dedicated for_each_cpu(), the rebalance_wq_table() can drop calling cpumask_weight(), together with some housekeeping code. This makes the overall complexity O(node * node_cpus), or simply speaking O(nr_cpu_ids). While there, fix opencoded for_each_possible_cpu() too. Signed-off-by: Yury Norov <yury.norov@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-14crypto: qat/qat_6xxx - Fix NULL vs IS_ERR() check in adf_probe()Dan Carpenter1-2/+2
The pcim_iomap_region() returns error pointers. It doesn't return NULL pointers. Update the check to match. Fixes: 17fd7514ae68 ("crypto: qat - add qat_6xxx driver") Signed-off-by: Dan Carpenter <dan.carpenter@linaro.org> Acked-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - add qat_6xxx driverLaurent M Coquerel13-0/+1328
Add a new driver, qat_6xxx, to support QAT GEN6 devices. QAT GEN6 devices are a follow-on generation of GEN4 devices and differently from the previous generation, they can support all three services (symmetric, asymmetric, and data compression) concurrently. In order to have the qat_6xxx driver to reuse some of the GEN4 logic, a new abstraction layer has been introduced to bridge the two implementations. This allows to avoid code duplication and to keep the qat_6xxx driver isolated from the GEN4 logic. This approach has been used for the PF to VF logic and the HW CSR access logic. Signed-off-by: Laurent M Coquerel <laurent.m.coquerel@intel.com> Co-developed-by: George Abraham P <george.abraham.p@intel.com> Signed-off-by: George Abraham P <george.abraham.p@intel.com> Co-developed-by: Karthikeyan Gopal <karthikeyan.gopal@intel.com> Signed-off-by: Karthikeyan Gopal <karthikeyan.gopal@intel.com> Co-developed-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - add firmware headers for GEN6 devicesSuman Kumar Chakraborty2-0/+417
Add firmware headers related to compression that define macros for building the hardware configuration word, along with bitfields related to algorithm settings. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - update firmware apiSuman Kumar Chakraborty2-4/+22
Update the firmware API to have partial decomp as an argument. Modify the firmware descriptor to support auto-select best and partial decompress. Define the maximal auto-select best value. Define the mask and bit position for the partial decompress field in the firmware descriptor. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - export adf_init_admin_pm()Suman Kumar Chakraborty1-0/+1
Export the function adf_init_admin_pm() as it will be used by the qat_6xxx driver to send the power management initialization messages to the firmware. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - expose configuration functionsSuman Kumar Chakraborty2-3/+6
The functions related to compression and crypto configurations were previously declared static, restricting the visibility to the defining source file. Remove the static qualifier, allowing it to be used in other files as needed. This is necessary for sharing this configuration functions with other QAT generations. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - export adf_get_service_mask()Giovanni Cabiddu2-1/+3
Export the function adf_get_service_mask() as it will be used by the qat_6xxx driver to configure the device. Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - add GEN6 firmware loaderJack Xu5-7/+176
Add support for the QAT GEN6 devices in the firmware loader. This includes handling firmware images signed with the RSA 3K and the XMSS algorithms. Co-developed-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Signed-off-by: Jack Xu <jack.xu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - refactor FW signing algorithmJack Xu1-70/+84
The current implementation is designed to support single FW signing authentication only. Refactor the implementation to support other FW signing methods. This does not include any functional change. Co-developed-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Signed-off-by: Jack Xu <jack.xu@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - use pr_fmt() in qat uclo.cSuman Kumar Chakraborty1-70/+65
Add pr_fmt() to qat uclo.c logging and update the debug and error messages to utilize it accordingly. This does not introduce any functional changes. Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2025-05-05crypto: qat - refactor compression template logicSuman Kumar Chakraborty22-146/+173
The logic that generates the compression templates, which are used by to submit compression requests to the QAT device, is very similar between QAT devices and diverges mainly on the HW generation-specific configuration word. This makes the logic that generates the compression and decompression templates common between GEN2 and GEN4 devices and abstracts the generation-specific logic to the generation-specific implementations. The adf_gen2_dc.c and adf_gen4_dc.c have been replaced by adf_dc.c, and the generation-specific logic has been reduced and moved to adf_gen2_hw_data.c and adf_gen4_hw_data.c. This does not introduce any functional change. Co-developed-by: Vijay Sundar Selvamani <vijay.sundar.selvamani@intel.com> Signed-off-by: Vijay Sundar Selvamani <vijay.sundar.selvamani@intel.com> Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>