kernel/linux.git/arch/x86/crypto, branch v7.0.10

crypto: x86/aes-gcm - Use new AES library API

2026-01-15T22:09:08+00:00

Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Since this changes the format of the AES-GCM key structures that are used by the AES-GCM assembly code, the offsets in the assembly code had to be updated to match. Note that the new key structures are smaller, since the decryption round keys are no longer unnecessarily included. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260112192035.10427-26-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: x86/aes - Remove the superseded AES-NI crypto_cipher

2026-01-15T22:09:07+00:00

Remove the "aes-aesni" crypto_cipher algorithm and the code specific to its implementation. It is no longer necessary because the AES library is now optimized with x86 AES-NI, and crypto/aes.c exposes the AES library via the crypto_cipher API. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260112192035.10427-19-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crypto: x86/nh: Migrate optimized code into library

2026-01-12T19:07:50+00:00

Migrate the x86_64 implementations of NH into lib/crypto/. This makes the nh() function be optimized on x86_64 kernels. Note: this temporarily makes the adiantum template not utilize the x86_64 optimized NH code. This is resolved in a later commit that converts the adiantum template to use nh() instead of "nhpoly1305". Link: https://lore.kernel.org/r/20251211011846.8179-6-ebiggers@kernel.org Signed-off-by: Eric Biggers

Merge tag 'v6.19-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

2025-12-03T19:28:38+00:00

Pull crypto updates from Herbert Xu: "API: - Rewrite memcpy_sglist from scratch - Add on-stack AEAD request allocation - Fix partial block processing in ahash Algorithms: - Remove ansi_cprng - Remove tcrypt tests for poly1305 - Fix EINPROGRESS processing in authenc - Fix double-free in zstd Drivers: - Use drbg ctr helper when reseeding xilinx-trng - Add support for PCI device 0x115A to ccp - Add support of paes in caam - Add support for aes-xts in dthev2 Others: - Use likely in rhashtable lookup - Fix lockdep false-positive in padata by removing a helper" * tag 'v6.19-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (71 commits) crypto: zstd - fix double-free in per-CPU stream cleanup crypto: ahash - Zero positive err value in ahash_update_finish crypto: ahash - Fix crypto_ahash_import with partial block data crypto: lib/mpi - use min() instead of min_t() crypto: ccp - use min() instead of min_t() hwrng: core - use min3() instead of nested min_t() crypto: aesni - ctr_crypt() use min() instead of min_t() crypto: drbg - Delete unused ctx from struct sdesc crypto: testmgr - Add missing DES weak and semi-weak key tests Revert "crypto: scatterwalk - Move skcipher walk and use it for memcpy_sglist" crypto: scatterwalk - Fix memcpy_sglist() to always succeed crypto: iaa - Request to add Kanchana P Sridhar to Maintainers. crypto: tcrypt - Remove unused poly1305 support crypto: ansi_cprng - Remove unused ansi_cprng algorithm crypto: asymmetric_keys - fix uninitialized pointers with free attribute KEYS: Avoid -Wflex-array-member-not-at-end warning crypto: ccree - Correctly handle return of sg_nents_for_len crypto: starfive - Correctly handle return of sg_nents_for_len crypto: iaa - Fix incorrect return value in save_iaa_wq() crypto: zstd - Remove unnecessary size_t cast ...

Merge tag 'aes-gcm-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux

2025-12-03T02:24:35+00:00

Pull AES-GCM optimizations from Eric Biggers: "More optimizations and cleanups for the x86_64 AES-GCM code: - Add a VAES+AVX2 optimized implementation of AES-GCM. This is very helpful on CPUs that have VAES but not AVX512, such as AMD Zen 3. - Make the VAES+AVX512 optimized implementation of AES-GCM handle large amounts of associated data efficiently. - Remove the "avx10_256" implementation of AES-GCM. It's superseded by the VAES+AVX2 optimized implementation. - Rename the "avx10_512" implementation to "avx512" Overall, this fills in a gap where AES-GCM wasn't fully optimized on some recent CPUs. It also drops code that won't be as useful as initially expected due to AVX10/256 being dropped from the AVX10 spec" * tag 'aes-gcm-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux: crypto: x86/aes-gcm-vaes-avx2 - initialize full %rax return register crypto: x86/aes-gcm - optimize long AAD processing with AVX512 crypto: x86/aes-gcm - optimize AVX512 precomputation of H^2 from H^1 crypto: x86/aes-gcm - revise some comments in AVX512 code crypto: x86/aes-gcm - reorder AVX512 precompute and aad_update functions crypto: x86/aes-gcm - clean up AVX512 code to assume 512-bit vectors crypto: x86/aes-gcm - rename avx10 and avx10_512 to avx512 crypto: x86/aes-gcm - remove VAES+AVX10/256 optimized code crypto: x86/aes-gcm - add VAES+AVX2 optimized code

crypto: aesni - ctr_crypt() use min() instead of min_t()

2025-11-24T09:43:40+00:00

min_t(unsigned int, a, b) casts an 'unsigned long' to 'unsigned int'. Use min(a, b) instead as it promotes any 'unsigned int' to 'unsigned long' and so cannot discard significant bits. In this case the 'unsigned long' value is small enough that the result is ok. Detected by an extra check added to min_t(). Signed-off-by: David Laight Signed-off-by: Herbert Xu

lib/crypto: x86/polyval: Migrate optimized code into library

2025-11-11T19:03:38+00:00

Migrate the x86_64 implementation of POLYVAL into lib/crypto/, wiring it up to the POLYVAL library interface. This makes the POLYVAL library be properly optimized on x86_64. This drops the x86_64 optimizations of polyval in the crypto_shash API. That's fine, since polyval will be removed from crypto_shash entirely since it is unneeded there. But even if it comes back, the crypto_shash API could just be implemented on top of the library API, as usual. Adjust the names and prototypes of the assembly functions to align more closely with the rest of the library code. Also replace a movaps instruction with movups to remove the assumption that the key struct is 16-byte aligned. Users can still align the key if they want (and at least in this case, movups is just as fast as movaps), but it's inconvenient to require it. Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20251109234726.638437-6-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: x86/aes-gcm-vaes-avx2 - initialize full %rax return register

2025-11-03T17:07:57+00:00

Update aes_gcm_dec_final_vaes_avx2() to be consistent with aes_gcm_dec_final_aesni() and aes_gcm_dec_final_vaes_avx512() by initializing the full %rax return register instead of just %al. Technically this is unnecessary, since these functions return bool. But I think it's worth being extra careful with the result of the tag comparison and also keeping the different implementations consistent. Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20251102015256.171536-1-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: x86/aes-gcm - optimize long AAD processing with AVX512

2025-10-27T03:37:41+00:00

Improve the performance of aes_gcm_aad_update_vaes_avx512() on large AAD (additional authenticated data) lengths by 4-8 times by making it use up to 512-bit vectors and a 4-vector-wide loop. Previously, it used only 256-bit vectors and a 1-vector-wide loop. Originally, I assumed that the case of large AADLEN was unimportant. Later, when reviewing the users of BoringSSL's AES-GCM code, I found that some callers use BoringSSL's AES-GCM API to just compute GMAC, authenticating lots of data but not en/decrypting any. Thus, I included a similar optimization in the BoringSSL port of this code. I believe it's wise to include this optimization in the kernel port too for similar reasons, and to align it more closely with the BoringSSL port. Another reason this function originally used 256-bit vectors was so that separate *_avx10_256 and *_avx10_512 versions of it wouldn't be needed. However, that's no longer applicable. To avoid a slight performance regression in the common case of AADLEN <= 16, also add a fast path for that case which uses 128-bit vectors. In fact, this case actually gets slightly faster too, since it saves a couple instructions over the original 256-bit code. Acked-by: Ard Biesheuvel Tested-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20251002023117.37504-9-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: x86/aes-gcm - optimize AVX512 precomputation of H^2 from H^1

2025-10-27T03:37:41+00:00

Squaring in GF(2^128) requires fewer instructions than a generic multiplication in GF(2^128). Take advantage of this when computing H^2 from H^1 in aes_gcm_precompute_vaes_avx512(). Note that aes_gcm_precompute_vaes_avx2() already uses this optimization. Acked-by: Ard Biesheuvel Tested-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20251002023117.37504-8-ebiggers@kernel.org Signed-off-by: Eric Biggers