summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2026-01-16lib/crypto: aesgcm: Use new AES library APIEric Biggers2-7/+7
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-34-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16lib/crypto: aescfb: Use new AES library APIEric Biggers3-22/+22
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-33-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: omap - Use new AES library APIEric Biggers2-4/+4
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-32-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: inside-secure - Use new AES library APIEric Biggers2-14/+12
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. This driver used crypto_aes_ctx::key_enc, but only to access the copy of the raw key that is stored at the beginning of the expanded key. To eliminate the dependency on this field, instead just access the raw key directly, which is already available in the relevant functions. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-31-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: drbg - Use new AES library APIEric Biggers4-31/+21
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-30-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: crypto4xx - Use new AES library APIEric Biggers1-5/+5
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-29-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: chelsio - Use new AES library APIEric Biggers1-5/+5
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_key and struct aes_enckey). In encryption-only use cases, this eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Likewise for decryption. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-28-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: ccp - Use new AES library APIEric Biggers1-2/+2
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-27-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: x86/aes-gcm - Use new AES library APIEric Biggers4-69/+67
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Since this changes the format of the AES-GCM key structures that are used by the AES-GCM assembly code, the offsets in the assembly code had to be updated to match. Note that the new key structures are smaller, since the decryption round keys are no longer unnecessarily included. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-26-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: arm64/ghash - Use new AES library APIEric Biggers1-20/+7
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-25-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: arm/ghash - Use new AES library APIEric Biggers1-5/+9
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-24-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16staging: rtl8723bs: core: Use new AES library APIEric Biggers1-10/+10
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-23-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16net: phy: mscc: macsec: Use new AES library APIEric Biggers1-4/+4
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-22-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16chelsio: Use new AES library APIEric Biggers3-8/+8
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-21-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16Bluetooth: SMP: Use new AES library APIEric Biggers1-4/+4
Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-20-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16crypto: x86/aes - Remove the superseded AES-NI crypto_cipherEric Biggers3-88/+1
Remove the "aes-aesni" crypto_cipher algorithm and the code specific to its implementation. It is no longer necessary because the AES library is now optimized with x86 AES-NI, and crypto/aes.c exposes the AES library via the crypto_cipher API. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-19-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16lib/crypto: x86/aes: Add AES-NI optimizationEric Biggers4-0/+348
Optimize the AES library with x86 AES-NI instructions. The relevant existing assembly functions, aesni_set_key(), aesni_enc(), and aesni_dec(), are a bit difficult to extract into the library: - They're coupled to the code for the AES modes. - They operate on struct crypto_aes_ctx. The AES library now uses different structs. - They assume the key is 16-byte aligned. The AES library only *prefers* 16-byte alignment; it doesn't require it. Moreover, they're not all that great in the first place: - They use unrolled loops, which isn't a great choice on x86. - They use the 'aeskeygenassist' instruction, which is unnecessary, is slow on Intel CPUs, and forces the loop to be unrolled. - They have special code for AES-192 key expansion, despite that being kind of useless. AES-128 and AES-256 are the ones used in practice. These are small functions anyway. Therefore, I opted to just write replacements of these functions for the library. They address all the above issues. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-18-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16lib/crypto: sparc/aes: Migrate optimized code into libraryEric Biggers8-137/+200
Move the SPARC64 AES assembly code into lib/crypto/, wire the key expansion and single-block en/decryption functions up to the AES library API, and remove the "aes-sparc64" crypto_cipher algorithm. The result is that both the AES library and crypto_cipher APIs use the SPARC64 AES opcodes, whereas previously only crypto_cipher did (and it wasn't enabled by default, which this commit fixes as well). Note that some of the functions in the SPARC64 AES assembly code are still used by the AES mode implementations in arch/sparc/crypto/aes_glue.c. For now, just export these functions. These exports will go away once the AES mode implementations are migrated to the library as well. (Trying to split up the assembly file seemed like much more trouble than it would be worth.) Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-17-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-16lib/crypto: s390/aes: Migrate optimized code into libraryEric Biggers5-115/+110
Implement aes_preparekey_arch(), aes_encrypt_arch(), and aes_decrypt_arch() using the CPACF AES instructions. Then, remove the superseded "aes-s390" crypto_cipher. The result is that both the AES library and crypto_cipher APIs use the CPACF AES instructions, whereas previously only crypto_cipher did (and it wasn't enabled by default, which this commit fixes as well). Note that this preserves the optimization where the AES key is stored in raw form rather than expanded form. CPACF just takes the raw key. Acked-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Holger Dengler <dengler@linux.ibm.com> Reviewed-by: Holger Dengler <dengler@linux.ibm.com> Link: https://lore.kernel.org/r/20260112192035.10427-16-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: riscv/aes: Migrate optimized code into libraryEric Biggers8-106/+166
Move the aes_encrypt_zvkned() and aes_decrypt_zvkned() assembly functions into lib/crypto/, wire them up to the AES library API, and remove the "aes-riscv64-zvkned" crypto_cipher algorithm. To make this possible, change the prototypes of these functions to take (rndkeys, key_len) instead of a pointer to crypto_aes_ctx, and change the RISC-V AES-XTS code to implement tweak encryption using the AES library instead of directly calling aes_encrypt_zvkned(). The result is that both the AES library and crypto_cipher APIs use RISC-V's AES instructions, whereas previously only crypto_cipher did (and it wasn't enabled by default, which this commit fixes as well). Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-15-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: powerpc/aes: Migrate POWER8 optimized code into libraryEric Biggers10-172/+226
Move the POWER8 AES assembly code into lib/crypto/, wire the key expansion and single-block en/decryption functions up to the AES library API, and remove the superseded "p8_aes" crypto_cipher algorithm. The result is that both the AES library and crypto_cipher APIs are now optimized for POWER8, whereas previously only crypto_cipher was (and optimizations weren't enabled by default, which this commit fixes too). Note that many of the functions in the POWER8 assembly code are still used by the AES mode implementations in arch/powerpc/crypto/. For now, just export these functions. These exports will go away once the AES modes are migrated to the library as well. (Trying to split up the assembly file seemed like much more trouble than it would be worth.) Another challenge with this code is that the POWER8 assembly code uses a custom format for the expanded AES key. Since that code is imported from OpenSSL and is also targeted to POWER8 (rather than POWER9 which has better data movement and byteswap instructions), that is not easily changed. For now I've just kept the custom format. To maintain full correctness, this requires executing some slow fallback code in the case where the usability of VSX changes between key expansion and use. This should be tolerable, as this case shouldn't happen in practice. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-14-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: powerpc/aes: Migrate SPE optimized code into libraryEric Biggers12-85/+122
Move the PowerPC SPE AES assembly code into lib/crypto/, wire the key expansion and single-block en/decryption functions up to the AES library API, and remove the superseded "aes-ppc-spe" crypto_cipher algorithm. The result is that both the AES library and crypto_cipher APIs are now optimized with SPE, whereas previously only crypto_cipher was (and optimizations weren't enabled by default, which this commit fixes too). Note that many of the functions in the PowerPC SPE assembly code are still used by the AES mode implementations in arch/powerpc/crypto/. For now, just export these functions. These exports will go away once the AES modes are migrated to the library as well. (Trying to split up the assembly files seemed like much more trouble than it would be worth.) Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-13-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: arm64/aes: Migrate optimized code into libraryEric Biggers13-290/+181
Move the ARM64 optimized AES key expansion and single-block AES en/decryption code into lib/crypto/, wire it up to the AES library API, and remove the superseded crypto_cipher algorithms. The result is that both the AES library and crypto_cipher APIs are now optimized for ARM64, whereas previously only crypto_cipher was (and the optimizations weren't enabled by default, which this fixes as well). Note: to see the diff from arch/arm64/crypto/aes-ce-glue.c to lib/crypto/arm64/aes.h, view this commit with 'git show -M10'. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-12-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: arm/aes: Migrate optimized code into libraryEric Biggers12-114/+63
Move the ARM optimized single-block AES en/decryption code into lib/crypto/, wire it up to the AES library API, and remove the superseded "aes-arm" crypto_cipher algorithm. The result is that both the AES library and crypto_cipher APIs are now optimized for ARM, whereas previously only crypto_cipher was (and the optimizations weren't enabled by default, which this fixes as well). Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-11-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: aes - Replace aes-generic with wrapper around libEric Biggers9-1353/+117
Now that the AES library's performance has been improved, replace aes_generic.c with a new file aes.c which wraps the AES library. In preparation for making the AES library actually utilize the kernel's existing architecture-optimized AES code including AES instructions, set the driver name to "aes-lib" instead of "aes-generic". This mirrors what's been done for the hash algorithms. Update testmgr.c accordingly. Since this removes the crypto_aes_set_key() helper function, add temporary replacements for it to arch/arm/crypto/aes-cipher-glue.c and arch/arm64/crypto/aes-cipher-glue.c. This is temporary, as that code will be migrated into lib/crypto/ in later commits. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-10-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: aes - Remove aes-fixed-time / CONFIG_CRYPTO_AES_TIEric Biggers17-119/+2
Remove aes-fixed-time, i.e. CONFIG_CRYPTO_AES_TI. This was a wrapper around the 256-byte-table-based AES implementation in lib/crypto/aes.c, with extra code to enable and disable IRQs for constant-time hardening. While nice in theory, in practice this had the following issues: - For bulk en/decryption it was 2-4 times slower than aes-generic. This resulted in aes-generic still being needed, creating fragmentation. - Having both aes-generic and aes-fixed-time punted an AES implementation decision to distros and users who are generally unprepared to handle it. In practice, whether aes-fixed-time gets used tends to be incidental and not match an explicit distro or user intent. (While aes-fixed-time has a higher priority than aes-generic, whether it actually gets enabled, loaded, and used depends on the kconfig and whether a modprobe of "aes" happens to be done. It also has a lower priority than aes-arm and aes-arm64.) - My changes to the generic AES code (in other commits) significantly close the gap with aes-fixed-time anyway. The table size is reduced from 8192 bytes to 1024 bytes, and prefetching is added. - While AES code *should* be constant-time, the real solutions for that are AES instructions (which most CPUs have now) or bit-slicing. arm and arm64 already have bit-sliced AES code for many modes; generic bit-sliced code could be written but would be very slow for single blocks. Overall, I suggest that trying to write constant-time table-based AES code is a bit futile anyway, and in the rare cases where a proper AES implementation is still unavailable it's reasonable to compromise with an implementation that simply prefetches the table. Thus, this commit removes aes-fixed-time and CONFIG_CRYPTO_AES_TI. The replacement is just the existing CONFIG_CRYPTO_AES, which for now maps to the existing aes-generic code, but I'll soon be changing to use the improved AES library code instead. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-9-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: aegis - Switch from crypto_ft_tab[] to aes_enc_tab[]Eric Biggers2-2/+2
Instead of crypto_ft_tab[0] from aes_generic.c, use aes_enc_tab from lib/crypto/aes.c. These contain the same data, so the result is the same. This will allow aes_generic.c to eventually be removed. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-8-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: arm64/aes - Select CRYPTO_LIB_SHA256 from correct placesEric Biggers1-1/+2
The call to sha256() occurs in code that is built when either CRYPTO_AES_ARM64_CE_BLK or CRYPTO_AES_ARM64_NEON_BLK. The option CRYPTO_AES_ARM64 is unrelated, notwithstanding its documentation. I'll be removing CRYPTO_AES_ARM64 soon anyway, but before doing that, fix where CRYPTO_LIB_SHA256 is selected from. Fixes: 01834444d972 ("crypto: arm64/aes - use SHA-256 library instead of crypto_shash") Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-7-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: arm64/aes - Switch to aes_enc_tab[] and aes_dec_tab[]Eric Biggers1-2/+2
Instead of crypto_ft_tab and crypto_it_tab from aes_generic.c, use aes_enc_tab and aes_dec_tab from lib/crypto/aes.c. These contain the same data in the first 1024 bytes (which is the part that this code uses), so the result is the same. This will allow aes_generic.c to eventually be removed. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-6-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: arm/aes - Switch to aes_enc_tab[] and aes_dec_tab[]Eric Biggers1-2/+2
Instead of crypto_ft_tab and crypto_it_tab from aes_generic.c, use aes_enc_tab and aes_dec_tab from lib/crypto/aes.c. These contain the same data in the first 1024 bytes (which is the part that this code uses), so the result is the same. This will allow aes_generic.c to eventually be removed. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-5-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: arm/aes-neonbs - Use AES library for single blocksEric Biggers2-14/+16
aes-neonbs-glue.c calls __aes_arm_encrypt() and __aes_arm_decrypt() to en/decrypt single blocks for CBC encryption, XTS tweak encryption, and XTS ciphertext stealing. In preparation for making the AES library use this same ARM-optimized single-block AES en/decryption code and making it an internal implementation detail of the AES library, replace the calls to these functions with calls to the AES library. Note that this reduces the size of the aesbs_cbc_ctx and aesbs_xts_ctx structs, since unnecessary decryption round keys are no longer included. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-4-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: aes: Introduce improved AES libraryEric Biggers4-70/+501
The kernel's AES library currently has the following issues: - It doesn't take advantage of the architecture-optimized AES code, including the implementations using AES instructions. - It's much slower than even the other software AES implementations: 2-4 times slower than "aes-generic", "aes-arm", and "aes-arm64". - It requires that both the encryption and decryption round keys be computed and cached. This is wasteful for users that need only the forward (encryption) direction of the cipher: the key struct is 484 bytes when only 244 are actually needed. This missed optimization is very common, as many AES modes (e.g. GCM, CFB, CTR, CMAC, and even the tweak key in XTS) use the cipher only in the forward (encryption) direction even when doing decryption. - It doesn't provide the flexibility to customize the prepared key format. The API is defined to do key expansion, and several callers in drivers/crypto/ use it specifically to expand the key. This is an issue when integrating the existing powerpc, s390, and sparc code, which is necessary to provide full parity with the traditional API. To resolve these issues, I'm proposing the following changes: 1. New structs 'aes_key' and 'aes_enckey' are introduced, with corresponding functions aes_preparekey() and aes_prepareenckey(). Generally these structs will include the encryption+decryption round keys and the encryption round keys, respectively. However, the exact format will be under control of the architecture-specific AES code. (The verb "prepare" is chosen over "expand" since key expansion isn't necessarily done. It's also consistent with hmac*_preparekey().) 2. aes_encrypt() and aes_decrypt() will be changed to operate on the new structs instead of struct crypto_aes_ctx. 3. aes_encrypt() and aes_decrypt() will use architecture-optimized code when available, or else fall back to a new generic AES implementation that unifies the existing two fragmented generic AES implementations. The new generic AES implementation uses tables for both SubBytes and MixColumns, making it almost as fast as "aes-generic". However, instead of aes-generic's huge 8192-byte tables per direction, it uses only 1024 bytes for encryption and 1280 bytes for decryption (similar to "aes-arm"). The cost is just some extra rotations. The new generic AES implementation also includes table prefetching, making it have some "constant-time hardening". That's an improvement from aes-generic which has no constant-time hardening. It does slightly regress in constant-time hardening vs. the old lib/crypto/aes.c which had smaller tables, and from aes-fixed-time which disabled IRQs on top of that. But I think this is tolerable. The real solutions for constant-time AES are AES instructions or bit-slicing. The table-based code remains a best-effort fallback for the increasingly-rare case where a real solution is unavailable. 4. crypto_aes_ctx and aes_expandkey() will remain for now, but only for callers that are using them specifically for the AES key expansion (as opposed to en/decrypting data with the AES library). This commit begins the migration process by introducing the new structs and functions, backed by the new generic AES implementation. To allow callers to be incrementally converted, aes_encrypt() and aes_decrypt() are temporarily changed into macros that use a _Generic expression to call either the old functions (which take crypto_aes_ctx) or the new functions (which take the new types). Once all callers have been updated, these macros will go away, the old functions will be removed, and the "_new" suffix will be dropped from the new functions. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-3-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: powerpc/aes - Rename struct aes_keyEric Biggers6-21/+22
Rename struct aes_key in aesp8-ppc.h and aes-gcm-p10-glue.c to p8_aes_key and p10_aes_key, respectively. This frees up the name to use in the library API in <crypto/aes.h>. Acked-by: Ard Biesheuvel <ardb@kernel.org> Link: https://lore.kernel.org/r/20260112192035.10427-2-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: mldsa: Add FIPS cryptographic algorithm self-testEric Biggers2-0/+489
Since ML-DSA is FIPS-approved, add the boot-time self-test which is apparently required. Just add a test vector manually for now, borrowed from lib/crypto/tests/mldsa-testvecs.h (where in turn it's borrowed from leancrypto). The SHA-* FIPS test vectors are generated by scripts/crypto/gen-fips-testvecs.py instead, but the common Python libraries don't support ML-DSA yet. Acked-by: Ard Biesheuvel <ardb@kernel.org> Reviewed-by: David Howells <dhowells@redhat.com> Link: https://lore.kernel.org/r/20260107044215.109930-1-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: nh: Restore dependency of arch code on !KMSANEric Biggers1-1/+1
Since the architecture-specific implementations of NH initialize memory in assembly code, they aren't compatible with KMSAN as-is. Fixes: 382de740759a ("lib/crypto: nh: Add NH library") Link: https://lore.kernel.org/r/20260105053652.1708299-1-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: md5: Use rol32() instead of open-coding itRusydi H. Makarim1-1/+1
For the bitwise left rotation in MD5STEP, use rol32() from <linux/bitops.h> instead of open-coding it. Signed-off-by: Rusydi H. Makarim <rusydi.makarim@kriptograf.id> Link: https://lore.kernel.org/r/20251214-rol32_in_md5-v1-1-20f5f11a92b2@kriptograf.id Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12fscrypt: Drop obsolete recommendation to enable optimized NHPoly1305Eric Biggers1-5/+0
CONFIG_CRYPTO_NHPOLY1305_NEON, CONFIG_CRYPTO_NHPOLY1305_SSE2, and CONFIG_CRYPTO_NHPOLY1305_AVX2 no longer exist. The equivalent optimizations are now just enabled automatically when Adiantum support is enabled. Update the fscrypt documentation accordingly. Link: https://lore.kernel.org/r/20251211011846.8179-13-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: testmgr - Remove nhpoly1305 testsEric Biggers2-1378/+0
These are no longer used, since nhpoly1305 support has been removed from the crypto_shash API. Acked-by: Herbert Xu <herbert@gondor.apana.org.au> Link: https://lore.kernel.org/r/20251211011846.8179-12-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: nhpoly1305 - Remove crypto_shash supportEric Biggers4-336/+0
Remove nhpoly1305 support from crypto_shash. It no longer has any user now that crypto/adiantum.c no longer uses it. Acked-by: Herbert Xu <herbert@gondor.apana.org.au> Link: https://lore.kernel.org/r/20251211011846.8179-11-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: adiantum - Drop support for asynchronous xchacha ciphersEric Biggers1-104/+70
This feature isn't useful in practice. Simplify and streamline the code in the synchronous case, i.e. the case that actually matters, instead. For example, by no longer having to support resuming the calculation after an asynchronous return of the xchacha cipher, we can just keep more of the state on the stack instead of in the request context. Link: https://lore.kernel.org/r/20251211011846.8179-10-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: adiantum - Use memcpy_{to,from}_sglist()Eric Biggers1-4/+4
Call the newer, easier-to-read functions memcpy_to_sglist() and memcpy_from_sglist() directly instead of calling scatterwalk_map_and_copy(). No change in behavior. Acked-by: Herbert Xu <herbert@gondor.apana.org.au> Link: https://lore.kernel.org/r/20251211011846.8179-9-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: adiantum - Use scatter_walk API instead of sg_miterEric Biggers1-18/+15
Make adiantum_hash_message() use the scatter_walk API instead of sg_miter. scatter_walk is a bit simpler and also more efficient. For example, unlike sg_miter, scatter_walk doesn't require that the number of scatterlist entries be calculated up-front. Link: https://lore.kernel.org/r/20251211011846.8179-8-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12crypto: adiantum - Convert to use NH libraryEric Biggers3-110/+191
Reimplement the Adiantum message hashing using the nh() library function, combined with some code which directly handles the Poly1305 stage. The latter code is derived from crypto/nhpoly1305.c. This eliminates the dependency on the "nhpoly1305" crypto_shash algorithm, which existed only to fit Adiantum message hashing into the traditional Linux crypto API paradigm. Now that simple, architecture-optimized library functions are a well-established option too, we can switch to this simpler implementation. Note: I've dropped the support for the optional third parameter of the adiantum template, which specified the nhpoly1305 implementation. We could keep accepting some strings in this parameter for backwards compatibility, but I don't think it's being used. I believe only "adiantum(xchacha12,aes)" and "adiantum(xchacha20,aes)" are used. Link: https://lore.kernel.org/r/20251211011846.8179-7-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: x86/nh: Migrate optimized code into libraryEric Biggers9-190/+49
Migrate the x86_64 implementations of NH into lib/crypto/. This makes the nh() function be optimized on x86_64 kernels. Note: this temporarily makes the adiantum template not utilize the x86_64 optimized NH code. This is resolved in a later commit that converts the adiantum template to use nh() instead of "nhpoly1305". Link: https://lore.kernel.org/r/20251211011846.8179-6-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: arm64/nh: Migrate optimized code into libraryEric Biggers7-94/+37
Migrate the arm64 NEON implementation of NH into lib/crypto/. This makes the nh() function be optimized on arm64 kernels. Note: this temporarily makes the adiantum template not utilize the arm64 optimized NH code. This is resolved in a later commit that converts the adiantum template to use nh() instead of "nhpoly1305". Link: https://lore.kernel.org/r/20251211011846.8179-5-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: arm/nh: Migrate optimized code into libraryEric Biggers7-92/+35
Migrate the arm32 NEON implementation of NH into lib/crypto/. This makes the nh() function be optimized on arm32 kernels. Note: this temporarily makes the adiantum template not utilize the arm32 optimized NH code. This is resolved in a later commit that converts the adiantum template to use nh() instead of "nhpoly1305". Link: https://lore.kernel.org/r/20251211011846.8179-4-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: tests: Add KUnit tests for NHEric Biggers5-0/+390
Add some simple KUnit tests for the nh() function. These replace the test coverage which will be lost by removing the nhpoly1305 crypto_shash. Note that the NH code also continues to be tested indirectly as well, via the tests for the "adiantum(xchacha12,aes)" crypto_skcipher. Link: https://lore.kernel.org/r/20251211011846.8179-3-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: nh: Add NH libraryEric Biggers4-0/+152
Add support for the NH "almost-universal hash function" to lib/crypto/, specifically the variant of NH used in Adiantum. This will replace the need for the "nhpoly1305" crypto_shash algorithm. All the implementations of "nhpoly1305" use architecture-optimized code only for the NH stage; they just use the generic C Poly1305 code for the Poly1305 stage. We can achieve the same result in a simpler way using an (architecture-optimized) nh() function combined with code in crypto/adiantum.c that passes the results to the Poly1305 library. This commit begins this cleanup by adding the nh() function. The code is derived from crypto/nhpoly1305.c and include/crypto/nhpoly1305.h. Link: https://lore.kernel.org/r/20251211011846.8179-2-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: tests: Add KUnit tests for ML-DSA verificationEric Biggers4-0/+2335
Add a KUnit test suite for ML-DSA verification, including the following for each ML-DSA parameter set (ML-DSA-44, ML-DSA-65, and ML-DSA-87): - Positive test (valid signature), using vector imported from leancrypto - Various negative tests: - Wrong length for signature, message, or public key - Out-of-range coefficients in z vector - Invalid encoded hint vector - Any bit flipped in signature, message, or public key - Unit test for the internal function use_hint() - A benchmark ML-DSA inputs and outputs are very large. To keep the size of the tests down, use just one valid test vector per parameter set, and generate the negative tests at runtime by mutating the valid test vector. I also considered importing the test vectors from Wycheproof. I've tested that mldsa_verify() indeed passes all of Wycheproof's ML-DSA test vectors that use an empty context string. However, importing these permanently would add over 6 MB of source. That's too much to be a reasonable addition to the Linux kernel tree for one algorithm. It also wouldn't actually provide much better test coverage than this commit. Another potential issue is that Wycheproof uses the Apache license. Similarly, this also differs from the earlier proposal to import a long list of test vectors from leancrypto. I retained only one valid signature for each algorithm, and I also added (runtime-generated) negative tests which were missing. I think this is a better tradeoff. Reviewed-by: David Howells <dhowells@redhat.com> Tested-by: David Howells <dhowells@redhat.com> Link: https://lore.kernel.org/r/20251214181712.29132-3-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
2026-01-12lib/crypto: Add ML-DSA verification supportEric Biggers4-0/+724
Add support for verifying ML-DSA signatures. ML-DSA (Module-Lattice-Based Digital Signature Algorithm) is specified in FIPS 204 and is the standard version of Dilithium. Unlike RSA and elliptic-curve cryptography, ML-DSA is believed to be secure even against adversaries in possession of a large-scale quantum computer. Compared to the earlier patch (https://lore.kernel.org/r/20251117145606.2155773-3-dhowells@redhat.com/) that was based on "leancrypto", this implementation: - Is about 700 lines of source code instead of 4800. - Generates about 4 KB of object code instead of 28 KB. - Uses 9-13 KB of memory to verify a signature instead of 31-84 KB. - Is at least about the same speed, with a microbenchmark showing 3-5% improvements on one x86_64 CPU and -1% to 1% changes on another. When memory is a bottleneck, it's likely much faster. - Correctly implements the RejNTTPoly step of the algorithm. The API just consists of a single function mldsa_verify(), supporting pure ML-DSA with any standard parameter set (ML-DSA-44, ML-DSA-65, or ML-DSA-87) as selected by an enum. That's all that's actually needed. The following four potential features are unneeded and aren't included. However, any that ever become needed could fairly easily be added later, as they only affect how the message representative mu is calculated: - Nonempty context strings - Incremental message hashing - HashML-DSA - External mu Signing support would, of course, be a larger and more complex addition. However, the kernel doesn't, and shouldn't, need ML-DSA signing support. Note that mldsa_verify() allocates memory, so it can sleep and can fail with ENOMEM. Unfortunately we don't have much choice about that, since ML-DSA needs a lot of memory. At least callers have to check for errors anyway, since the signature could be invalid. Note that verification doesn't require constant-time code, and in fact some steps are inherently variable-time. I've used constant-time patterns in some places anyway, but technically they're not needed. Reviewed-by: David Howells <dhowells@redhat.com> Tested-by: David Howells <dhowells@redhat.com> Link: https://lore.kernel.org/r/20251214181712.29132-2-ebiggers@kernel.org Signed-off-by: Eric Biggers <ebiggers@kernel.org>
generated by cgit v1.2.3 (git 2.39.0) at 2026-05-01 22:21:56 +0300