kernel/linux.git/lib/crc/Makefile, branch v7.2-rc1

lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64

2026-05-28T20:14:23+00:00

Tweak the NEON intrinsics crc64 code written for arm64 so it can be built for 32-bit ARM as well. The only workaround needed is to provide alternatives for vmull_p64() and vmull_high_p64() on Clang, which only defines those when building for the AArch64 or arm64ec ISA. Use the same helpers for GCC too, to avoid doubling the size of the test/validation matrix. KUnit benchmark results (Cortex-A53 @ 1 Ghz) Before: # crc64_nvme_benchmark: len=1: 35 MB/s # crc64_nvme_benchmark: len=16: 78 MB/s # crc64_nvme_benchmark: len=64: 87 MB/s # crc64_nvme_benchmark: len=127: 88 MB/s # crc64_nvme_benchmark: len=128: 88 MB/s # crc64_nvme_benchmark: len=200: 89 MB/s # crc64_nvme_benchmark: len=256: 89 MB/s # crc64_nvme_benchmark: len=511: 89 MB/s # crc64_nvme_benchmark: len=512: 89 MB/s # crc64_nvme_benchmark: len=1024: 90 MB/s # crc64_nvme_benchmark: len=3173: 90 MB/s # crc64_nvme_benchmark: len=4096: 90 MB/s # crc64_nvme_benchmark: len=16384: 90 MB/s After: # crc64_nvme_benchmark: len=1: 32 MB/s # crc64_nvme_benchmark: len=16: 76 MB/s # crc64_nvme_benchmark: len=64: 71 MB/s # crc64_nvme_benchmark: len=127: 88 MB/s # crc64_nvme_benchmark: len=128: 618 MB/s # crc64_nvme_benchmark: len=200: 542 MB/s # crc64_nvme_benchmark: len=256: 920 MB/s # crc64_nvme_benchmark: len=511: 836 MB/s # crc64_nvme_benchmark: len=512: 1261 MB/s # crc64_nvme_benchmark: len=1024: 1531 MB/s # crc64_nvme_benchmark: len=3173: 1731 MB/s # crc64_nvme_benchmark: len=4096: 1851 MB/s # crc64_nvme_benchmark: len=16384: 1858 MB/s Don't bother with big-endian, as it doesn't work correctly on Clang, and is barely used these days. Note that ARM disables preemption and softirq processing when using kernel mode SIMD, so take care not to hog the CPU for too long. Signed-off-by: Ard Biesheuvel Link: https://patch.msgid.link/20260422171655.3437334-15-ardb+git@google.com Signed-off-by: Eric Biggers

lib/crc: Turn NEON intrinsics crc64 implementation into common code

2026-05-28T20:14:22+00:00

Move and rename the CRC64 NEON intrinsics implementation source file and rename the function name to reflect that it is NEON code that can be shared. This will be wired up for 32-bit ARM in a subsequent patch. Signed-off-by: Ard Biesheuvel Link: https://patch.msgid.link/20260422171655.3437334-14-ardb+git@google.com Signed-off-by: Eric Biggers

lib/crc: arm64: Use existing macros for kernel-mode FPU cflags

2026-04-02T23:14:53+00:00

Use the existing CC_FPU_CFLAGS and CC_NO_FPU_CFLAGS to pass the appropriate compiler command line options for building kernel mode NEON intrinsics code. This is tidier, and will make it easier to reuse the code for 32-bit ARM. Signed-off-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260330144630.33026-9-ardb@kernel.org Signed-off-by: Eric Biggers

lib/crc: arm64: add NEON accelerated CRC64-NVMe implementation

2026-03-29T20:22:13+00:00

Implement an optimized CRC64 (NVMe) algorithm for ARM64 using NEON Polynomial Multiply Long (PMULL) instructions. The generic shift-and-XOR software implementation is slow, which creates a bottleneck in NVMe and other storage subsystems. The acceleration is implemented using C intrinsics () rather than raw assembly for better readability and maintainability. Key highlights of this implementation: - Uses 4KB chunking inside scoped_ksimd() to avoid preemption latency spikes on large buffers. - Pre-calculates and loads fold constants via vld1q_u64() to minimize register spilling. - Benchmarks show the break-even point against the generic implementation is around 128 bytes. The PMULL path is enabled only for len >= 128. Performance results (kunit crc_benchmark on Cortex-A72): - Generic (len=4096): ~268 MB/s - PMULL (len=4096): ~1556 MB/s (nearly 6x improvement) Signed-off-by: Demian Shulhan Link: https://lore.kernel.org/r/20260329074338.1053550-1-demyansh@gmail.com Signed-off-by: Eric Biggers

lib/crc: x86: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the x86-optimized CRC code from arch/x86/lib/crc* into its new location in lib/crc/x86/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-12-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: sparc: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the sparc-optimized CRC code from arch/sparc/lib/crc* into its new location in lib/crc/sparc/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-11-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: s390: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the s390-optimized CRC code from arch/s390/lib/crc* into its new location in lib/crc/s390/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-10-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: riscv: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the riscv-optimized CRC code from arch/riscv/lib/crc* into its new location in lib/crc/riscv/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-9-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: powerpc: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the powerpc-optimized CRC code from arch/powerpc/lib/crc* into its new location in lib/crc/powerpc/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-8-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: arm64: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the arm64-optimized CRC code from arch/arm64/lib/crc* into its new location in lib/crc/arm64/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-5-ebiggers@kernel.org Signed-off-by: Eric Biggers