kernel/linux.git/lib/crc/Makefile, branch master

lib/crc: arm64: Use existing macros for kernel-mode FPU cflags

2026-04-02T23:14:53+00:00

Use the existing CC_FPU_CFLAGS and CC_NO_FPU_CFLAGS to pass the appropriate compiler command line options for building kernel mode NEON intrinsics code. This is tidier, and will make it easier to reuse the code for 32-bit ARM. Signed-off-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260330144630.33026-9-ardb@kernel.org Signed-off-by: Eric Biggers

lib/crc: arm64: add NEON accelerated CRC64-NVMe implementation

2026-03-29T20:22:13+00:00

Implement an optimized CRC64 (NVMe) algorithm for ARM64 using NEON Polynomial Multiply Long (PMULL) instructions. The generic shift-and-XOR software implementation is slow, which creates a bottleneck in NVMe and other storage subsystems. The acceleration is implemented using C intrinsics () rather than raw assembly for better readability and maintainability. Key highlights of this implementation: - Uses 4KB chunking inside scoped_ksimd() to avoid preemption latency spikes on large buffers. - Pre-calculates and loads fold constants via vld1q_u64() to minimize register spilling. - Benchmarks show the break-even point against the generic implementation is around 128 bytes. The PMULL path is enabled only for len >= 128. Performance results (kunit crc_benchmark on Cortex-A72): - Generic (len=4096): ~268 MB/s - PMULL (len=4096): ~1556 MB/s (nearly 6x improvement) Signed-off-by: Demian Shulhan Link: https://lore.kernel.org/r/20260329074338.1053550-1-demyansh@gmail.com Signed-off-by: Eric Biggers

lib/crc: x86: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the x86-optimized CRC code from arch/x86/lib/crc* into its new location in lib/crc/x86/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-12-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: sparc: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the sparc-optimized CRC code from arch/sparc/lib/crc* into its new location in lib/crc/sparc/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-11-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: s390: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the s390-optimized CRC code from arch/s390/lib/crc* into its new location in lib/crc/s390/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-10-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: riscv: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the riscv-optimized CRC code from arch/riscv/lib/crc* into its new location in lib/crc/riscv/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-9-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: powerpc: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the powerpc-optimized CRC code from arch/powerpc/lib/crc* into its new location in lib/crc/powerpc/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-8-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: arm64: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the arm64-optimized CRC code from arch/arm64/lib/crc* into its new location in lib/crc/arm64/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-5-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: arm: Migrate optimized CRC code into lib/crc/

2025-06-30T16:31:57+00:00

Move the arm-optimized CRC code from arch/arm/lib/crc* into its new location in lib/crc/arm/, and wire it up in the new way. This new way of organizing the CRC code eliminates the need to artificially split the code for each CRC variant into separate arch and generic modules, enabling better inlining and dead code elimination. For more details, see "lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/". Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-4-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crc: Prepare for arch-optimized code in subdirs of lib/crc/

2025-06-30T16:31:57+00:00

Rework how lib/crc/ supports arch-optimized code. First, instead of the arch-optimized CRC code being in arch/$(SRCARCH)/lib/, it will now be in lib/crc/$(SRCARCH)/. Second, the API functions (e.g. crc32c()), arch-optimized functions (e.g. crc32c_arch()), and generic functions (e.g. crc32c_base()) will now be part of a single module for each CRC type, allowing better inlining and dead code elimination. The second change is made possible by the first. As an example, consider CONFIG_CRC32=m on x86. We'll now have just crc32.ko instead of both crc32-x86.ko and crc32.ko. The two modules were already coupled together and always both got loaded together via direct symbol dependency, so the separation provided no benefit. Note: later I'd like to apply the same design to lib/crypto/ too, where often the API functions are out-of-line so this will work even better. In those cases, for each algorithm we currently have 3 modules all coupled together, e.g. libsha256.ko, libsha256-generic.ko, and sha256-x86.ko. We should have just one, inline things properly, and rely on the compiler's dead code elimination to decide the inclusion of the generic code instead of manually setting it via kconfig. Having arch-specific code outside arch/ was somewhat controversial when Zinc proposed it back in 2018. But I don't think the concerns are warranted. It's better from a technical perspective, as it enables the improvements mentioned above. This model is already successfully used in other places in the kernel such as lib/raid6/. The community of each architecture still remains free to work on the code, even if it's not in arch/. At the time there was also a desire to put the library code in the same files as the old-school crypto API, but that was a mistake; now that the library is separate, that's no longer a constraint either. Reviewed-by: "Martin K. Petersen" Acked-by: Ingo Molnar Acked-by: "Jason A. Donenfeld" Link: https://lore.kernel.org/r/20250607200454.73587-3-ebiggers@kernel.org Link: https://lore.kernel.org/r/20250612054514.142728-1-ebiggers@kernel.org Link: https://lore.kernel.org/r/20250621012221.4351-1-ebiggers@kernel.org Signed-off-by: Eric Biggers