kernel/linux.git/tools/accounting, branch master

tools/accounting: handle truncated taskstats netlink messages

2026-04-15T09:15:02+00:00

procacct and getdelays use a fixed receive buffer for taskstats generic netlink messages. A multi-threaded process exit can emit a single PID+TGID notification large enough to exceed that buffer on newer kernels. Switch to recvmsg() so MSG_TRUNC is detected explicitly, increase the message buffer size, and report truncated datagrams clearly instead of misparsing them as fatal netlink errors. Also print the taskstats version in debug output to make version mismatches easier to diagnose while inspecting taskstats traffic. Link: https://lkml.kernel.org/r/520308bb4cbbaf8dc2c7296b5f60f11e12fb30a5.1774810498.git.cyyzero16@gmail.com Signed-off-by: Yiyang Chen Cc: Balbir Singh Cc: Dr. Thomas Orgis Cc: Fan Yu Cc: Wang Yaxin Cc: Signed-off-by: Andrew Morton

tools/getdelays: use the static UAPI headers from tools/include/uapi

2026-03-28T04:19:43+00:00

The include directory ../../usr/include is only present if an in-tree kernel build with CONFIG_HEADERS_INSTALL was done before. Otherwise the system UAPI headers are used, which most likely are not the most recent ones. To make sure to always have access to up-to-date UAPI headers, use the static copy in tools/include/uapi. Link: https://lkml.kernel.org/r/20260307-accounting-taskstats-h-v1-2-0b75915c6ce5@weissschuh.net Reported-by: kernel test robot Closes: https://lore.kernel.org/r/202603062103.Z5fecwZD-lkp@intel.com/ Signed-off-by: Thomas Weißschuh Cc: Arnd Bergmann Cc: Balbir Singh Cc: Jiang Kun Cc: Wang Yaxin Cc: xu xin Cc: Yang Yang Signed-off-by: Andrew Morton

delayacct: fix build regression on accounting tool

2026-02-12T23:45:56+00:00

The accounting tool was modified for the original ABI using a custom 'timespec64' type in linux/taskstats.h, which I changed to use the regular __kernel_timespec type, causing a build failure: getdelays.c:202:45: warning: 'struct timespec64' declared inside parameter list will not be visible outside of this definition or declaration 202 | static const char *format_timespec64(struct timespec64 *ts) | ^~~~~~~~~~ Change the tool to match the updated header. Link: https://lkml.kernel.org/r/20260210103427.2984963-1-arnd@kernel.org Fixes: 503efe850c74 ("delayacct: add timestamp of delay max") Fixes: f06e31eef4c1 ("delayacct: fix uapi timespec64 definition") Signed-off-by: Arnd Bergmann Reported-by: kernel test robot Closes: https://lore.kernel.org/r/202602091611.lxgINqXp-lkp@intel.com/ Signed-off-by: Andrew Morton

delayacct: add timestamp of delay max

2026-02-01T00:16:06+00:00

Problem ======= Commit 658eb5ab916d ("delayacct: add delay max to record delay peak") introduced the delay max for getdelays, which records abnormal latency peaks and helps us understand the magnitude of such delays. However, the peak latency value alone is insufficient for effective root cause analysis. Without the precise timestamp of when the peak occurred, we still lack the critical context needed to correlate it with other system events. Solution ======== To address this, we need to additionally record a precise timestamp when the maximum latency occurs. By correlating this timestamp with system logs and monitoring metrics, we can identify processes with abnormal resource usage at the same moment, which can help us to pinpoint root causes. Use Case ======== bash-4.4# ./getdelays -d -t 227 print delayacct stats ON TGID 227 CPU count real total virtual total delay total delay average delay max delay min delay max timestamp 46 188000000 192348334 4098012 0.089ms 0.429260ms 0.051205ms 2026-01-15T15:06:58 IO count delay total delay average delay max delay min delay max timestamp 0 0 0.000ms 0.000000ms 0.000000ms N/A SWAP count delay total delay average delay max delay min delay max timestamp 0 0 0.000ms 0.000000ms 0.000000ms N/A RECLAIM count delay total delay average delay max delay min delay max timestamp 0 0 0.000ms 0.000000ms 0.000000ms N/A THRAS HING count delay total delay average delay max delay min delay max timestamp 0 0 0.000ms 0.000000ms 0.000000ms N/A COMPACT count delay total delay average delay max delay min delay max timestamp 0 0 0.000ms 0.000000ms 0.000000ms N/A WPCOPY count delay total delay average delay max delay min delay max timestamp 182 19413338 0.107ms 0.547353ms 0.022462ms 2026-01-15T15:05:24 IRQ count delay total delay average delay max delay min delay max timestamp 0 0 0.000ms 0.000000ms 0.000000ms N/A Link: https://lkml.kernel.org/r/20260119100241520gWubW8-5QfhSf9gjqcc_E@zte.com.cn Signed-off-by: Wang Yaxin Cc: Fan Yu Cc: Jonathan Corbet Cc: xu xin Cc: Yang Yang Signed-off-by: Andrew Morton

tools/delaytop: improve error handling for missing PSI support

2025-09-14T00:32:56+00:00

Enhanced display logic to conditionally show PSI information only when successfully read, with helpful guidance for users to enable PSI support (psi=1 cmdline parameter). Link: https://lkml.kernel.org/r/20250907001417537vSx6nUsb3ILqI0iQ-WnGp@zte.com.cn Signed-off-by: Fan Yu Reviewed-by: xu xin Signed-off-by: Andrew Morton

tools/delaytop: add interactive mode with keyboard controls

2025-09-14T00:32:56+00:00

The original delaytop only supported static output with limited interaction. Users had to restart the tool with different command-line options to change sorting or display modes, which disrupted continuous monitoring and reduced productivity during performance investigations. Adds real-time interactive controls through keyboard input: 1) Add interactive menu system with visual prompts 2) Support dynamic sorting changes without restarting 3) Enable toggle of memory verbose mode with 'M' key The interactive mode transforms delaytop from a static monitoring tool into a dynamic investigation platform, allowing users to adapt the view in real-time based on observed performance patterns. Link: https://lkml.kernel.org/r/20250907001338580EURha20BxWFmBSrUpS8D1@zte.com.cn Signed-off-by: Fan Yu Reviewed-by: xu xin Signed-off-by: Andrew Morton

tools/delaytop: add memory verbose mode support

2025-09-14T00:32:55+00:00

The original delaytop tool always displayed detailed memory subsystem breakdown, which could be overwhelming for users who only need high-level overview. Add flexible display control allowing users to choose their preferred information granularity. The new flexibility provides: 1) For quick monitoring: use normal mode to reduce visual clutter 2) For deep analysis: use verbose mode to see all memory subsystem details Link: https://lkml.kernel.org/r/202509070012527934u0ySb3teQ4gOYKnocyNO@zte.com.cn Signed-off-by: Fan Yu Reviewed-by: xu xin Signed-off-by: Andrew Morton

tools/delaytop: add flexible sorting by delay field

2025-09-14T00:32:55+00:00

Patch series "tools/delaytop: implement real-time keyboard interaction support", v2. Current Limitations =================== The current delaytop implementation has two main limitations: 1) Static sorting only by CPU delay Forcing users to restart with different parameters to analyze other resource bottlenecks. 2) Memory delay information is always expanded Causing information overload when only high-level memory pressure monitoring is needed. Improvements ============ 1) Implemented dynamic sorting capability - Interactive key 'o' triggers sort mode. - Supports sorting by CPU/IO/Memory/IRQ delays. - Memory subcategories available in verbose mode. * c - CPU delay (default) * i - IO delay * m - Total memory delay * q - IRQ delay * s/r/t/p/w - Memory subcategories (in verbose mode) 2) Added memory display modes - Compact view (default): shows aggregated memory delays. - Verbose view ('M' key): breaks down into memory sub-delays. * SWAP - swapin delays * RCL - freepages reclaim delays * THR - thrashing delays * CMP - compaction delays * WP - write-protect copy delays Practical benefits ================== 1) Dynamic Sorting for Real-Time Bottleneck Detection System administrators can now dynamically change sorting to identify different types of resource bottlenecks without restarting. 2) Enhanced Usability with On-Screen Keybindings More intuitive interactive usage with on-screen keybindings help. Reduced screen clutter when only memory overview is needed. Use Case ======== # ./delaytop System Pressure Information: (avg10/avg60vg300/total) CPU some: 0.0%/ 0.0%/ 0.0%/ 106817(ms) CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) IO full: 0.0%/ 0.0%/ 0.0%/ 2245(ms) IO some: 0.0%/ 0.0%/ 0.0%/ 2791(ms) IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) [o]sort [M]memverbose [q]quit Top 20 processes (sorted by cpu delay): PID TGID COMMAND CPU(ms) IO(ms) IRQ(ms) MEM(ms) ------------------------------------------------------------------------ 110 110 kworker/15:0H-s 27.91 0.00 0.00 0.00 57 57 cpuhp/7 3.18 0.00 0.00 0.00 99 99 cpuhp/14 2.97 0.00 0.00 0.00 51 51 cpuhp/6 0.90 0.00 0.00 0.00 44 44 kworker/4:0H-sy 0.80 0.00 0.00 0.00 76 76 idle_inject/10 0.31 0.00 0.00 0.00 100 100 idle_inject/14 0.30 0.00 0.00 0.00 1309 1309 systemsettings 0.29 0.00 0.00 0.00 60 60 ksoftirqd/7 0.28 0.00 0.00 0.00 45 45 cpuhp/5 0.22 0.00 0.00 0.00 63 63 cpuhp/8 0.20 0.00 0.00 0.00 87 87 cpuhp/12 0.18 0.00 0.00 0.00 93 93 cpuhp/13 0.17 0.00 0.00 0.00 1265 1265 acpid 0.17 0.00 0.00 0.00 1552 1552 sshd 0.17 0.00 0.00 0.00 2584 2584 sddm-helper 0.16 0.00 0.00 0.00 1284 1284 rtkit-daemon 0.15 0.00 0.00 0.00 1326 1326 nde-netfilter 0.14 0.00 0.00 0.00 27 27 cpuhp/2 0.13 0.00 0.00 0.00 631 631 kworker/11:2-rc 0.11 0.00 0.00 0.00 # ./delaytop -M System Pressure Information: (avg10/avg60vg300/total) CPU some: 0.0%/ 0.0%/ 0.0%/ 106827(ms) CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) IO full: 0.0%/ 0.0%/ 0.0%/ 2245(ms) IO some: 0.0%/ 0.0%/ 0.0%/ 2791(ms) IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) [o]sort [M]memverbose [q]quit Top 20 processes (sorted by mem delay): PID TGID COMMAND MEM(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) ------------------------------------------------------------------------------------------ 121732 121732 delaytop 0.01 0.00 0.00 0.00 0.00 0.01 95876 95876 top 0.00 0.00 0.00 0.00 0.00 0.00 121641 121641 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00 121693 121693 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00 121661 121661 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00 1 1 systemd 0.00 0.00 0.00 0.00 0.00 0.00 2 2 kthreadd 0.00 0.00 0.00 0.00 0.00 0.00 3 3 pool_workqueue_ 0.00 0.00 0.00 0.00 0.00 0.00 4 4 kworker/R-rcu_g 0.00 0.00 0.00 0.00 0.00 0.00 5 5 kworker/R-rcu_p 0.00 0.00 0.00 0.00 0.00 0.00 6 6 kworker/R-slub_ 0.00 0.00 0.00 0.00 0.00 0.00 7 7 kworker/R-netns 0.00 0.00 0.00 0.00 0.00 0.00 9 9 kworker/0:0H-sy 0.00 0.00 0.00 0.00 0.00 0.00 11 11 kworker/u32:0-n 0.00 0.00 0.00 0.00 0.00 0.00 12 12 kworker/R-mm_pe 0.00 0.00 0.00 0.00 0.00 0.00 13 13 rcu_tasks_kthre 0.00 0.00 0.00 0.00 0.00 0.00 14 14 rcu_tasks_rude_ 0.00 0.00 0.00 0.00 0.00 0.00 15 15 rcu_tasks_trace 0.00 0.00 0.00 0.00 0.00 0.00 16 16 ksoftirqd/0 0.00 0.00 0.00 0.00 0.00 0.00 17 17 rcu_preempt 0.00 0.00 0.00 0.00 0.00 0.00 When psi is not enabled: # ./delaytop System Pressure Information: (avg10/avg60vg300/total) PSI not found: check if psi=1 enabled in cmdline This patch (of 5): The delaytop tool only supported sorting by CPU delay, which limited its usefulness when users needed to identify bottlenecks in other subsystems. Users had no way to sort processes by IO, IRQ, or other delay types to quickly pinpoint specific performance issues. Add -s/--sort option to allow sorting by different delay types. Users can now quickly identify bottlenecks in specific subsystems by sorting processes by the relevant delay metric. Link: https://lkml.kernel.org/r/20250907001101305vrTGnXaRNvtmsGkp-Ljk_@zte.com.cn Link: https://lkml.kernel.org/r/20250907001205573L3XpsQMIQnLgDqiiKYd3H@zte.com.cn Signed-off-by: Fan Yu Reviewed-by: xu xin Cc: Jonathan Corbet Cc: Wang Yaxin Cc: Yang Yang Signed-off-by: Andrew Morton

tools/getdelays: add backward compatibility for taskstats version

2025-08-02T19:01:41+00:00

Add version checks to print_delayacct() to handle differences in struct taskstats across kernel versions. Field availability depends on taskstats version (t->version), corresponding to TASKSTATS_VERSION in kernel headers (see include/uapi/linux/taskstats.h). Version feature mapping: - version >= 11 - supports COMPACT statistics - version >= 13 - supports WPCOPY statistics - version >= 14 - supports IRQ statistics - version >= 16 - supports *_max and *_min delay statistics This ensures the tool works correctly with both older and newer kernel versions by conditionally printing fields based on the reported version. eg.1 bash# grep -r "#define TASKSTATS_VERSION" /usr/include/linux/taskstats.h "#define TASKSTATS_VERSION 10" bash# ./getdelays -d -p 1 CPU count real total virtual total delay total delay average 7481 3786181709 3807098291 36393725 0.005ms IO count delay total delay average 369 1116046035 3.025ms SWAP count delay total delay average 0 0 0.000ms RECLAIM count delay total delay average 0 0 0.000ms THRASHING count delay total delay average 0 0 0.000ms eg.2 bash# grep -r "#define TASKSTATS_VERSION" /usr/include/linux/taskstats.h "#define TASKSTATS_VERSION 14" bash# ./getdelays -d -p 1 CPU count real total virtual total delay total delay average 68862 163474790046 174584722267 19962496806 0.290ms IO count delay total delay average 0 0 0.000ms SWAP count delay total delay average 0 0 0.000ms RECLAIM count delay total delay average 0 0 0.000ms THRASHING count delay total delay average 0 0 0.000ms COMPACT count delay total delay average 0 0 0.000ms WPCOPY count delay total delay average 0 0 0.000ms IRQ count delay total delay average 0 0 0.000ms Link: https://lkml.kernel.org/r/20250731225326549CttJ7g9NfjTlaqBwl015T@zte.com.cn Signed-off-by: Fan Yu Cc: Fan Yu Cc: Jonathan Corbet Cc: Wang Yaxin Cc: xu xin Cc: Yang Yang Signed-off-by: Andrew Morton

delaytop: enhance error logging and add PSI feature description

2025-08-02T19:01:41+00:00

This patch improves error diagnostics and documentation for delaytop: 1) Enhanced error logging: - Added explicit error messages in critical failure paths - Implemented BOOL_FPRINT macro for robust output handling 2) PSI feature documentation: - Updated header comment to reflect PSI monitoring capability - Improved output formatting for PSI information System Pressure Information: (avg10/avg60/avg300/total) CPU some: 0.0%/ 0.0%/ 0.0%/ 345(ms) CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) IO full: 0.0%/ 0.0%/ 0.0%/ 65(ms) IO some: 0.0%/ 0.0%/ 0.0%/ 79(ms) IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Link: https://lkml.kernel.org/r/202507281628341752gMXCMN7S-Vz_LHYHum9r@zte.com.cn Signed-off-by: Fan Yu Signed-off-by: Wang Yaxin Acked-by: Yang Yang Cc: Fan Yu Cc: Jonathan Corbet Cc: xu xin Signed-off-by: Andrew Morton