kernel/linux.git/tools/accounting, branch v6.18.21

tools/delaytop: improve error handling for missing PSI support

2025-09-14T00:32:56+00:00

Enhanced display logic to conditionally show PSI information only when successfully read, with helpful guidance for users to enable PSI support (psi=1 cmdline parameter). Link: https://lkml.kernel.org/r/20250907001417537vSx6nUsb3ILqI0iQ-WnGp@zte.com.cn Signed-off-by: Fan Yu Reviewed-by: xu xin Signed-off-by: Andrew Morton

tools/delaytop: add interactive mode with keyboard controls

2025-09-14T00:32:56+00:00

The original delaytop only supported static output with limited interaction. Users had to restart the tool with different command-line options to change sorting or display modes, which disrupted continuous monitoring and reduced productivity during performance investigations. Adds real-time interactive controls through keyboard input: 1) Add interactive menu system with visual prompts 2) Support dynamic sorting changes without restarting 3) Enable toggle of memory verbose mode with 'M' key The interactive mode transforms delaytop from a static monitoring tool into a dynamic investigation platform, allowing users to adapt the view in real-time based on observed performance patterns. Link: https://lkml.kernel.org/r/20250907001338580EURha20BxWFmBSrUpS8D1@zte.com.cn Signed-off-by: Fan Yu Reviewed-by: xu xin Signed-off-by: Andrew Morton

tools/delaytop: add memory verbose mode support

2025-09-14T00:32:55+00:00

The original delaytop tool always displayed detailed memory subsystem breakdown, which could be overwhelming for users who only need high-level overview. Add flexible display control allowing users to choose their preferred information granularity. The new flexibility provides: 1) For quick monitoring: use normal mode to reduce visual clutter 2) For deep analysis: use verbose mode to see all memory subsystem details Link: https://lkml.kernel.org/r/202509070012527934u0ySb3teQ4gOYKnocyNO@zte.com.cn Signed-off-by: Fan Yu Reviewed-by: xu xin Signed-off-by: Andrew Morton

tools/delaytop: add flexible sorting by delay field

2025-09-14T00:32:55+00:00

Patch series "tools/delaytop: implement real-time keyboard interaction support", v2. Current Limitations =================== The current delaytop implementation has two main limitations: 1) Static sorting only by CPU delay Forcing users to restart with different parameters to analyze other resource bottlenecks. 2) Memory delay information is always expanded Causing information overload when only high-level memory pressure monitoring is needed. Improvements ============ 1) Implemented dynamic sorting capability - Interactive key 'o' triggers sort mode. - Supports sorting by CPU/IO/Memory/IRQ delays. - Memory subcategories available in verbose mode. * c - CPU delay (default) * i - IO delay * m - Total memory delay * q - IRQ delay * s/r/t/p/w - Memory subcategories (in verbose mode) 2) Added memory display modes - Compact view (default): shows aggregated memory delays. - Verbose view ('M' key): breaks down into memory sub-delays. * SWAP - swapin delays * RCL - freepages reclaim delays * THR - thrashing delays * CMP - compaction delays * WP - write-protect copy delays Practical benefits ================== 1) Dynamic Sorting for Real-Time Bottleneck Detection System administrators can now dynamically change sorting to identify different types of resource bottlenecks without restarting. 2) Enhanced Usability with On-Screen Keybindings More intuitive interactive usage with on-screen keybindings help. Reduced screen clutter when only memory overview is needed. Use Case ======== # ./delaytop System Pressure Information: (avg10/avg60vg300/total) CPU some: 0.0%/ 0.0%/ 0.0%/ 106817(ms) CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) IO full: 0.0%/ 0.0%/ 0.0%/ 2245(ms) IO some: 0.0%/ 0.0%/ 0.0%/ 2791(ms) IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) [o]sort [M]memverbose [q]quit Top 20 processes (sorted by cpu delay): PID TGID COMMAND CPU(ms) IO(ms) IRQ(ms) MEM(ms) ------------------------------------------------------------------------ 110 110 kworker/15:0H-s 27.91 0.00 0.00 0.00 57 57 cpuhp/7 3.18 0.00 0.00 0.00 99 99 cpuhp/14 2.97 0.00 0.00 0.00 51 51 cpuhp/6 0.90 0.00 0.00 0.00 44 44 kworker/4:0H-sy 0.80 0.00 0.00 0.00 76 76 idle_inject/10 0.31 0.00 0.00 0.00 100 100 idle_inject/14 0.30 0.00 0.00 0.00 1309 1309 systemsettings 0.29 0.00 0.00 0.00 60 60 ksoftirqd/7 0.28 0.00 0.00 0.00 45 45 cpuhp/5 0.22 0.00 0.00 0.00 63 63 cpuhp/8 0.20 0.00 0.00 0.00 87 87 cpuhp/12 0.18 0.00 0.00 0.00 93 93 cpuhp/13 0.17 0.00 0.00 0.00 1265 1265 acpid 0.17 0.00 0.00 0.00 1552 1552 sshd 0.17 0.00 0.00 0.00 2584 2584 sddm-helper 0.16 0.00 0.00 0.00 1284 1284 rtkit-daemon 0.15 0.00 0.00 0.00 1326 1326 nde-netfilter 0.14 0.00 0.00 0.00 27 27 cpuhp/2 0.13 0.00 0.00 0.00 631 631 kworker/11:2-rc 0.11 0.00 0.00 0.00 # ./delaytop -M System Pressure Information: (avg10/avg60vg300/total) CPU some: 0.0%/ 0.0%/ 0.0%/ 106827(ms) CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) IO full: 0.0%/ 0.0%/ 0.0%/ 2245(ms) IO some: 0.0%/ 0.0%/ 0.0%/ 2791(ms) IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) [o]sort [M]memverbose [q]quit Top 20 processes (sorted by mem delay): PID TGID COMMAND MEM(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) ------------------------------------------------------------------------------------------ 121732 121732 delaytop 0.01 0.00 0.00 0.00 0.00 0.01 95876 95876 top 0.00 0.00 0.00 0.00 0.00 0.00 121641 121641 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00 121693 121693 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00 121661 121661 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00 1 1 systemd 0.00 0.00 0.00 0.00 0.00 0.00 2 2 kthreadd 0.00 0.00 0.00 0.00 0.00 0.00 3 3 pool_workqueue_ 0.00 0.00 0.00 0.00 0.00 0.00 4 4 kworker/R-rcu_g 0.00 0.00 0.00 0.00 0.00 0.00 5 5 kworker/R-rcu_p 0.00 0.00 0.00 0.00 0.00 0.00 6 6 kworker/R-slub_ 0.00 0.00 0.00 0.00 0.00 0.00 7 7 kworker/R-netns 0.00 0.00 0.00 0.00 0.00 0.00 9 9 kworker/0:0H-sy 0.00 0.00 0.00 0.00 0.00 0.00 11 11 kworker/u32:0-n 0.00 0.00 0.00 0.00 0.00 0.00 12 12 kworker/R-mm_pe 0.00 0.00 0.00 0.00 0.00 0.00 13 13 rcu_tasks_kthre 0.00 0.00 0.00 0.00 0.00 0.00 14 14 rcu_tasks_rude_ 0.00 0.00 0.00 0.00 0.00 0.00 15 15 rcu_tasks_trace 0.00 0.00 0.00 0.00 0.00 0.00 16 16 ksoftirqd/0 0.00 0.00 0.00 0.00 0.00 0.00 17 17 rcu_preempt 0.00 0.00 0.00 0.00 0.00 0.00 When psi is not enabled: # ./delaytop System Pressure Information: (avg10/avg60vg300/total) PSI not found: check if psi=1 enabled in cmdline This patch (of 5): The delaytop tool only supported sorting by CPU delay, which limited its usefulness when users needed to identify bottlenecks in other subsystems. Users had no way to sort processes by IO, IRQ, or other delay types to quickly pinpoint specific performance issues. Add -s/--sort option to allow sorting by different delay types. Users can now quickly identify bottlenecks in specific subsystems by sorting processes by the relevant delay metric. Link: https://lkml.kernel.org/r/20250907001101305vrTGnXaRNvtmsGkp-Ljk_@zte.com.cn Link: https://lkml.kernel.org/r/20250907001205573L3XpsQMIQnLgDqiiKYd3H@zte.com.cn Signed-off-by: Fan Yu Reviewed-by: xu xin Cc: Jonathan Corbet Cc: Wang Yaxin Cc: Yang Yang Signed-off-by: Andrew Morton

tools/getdelays: add backward compatibility for taskstats version

2025-08-02T19:01:41+00:00

Add version checks to print_delayacct() to handle differences in struct taskstats across kernel versions. Field availability depends on taskstats version (t->version), corresponding to TASKSTATS_VERSION in kernel headers (see include/uapi/linux/taskstats.h). Version feature mapping: - version >= 11 - supports COMPACT statistics - version >= 13 - supports WPCOPY statistics - version >= 14 - supports IRQ statistics - version >= 16 - supports *_max and *_min delay statistics This ensures the tool works correctly with both older and newer kernel versions by conditionally printing fields based on the reported version. eg.1 bash# grep -r "#define TASKSTATS_VERSION" /usr/include/linux/taskstats.h "#define TASKSTATS_VERSION 10" bash# ./getdelays -d -p 1 CPU count real total virtual total delay total delay average 7481 3786181709 3807098291 36393725 0.005ms IO count delay total delay average 369 1116046035 3.025ms SWAP count delay total delay average 0 0 0.000ms RECLAIM count delay total delay average 0 0 0.000ms THRASHING count delay total delay average 0 0 0.000ms eg.2 bash# grep -r "#define TASKSTATS_VERSION" /usr/include/linux/taskstats.h "#define TASKSTATS_VERSION 14" bash# ./getdelays -d -p 1 CPU count real total virtual total delay total delay average 68862 163474790046 174584722267 19962496806 0.290ms IO count delay total delay average 0 0 0.000ms SWAP count delay total delay average 0 0 0.000ms RECLAIM count delay total delay average 0 0 0.000ms THRASHING count delay total delay average 0 0 0.000ms COMPACT count delay total delay average 0 0 0.000ms WPCOPY count delay total delay average 0 0 0.000ms IRQ count delay total delay average 0 0 0.000ms Link: https://lkml.kernel.org/r/20250731225326549CttJ7g9NfjTlaqBwl015T@zte.com.cn Signed-off-by: Fan Yu Cc: Fan Yu Cc: Jonathan Corbet Cc: Wang Yaxin Cc: xu xin Cc: Yang Yang Signed-off-by: Andrew Morton

delaytop: enhance error logging and add PSI feature description

2025-08-02T19:01:41+00:00

This patch improves error diagnostics and documentation for delaytop: 1) Enhanced error logging: - Added explicit error messages in critical failure paths - Implemented BOOL_FPRINT macro for robust output handling 2) PSI feature documentation: - Updated header comment to reflect PSI monitoring capability - Improved output formatting for PSI information System Pressure Information: (avg10/avg60/avg300/total) CPU some: 0.0%/ 0.0%/ 0.0%/ 345(ms) CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) IO full: 0.0%/ 0.0%/ 0.0%/ 65(ms) IO some: 0.0%/ 0.0%/ 0.0%/ 79(ms) IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Link: https://lkml.kernel.org/r/202507281628341752gMXCMN7S-Vz_LHYHum9r@zte.com.cn Signed-off-by: Fan Yu Signed-off-by: Wang Yaxin Acked-by: Yang Yang Cc: Fan Yu Cc: Jonathan Corbet Cc: xu xin Signed-off-by: Andrew Morton

delaytop: add psi info to show system delay

2025-07-20T02:08:29+00:00

Support showing whole delay of system by reading PSI, just like the first few lines of information output by the top command. the output of delaytop includes both system-wide delay and delay of individual tasks, providing a more comprehensive reflection of system latency status. Use case ======== bash# ./delaytop System Pressure Information: (avg10/avg60/avg300/total) CPU: full: 0.0%/ 0.0%/ 0.0%/0 some: 0.1%/ 0.0%/ 0.0%/14216596 Memory: full: 0.0%/ 0.0%/ 0.0%/34010659 some: 0.0%/ 0.0%/ 0.0%/35406492 IO: full: 0.1%/ 0.0%/ 0.0%/51029453 some: 0.1%/ 0.0%/ 0.0%/55330465 IRQ: full: 0.0%/ 0.0%/ 0.0%/0 Top 20 processes (sorted by CPU delay): PID TGID COMMAND CPU(ms) IO(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) IRQ(ms) --------------------------------------------------------------------------------------------- 32 32 kworker/2:0H-sy 23.65 0.00 0.00 0.00 0.00 0.00 0.00 0.00 497 497 kworker/R-scsi_ 1.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 495 495 kworker/R-scsi_ 1.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 494 494 scsi_eh_0 1.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 485 485 kworker/R-ata_s 0.90 0.00 0.00 0.00 0.00 0.00 0.00 0.00 574 574 kworker/R-kdmfl 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 34 34 idle_inject/3 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1123 1123 nde-netfilter 0.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 60 60 ksoftirqd/7 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 114 114 kworker/0:2-cgr 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 496 496 scsi_eh_1 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 51 51 cpuhp/6 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1667 1667 atd 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 45 45 cpuhp/5 0.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1102 1102 nde-backupservi 0.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1098 1098 systemsettings 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1100 1100 audit-monitor 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 53 53 migration/6 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1482 1482 sshd 0.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 39 39 cpuhp/4 0.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Link: https://lkml.kernel.org/r/20250710135451340_5pOgpIFi0M5AE7H44W1D@zte.com.cn Co-developed-by: Fan Yu Signed-off-by: Fan Yu Signed-off-by: Wang Yaxin Signed-off-by: Jiang Kun Cc: Balbir Singh Cc: David Hildenbrand Cc: Peilin He Cc: Qiang Tu Cc: wangyong Cc: xu xin Cc: Yang Yang Cc: Yunkai Zhang Signed-off-by: Andrew Morton

tools/accounting/delaytop: add delaytop to record top-n task delay

2025-07-10T05:57:56+00:00

Problem ======= The "getdelays" can only display the latency of a single task by specifying a PID, but it has the following limitations: 1. single-task perspective: only supports querying the latency (CPU, I/O, memory, etc.) of an individual task via PID and cannot provide a global analysis of high-latency processes across the system. 2. lack of High-Latency process awareness: when the overall system latency is high (e.g., a spike in CPU latency), there is no way to quickly identify the top N processes contributing to the highest latency. 3. poor interactivity: It lacks dynamic sorting and refresh capabilities (similar to top), making it difficult to monitor latency changes in real time. Solution ======== To address these limitations, we introduce the "delaytop" with the following capabilities: 1. system view: monitors latency metrics (CPU, I/O, memory, IRQ, etc.) for all system processes 2. supports field-based sorting (e.g., default sort by CPU latency in descending order) 3. dynamic interactive interface: focus on specific processes with --pid; limit displayed entries with --processes 20; control monitoring duration with --iterations; Use case ======== bash# ./delaytop Top 20 processes (sorted by CPU delay): PID TGID COMMAND CPU(ms) IO(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) IRQ(ms) --------------------------------------------------------------------------------------------- 26 26 kworker/1:0H 5.55 0.00 0.00 0.00 0.00 0.00 0.00 0.00 32 32 kworker/2:0H-kb 2.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 38 38 kworker/3:0H-ev 2.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00 84 84 kworker/R-vfio- 1.62 0.00 0.00 0.00 0.00 0.00 0.00 0.00 24 24 ksoftirqd/1 1.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19 19 idle_inject/0 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16 16 rcu_exp_par_gp_ 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11 11 kworker/0:1 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22 22 idle_inject/1 0.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3 3 pool_workqueue_ 0.74 0.00 0.00 0.00 0.00 0.00 0.00 0.00 81 81 scsi_eh_1 0.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00 30 30 ksoftirqd/2 0.42 0.00 0.00 0.00 0.00 0.00 0.00 0.00 36 36 ksoftirqd/3 0.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9 9 kworker/0:0-eve 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8 8 kworker/R-netns 0.34 0.00 0.00 0.00 0.00 0.00 0.00 0.00 76 76 kworker/1:1-pm 0.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21 21 cpuhp/1 0.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4 4 kworker/R-rcu_g 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12 12 kworker/u16:0-i 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1 1 init 0.18 0.00 0.00 0.00 0.00 0.00 0.08 0.00 Link: https://lkml.kernel.org/r/20250619211843633h05gWrBDMFkEH6xAVm_5y@zte.com.cn Co-developed-by: Fan Yu Signed-off-by: Fan Yu Signed-off-by: Yaxin Wang Cc: Balbir Singh Cc: David Hildenbrand Cc: Peilin He Cc: Qiang Tu Cc: wangyong Cc: xu xin Cc: Yang Yang Cc: ye xingchen Cc: Yunkai Zhang Signed-off-by: Andrew Morton

getdelays: fix error format characters

2025-02-18T06:40:02+00:00

getdelays had a compilation issue because the format string was not updated when the "delay min" was added. For example, after adding the "delay min" in printf, there were 7 strings but only 6 "%s" format specifiers. Similarly, after adding the 't->cpu_delay_total', there were 7 variables but only 6 format characters specifiers, causing compilation issues as follows. This commit fixes these issues to ensure that getdelays compiles correctly. root@xx:~/linux-next/tools/accounting$ make getdelays.c:199:9: warning: format `%llu' expects argument of type `long long unsigned int', but argument 8 has type `char *' [-Wformat=] 199 | printf("\n\nCPU %15s%15s%15s%15s%15s%15s\n" | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ..... 216 | "delay total", "delay average", "delay max", "delay min", | ~~~~~~~~~~~ | | | char * getdelays.c:200:21: note: format string is defined here 200 | " %15llu%15llu%15llu%15llu%15.3fms%13.6fms\n" | ~~~~~^ | | | long long unsigned int | %15s getdelays.c:199:9: warning: format `%f' expects argument of type `double', but argument 12 has type `long long unsigned int' [-Wformat=] 199 | printf("\n\nCPU %15s%15s%15s%15s%15s%15s\n" | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ..... 220 | (unsigned long long)t->cpu_delay_total, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | | | long long unsigned int ..... Link: https://lkml.kernel.org/r/20250208144400544RduNRhwIpT3m2JyRBqskZ@zte.com.cn Fixes: f65c64f311ee ("delayacct: add delay min to record delay peak") Reviewed-by: xu xin Signed-off-by: Wang Yaxin Signed-off-by: Kun Jiang Cc: Balbir Singh Cc: David Hildenbrand Cc: Fan Yu Cc: Peilin He Cc: Qiang Tu Cc: wangyong Cc: ye xingchen Cc: Yunkai Zhang Signed-off-by: Andrew Morton

delayacct: add delay min to record delay peak

2025-01-13T04:21:16+00:00

Delay accounting can now calculate the average delay of processes, detect the overall system load, and also record the 'delay max' to identify potential abnormal delays. However, 'delay min' can help us identify another useful delay peak. By comparing the difference between 'delay max' and 'delay min', we can understand the optimization space for latency, providing a reference for the optimization of latency performance. Use case ========= bash-4.4# ./getdelays -d -t 242 print delayacct stats ON TGID 242 CPU count real total virtual total delay total delay average delay max delay min 39 156000000 156576579 2111069 0.054ms 0.212296ms 0.031307ms IO count delay total delay average delay max delay min 0 0 0.000ms 0.000000ms 0.000000ms SWAP count delay total delay average delay max delay min 0 0 0.000ms 0.000000ms 0.000000ms RECLAIM count delay total delay average delay max delay min 0 0 0.000ms 0.000000ms 0.000000ms THRASHING count delay total delay average delay max delay min 0 0 0.000ms 0.000000ms 0.000000ms COMPACT count delay total delay average delay max delay min 0 0 0.000ms 0.000000ms 0.000000ms WPCOPY count delay total delay average delay max delay min 156 11215873 0.072ms 0.207403ms 0.033913ms IRQ count delay total delay average delay max delay min 0 0 0.000ms 0.000000ms 0.000000ms Link: https://lkml.kernel.org/r/20241220173105906EOdsPhzjMLYNJJBqgz1ga@zte.com.cn Co-developed-by: Wang Yong Signed-off-by: Wang Yong Co-developed-by: xu xin Signed-off-by: xu xin Signed-off-by: Wang Yaxin Co-developed-by: Kun Jiang Signed-off-by: Kun Jiang Cc: Balbir Singh Cc: David Hildenbrand Cc: Fan Yu Cc: Peilin He Cc: tuqiang Cc: ye xingchen Cc: Yunkai Zhang Signed-off-by: Andrew Morton