summaryrefslogtreecommitdiff
path: root/tools/tracing/rtla/src
AgeCommit message (Collapse)AuthorFilesLines
2025-03-07rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threadsTomas Glozar1-6/+9
commit 217f0b1e990e30a1f06f6d531fdb4530f4788d48 upstream. When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/options. This option is not re-enabled in a subsequent run with kernel-space threads, leading to rtla collecting no results if the previous run exited abnormally: $ rtla timerlat top -u ^\Quit (core dumped) $ rtla timerlat top -k -d 1s Timer Latency 0 00:00:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max The issue persists until OSNOISE_WORKLOAD is set manually by running: $ echo OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options Set OSNOISE_WORKLOAD when running rtla with kernel-space threads if available to fix the issue. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250107144823.239782-4-tglozar@redhat.com Fixes: cdca4f4e5e8e ("rtla/timerlat_top: Add timerlat user-space support") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> [ params->kernel_workload does not exist in 6.6, use !params->user_top ] Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-03-07rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threadsTomas Glozar1-6/+9
commit d8d866171a414ed88bd0d720864095fd75461134 upstream. When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/options. This option is not re-enabled in a subsequent run with kernel-space threads, leading to rtla collecting no results if the previous run exited abnormally: $ rtla timerlat hist -u ^\Quit (core dumped) $ rtla timerlat hist -k -d 1s Index over: count: min: avg: max: ALL: IRQ Thr Usr count: 0 0 0 min: - - - avg: - - - max: - - - The issue persists until OSNOISE_WORKLOAD is set manually by running: $ echo OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options Set OSNOISE_WORKLOAD when running rtla with kernel-space threads if available to fix the issue. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250107144823.239782-3-tglozar@redhat.com Fixes: ed774f7481fa ("rtla/timerlat_hist: Add timerlat user-space support") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> [ params->kernel_workload does not exist in 6.6, use !params->user_hist ] Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-03-07Revert "rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threads"Tomas Glozar1-9/+6
This reverts commit 83b74901bdc9b58739193b8ee6989254391b6ba7. The commit breaks rtla build, since params->kernel_workload is not present on 6.6-stable. Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-03-07Revert "rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threads"Tomas Glozar1-9/+6
This reverts commit 41955b6c268154f81e34f9b61cf8156eec0730c0. The commit breaks rtla build, since params->kernel_workload is not present on 6.6-stable. Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-02-21rtla/timerlat_top: Abort event processing on second signalTomas Glozar1-0/+8
[ Upstream commit 80967b354a76b360943af384c10d807d98bea5c4 ] If either SIGINT is received twice, or after a SIGALRM (that is, after timerlat was supposed to stop), abort processing events currently left in the tracefs buffer and exit immediately. This allows the user to exit rtla without waiting for processing all events, should that take longer than wanted, at the cost of not processing all samples. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-6-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Sasha Levin <sashal@kernel.org>
2025-02-21rtla/timerlat_hist: Abort event processing on second signalTomas Glozar1-0/+8
[ Upstream commit d6899e560366e10141189697502bc5521940c588 ] If either SIGINT is received twice, or after a SIGALRM (that is, after timerlat was supposed to stop), abort processing events currently left in the tracefs buffer and exit immediately. This allows the user to exit rtla without waiting for processing all events, should that take longer than wanted, at the cost of not processing all samples. Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-5-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Sasha Levin <sashal@kernel.org>
2025-02-17rtla/timerlat_top: Stop timerlat tracer on signalTomas Glozar1-1/+11
commit a4dfce7559d75430c464294ddee554be2a413c4a upstream. Currently, when either SIGINT from the user or SIGALRM from the duration timer is caught by rtla-timerlat, stop_tracing is set to break out of the main loop. This is not sufficient for cases where the timerlat tracer is producing more data than rtla can consume, since in that case, rtla is looping indefinitely inside tracefs_iterate_raw_events, never reaches the check of stop_tracing and hangs. In addition to setting stop_tracing, also stop the timerlat tracer on received signal (SIGINT or SIGALRM). This will stop new samples so that the existing samples may be processed and tracefs_iterate_raw_events eventually exits. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-4-tglozar@redhat.com Fixes: a828cd18bc4a ("rtla: Add timerlat tool and timelart top mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-02-17rtla/timerlat_hist: Stop timerlat tracer on signalTomas Glozar1-1/+10
commit c73cab9dbed04d8f65ca69177b4b21ed3e09dfa7 upstream. Currently, when either SIGINT from the user or SIGALRM from the duration timer is caught by rtla-timerlat, stop_tracing is set to break out of the main loop. This is not sufficient for cases where the timerlat tracer is producing more data than rtla can consume, since in that case, rtla is looping indefinitely inside tracefs_iterate_raw_events, never reaches the check of stop_tracing and hangs. In addition to setting stop_tracing, also stop the timerlat tracer on received signal (SIGINT or SIGALRM). This will stop new samples so that the existing samples may be processed and tracefs_iterate_raw_events eventually exits. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-3-tglozar@redhat.com Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-02-17rtla: Add trace_instance_stopTomas Glozar2-0/+9
commit e879b5dcf8d044f3865a32d95cc5b213f314c54f upstream. Support not only turning trace on for the timerlat tracer, but also turning it off. This will be used in subsequent patches to stop the timerlat tracer without also wiping the trace buffer. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Cc: Gabriele Monaco <gmonaco@redhat.com> Link: https://lore.kernel.org/20250116144931.649593-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-02-17rtla/timerlat_top: Set OSNOISE_WORKLOAD for kernel threadsTomas Glozar1-6/+9
commit 217f0b1e990e30a1f06f6d531fdb4530f4788d48 upstream. When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/options. This option is not re-enabled in a subsequent run with kernel-space threads, leading to rtla collecting no results if the previous run exited abnormally: $ rtla timerlat top -u ^\Quit (core dumped) $ rtla timerlat top -k -d 1s Timer Latency 0 00:00:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max The issue persists until OSNOISE_WORKLOAD is set manually by running: $ echo OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options Set OSNOISE_WORKLOAD when running rtla with kernel-space threads if available to fix the issue. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250107144823.239782-4-tglozar@redhat.com Fixes: cdca4f4e5e8e ("rtla/timerlat_top: Add timerlat user-space support") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-02-17rtla/timerlat_hist: Set OSNOISE_WORKLOAD for kernel threadsTomas Glozar1-6/+9
commit d8d866171a414ed88bd0d720864095fd75461134 upstream. When using rtla timerlat with userspace threads (-u or -U), rtla disables the OSNOISE_WORKLOAD option in /sys/kernel/tracing/osnoise/options. This option is not re-enabled in a subsequent run with kernel-space threads, leading to rtla collecting no results if the previous run exited abnormally: $ rtla timerlat hist -u ^\Quit (core dumped) $ rtla timerlat hist -k -d 1s Index over: count: min: avg: max: ALL: IRQ Thr Usr count: 0 0 0 min: - - - avg: - - - max: - - - The issue persists until OSNOISE_WORKLOAD is set manually by running: $ echo OSNOISE_WORKLOAD > /sys/kernel/tracing/osnoise/options Set OSNOISE_WORKLOAD when running rtla with kernel-space threads if available to fix the issue. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250107144823.239782-3-tglozar@redhat.com Fixes: ed774f7481fa ("rtla/timerlat_hist: Add timerlat user-space support") Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-02-17rtla/osnoise: Distinguish missing workload optionTomas Glozar1-1/+1
commit 80d3ba1cf51bfbbb3b098434f2b2c95cd7c0ae5c upstream. osnoise_set_workload returns -1 for both missing OSNOISE_WORKLOAD option and failure in setting the option. Return -1 for missing and -2 for failure to distinguish them. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: Luis Goncalves <lgoncalv@redhat.com> Link: https://lore.kernel.org/20250107144823.239782-2-tglozar@redhat.com Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-12-19rtla/timerlat: Make timerlat_hist_cpu->*_count unsigned long longTomas Glozar1-6/+6
commit 76b3102148135945b013797fac9b206273f0f777 upstream. Do the same fix as in previous commit also for timerlat-hist. Link: https://lore.kernel.org/20241011121015.2868751-2-tglozar@redhat.com Reported-by: Attila Fazekas <afazekas@redhat.com> Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> [ Drop hunk fixing printf in timerlat_print_stats_all since that is not in 6.6 ] Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-12-14rtla/timerlat: Make timerlat_top_cpu->*_count unsigned long longTomas Glozar1-4/+4
[ Upstream commit 4eba4723c5254ba8251ecb7094a5078d5c300646 ] Most fields of struct timerlat_top_cpu are unsigned long long, but the fields {irq,thread,user}_count are int (32-bit signed). This leads to overflow when tracing on a large number of CPUs for a long enough time: $ rtla timerlat top -a20 -c 1-127 -d 12h ... 0 12:00:00 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max 1 #43200096 | 0 0 1 2 | 3 2 6 12 ... 127 #43200096 | 0 0 1 2 | 3 2 5 11 ALL #119144 e4 | 0 5 4 | 2 28 16 The average latency should be 0-1 for IRQ and 5-6 for thread, but is reported as 5 and 28, about 4 to 5 times more, due to the count overflowing when summed over all CPUs: 43200096 * 127 = 5486412192, however, 1191444898 (= 5486412192 mod MAX_INT) is reported instead, as seen on the last line of the output, and the averages are thus ~4.6 times higher than they should be (5486412192 / 1191444898 = ~4.6). Fix the issue by changing {irq,thread,user}_count fields to unsigned long long, similarly to other fields in struct timerlat_top_cpu and to the count variable in timerlat_top_print_sum. Link: https://lore.kernel.org/20241011121015.2868751-1-tglozar@redhat.com Reported-by: Attila Fazekas <afazekas@redhat.com> Signed-off-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-12-14tools/rtla: fix collision with glibc sched_attr/sched_set_attrJan Stancek2-2/+4
[ Upstream commit 0eecee340672c4b512f6f4a8c6add26df05d130c ] glibc commit 21571ca0d703 ("Linux: Add the sched_setattr and sched_getattr functions") now also provides 'struct sched_attr' and sched_setattr() which collide with the ones from rtla. In file included from src/trace.c:11: src/utils.h:49:8: error: redefinition of ‘struct sched_attr’ 49 | struct sched_attr { | ^~~~~~~~~~ In file included from /usr/include/bits/sched.h:60, from /usr/include/sched.h:43, from /usr/include/tracefs/tracefs.h:10, from src/trace.c:4: /usr/include/linux/sched/types.h:98:8: note: originally defined here 98 | struct sched_attr { | ^~~~~~~~~~ Define 'struct sched_attr' conditionally, similar to what strace did: https://lore.kernel.org/all/20240930222913.3981407-1-raj.khem@gmail.com/ and rename rtla's version of sched_setattr() to avoid collision. Link: https://lore.kernel.org/8088f66a7a57c1b209cd8ae0ae7c336a7f8c930d.1728572865.git.jstancek@redhat.com Signed-off-by: Jan Stancek <jstancek@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-10-10rtla: Fix the help text in osnoise and timerlat top toolsEder Zulian2-3/+3
commit 3d7b8ea7a8a20a45d019382c4dc6ed79e8bb95cf upstream. The help text in osnoise top and timerlat top had some minor errors and omissions. The -d option was missing the 's' (second) abbreviation and the error message for '-d' used '-D'. Cc: stable@vger.kernel.org Fixes: 1eceb2fc2ca54 ("rtla/osnoise: Add osnoise top mode") Fixes: a828cd18bc4ad ("rtla: Add timerlat tool and timelart top mode") Link: https://lore.kernel.org/20240813155831.384446-1-ezulian@redhat.com Suggested-by: Tomas Glozar <tglozar@redhat.com> Reviewed-by: Tomas Glozar <tglozar@redhat.com> Signed-off-by: Eder Zulian <ezulian@redhat.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-08-29rtla/osnoise: Prevent NULL dereference in error handlingDan Carpenter1-7/+4
commit 90574d2a675947858b47008df8d07f75ea50d0d0 upstream. If the "tool->data" allocation fails then there is no need to call osnoise_free_top() and, in fact, doing so will lead to a NULL dereference. Cc: stable@vger.kernel.org Cc: John Kacur <jkacur@redhat.com> Cc: "Luis Claudio R. Goncalves" <lgoncalv@redhat.com> Cc: Clark Williams <williams@redhat.com> Fixes: 1eceb2fc2ca5 ("rtla/osnoise: Add osnoise top mode") Link: https://lore.kernel.org/f964ed1f-64d2-4fde-ad3e-708331f8f358@stanley.mountain Signed-off-by: Dan Carpenter <dan.carpenter@linaro.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-06-21rtla/auto-analysis: Replace \t with spacesDaniel Bristot de Oliveira1-46/+63
commit a40e5e4dd0207485dee75e2b8e860d5853bcc5f7 upstream. When copying timerlat auto-analysis from a terminal to some web pages or chats, the \t are being replaced with a single ' ' or ' ', breaking the output. For example: ## CPU 3 hit stop tracing, analyzing it ## IRQ handler delay: 1.30 us (0.11 %) IRQ latency: 1.90 us Timerlat IRQ duration: 3.00 us (0.24 %) Blocking thread: 1223.16 us (99.00 %) insync:4048 1223.16 us IRQ interference 4.93 us (0.40 %) local_timer:236 4.93 us ------------------------------------------------------------------------ Thread latency: 1235.47 us (100%) Replace \t with spaces to avoid this problem. Link: https://lkml.kernel.org/r/ec7ed2b2809c22ab0dfc8eb7c805ab9cddc4254a.1713968967.git.bristot@kernel.org Cc: stable@vger.kernel.org Cc: Jonathan Corbet <corbet@lwn.net> Cc: Juri Lelli <juri.lelli@redhat.com> Fixes: 27e348b221f6 ("rtla/timerlat: Add auto-analysis core") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-06-21rtla/timerlat: Simplify "no value" printing on topDaniel Bristot de Oliveira1-12/+5
commit 5f0769331a965675cdfec97c09f3f6e875d7c246 upstream. Instead of printing three times the same output, print it only once, reducing lines and being sure that all no values have the same length. It also fixes an extra '\n' when running the with kernel threads, like here: =============== %< ============== Timer Latency 0 00:00:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max 2 #0 | - - - - | 161 161 161 161 3 #0 | - - - - | 161 161 161 161 8 #1 | 54 54 54 54 | - - - -'\n' ---------------|----------------------------------------|--------------------------------------- ALL #1 e0 | 54 54 54 | 161 161 161 =============== %< ============== This '\n' should have been removed with the user-space support that added another '\n' if not running with kernel threads. Link: https://lkml.kernel.org/r/0a4d8085e7cd706733a5dc10a81ca38b82bd4992.1713968967.git.bristot@kernel.org Cc: stable@vger.kernel.org Cc: Jonathan Corbet <corbet@lwn.net> Cc: Juri Lelli <juri.lelli@redhat.com> Fixes: cdca4f4e5e8e ("rtla/timerlat_top: Add timerlat user-space support") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-06-16rtla/timerlat: Fix histogram report when a cpu count is 0John Kacur1-18/+42
commit 01b05fc0e5f3aec443a9a8ffa0022cbca2fd3608 upstream. On short runs it is possible to get no samples on a cpu, like this: # rtla timerlat hist -u -T50 Index IRQ-001 Thr-001 Usr-001 IRQ-002 Thr-002 Usr-002 2 1 0 0 0 0 0 33 0 1 0 0 0 0 36 0 0 1 0 0 0 49 0 0 0 1 0 0 52 0 0 0 0 1 0 over: 0 0 0 0 0 0 count: 1 1 1 1 1 0 min: 2 33 36 49 52 18446744073709551615 avg: 2 33 36 49 52 - max: 2 33 36 49 52 0 rtla timerlat hit stop tracing IRQ handler delay: (exit from idle) 48.21 us (91.09 %) IRQ latency: 49.11 us Timerlat IRQ duration: 2.17 us (4.09 %) Blocking thread: 1.01 us (1.90 %) swapper/2:0 1.01 us ------------------------------------------------------------------------ Thread latency: 52.93 us (100%) Max timerlat IRQ latency from idle: 49.11 us in cpu 2 Note, the value 18446744073709551615 is the same as ~0. Fix this by reporting no results for the min, avg and max if the count is 0. Link: https://lkml.kernel.org/r/20240510190318.44295-1-jkacur@redhat.com Cc: stable@vger.kernel.org Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Suggested-by: Daniel Bristot de Oliveria <bristot@kernel.org> Signed-off-by: John Kacur <jkacur@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-02-23tools/rtla: Fix uninitialized bucket/data->bucket_size warningDaniel Bristot de Oliveira2-4/+2
commit 64dc40f7523369912d7adb22c8cb655f71610505 upstream. When compiling rtla with clang, I am getting the following warnings: $ make HOSTCC=clang CC=clang LLVM_IAS=1 [..] clang -O -g -DVERSION=\"6.8.0-rc3\" -flto=auto -fexceptions -fstack-protector-strong -fasynchronous-unwind-tables -fstack-clash-protection -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS $(pkg-config --cflags libtracefs) -c -o src/osnoise_hist.o src/osnoise_hist.c src/osnoise_hist.c:138:6: warning: variable 'bucket' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized] 138 | if (data->bucket_size) | ^~~~~~~~~~~~~~~~~ src/osnoise_hist.c:149:6: note: uninitialized use occurs here 149 | if (bucket < entries) | ^~~~~~ src/osnoise_hist.c:138:2: note: remove the 'if' if its condition is always true 138 | if (data->bucket_size) | ^~~~~~~~~~~~~~~~~~~~~~ 139 | bucket = duration / data->bucket_size; src/osnoise_hist.c:132:12: note: initialize the variable 'bucket' to silence this warning 132 | int bucket; | ^ | = 0 1 warning generated. [...] clang -O -g -DVERSION=\"6.8.0-rc3\" -flto=auto -fexceptions -fstack-protector-strong -fasynchronous-unwind-tables -fstack-clash-protection -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS $(pkg-config --cflags libtracefs) -c -o src/timerlat_hist.o src/timerlat_hist.c src/timerlat_hist.c:181:6: warning: variable 'bucket' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized] 181 | if (data->bucket_size) | ^~~~~~~~~~~~~~~~~ src/timerlat_hist.c:204:6: note: uninitialized use occurs here 204 | if (bucket < entries) | ^~~~~~ src/timerlat_hist.c:181:2: note: remove the 'if' if its condition is always true 181 | if (data->bucket_size) | ^~~~~~~~~~~~~~~~~~~~~~ 182 | bucket = latency / data->bucket_size; src/timerlat_hist.c:175:12: note: initialize the variable 'bucket' to silence this warning 175 | int bucket; | ^ | = 0 1 warning generated. This is a legit warning, but data->bucket_size is always > 0 (see timerlat_hist_parse_args()), so the if is not necessary. Remove the unneeded if (data->bucket_size) to avoid the warning. Link: https://lkml.kernel.org/r/6e1b1665cd99042ae705b3e0fc410858c4c42346.1707217097.git.bristot@kernel.org Cc: stable@vger.kernel.org Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Nathan Chancellor <nathan@kernel.org> Cc: Nick Desaulniers <ndesaulniers@google.com> Cc: Bill Wendling <morbo@google.com> Cc: Justin Stitt <justinstitt@google.com> Cc: Donald Zickus <dzickus@redhat.com> Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-02-23tools/rtla: Exit with EXIT_SUCCESS when help is invokedJohn Kacur4-4/+20
commit b5f319360371087d52070d8f3fc7789e80ce69a6 upstream. Fix rtla so that the following commands exit with 0 when help is invoked rtla osnoise top -h rtla osnoise hist -h rtla timerlat top -h rtla timerlat hist -h Link: https://lore.kernel.org/linux-trace-devel/20240203001607.69703-1-jkacur@redhat.com Cc: stable@vger.kernel.org Fixes: 1eeb6328e8b3 ("rtla/timerlat: Add timerlat hist mode") Signed-off-by: John Kacur <jkacur@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-02-23tools/rtla: Fix clang warning about mount_point var sizeDaniel Bristot de Oliveira1-1/+1
commit 30369084ac6e27479a347899e74f523e6ca29b89 upstream. clang is reporting this warning: $ make HOSTCC=clang CC=clang LLVM_IAS=1 [...] clang -O -g -DVERSION=\"6.8.0-rc3\" -flto=auto -fexceptions -fstack-protector-strong -fasynchronous-unwind-tables -fstack-clash-protection -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS $(pkg-config --cflags libtracefs) -c -o src/utils.o src/utils.c src/utils.c:548:66: warning: 'fscanf' may overflow; destination buffer in argument 3 has size 1024, but the corresponding specifier may require size 1025 [-Wfortify-source] 548 | while (fscanf(fp, "%*s %" STR(MAX_PATH) "s %99s %*s %*d %*d\n", mount_point, type) == 2) { | ^ Increase mount_point variable size to MAX_PATH+1 to avoid the overflow. Link: https://lkml.kernel.org/r/1b46712e93a2f4153909514a36016959dcc4021c.1707217097.git.bristot@kernel.org Cc: stable@vger.kernel.org Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Nathan Chancellor <nathan@kernel.org> Cc: Nick Desaulniers <ndesaulniers@google.com> Cc: Bill Wendling <morbo@google.com> Cc: Justin Stitt <justinstitt@google.com> Cc: Donald Zickus <dzickus@redhat.com> Fixes: a957cbc02531 ("rtla: Add -C cgroup support") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-02-23tools/rtla: Replace setting prio with nice for SCHED_OTHERlimingming32-3/+5
commit 14f08c976ffe0d2117c6199c32663df1cbc45c65 upstream. Since the sched_priority for SCHED_OTHER is always 0, it makes no sence to set it. Setting nice for SCHED_OTHER seems more meaningful. Link: https://lkml.kernel.org/r/20240207065142.1753909-1-limingming3@lixiang.com Cc: stable@vger.kernel.org Fixes: b1696371d865 ("rtla: Helper functions for rtla") Signed-off-by: limingming3 <limingming3@lixiang.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-02-23tools/rtla: Remove unused sched_getattr() functionDaniel Bristot de Oliveira1-6/+0
commit 084ce16df0f060efd371092a09a7ae74a536dc11 upstream. Clang is reporting: $ make HOSTCC=clang CC=clang LLVM_IAS=1 [...] clang -O -g -DVERSION=\"6.8.0-rc3\" -flto=auto -fexceptions -fstack-protector-strong -fasynchronous-unwind-tables -fstack-clash-protection -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS $(pkg-config --cflags libtracefs) -c -o src/utils.o src/utils.c src/utils.c:241:19: warning: unused function 'sched_getattr' [-Wunused-function] 241 | static inline int sched_getattr(pid_t pid, struct sched_attr *attr, | ^~~~~~~~~~~~~ 1 warning generated. Which is correct, so remove the unused function. Link: https://lkml.kernel.org/r/eaed7ba122c4ae88ce71277c824ef41cbf789385.1707217097.git.bristot@kernel.org Cc: stable@vger.kernel.org Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Nathan Chancellor <nathan@kernel.org> Cc: Nick Desaulniers <ndesaulniers@google.com> Cc: Bill Wendling <morbo@google.com> Cc: Justin Stitt <justinstitt@google.com> Cc: Donald Zickus <dzickus@redhat.com> Fixes: b1696371d865 ("rtla: Helper functions for rtla") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2023-11-20rtla: Fix uninitialized variable foundColin Ian King1-1/+1
[ Upstream commit 696444a544ecd6d62c1edc89516b376cefb28929 ] Variable found is not being initialized, in the case where the desired mount is not found the variable contains garbage. Fix this by initializing it to zero. Link: https://lore.kernel.org/all/20230727150117.627730-1-colin.i.king@gmail.com/ Fixes: a957cbc02531 ("rtla: Add -C cgroup support") Signed-off-by: Colin Ian King <colin.i.king@gmail.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-09-22rtla/timerlat: Do not stop user-space if a cpu is offlineDaniel Bristot de Oliveira1-2/+4
If no CPU list is passed, timerlat in user-space will dispatch one thread per sysconf(_SC_NPROCESSORS_CONF). However, not all CPU might be available, for instance, if HT is disabled. Currently, rtla timerlat is stopping the session if an user-space thread cannot set affinity to a CPU, or if a running user-space thread is killed. However, this is too restrictive. So, reduce the error to a debug message, and rtla timerlat run as long as there is at least one user-space thread alive. Link: https://lore.kernel.org/lkml/59cf2c882900ab7de91c6ee33b382ac7fa6b4ed0.1694781909.git.bristot@kernel.org Fixes: cdca4f4e5e8e ("rtla/timerlat_top: Add timerlat user-space support") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
2023-09-12rtla/timerlat_aa: Fix previous IRQ delay for IRQs that happens after thread ↵Daniel Bristot de Oliveira1-5/+8
sample timerlat auto-analysis takes note of all IRQs, before or after the execution of the timerlat thread. Because we cannot go backward in the trace (we will fix it when moving to trace-cmd lib?), timerlat aa take note of the last IRQ execution in the waiting for the IRQ state, and then print it if it is executed after the expected timer IRQ starting time. After the thread sample, the timerlat starts recording the next IRQs as "previous" irq for the next occurrence. However, if an IRQ happens after the thread measurement but before the tracing stops, it is classified as a previous IRQ. That is not wrong, as it can be "previous" for the subsequent activation. What is wrong is considering it as a potential source for the last activation. Ignore the IRQ interference that happens after the IRQ starting time for now. A future improvement for timerlat can be either keeping a list of previous IRQ execution or using the trace-cmd library. Still, it requires further investigation - it is a new feature. Link: https://lore.kernel.org/lkml/a44a3f5c801dcc697bacf7325b65d4a5b0460537.1691162043.git.bristot@kernel.org Fixes: 27e348b221f6 ("rtla/timerlat: Add auto-analysis core") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
2023-09-12rtla/timerlat_aa: Fix negative IRQ delayDaniel Bristot de Oliveira1-1/+17
When estimating the IRQ timer delay, we are dealing with two different clock sources: the external clock source that timerlat uses as a reference and the clock used by the tracer. There are also two moments: the time reading the clock and the timer in which the event is placed in the buffer (the trace event timestamp). If the processor is slow or there is some hardware noise, the difference between the timestamp and the external clock, read can be longer than the IRQ handler delay, resulting in a negative time. If so, set IRQ to start delay as 0. In the end, it is less near-zero and relevant then the noise. Link: https://lore.kernel.org/lkml/a066fb667c7136d86dcddb3c7ccd72587db3e7c7.1691162043.git.bristot@kernel.org Fixes: 27e348b221f6 ("rtla/timerlat: Add auto-analysis core") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
2023-09-12rtla/timerlat_aa: Zero thread sum after every sample analysisDaniel Bristot de Oliveira1-0/+1
The thread thread_thread_sum accounts for thread interference during a single activation. It was not being zeroed, so it was accumulating thread interference over all activations. It was not that visible when timerlat was the highest priority. Link: https://lore.kernel.org/lkml/97bff55b0141f2d01b47d9450a5672fde147b89a.1691162043.git.bristot@kernel.org Fixes: 27e348b221f6 ("rtla/timerlat: Add auto-analysis core") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org>
2023-06-13rtla/timerlat_hist: Add timerlat user-space supportDaniel Bristot de Oliveira1-11/+118
Add the support for running timerlat threads in user-space. In this mode, enabled with -u/--user-threads, timerlat dispatches user-space processes that will loop in the timerlat_fd, measuring the overhead for going to user-space and then returning to the kernel - in addition to the existing measurements. Here is one example of the tool's output with -u enabled: $ sudo timerlat hist -u -c 1-3 -d 600 # RTLA timerlat histogram # Time unit is microseconds (us) # Duration: 0 00:10:01 Index IRQ-001 Thr-001 Usr-001 IRQ-002 Thr-002 Usr-002 IRQ-003 Thr-003 Usr-003 0 477555 0 0 425287 0 0 474357 0 0 1 122385 7998 0 174616 1921 0 125412 3138 0 2 47 587376 492150 89 594717 447830 147 593463 454872 3 11 2549 101930 7 2682 145580 64 2530 138680 4 3 1954 2833 1 463 4917 11 548 4656 5 0 60 1037 0 138 1117 6 179 1130 6 0 26 1837 0 38 277 1 76 339 7 0 15 143 0 28 147 2 37 156 8 0 10 23 0 11 75 0 12 80 9 0 7 17 0 0 26 0 11 42 10 0 2 11 0 0 18 0 2 20 11 0 0 7 0 1 8 0 2 12 12 0 0 6 0 1 4 0 2 8 13 0 1 3 0 0 0 0 0 1 14 0 1 0 0 0 1 0 0 2 15 0 1 0 0 0 0 0 0 2 16 0 1 2 0 0 0 0 0 0 17 0 0 1 0 0 0 0 0 0 19 0 0 1 0 0 0 0 0 0 over: 0 0 0 0 0 0 0 0 0 count: 600001 600001 600001 600000 600000 600000 600000 600000 600000 min: 0 1 2 0 1 2 0 1 2 avg: 0 1 2 0 2 2 0 2 2 max: 4 16 19 4 12 14 7 12 15 The tuning setup like -p or -C work for the user-space threads as well. Link: https://lkml.kernel.org/r/b6a042d55003c4a67ff7dce28d96044b7044f00d.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla/timerlat_top: Add timerlat user-space supportDaniel Bristot de Oliveira7-6/+474
Add the support for running timerlat threads in user-space. In this mode, enabled with -u/--user-threads, timerlat dispatches user-space processes that will loop in the timerlat_fd, measuring the overhead for going to user-space and then returning to the kernel - in addition to the existing measurements. Here is one example of the tool's output with -u enabled: $ sudo timerlat top -u -d 600 -q Timer Latency 0 00:10:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) | Ret user Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max | cur min avg max 0 #600001 | 0 0 0 3 | 2 1 2 9 | 3 2 3 15 1 #600001 | 0 0 0 2 | 2 1 2 13 | 2 2 3 18 2 #600001 | 0 0 0 10 | 2 1 2 16 | 3 2 3 20 3 #600001 | 0 0 0 7 | 2 1 2 10 | 3 2 3 11 4 #600000 | 0 0 0 16 | 2 1 2 41 | 3 2 3 58 5 #600000 | 0 0 0 3 | 2 1 2 10 | 3 2 3 13 6 #600000 | 0 0 0 5 | 2 1 2 7 | 3 2 3 10 7 #600000 | 0 0 0 1 | 2 1 2 7 | 3 2 3 10 The tuning setup like -p or -C work for the user-space threads as well. Link: https://lkml.kernel.org/r/758ad2292a0a1d884138d08219e1a0f572d257a2.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla/hwnoise: Reduce runtime to 75%Daniel Bristot de Oliveira1-1/+7
osnoise runs 100% of time by default. It makes sense because osnoise is preemptive. hwnoise checks preemption once a second, so it reduces system progress. Reduce runtime to 75% to avoid problems by default. I added a Fixes as it might avoid problems for first time users as it lands on distros. Link: https://lkml.kernel.org/r/af0b7113ffc00031b9af4bb40ef5889a27dadf8c.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Fixes: 1f428356c38d ("rtla: Add hwnoise tool") Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla: Start the tracers after creating all instancesDaniel Bristot de Oliveira4-17/+46
Group all start tracing after finishing creating all instances. The tracing instance starts first for the case of hitting a stop tracing while enabling other instances. The trace instance is the one with most valuable information. Link: https://lkml.kernel.org/r/67da7a703a56f75d7cd46568525145a65501a7e8.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla/timerlat_hist: Add auto-analysis supportDaniel Bristot de Oliveira1-2/+49
Add auto-analysis to timerlat hist, including the --no-aa option to reduce overhead and --dump-task. --aa-only was not added as it is already on timerlat top. Link: https://lkml.kernel.org/r/c2693f47ee83e659a7723fed8035f5d2534f528e.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla/timerlat: Give timerlat auto analysis its own instanceDaniel Bristot de Oliveira3-21/+67
Currently, the auto-analysis is attached to the timerlat top instance. The idea was to avoid creating another instance just for that, so one instance could be reused. The drawback is that, by doing so, the auto-analysis run for the entire session, consuming CPU time. On my 24 box CPUs for timerlat with a 100 us period consumed 50 % with auto analysis, but only 16 % without. By creating an instance for auto-analysis, we can keep the processing stopped until a stop tracing condition is hit. Once it happens, timerlat auto-analysis can use its own trace instance to parse only the end of the trace. By doing so, auto-analysis stop consuming cpu time when it is not needed. If the --aa-only is passed, the timerlat top instance is reused for auto analysis. Link: https://lkml.kernel.org/r/346b7168c1bae552a415715ec6d23c129a43bdb7.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla: Automatically move rtla to a house-keeping cpuDaniel Bristot de Oliveira6-0/+87
When the user sets -c <cpu-list> try to move rtla out of the <cpu-list>, even without an -H option. This is useful to avoid having rtla interfering with the workload. This works by removing <cpu-list> from rtla's current affinity. If rtla fails to move itself away it is not that of a problem as this is an automatic measure. Link: https://lkml.kernel.org/r/c54304d90c777310fb85a3e658d1449173759aab.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla: Change monitored_cpus from char * to cpu_set_tDaniel Bristot de Oliveira5-87/+24
Use a cpumask instead of a char *, reducing memory footprint and code. No functional change, and in preparation for auto house-keeping. Link: https://lkml.kernel.org/r/54c46293261d13cb1042d0314486539eeb45fe5d.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla: Add --house-keeping optionDaniel Bristot de Oliveira6-9/+162
To avoid having rtla interfering with the measurement threads, add an option for the user to set the CPUs in which rtla should run. For instance: # rtla timerlat top -H 0 -c 1-7 Will place rtla in the CPU 0, while running the measurement threads in the CPU 1-7. Link: https://lkml.kernel.org/r/6a6c78a579a96ba8b02ae67ee1e0ba2cb5e03c4a.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Suggested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-06-13rtla: Add -C cgroup supportDaniel Bristot de Oliveira6-9/+282
The -C option sets a cgroup to the tracer's threads. If the -C option is passed without arguments, the tracer's thread will inherit rtla's cgroup. Otherwise, the threads will be placed on the cgroup passed to the option. Link: https://lkml.kernel.org/r/cb051477331d292f17c08bf1d66f0e0384bbe5a5.1686066600.git.bristot@kernel.org Cc: William White <chwhite@redhat.com> Cc: Jonathan Corbet <corbet@lwn.net> Tested-by: Juri Lelli <juri.lelli@redhat.com> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-04-26rtla/timerlat: Fix "Previous IRQ" auto analysis' lineDaniel Bristot de Oliveira1-1/+1
The "Previous IRQ interference" line is misaligned and without a \n, breaking the tool's output: ## CPU 12 hit stop tracing, analyzing it ## Previous IRQ interference: up to 2.22 us IRQ handler delay: 18.06 us (0.00 %) IRQ latency: 18.52 us Timerlat IRQ duration: 4.41 us (0.00 %) Blocking thread: 216.93 us (0.03 %) Fix the output: ## CPU 7 hit stop tracing, analyzing it ## Previous IRQ interference: up to 8.93 us IRQ handler delay: 0.98 us (0.00 %) IRQ latency: 2.95 us Timerlat IRQ duration: 11.26 us (0.03 %) Link: https://lore.kernel.org/linux-trace-devel/8b5819077f15ccf24745c9bf3205451e16ee32d9.1679685525.git.bristot@kernel.org Fixes: 27e348b221f6 ("rtla/timerlat: Add auto-analysis core") Cc: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-04-26rtla/timerlat: Add auto-analysis only optionDaniel Bristot de Oliveira1-5/+44
Parsing and formating timerlat data might consume a reasonable amount of CPU time on very large systems, or when timerlat has a too short period. Add an option to run timerlat with auto-analysis enabled while skipping the statistics parsing. In this mode, rtla timerlat periodically checks if the tracing is on, going to sleep waiting for the stop tracing condition to stop tracing, or for the tracing session to finish. If the stop tracing condition is hit, the tool prints the auto analysis. Otherwise, the tool prints the max observed latency and exit. The max observed latency is captured via tracing_max_latency. Link: https://lore.kernel.org/linux-trace-devel/4dc514d1d5dc353c537a466a9b5af44c266b6da2.1680106912.git.bristot@kernel.org Cc: Jonathan Corbet <corbet@lwn.net> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-02-14rtla: Add hwnoise toolDaniel Bristot de Oliveira4-16/+196
The hwnoise tool is a special mode for the osnoise top tool. hwnoise dispatches the osnoise tracer and displays a summary of the noise. The difference is that it runs the tracer with the OSNOISE_IRQ_DISABLE option set, thus only allowing only hardware-related noise, resulting in a simplified output. hwnoise has the same features of osnoise. An example of the tool's output: # rtla hwnoise -c 1-11 -T 1 -d 10m -q Hardware-related Noise duration: 0 00:10:00 | time is in us CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI 1 #599 599000000 138 99.99997 3 3 4 74 2 #599 599000000 85 99.99998 3 3 4 75 3 #599 599000000 86 99.99998 4 3 6 75 4 #599 599000000 81 99.99998 4 4 2 75 5 #599 599000000 85 99.99998 2 2 2 75 Link: https://lkml.kernel.org/r/2d6f49a6f3a4f8b51b2c806458b1cff71ad4d014.1675805361.git.bristot@kernel.org Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Cc: Daniel Bristot de Oliveira <bristot@kernel.org> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Clark Williams <williams@redhat.com> Cc: Bagas Sanjaya <bagasdotme@gmail.com> Cc: Jonathan Corbet <corbet@lwn.net> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-02-02rtla/timerlat: Add auto-analysis support to timerlat topDaniel Bristot de Oliveira1-2/+44
Currently, timerlat top displays the timerlat tracer latency results, saving the intuitive timerlat trace for the developer to analyze. This patch goes a step forward in the automaton of the scheduling latency analysis by providing a summary of the root cause of a latency higher than the passed "stop tracing" parameter if the trace stops. The output is intuitive enough for non-expert users to have a general idea of the root cause by looking at each factor's contribution percentage while keeping the technical detail in the output for more expert users to start an in dept debug or to correlate a root cause with an existing one. The terminology is in line with recent industry and academic publications to facilitate the understanding of both audiences. Here is one example of tool output: ----------------------------------------- %< ----------------------------------------------------- # taskset -c 0 timerlat -a 40 -c 1-23 -q Timer Latency 0 00:00:12 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max 1 #12322 | 0 0 1 15 | 10 3 9 31 2 #12322 | 3 0 1 12 | 10 3 9 23 3 #12322 | 1 0 1 21 | 8 2 8 34 4 #12322 | 1 0 1 17 | 10 2 11 33 5 #12322 | 0 0 1 12 | 8 3 8 25 6 #12322 | 1 0 1 14 | 16 3 11 35 7 #12322 | 0 0 1 14 | 9 2 8 29 8 #12322 | 1 0 1 22 | 9 3 9 34 9 #12322 | 0 0 1 14 | 8 2 8 24 10 #12322 | 1 0 0 12 | 9 3 8 24 11 #12322 | 0 0 0 15 | 6 2 7 29 12 #12321 | 1 0 0 13 | 5 3 8 23 13 #12319 | 0 0 1 14 | 9 3 9 26 14 #12321 | 1 0 0 13 | 6 2 8 24 15 #12321 | 1 0 1 15 | 12 3 11 27 16 #12318 | 0 0 1 13 | 7 3 10 24 17 #12319 | 0 0 1 13 | 11 3 9 25 18 #12318 | 0 0 0 12 | 8 2 8 20 19 #12319 | 0 0 1 18 | 10 2 9 28 20 #12317 | 0 0 0 20 | 9 3 8 34 21 #12318 | 0 0 0 13 | 8 3 8 28 22 #12319 | 0 0 1 11 | 8 3 10 22 23 #12320 | 28 0 1 28 | 41 3 11 41 rtla timerlat hit stop tracing ## CPU 23 hit stop tracing, analyzing it ## IRQ handler delay: 27.49 us (65.52 %) IRQ latency: 28.13 us Timerlat IRQ duration: 9.59 us (22.85 %) Blocking thread: 3.79 us (9.03 %) objtool:49256 3.79 us Blocking thread stacktrace -> timerlat_irq -> __hrtimer_run_queues -> hrtimer_interrupt -> __sysvec_apic_timer_interrupt -> sysvec_apic_timer_interrupt -> asm_sysvec_apic_timer_interrupt -> _raw_spin_unlock_irqrestore -> cgroup_rstat_flush_locked -> cgroup_rstat_flush_irqsafe -> mem_cgroup_flush_stats -> mem_cgroup_wb_stats -> balance_dirty_pages -> balance_dirty_pages_ratelimited_flags -> btrfs_buffered_write -> btrfs_do_write_iter -> vfs_write -> __x64_sys_pwrite64 -> do_syscall_64 -> entry_SYSCALL_64_after_hwframe ------------------------------------------------------------------------ Thread latency: 41.96 us (100%) The system has exit from idle latency! Max timerlat IRQ latency from idle: 17.48 us in cpu 4 Saving trace to timerlat_trace.txt ----------------------------------------- >% ----------------------------------------------------- In this case, the major factor was the delay suffered by the IRQ handler that handles timerlat wakeup: 65.52 %. This can be caused by the current thread masking interrupts, which can be seen in the blocking thread stacktrace: the current thread (objtool:49256) disabled interrupts via raw spin lock operations inside mem cgroup, while doing write syscall in a btrfs file system. A simple search for the function name on Google shows that this is a legit case for disabling the interrupts: cgroup: Use irqsave in cgroup_rstat_flush_locked() lore.kernel.org/linux-mm/20220301122143.1521823-2-bigeasy@linutronix.de/ The output also prints other reasons for the latency root cause, such as: - an IRQ that happened before the IRQ handler that caused delays - The interference from NMI, IRQ, Softirq, and Threads The details about how these factors affect the scheduling latency can be found here: https://bristot.me/demystifying-the-real-time-linux-latency/ Link: https://lkml.kernel.org/r/3d45f40e630317f51ac6d678e2d96d310e495729.1675179318.git.bristot@kernel.org Cc: Daniel Bristot de Oliveira <bristot@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-02-02rtla/timerlat: Add auto-analysis coreDaniel Bristot de Oliveira3-0/+1005
Currently, timerlat displays a summary of the timerlat tracer results saving the trace if the system hits a stop condition. While this represented a huge step forward, the root cause was not that is accessible to non-expert users. The auto-analysis fulfill this gap by parsing the trace timerlat runs, printing an intuitive auto-analysis. Link: https://lkml.kernel.org/r/1ee073822f6a2cbb33da0c817331d0d4045e837f.1675179318.git.bristot@kernel.org Cc: Daniel Bristot de Oliveira <bristot@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Signed-off-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-02-01tools/tracing/rtla: osnoise_hist: display average with two-digit precisionAndreas Ziegler1-2/+2
Calculate average value in osnoise-hist summary with two-digit precision to avoid displaying too optimitic results. Link: https://lkml.kernel.org/r/20230103103400.275566-3-br015@umbiko.net Signed-off-by: Andreas Ziegler <br015@umbiko.net> Acked-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2023-02-01tools/tracing/rtla: osnoise_hist: use total duration for average calculationAndreas Ziegler1-1/+4
Sampled durations must be weighted by observed quantity, to arrive at a correct average duration value. Perform calculation of total duration by summing (duration * count). Link: https://lkml.kernel.org/r/20230103103400.275566-2-br015@umbiko.net Fixes: 829a6c0b5698 ("rtla/osnoise: Add the hist mode") Signed-off-by: Andreas Ziegler <br015@umbiko.net> Acked-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2022-12-10rtla: Fix exit status when returning from calls to usage()John Kacur3-17/+13
rtla_usage(), osnoise_usage() and timerlat_usage() all exit with an error status. However when these are called from help, they should exit with a non-error status. Fix this by passing the exit status to the functions. Note, although we remove the subsequent call to exit after calling usage, we leave it in at the end of a function to suppress the compiler warning "control reaches end of a non-void function". Link: https://lkml.kernel.org/r/20221107144313.22470-1-jkacur@redhat.com Signed-off-by: John Kacur <jkacur@redhat.com> Acked-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2022-08-10rtla: Fix tracer nameAlexandre Vicenzi2-2/+2
The correct tracer name is timerlat and not timelat. Link: https://lore.kernel.org/linux-trace-devel/20220808180343.22262-1-alexandre.vicenzi@suse.com Signed-off-by: Alexandre Vicenzi <alexandre.vicenzi@suse.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
2022-08-01rtla: Define syscall numbers for riscvAndreas Schwab1-1/+1
RISC-V uses the same (generic) syscall numbers as ARM64. Link: https://lkml.kernel.org/r/mvma68wl2ul.fsf@suse.de Signed-off-by: Andreas Schwab <schwab@suse.de> Acked-by: Daniel Bristot de Oliveira <bristot@kernel.org> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
generated by cgit v1.2.3 (git 2.39.0) at 2026-01-21 03:32:00 +0300