kernel/linux.git/tools/testing/selftests/tc-testing/tdc.py, branch v6.7

selftests/tc-testing: implement tdc parallel test run

2023-09-28T07:51:07+00:00

Use a Python process pool to run the tests in parallel. Not all tests can run in parallel, for instance tests that are not namespaced and tests that use netdevsim, as they can conflict with one another. The code logic will split the tests into serial and parallel. For the parallel tests, we build batches of 32 tests and queue each batch on the process pool. For the serial tests, they are queued as a whole into the process pool, which in turn executes them concurrently with the parallel tests. Even though the tests serialize on rtnl_lock in the kernel, this feature showed results with a ~3x speedup on the wall time for the entire test suite running in a VM: Before - 4m32.502s After - 1m19.202s Examples: In order to run tdc using 4 processes: ./tdc.py -J4 <...> In order to run tdc using 1 process: ./tdc.py -J1 <...> || ./tdc.py <...> Note that the kernel configuration will affect the speed of the tests, especially if such configuration slows down process creation and/or fork(). Tested-by: Davide Caratti Signed-off-by: Pedro Tammela Acked-by: Jamal Hadi Salim Signed-off-by: Paolo Abeni

selftests/tc-testing: localize test resources

2023-09-28T07:51:07+00:00

As of today, the current tdc architecture creates one netns and uses it to run all tests. This assumption was embedded into the nsPlugin which carried over as how the tests were written. The tdc tests are by definition self contained and can, theoretically, run in parallel. Even though in the kernel they will serialize over the rtnl lock, we should expect a significant speedup of the total wall time for the entire test suite, which is hitting close to 1100 tests at this point. A first step to achieve this goal is to remove sharing of global resources like veth/dummy interfaces and the netns. In this patch we 'localize' these resources on a per test basis. Each test gets it's own netns, VETH/dummy interfaces. The resources are spawned in the pre_suite phase, where tdc will prepare all netns and interfaces for all tests. This is done in order to avoid concurrency issues with netns / interfaces spawning and commands using them. As tdc progresses, the resources are deleted after each test finishes executing. Tests that don't use the nsPlugin still run under the root namespace, but are now required to manage any external resources like interfaces. These cannot be parallelized as their definition doesn't allow it. On the other hand, when using the nsPlugin, tests don't need to create dummy/veth interfaces as these are handled already. Tested-by: Davide Caratti Signed-off-by: Pedro Tammela Acked-by: Jamal Hadi Salim Signed-off-by: Paolo Abeni

selftests: tc-testing: add "depends_on" property to skip tests

2023-03-31T06:24:24+00:00

currently, users can skip individual test cases by means of writing "skip": "yes" in the scenario file. Extend this functionality, introducing 'dependsOn': it's optional property like "skip", but the value contains a command (for example, a probe on iproute2 to check if it supports a specific feature). If such property is present, tdc executes that command and skips the test when the return value is non-zero. Reviewed-by: Pedro Tammela Signed-off-by: Davide Caratti Signed-off-by: Jakub Kicinski

selftests: tc-testing: Add matchJSON to tdc

2022-10-27T03:22:33+00:00

This allows the use of a matchJSON field in tests to match against JSON output from the command under test, if that command outputs JSON. You specify what you want to match against as a JSON array or object in the test's matchJSON field. You can leave out any fields you don't want to match against that are present in the output and they will be skipped. An example matchJSON value would look like this: "matchJSON": [ { "Value": { "neighIP": { "family": 4, "addr": "AQIDBA==", "width": 32 }, "nsflags": 142, "ncflags": 0, "LLADDR": "ESIzRFVm" } } ] The real output from the command under test might have some extra fields that we don't care about for matching, and since we didn't include them in our matchJSON value, those fields will not be attempted to be matched. If everything we included above has the same values as the real command output, the test will pass. The matchJSON field's type must be the same as the command output's type, otherwise the test will fail. So if the command outputs an array, then the value of matchJSON must also be an array. If matchJSON is an array, it must not contain more elements than the command output's array, otherwise the test will fail. Signed-off-by: Jeremy Carter Signed-off-by: Victor Nogueira Acked-by: Jamal Hadi Salim Link: https://lore.kernel.org/r/20221024111603.2185410-1-victor@mojatatu.com Signed-off-by: Jakub Kicinski

selftests/tc-testing: add exit code

2021-12-03T11:46:41+00:00

Mark the summary result as FAIL to prevent from confusing the selftest framework if some of them are failed. Previously, the selftest framework always treats it as *ok* even though some of them are failed actually. That's because the script tdc.sh always return 0. # All test results: # # 1..97 # ok 1 83be - Create FQ-PIE with invalid number of flows # ok 2 8b6e - Create RED with no flags [...snip] # ok 6 5f15 - Create RED with flags ECN, harddrop # ok 7 53e8 - Create RED with flags ECN, nodrop # ok 8 d091 - Fail to create RED with only nodrop flag # ok 9 af8e - Create RED with flags ECN, nodrop, harddrop # not ok 10 ce7d - Add mq Qdisc to multi-queue device (4 queues) # Could not match regex pattern. Verify command output: # qdisc mq 1: root # qdisc fq_codel 0: parent 1:4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 # qdisc fq_codel 0: parent 1:3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 [...snip] # ok 96 6979 - Change quantum of a strict ETS band # ok 97 9a7d - Change ETS strict band without quantum # # # # ok 1 selftests: tc-testing: tdc.sh <<< summary result CC: Philip Li Reported-by: kernel test robot Signed-off-by: Li Zhijian Acked-by: Davide Caratti Acked-by: Jamal Hadi Salim Signed-off-by: David S. Miller

tc-testing: remove duplicate code in tdc.py

2020-04-08T01:38:55+00:00

In set_operation_mode() function remove duplicated check for args.list parameter, which is already done one line before. Signed-off-by: Roman Mashak Signed-off-by: David S. Miller

tc-testing: Clarify the use of tdc's -d option

2019-07-31T22:54:34+00:00

The -d command line argument to tdc requires the name of a physical device on the system where the tests will be run. If -d has not been used, tdc will skip tests that require a physical device. This patch is intended to better document what the -d option does and how it is used. Signed-off-by: Lucas Bates Signed-off-by: David S. Miller

tc-testing: Allow tdc plugins to see test case data

2019-07-09T21:07:20+00:00

Instead of only passing the test case name and ID, pass the entire current test case down to the plugins. This change allows plugins to start accepting commands and directives from the test cases themselves, for greater flexibility in testing. Signed-off-by: Lucas Bates Signed-off-by: David S. Miller

tc-testing: Restore original behaviour for namespaces in tdc

2019-06-25T19:57:29+00:00

This patch restores the original behaviour for tdc prior to the introduction of the plugin system, where the network namespace functionality was split from the main script. It introduces the concept of required plugins for testcases, and will automatically load any plugin that isn't already enabled when said plugin is required by even one testcase. Additionally, the -n option for the nsPlugin is deprecated so the default action is to make use of the namespaces. Instead, we introduce -N to not use them, but still create the veth pair. buildebpfPlugin's -B option is also deprecated. If a test cases requires the features of a specific plugin in order to pass, it should instead include a new key/value pair describing plugin interactions: "plugins": { "requires": "buildebpfPlugin" }, A test case can have more than one required plugin: a list can be inserted as the value for 'requires'. Signed-off-by: Lucas Bates Acked-by: Davide Caratti Tested-by: Nicolas Dichtel Signed-off-by: David S. Miller

tc-testing: Allow test cases to be skipped

2019-03-02T07:05:06+00:00

By adding a check for an optional key/value pair to the test case data, individual test cases may be skipped to prevent tdc from aborting a test run due to setup or teardown failure. If a test case is skipped, it will still appear in the results output to allow for a consistent number of executed tests in each run. However, the test will be marked as skipped. This support for skipping extends to any plugins that may generate additional results for each executed test. Signed-off-by: Lucas Bates Signed-off-by: David S. Miller