lkml.org 
[lkml]   [2021]   [Feb]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: [PATCH v3 0/3] kunit: support running subsets of test suites from kunit.py
On Thu, Feb 4, 2021 at 2:54 PM Daniel Latypov <dlatypov@google.com> wrote:
>
> When using `kunit.py run` to run tests, users must populate a
> `kunitconfig` file to select the options the tests are hidden behind and
> all their dependencies.
>
> The patch [1] to allow specifying a path to kunitconfig promises to make
> this nicer as we can have checked in files corresponding to different
> sets of tests.
>
> But it's still annoying
> 1) when trying to run a subet of tests
> 2) when you want to run tests that don't have such a pre-existing
> kunitconfig and selecting all the necessary options is tricky.
>
> This patch series aims to alleviate both:
> 1) `kunit.py run 'my-suite-*'`
> I.e. use my current kunitconfig, but just run suites that match this glob
> 2) `kunit.py run --alltests 'my-suite-*'`
> I.e. use allyesconfig so I don't have to worry about writing a
> kunitconfig at all.
>
> See the first commit message for more details and discussion about
> future work.
>
> This patch series also includes a bugfix for a latent bug that can't be
> triggered right now but has worse consequences as a result of the
> changes needed to plumb in this suite name glob.
>
> [1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlatypov@google.com/

I tried applying this series on top of several of your other patches
over the past cycle and running the kunit_tool unit tests
(kunit_tool_test.py) and I got several failures:

======================================================================
FAIL: test_exec_builddir (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tools/testing/kunit/kunit_tool_test.py", line 417, in test_exec_builddir
self.linux_source_mock.run_kernel.assert_called_once_with(build_dir=build_dir,
timeout=300)
File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
return self.assert_called_with(*args, **kwargs)
File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_exec_passes_args_pass (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tools/testing/kunit/kunit_tool_test.py", line 337, in
test_exec_passes_args_pass
self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit',
timeout=300)
File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
return self.assert_called_with(*args, **kwargs)
File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_exec_timeout (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tools/testing/kunit/kunit_tool_test.py", line 385, in test_exec_timeout
self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit',
timeout=timeout)
File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
return self.assert_called_with(*args, **kwargs)
File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=3453)
Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_builddir (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tools/testing/kunit/kunit_tool_test.py", line 400, in test_run_builddir
self.linux_source_mock.run_kernel.assert_called_once_with(
File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
return self.assert_called_with(*args, **kwargs)
File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_passes_args_pass (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tools/testing/kunit/kunit_tool_test.py", line 344, in
test_run_passes_args_pass
self.linux_source_mock.run_kernel.assert_called_once_with(
File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
return self.assert_called_with(*args, **kwargs)
File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=300)
Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')

======================================================================
FAIL: test_run_timeout (__main__.KUnitMainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tools/testing/kunit/kunit_tool_test.py", line 392, in test_run_timeout
self.linux_source_mock.run_kernel.assert_called_once_with(
File "/usr/lib/python3.8/unittest/mock.py", line 925, in
assert_called_once_with
return self.assert_called_with(*args, **kwargs)
File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with
raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: run_kernel(build_dir='.kunit', timeout=3453)
Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit')

----------------------------------------------------------------------
Ran 37 tests in 0.082s

FAILED (failures=6)

\
 
 \ /
  Last update: 2021-02-06 04:37    [W:0.046 / U:0.420 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site