lkml.org 
[lkml]   [2020]   [Jun]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: RFC: KTAP documentation - expected messages
On Mon, Jun 22, 2020 at 7:47 PM David Gow <davidgow@google.com> wrote:
>
> On Mon, Jun 22, 2020 at 6:45 AM Frank Rowand <frowand.list@gmail.com> wrote:
> >
> > Tim Bird started a thread [1] proposing that he document the selftest result
> > format used by Linux kernel tests.
> >
> > [1] https://lore.kernel.org/r/CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com
> >
> > The issue of messages generated by the kernel being tested (that are not
> > messages directly created by the tests, but are instead triggered as a
> > side effect of the test) came up. In this thread, I will call these
> > messages "expected messages". Instead of sidetracking that thread with
> > a proposal to handle expected messages, I am starting this new thread.
>
> Thanks for doing this: I think there are quite a few tests which could
> benefit from something like this.
>
> I think there were actually two separate questions: what do we do with
> unexpected messages (most of which I expect are useless, but some of
> which may end up being related to an unexpected test failure), and how
> to have tests "expect" a particular message to appear. I'll stick to
> talking about the latter for this thread, but even there there's two
> possible interpretations of "expected messages" we probably want to
> explicitly distinguish between: a message which must be present for
> the test to pass (which I think best fits the "expected message"
> name), and a message which the test is likely to produce, but which
> shouldn't alter the result (an "ignored message"). I don't see much
> use for the latter at present, but if we wanted to do more things with
> messages and had some otherwise very verbose tests, it could
> potentially be useful.

+Dmitry Vyukov, I think you were interested in this for KASAN before
we went with the signalling approach. Any thoughts?

> The other thing I'd note here is that this proposal seems to be doing
> all of the actual message filtering in userspace, which makes a lot of
> sense for kselftest tests, but does mean that the kernel can't know if
> the test has passed or failed. There's definitely a tradeoff between
> trying to put too much needless string parsing in the kernel and
> having to have a userland tool determine the test results. The
> proposed KCSAN test suite[1] is using tracepoints to do this in the
> kernel. It's not the cleanest thing, but there's no reason KUnit or
> similar couldn't implement a nicer API around it.
>
> [1]: https://lkml.org/lkml/2020/6/22/1506
>
> > I implemented an API for expected messages that are triggered by tests
> > in the Devicetree unittest code, with the expectation that the specific
> > details may change when the Devicetree unittest code adapts the KUnit
> > API. It seems appropriate to incorporate the concept of expected
> > messages in Tim's documentation instead of waiting to address the
> > subject when the Devicetree unittest code adapts the KUnit API, since
> > Tim's document may become the kernel selftest standard.
>
> Is having a nice way to handle expected messages the only thing
> holding up porting this to KUnit?
>
> > Instead of creating a very long email containing multiple objects,
> > I will reply to this email with a separate reply for each of:
> >
> > The "expected messages" API implemention and use can be from
> > drivers/of/unittest.c in the mainline kernel.
> >
> > of_unittest_expect - A proof of concept perl program to filter console
> > output containing expected messages output
> >
> > of_unittest_expect is also available by cloning
> > https://github.com/frowand/dt_tools.git
> >
> > An example raw console output with timestamps and expect messages.
> >
> > An example of console output processed by filter program
> > of_unittest_expect to be more human readable. The expected
> > messages are not removed, but are flagged.
> >
> > An example of console output processed by filter program
> > of_unittest_expect to be more human readable. The expected
> > messages are removed instead of being flagged.
>
> Cheers,
> -- David

\
 
 \ /
  Last update: 2020-06-23 23:39    [W:1.737 / U:1.056 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site