Messages in this thread | | | From | Thorsten Leemhuis <> | Subject | Insights from doing regression tracking for Linux 4.7 | Date | Mon, 1 Aug 2016 21:41:53 +0200 |
| |
In case anyone wonders if I regret doing regression tracking for Linux 4.7: No, that is not the case. It isn't really fun, but well, I didn't expect it to be ;-) But FWIW, find below a few thoughts about the whole regression tracking thing I thought might be good to write down and share while they are still fresh in my head.
The TLDR version: I currently think it would help a lot to have something like patchwork (https://github.com/getpatchwork/patchwork ) that is able to track regressions instead.
This is just a vague idea of mine right now I plan to investigate further into sooner or later (I'm just coming back from a holiday and have various conference talks to prepare -- besides doing my real job and regression tracking for 4.8).
Here is the long version with the details that got me thinking into above mentioned direction (in no particular order):
* Some regression tracking work (track date and place of initial report; are things proceeding, compile a report at least once a week, ...) feel time consuming and boring, because most of it is mechanical work a computer should do, as that's what they were designed for ;-)
* My work afaics helped to get a few regressions fixed that otherwise might have been forgotten; but quite a few times I noticed pull request mentioning fixes for regression that I had not been aware of; I wonder how many regressions are still out there because both me and the relevant subsystem maintainer missed them.
* To make regression tracking really work well the workload must be spread among multiple people. It would afaics be best to get developers and maintainers involved more and give them something at hand to make them track regressions in their area of work. But to make that happen we need to have some kind of regression tracking software somewhere; and to make developers and maintainers actually use it is has to make their life easier somehow (similar to how patch tracking in patchwork makes things easier at least for some people).
* It's hard to get aware of all regression, as they are reported to various places (LKML, bugzilla.kernel.org, as well as lots of other mailing lists and a few other bugzillas). So a human or automatic regression tracker need to get told about regressions. That currently only sometimes happens for various reasons (I'm new to this; some people prefer if their regressions are not mentioned in the spotlight; some developers don't like to deal with bugzilla; ...). An email alias or a dedicated mailing might be a step in the right direction; a better solution might be a computer program that semi-automatically picks up regressions in those places where they are reported currently.
* Manually created regression reports (that's how I did them) sometimes quickly get out of date; a more automated solution that compiles and publishes up2date reports on the web would be better.
* bugzilla.kernel.org is afaics not a good solution when it comes to track regressions that are reported in other places. Main reason: Make someone (maintainers or someone that tracks regression) create and update bug entries for regressions mentioned in other places is a solution that to me seems unlikely to fly due to the overhead bugzilla has. Yes, Rafael did it like that when he did regression tracking, but I'm currently think that time is better spend elsewhere; and having one issue discussed in two places can quickly lead to confusion. Another reason why bugzilla.kernel.org is afaics not a good solution: it sometimes confusing to have tracked regressions and new, not yet verified regressions in one place.
Those were my main thoughts on the whole thing; here are a few other:
* I'll try to do regression tracking for the 4.7 stable series (until 4.8 comes out), but doing it for mainline at least for 4.8 (not sure if I can continue my work after that point) has higher priority for now, as weeks only have 168 hours :-/
* For now I won't get into regression tracking for longterm kernels, as the time afaics is better spend on getting regression tracking on a more solid track.
* Right now I only poke people manually when I think something needs to be done to get things rolling again. Rafael sent automatic reminder mails (together with the weekly reports I think). Maybe that is something I should do, too; but OTOH it can quickly result in people ignoring those mails.
* I should have helped to get https://git.kernel.org/torvalds/c/262e2bfd7d1e1f1ee48b870e5dfabb87c06b975e more quickly to mainline as the commit that introduced the regression already had made it to the stable and broke things there for a few weeks (and there was an earlier patch from jthumshirn that should have fixed the issue, too) :-/
* I had hoped to spend more time helping users to (a) identify if their problems actually are regressions and (b) getting their regressions heard by the right people. :-/
* I didn't do any statistics yet on how much my work helped (some regressions I noticed were added and removed from my list between two reports and thus for outsiders never really were on the reports); for now that's not worth the work afaics.
* Something I avoided doing for now, because it's afaics quite a bit of work: make bugzilla work better. It works well for some issues, but not at all for others (most are somewhere in between). I for example suspect the list of default assignees needs a major overhaul.
* Something else I'd like to do some when: Make it easier for novice users to test for regressions. That among other means: improve the documentation so people that want to test find their way in; for example create docs that explain "how to test mainline rc kernels on the major distributions easily and without risk", "how to use localmodconfig" or "bisect regressions for dummies" in a central place would be nice to have. And maybe even form groups of people that do regression testing regularly; ideally together with Arch Linux, Fedora, openSUSE Tumbleweed, and other distros that regularly ship the latest Linux versions (and thus have a interest that new mainline versions do not have to many regressions).
That it for now. HTH.
Ciao, Thorsten
| |