lkml.org 
[lkml]   [2012]   [Apr]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH] clocksource, prevent overflow in clocksource_cyc2ns


On 04/04/2012 09:08 PM, John Stultz wrote:
> On 04/04/2012 11:33 AM, Prarit Bhargava wrote:
>>> One idea might be to replace the cyc2ns w/ mult_frac in only the watchdog code.
>>> I need to think on that some more (and maybe have you provide some debug output)
>>> to really understand how that's solving the issue for you, but it would be able
>>> to be done w/o affecting the other assumptions of the timekeeping core.
>>>
>> Hey John,
>>
>> After reading the initial part of your reply I was thinking about calling
>> mult_frac() directly from the watchdog code as well.
>>
>> Here's some debug output I cobbled together to get an idea of how quickly the
>> overflow was happening.
>>
>> [ 5.435323] clocksource_watchdog: {0} cs tsc csfirst 227349443638728 mask
>> 0xFFFFFFFFFFFFFFFF mult 797281036 shift 31
>> [ 5.444930] clocksource_watchdog: {0} wd hpet wdfirst 78332535 mask
>> 0xFFFFFFFF mult 292935555 shift 22
>>
>> These, of course, are just the basic data from the clocksources tsc and hpet.
>
> If I'm doing the math right, these are ~2.7 Ghz cpus?

Yes.

>
> So what kernel version are you using?

I was on an earlier version of Fedora (F16) ... but I'll jump forward and see if
I can still hit it.

>
> In trying to reproduce this locally against Linus' HEAD on a much smaller system
> (single core + HT 1.6Ghz), I got:
> [ 6.611366] clocksource_watchdog: {0} cs tsc csfirst 36177888648 mask
> ffffffffffffffff mult 10485747 shift 24
> [ 6.611596] clocksource_watchdog: {0} wd hpet wdfirst 169168400 mask ffffffff
> mult 2684354560 shift 26
>
> Note the smaller shift values. Not too long ago the shift calculation was
> adjusted to allow for longer periods between interrupts, so I suspect you're on
> an older kernel.
>
> Further, using your debug patch on my system, it was well beyond 10 minutes
> before the debug overflow occurred. And similarly I couldn't trip the watchdog
> trigger using sysrq-t (but again, only two threads here, so not nearly as much
> data to print as you have).

I'm going to try this on a 32-cpu system (running the previously mentioned test)
with linux.git HEAD.

>
> Could you verify that the issue you're seeing is still is present w/ current
> mainline? Please don't take this as me dismissing your problem! As I mentioned

Absolutely :) I didn't take it that way at all. .... when I get in this AM I'll
bang out a test and see if I can cause this to happen with sysrq-t. Keep in
mind that 10000 threads is the *minimum* I was able to cause this with, which is
only ~315 threads/cpu, which isn't a lot :/. At that number of threads the dump
takes about 6 mins. Doubling it, IIRC, exceeded 10 mins.
> earlier there are some known issues w/ the clocksource watchdog code. But I want
> to narrow down if you're problem is currently present in mainline or only in
> older kernels, as that will help us find the proper fix.

Thanks John,

P.

>
> thanks
> -john
>


\
 
 \ /
  Last update: 2012-04-05 13:03    [from the cache]
©2003-2011 Jasper Spaans