Messages in this thread | | | Date | Fri, 12 Jul 2013 13:00:18 -0400 | From | Dave Jones <> | Subject | Re: Yet more softlockups. |
| |
On Fri, Jul 12, 2013 at 08:55:31AM -0700, Dave Hansen wrote:
> I mean that somebody turned 'active_events' on without actually wanting > perf to be on. I'd be curious how it got set to something nonzero. > Could you stick a WARN_ONCE() or printk_ratelimit() on the three sites > that modify it?
I'll add to my list of things to get to, but it probably won't be until post-weekend.
> > Here's a fun trick: > > > > trinity -c perf_event_open -C4 -q -l off > > > > Within about a minute, that brings any of my boxes to its knees. > > The softlockup detector starts going nuts, and then the box wedges solid. > > *But* dropping the perf sample rate has been really effective at keeping > me from hitting it, and I've had to use _lots_ of CPUs (60-160) doing > those NMIs at once to trigger the lockups. > > Being able to trigger it with so few CPUs is interesting though. I'll > try on my hardware.
I hacked trinity to pause for 24s after each call, and changed the kernel to taint on lockup (so that trinity would immediatly stop). My hope was to find the combination of perf_event_open calls that triggered this. After 12 hours, it's still ticking along. It's now done about a thousand more calls than is usually necessary to hit the bug. Which makes me wonder if this is timing related somehow.
Perhaps also worth noting that the majority of calls trinity makes will -EINVAL
Dave
| |