This message generated a parse failure. Raw output follows here. Please use 'back' to navigate. From devnull@lkml.org Fri Apr 19 07:34:20 2024 >From spaans Mon May 23 16:12:54 2016 Envelope-to: lkml@grols.ch Delivery-date: Mon, 23 May 2016 16:04:21 +0200 Received: from srv.grols.ch [2a00:d10:4002:1::101] by squeeze.vs19.net with IMAP (fetchmail-6.3.21) for (single-drop); Mon, 23 May 2016 16:12:54 +0200 (CEST) Received: from vger.kernel.org ([209.132.180.67]) by home.grols.ch with esmtp (Exim 4.87) (envelope-from ) id 1b4qSx-0002CK-G4 for lkml@grols.ch; Mon, 23 May 2016 16:04:21 +0200 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932500AbcEWOER (ORCPT ); Mon, 23 May 2016 10:04:17 -0400 Received: from e18.ny.us.ibm.com ([129.33.205.208]:46245 "EHLO e18.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932440AbcEWOEQ convert rfc822-to-8bit (ORCPT ); Mon, 23 May 2016 10:04:16 -0400 Received: from localhost by e18.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 23 May 2016 10:04:15 -0400 Received: from d01dlp03.pok.ibm.com (9.56.250.168) by e18.ny.us.ibm.com (146.89.104.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Mon, 23 May 2016 10:04:12 -0400 X-IBM-Helo: d01dlp03.pok.ibm.com X-IBM-MailFrom: paulmck@linux.vnet.ibm.com X-IBM-RcptTo: linux-kernel@vger.kernel.org Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id 491B6C90045 for ; Mon, 23 May 2016 10:04:05 -0400 (EDT) Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u4NE4Cja20709460 for ; Mon, 23 May 2016 14:04:12 GMT Received: from d01av01.pok.ibm.com (localhost [127.0.0.1]) by d01av01.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u4NE4ARm003331 for ; Mon, 23 May 2016 10:04:12 -0400 Received: from paulmck-ThinkPad-W541 (sig-9-48-104-62.ibm.com [9.48.104.62]) by d01av01.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u4NE49w6003304; Mon, 23 May 2016 10:04:10 -0400 Received: by paulmck-ThinkPad-W541 (Postfix, from userid 1000) id 6B5D016C1384; Mon, 23 May 2016 07:04:10 -0700 (PDT) Date: Mon, 23 May 2016 07:04:10 -0700 From: "Paul E. McKenney" To: peterz@infradead.org Cc: linux-kernel@vger.kernel.org, mgalbraith@suse.de, ahh@google.com, bsegall@google.com, torvalds@linux-foundation.org, matt@codeblueprint.co.uk, morten.rasmussen@arm.com, pjt@google.com, pkondeti@infradead.org, tglx@linutronix.de, byungchul.park@lge.com, m Subject: Bisection: Lost wakeups from b5179ac70de8 Message-Id: <20160523140410.GA26287@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: 8BIT User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16052314-0045-0000-0000-00000444E03F X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-Id: X-Mailing-List: linux-kernel@vger.kernel.org Received-SPF: none client-ip=209.132.180.67; envelope-from=linux-kernel-owner@vger.kernel.org; helo=vger.kernel.org X-Spam-Score: 2.4 X-Spam-Score-Bar: ++ X-Spam-Action: no action X-Spam-Report: Action: no action Symbol: URIBL_BLOCKED(0.00) Symbol: FORGED_RECIPIENTS(2.00) Symbol: BROKEN_HEADERS(1.00) Symbol: FORGED_SENDER(0.30) Symbol: BAYES_HAM(-0.83) Symbol: MIME_GOOD(-0.10) Message-ID: 20160523140410.GA26287@linux.vnet.ibm.com Hello, Peter, Current mainline doesn't do well with RCU torture testing, and the symptom once again looks like lost wakeups. Thankfully, this time each run takes only about an hour, and the false-positive/-negative rate is negligible. This means that for the first time ever, "git bisect" actually did something useful for me. The first bad commit is: b5179ac70de8 ("sched/fair: Prepare to fix fairness problems on migration"). This patch also seems to result in softlockups: http://permalink.gmane.org/gmane.linux.kernel/2225784 As always, this was on a virtualized environment, but the exact same environment was used for both the good and the bad runs. Sadly, the much lower-probability bugs appear to be alive and well, but I guess you can't have everything. Please let me know if there is additional debug information that would be useful to you. Thanx, Paul PS: For whatever it is worth, the summary output from a good run and a bad run is shown below. ------------------------------------------------------------------------ Good run: ------------------------------------------------------------------------ Results directory: /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-21:12:31 tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 54 --duration 20 SRCU-N ------- 31383 grace periods (26.1525 per second) SRCU-P ------- 451 grace periods (0.375833 per second) TASKS01 ------- 820 grace periods (0.683333 per second) TASKS02 ------- 845 grace periods (0.704167 per second) TASKS03 ------- 766 grace periods (0.638333 per second) TINY01 ------- 137261 grace periods (114.384 per second) TINY02 ------- 386191 grace periods (321.826 per second) TREE01 ------- 60501 grace periods (50.4175 per second) TREE02 ------- 34814 grace periods (29.0117 per second) TREE03 ------- 14581 grace periods (12.1508 per second) no success message, 81 successful version messages !!! PID 40003 hung at 1380 vs. 1200 seconds TREE04 ------- 7856 grace periods (6.54667 per second) TREE05 ------- 39473 grace periods (32.8942 per second) TREE06 ------- 38812 grace periods (32.3433 per second) TREE07 ------- 6874 grace periods (5.72833 per second) TREE08 ------- 27803 grace periods (23.1692 per second) TREE09 ------- 88783 grace periods (73.9858 per second) ------------------------------------------------------------------------ Bad run: ------------------------------------------------------------------------ Results directory: /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11 tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 54 --duration 20 SRCU-N ------- 5108 grace periods (4.25667 per second) no success message, 14 successful version messages WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/SRCU-N/console.log WARNING: Summary: Call Traces: 116 Stalls: 58 !!! PID 40837 hung at 1380 vs. 1200 seconds SRCU-P ------- 580 grace periods (0.483333 per second) WARNING: BAD SEQ 384:384 last:578 version 33 /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/SRCU-P/console.log WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/SRCU-P/console.log WARNING: Summary: Call Traces: 1 Stalls: 11 TASKS01 ------- [ grace periods (0 per second) no success message, 1 successful version messages WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TASKS01/console.log WARNING: Summary: Call Traces: 36 !!! PID 40965 hung at 1380 vs. 1200 seconds TASKS02 ------- 831 grace periods (0.6925 per second) TASKS03 ------- 14 grace periods (0.0116667 per second) no success message, 4 successful version messages WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TASKS03/console.log WARNING: Summary: Call Traces: 10 !!! PID 15654 hung at 1380 vs. 1200 seconds TINY01 ------- 140482 grace periods (117.068 per second) TINY02 ------- 416484 grace periods (347.07 per second) TREE01 ------- 17599 grace periods (14.6658 per second) WARNING: BAD SEQ 2251:2251 last:17598 version 19 /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE01/console.log WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE01/console.log WARNING: Summary: Stalls: 4 TREE02 ------- 15117 grace periods (12.5975 per second) TREE03 ------- 4460 grace periods (3.71667 per second) WARNING: BAD SEQ 1846:1846 last:4456 version 29 /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE03/console.log WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE03/console.log WARNING: Summary: Stalls: 2 !!! PID 40927 hung at 1380 vs. 1200 seconds TREE04 ------- 1247 grace periods (1.03917 per second) WARNING: BAD SEQ 134:134 last:1247 version 6 /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE04/console.log WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE04/console.log WARNING: Summary: Stalls: 7 TREE05 ------- 5543 grace periods (4.61917 per second) WARNING: BAD SEQ 1854:1854 last:5542 version 28 /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE05/console.log WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE05/console.log WARNING: Summary: Call Traces: 2 Stalls: 28 TREE06 ------- 15510 grace periods (12.925 per second) WARNING: BAD SEQ 5445:5445 last:15457 version 29 /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE06/console.log WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE06/console.log WARNING: Summary: Stalls: 2 TREE07 ------- 857 grace periods (0.714167 per second) no success message, 78 successful version messages WARNING: BAD SEQ 211:211 last:857 version 9 WARNING: Assertion failure in /home/paulmck/public_git/bisect-linux-rcu/tools/testing/selftests/rcutorture/res/2016.05.22-17:22:11/TREE07/console.log WARNING: Summary: Stalls: 36 !!! PID 40988 hung at 1380 vs. 1200 seconds TREE08 ------- 7868 grace periods (6.55667 per second) TREE09 ------- 89784 grace periods (74.82 per second)