This message generated a parse failure. Raw output follows here. Please use 'back' to navigate. From devnull@lkml.org Sun May 19 13:29:41 2024 >From spaans Wed Dec 2 11:10:30 2015 Received: from pop.ziggo.nl [212.54.42.19] by squeeze.vs19.net with POP3 (fetchmail-6.3.21) for (single-drop); Wed, 02 Dec 2015 11:10:30 +0100 (CET) Received: from md4.tb.mail.iss.local ([212.54.42.148]) by mc11.tb.mail.iss.local (Dovecot) with LMTP id KegpAQ/DXlaaGQAA6JE+RA for ; Wed, 02 Dec 2015 11:08:51 +0100 Received: from mx20.tb.mail.iss.as9143.net ([212.54.42.148]) by md4.tb.mail.iss.local (Dovecot) with LMTP id cP/xPV0W1VVCZQAAkz+7wA ; Wed, 02 Dec 2015 11:09:53 +0100 Received: from vger.kernel.org ([209.132.180.67]) by mx20.tb.mail.iss.as9143.net with esmtp (Exim 4.82) (envelope-from ) id 1a44LC-0000pE-Nm for bmw12-lkml@ziggo.nl; Wed, 02 Dec 2015 11:08:51 +0100 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932642AbbLBKIj (ORCPT ); Wed, 2 Dec 2015 05:08:39 -0500 Received: from vserver.eikelenboom.it ([84.200.39.61]:43505 "EHLO smtp.eikelenboom.it" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932561AbbLBKIf (ORCPT ); Wed, 2 Dec 2015 05:08:35 -0500 Received: from localhost ([127.0.0.1]:47088 helo=roundcube.eikelenboom.it) by smtp.eikelenboom.it with esmtps (TLS1.2:DHE_RSA_3DES_EDE_CBC_SHA1:192) (Exim 4.80) (envelope-from ) id 1a44Gy-00063J-FU; Wed, 02 Dec 2015 11:04:28 +0100 Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=_b97d9e57cfeec0cbbc0ab78af02c5e7c" Date: Wed, 02 Dec 2015 11:04:28 +0100 From: Sander Eikelenboom To: Boris Ostrovsky Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xen.org, david.vrabel@citrix.com Subject: Re: [Xen-devel] linux 4.4 Regression: 100% cpu usage on idle pv guest under Xen with single vcpu. In-Reply-To: <565E301D.7020501@oracle.com> References: <80dcf47eb772c65a62652d7a56c8ed26@eikelenboom.it> <20151130214513.GF14317@char.us.oracle.com> <16a638d3c5e24963599e621f265181f8@eikelenboom.it> <565CD3A7.1060006@oracle.com> <1454b921884da94b7c1e9f434c13eb4a@eikelenboom.it> <565E239E.8070507@oracle.com> Message-Id: X-Sender: linux@eikelenboom.it User-Agent: Roundcube Webmail/0.9.5 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-Id: X-Mailing-List: linux-kernel@vger.kernel.org X-Ziggo-spamsetting: Instelling=Gemiddeld Scorelimiet=14 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.1 cv=YofyPNsX c=1 sm=0 tr=0 a=UK1r566ZdBxH71SXbqIOeA==:17 a=VwQbUJbxAAAA:8 a=cIF5Tx0qAAAA:8 a=wUQvQvOEmiQA:10 a=mLnsDVdbAAAA:8 a=F3vXizE51qwE89aUh7UA:9 a=QEXdDO2ut3YA:10 a=W4fCTVhc0GXqCqrhu5MA:9 a=l01A1z2Oxs-Fv8mRpLYA:9 a=W0vUJOdyAAAA: X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No --=_b97d9e57cfeec0cbbc0ab78af02c5e7c Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=UTF-8; format=flowed On 2015-12-02 00:41, Boris Ostrovsky wrote: > On 12/01/2015 06:30 PM, Sander Eikelenboom wrote: >> On 2015-12-02 00:19, Boris Ostrovsky wrote: >>> On 12/01/2015 06:00 PM, Sander Eikelenboom wrote: >>>> On 2015-12-01 23:47, Boris Ostrovsky wrote: >>>>> On 11/30/2015 05:55 PM, Sander Eikelenboom wrote: >>>>>> On 2015-11-30 23:54, Boris Ostrovsky wrote: >>>>>>> On 11/30/2015 04:46 PM, Sander Eikelenboom wrote: >>>>>>>> On 2015-11-30 22:45, Konrad Rzeszutek Wilk wrote: >>>>>>>>> On Sat, Nov 28, 2015 at 04:47:43PM +0100, Sander Eikelenboom >>>>>>>>> wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I have just tested a 4.4-rc2 kernel (current linus tree) + the >>>>>>>>>> tip tree >>>>>>>>>> pulled on top. >>>>>>>>>> >>>>>>>>>> Running this kernel under Xen on PV-guests with multiple vcpus >>>>>>>>>> goes well (on >>>>>>>>>> idle < 10% cpu usage), >>>>>>>>>> but a guest with only a single vcpu doesn't idle at all, it >>>>>>>>>> seems a kworker >>>>>>>>>> thread is stuck: >>>>>>>>>> root 569 98.0 0.0 0 0 ? R 16:02 12:47 >>>>>>>>>> [kworker/0:1] >>>>>>>>>> >>>>>>>>>> Running a 4.3 kernel works fine with a single vpcu, bisecting >>>>>>>>>> would probably >>>>>>>>>> quite painful since there were some breakages this merge >>>>>>>>>> window with respect >>>>>>>>>> to Xen pv-guests. >>>>>>>>>> >>>>>>>>>> There are some differences in the diff's from booting a 4.3, >>>>>>>>>> 4.4-single, >>>>>>>>>> 4.4-multi cpu boot: >>>>>>>>> >>>>>>>>> Boris has been tracking a bunch of them. I am attaching the >>>>>>>>> latest set of >>>>>>>>> patches I've to carry on top of v4.4-rc3. >>>>>>>> >>>>>>>> Hi Konrad, >>>>>>>> >>>>>>>> i will test those, see if it fixes all my issues and report back >>>>>>> >>>>>>> They shouldn't help you ;-( (and I just saw a message from you >>>>>>> confirming this) >>>>>>> >>>>>>> The first one fixes a 32-bit bug (on bare metal too). The second >>>>>>> fixes >>>>>>> a fatal bug for 32-bit PV guests. The other two are code >>>>>>> improvements/cleanup. >>>>>> >>>>>> One of these patches also fixes a bug i was having with a >>>>>> pci-passthrough device in >>>>>> a HVM that wasn't working (depending on which dom0-kernel i was >>>>>> using (4.3 or 4.4)), >>>>>> but didn't report yet. >>>>>> >>>>>> Fingers crossed but i think this pv-guest single vcpu issue is the >>>>>> last i'm troubled by for now ;) >>>>> >>>>> I could not reproduce this, including with your kernel config file. >>>> >>>> Hmm that's unpleasant :-\ >>>> >>>> Hmm other strange thing is it doesn't seem to affect dom0 (which is >>>> also a PV guest), but only unprivileged ones >>>> All unprivileged pv-guests seem to have the irq issue, but only with >>>> a single vcpu i see to get the stuck kworker thread that got my >>>> attention, with a 2 vcpu that doesn't seem to happen, but you still >>>> get the dmesg output and warnings about hvc) >>>> >>>> Could it be that: >>>> >>>> arch/x86/include/asm/i8259.h >>>> static inline int nr_legacy_irqs(void) >>>> { >>>> return legacy_pic->nr_legacy_irqs; >>>> } >>>> >>>> returns something different in some circumstances ? >>> >>> It should return 16 pre-8c058b0b9c34d8c8d7912880956543769323e2d8 and >>> 0 >>> after that commit. >>> >>> This is the last number that you see in >>> NR_IRQS:4352 nr_irqs:48 0 >>> line. >>> >>> I think you should be able to safely revert both >>> b4ff8389ed14b849354b59ce9b360bdefcdbf99c and >>> 8c058b0b9c34d8c8d7912880956543769323e2d8 and see if it makes any >>> difference. >>> >>> >>> -boris >>> >> >> That was already underway compiling :) >> >> And it does reveal that reverting both fixes the issue, no stuck >> kworker thread .. and no: >> genirq: Flags mismatch irq 8. 00000000 (hvc_console) vs. 00000000 >> (rtc0) >> hvc_open: request_irq failed with rc -16. > > > Let me try it again tomorrow. Can you post your guest config file, Xen > version and host HW (Intel or AMD)? 'xl info' maybe? > > -boris Hi Boris, A fresh new day .. a fresh new thought. If i look at the /proc/interrupts from a broken and a kernel with both commits the thing that catches the eye is irq8, just as the dmesg message was telling. In my PV guest rtc0 now seems to try and take irq8 that was already assigned to HVC ? Sounds like some assumptions around the legacy range are broken somewhere. What is the benefit of not just reserving the legacy range ? Attached the /proc/interrupts from both boots. -- Sander > >> >> What i did get was an conflict reverting >> b4ff8389ed14b849354b59ce9b360bdefcdbf99c: >> arch/arm64/include/asm/irq.h, although that shouldn't matter because >> we are on x86 and not on arm. >> >> -- Sander >> >> >>>> >>>> -- Sander >>>> >>>>> >>>>> -boris >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xen.org >>>> http://lists.xen.org/xen-devel --=_b97d9e57cfeec0cbbc0ab78af02c5e7c Content-Transfer-Encoding: base64 Content-Type: text/plain; name=interrupts-after-reverts.txt Content-Disposition: attachment; filename=interrupts-after-reverts.txt; size=1304 ICAgICAgICAgICBDUFUwICAgICAgIAogMTY6ICAgICAzMTU1MzYgIHhlbi1wZXJjcHUtdmlycSAg ICAgIHRpbWVyMAogMTc6ICAgICAgICAgIDAgIHhlbi1wZXJjcHUtaXBpICAgICAgIHNwaW5sb2Nr MAogMTg6ICAgICAgICAgIDAgIHhlbi1wZXJjcHUtaXBpICAgICAgIHJlc2NoZWQwCiAxOTogICAg ICAgICAgMCAgeGVuLXBlcmNwdS1pcGkgICAgICAgY2FsbGZ1bmMwCiAyMDogICAgICAgICAgMCAg eGVuLXBlcmNwdS12aXJxICAgICAgZGVidWcwCiAyMTogICAgICAgICAgMCAgeGVuLXBlcmNwdS1p cGkgICAgICAgY2FsbGZ1bmNzaW5nbGUwCiAyMjogICAgICAgICAgMCAgeGVuLXBlcmNwdS1pcGkg ICAgICAgaXJxd29yazAKIDIzOiAgICAgICAgMzQ2ICAgeGVuLWR5bi1ldmVudCAgICAgeGVuYnVz CiAyNDogICAgICAgIDEzNCAgIHhlbi1keW4tZXZlbnQgICAgIGh2Y19jb25zb2xlCiAyNTogICAg ICAxMTQ2NCAgIHhlbi1keW4tZXZlbnQgICAgIGJsa2lmCiAyNjogICAgICAyODcxMCAgIHhlbi1k eW4tZXZlbnQgICAgIGV0aDAtcTAtdHgKIDI3OiAgICAgIDQwMTM2ICAgeGVuLWR5bi1ldmVudCAg ICAgZXRoMC1xMC1yeApOTUk6ICAgICAgICAgIDAgICBOb24tbWFza2FibGUgaW50ZXJydXB0cwpM T0M6ICAgICAgICAgIDAgICBMb2NhbCB0aW1lciBpbnRlcnJ1cHRzClNQVTogICAgICAgICAgMCAg IFNwdXJpb3VzIGludGVycnVwdHMKUE1JOiAgICAgICAgICAwICAgUGVyZm9ybWFuY2UgbW9uaXRv cmluZyBpbnRlcnJ1cHRzCklXSTogICAgICAgICAgMCAgIElSUSB3b3JrIGludGVycnVwdHMKUlRS OiAgICAgICAgICAwICAgQVBJQyBJQ1IgcmVhZCByZXRyaWVzClJFUzogICAgICAgICAgMCAgIFJl c2NoZWR1bGluZyBpbnRlcnJ1cHRzCkNBTDogICAgICAgICAgMCAgIEZ1bmN0aW9uIGNhbGwgaW50 ZXJydXB0cwpUTEI6ICAgICAgICAgIDAgICBUTEIgc2hvb3Rkb3ducwpUUk06ICAgICAgICAgIDAg ICBUaGVybWFsIGV2ZW50IGludGVycnVwdHMKVEhSOiAgICAgICAgICAwICAgVGhyZXNob2xkIEFQ SUMgaW50ZXJydXB0cwpERlI6ICAgICAgICAgIDAgICBEZWZlcnJlZCBFcnJvciBBUElDIGludGVy cnVwdHMKTUNFOiAgICAgICAgICAwICAgTWFjaGluZSBjaGVjayBleGNlcHRpb25zCk1DUDogICAg ICAgICAgMCAgIE1hY2hpbmUgY2hlY2sgcG9sbHMKRVJSOiAgICAgICAgICAwCk1JUzogICAgICAg ICAgMApQSU46ICAgICAgICAgIDAgICBQb3N0ZWQtaW50ZXJydXB0IG5vdGlmaWNhdGlvbiBldmVu dApQSVc6ICAgICAgICAgIDAgICBQb3N0ZWQtaW50ZXJydXB0IHdha2V1cCBldmVudAo= --=_b97d9e57cfeec0cbbc0ab78af02c5e7c Content-Transfer-Encoding: base64 Content-Type: text/plain; name=interrupts-broken.txt Content-Disposition: attachment; filename=interrupts-broken.txt; size=1297 ICAgICAgICAgICBDUFUwICAgICAgIAogIDA6ICAgICAgMjkwNTMgIHhlbi1wZXJjcHUtdmlycSAg ICAgIHRpbWVyMAogIDE6ICAgICAgICAgIDAgIHhlbi1wZXJjcHUtaXBpICAgICAgIHNwaW5sb2Nr MAogIDI6ICAgICAgICAgIDAgIHhlbi1wZXJjcHUtaXBpICAgICAgIHJlc2NoZWQwCiAgMzogICAg ICAgICAgMCAgeGVuLXBlcmNwdS1pcGkgICAgICAgY2FsbGZ1bmMwCiAgNDogICAgICAgICAgMCAg eGVuLXBlcmNwdS12aXJxICAgICAgZGVidWcwCiAgNTogICAgICAgICAgMCAgeGVuLXBlcmNwdS1p cGkgICAgICAgY2FsbGZ1bmNzaW5nbGUwCiAgNjogICAgICAgICAgMCAgeGVuLXBlcmNwdS1pcGkg ICAgICAgaXJxd29yazAKICA3OiAgICAgICAgMzQyICAgeGVuLWR5bi1ldmVudCAgICAgeGVuYnVz CiAgODogICAgICAgIDE0MCAgIHhlbi1keW4tZXZlbnQgICAgIHJ0YzAKICA5OiAgICAgICAzMjM4 ICAgeGVuLWR5bi1ldmVudCAgICAgYmxraWYKIDEwOiAgICAgICAgMTE5ICAgeGVuLWR5bi1ldmVu dCAgICAgZXRoMC1xMC10eAogMTE6ICAgICAgICAxOTkgICB4ZW4tZHluLWV2ZW50ICAgICBldGgw LXEwLXJ4Ck5NSTogICAgICAgICAgMCAgIE5vbi1tYXNrYWJsZSBpbnRlcnJ1cHRzCkxPQzogICAg ICAgICAgMCAgIExvY2FsIHRpbWVyIGludGVycnVwdHMKU1BVOiAgICAgICAgICAwICAgU3B1cmlv dXMgaW50ZXJydXB0cwpQTUk6ICAgICAgICAgIDAgICBQZXJmb3JtYW5jZSBtb25pdG9yaW5nIGlu dGVycnVwdHMKSVdJOiAgICAgICAgICAwICAgSVJRIHdvcmsgaW50ZXJydXB0cwpSVFI6ICAgICAg ICAgIDAgICBBUElDIElDUiByZWFkIHJldHJpZXMKUkVTOiAgICAgICAgICAwICAgUmVzY2hlZHVs aW5nIGludGVycnVwdHMKQ0FMOiAgICAgICAgICAwICAgRnVuY3Rpb24gY2FsbCBpbnRlcnJ1cHRz ClRMQjogICAgICAgICAgMCAgIFRMQiBzaG9vdGRvd25zClRSTTogICAgICAgICAgMCAgIFRoZXJt YWwgZXZlbnQgaW50ZXJydXB0cwpUSFI6ICAgICAgICAgIDAgICBUaHJlc2hvbGQgQVBJQyBpbnRl cnJ1cHRzCkRGUjogICAgICAgICAgMCAgIERlZmVycmVkIEVycm9yIEFQSUMgaW50ZXJydXB0cwpN Q0U6ICAgICAgICAgIDAgICBNYWNoaW5lIGNoZWNrIGV4Y2VwdGlvbnMKTUNQOiAgICAgICAgICAw ICAgTWFjaGluZSBjaGVjayBwb2xscwpFUlI6ICAgICAgICAgIDAKTUlTOiAgICAgICAgICAwClBJ TjogICAgICAgICAgMCAgIFBvc3RlZC1pbnRlcnJ1cHQgbm90aWZpY2F0aW9uIGV2ZW50ClBJVzog ICAgICAgICAgMCAgIFBvc3RlZC1pbnRlcnJ1cHQgd2FrZXVwIGV2ZW50Cg== --=_b97d9e57cfeec0cbbc0ab78af02c5e7c-- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/