lkml.org 
[lkml]   [2018]   [Jul]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectOOM in secondary cgroup leading to networking lose
<div dir="ltr"><div><div class="gmail_signature"><div dir="ltr"><div>Hi friends,</div><div><br></div><div>We observed a sequence of events starting from an OOM in a secondary cgroup</div><div>which ends up in the bond interface being down for a period of up to 12</div><div>seconds. Below is some piece of dmesg about when the bond interface went down:</div><div><br></div><div>[Wed Jul 25 19:20:45 2018] Call Trace:</div><div>[Wed Jul 25 19:20:45 2018]  &lt;IRQ&gt;</div><div>[Wed Jul 25 19:20:45 2018]  ? dev_deactivate_queue.constprop.29+0x60/0x60</div><div>[Wed Jul 25 19:20:45 2018]  call_timer_fn+0x30/0x120</div><div>[Wed Jul 25 19:20:45 2018]  run_timer_softirq+0x3c8/0x420</div><div>[Wed Jul 25 19:20:45 2018]  ? timerqueue_add+0x52/0x80</div><div>[Wed Jul 25 19:20:45 2018]  ? enqueue_hrtimer+0x37/0x80</div><div>[Wed Jul 25 19:20:45 2018]  ? recalibrate_cpu_khz+0x10/0x10</div><div>[Wed Jul 25 19:20:45 2018]  __do_softirq+0xde/0x2b3</div><div>[Wed Jul 25 19:20:45 2018]  irq_exit+0xae/0xb0</div><div>[Wed Jul 25 19:20:45 2018]  smp_apic_timer_interrupt+0x70/0x120</div><div>[Wed Jul 25 19:20:45 2018]  apic_timer_interrupt+0x7d/0x90</div><div>[Wed Jul 25 19:20:45 2018]  &lt;/IRQ&gt;</div><div>[Wed Jul 25 19:20:45 2018] RIP: 0010:cpuidle_enter_state+0xa2/0x2e0</div><div>[Wed Jul 25 19:20:45 2018] RSP: 0018:ffffffff9c403eb0 EFLAGS: 00000246</div><div>ORIG_RAX: ffffffffffffff10</div><div>[Wed Jul 25 19:20:45 2018] RAX: ffff9075c0821a40 RBX: 0006cdb159b6000e RCX:</div><div>000000000000001f</div><div>[Wed Jul 25 19:20:45 2018] RDX: 0006cdb159b6000e RSI: ffed6f2696159a35 RDI:</div><div>0000000000000000</div><div>[Wed Jul 25 19:20:45 2018] RBP: ffffd3edc100b900 R08: 0000000000000f48 R09:</div><div>0000000000000cfe</div><div>[Wed Jul 25 19:20:45 2018] R10: ffffffff9c403e90 R11: 0000000000000f12 R12:</div><div>0000000000000003</div><div>[Wed Jul 25 19:20:45 2018] R13: ffffffff9c4b03d8 R14: 0000000000000000 R15:</div><div>0006cdb159783c8e</div><div>[Wed Jul 25 19:20:45 2018]  do_idle+0x181/0x1e0</div><div>[Wed Jul 25 19:20:45 2018]  cpu_startup_entry+0x19/0x20</div><div>[Wed Jul 25 19:20:45 2018]  start_kernel+0x400/0x408</div><div>[Wed Jul 25 19:20:45 2018]  secondary_startup_64+0xa5/0xb0</div><div>[Wed Jul 25 19:20:45 2018] Code: 63 8e 60 04 00 00 eb 8f 4c 89 f7 c6 05 79 c7</div><div>b8 00 01 e8 00 7c fd ff 89 d9 48 89 c2 4c 89 f6 48 c7 c7 f0 38 28 9c e8 c7 a3</div><div>b6 ff &lt;0f&gt; 0b eb bd 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 57 41 56 </div><div>[Wed Jul 25 19:20:45 2018] ---[ end trace 2ad2942fe3431402 ]---</div><div>[Wed Jul 25 19:20:45 2018] ixgbe 0000:19:00.1 eno2: initiating reset due to tx</div><div>timeout</div><div>[Wed Jul 25 19:20:45 2018] ixgbe 0000:19:00.1 eno2: Reset adapter</div><div>[Wed Jul 25 19:20:48 2018] ixgbe 0000:19:00.0 eno1: initiating reset due to tx</div><div>timeout</div><div>[Wed Jul 25 19:20:53 2018] ixgbe 0000:19:00.0 eno1: initiating reset due to tx</div><div>timeout</div><div><br></div><div>We have observed a similar behavior on a 4.15.11 kernel we were running on a</div><div>different machine, the current machine these logs are from runs a 4.14.52</div><div>kernel. A more detailed dmesg content can be found here:</div><div><br></div><div><a href="https://gist.github.com/bjhaid/49a1c58742ef2458984339503290ef9a">https://gist.github.com/bjhaid/49a1c58742ef2458984339503290ef9a</a></div><div><br></div><div>I will appreciate any help in figuring out the cause and fix of this issue.</div><div><br></div><div>Extra bit of information is the machines are Kubernetes nodes running docker.</div><div><br></div><div>Thanks!</div><div dir="ltr"><br></div>Abejide Ayodele<br><span style="color:rgb(85,85,85);font-family:&quot;Helvetica Neue&quot;,Helvetica,Arial,sans-serif;font-size:15px;line-height:22px">It always seems impossible until it&#39;s done. --Nelson Mandela</span><br></div></div></div>
</div>
\
 
 \ /
  Last update: 2018-07-26 00:18    [W:0.205 / U:0.004 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site