Messages in this thread | | | From | Eric Dumazet <> | Date | Wed, 10 Apr 2019 07:16:21 -0700 | Subject | Re: [PATCH] net: add big honking pfmemalloc OOM warning |
| |
On Wed, Apr 10, 2019 at 3:20 AM Juha-Matti Tilli <juha-matti.tilli@foreca.com> wrote: > > A system administrator is not notified (except via an obscure SNMP > counter that most sysadmins don't know to look for) if packets are > dropped due to out-of-memory condition when SKBs use pfmemalloc > reserves. This can for example lead to NFS connections hanging on > high-volume systems. > > Implement a ratelimited big honking out of memory warning that directs > the sysadmin to bump up vm.min_free_kbytes in case this problem happens. > Our experience shows that with default vm.min_free_kbytes (90112, or > about 90 megabytes), NFS connections hang approximately once per day, > whereas with 901120 (default multiplied by 10) NFS connections never > hang. >
If NFS sessions hang, then there is a bug to eventually root cause and fix.
Just telling the user : Increase the limit is the same thing than admitting :
Our limit system or TCP or NFS stacks are broken and unable to recover, so lets disable the limit system and work around a more serious bug.
Maybe the bug is in a NIC driver, please share more details before adding yet another noisy signal in syslog
SNMP counters are per netns, and more useful in the modern computing era, where a host is shared by many different containers.
> > Signed-off-by: Juha-Matti Tilli <juha-matti.tilli@foreca.com> > Cc: Eric Dumazet <edumazet@google.com> > Cc: Rafael Aquini <aquini@redhat.com> > Cc: Murphy Zhou <xzhou@redhat.com> > Cc: Yongcheng Yang <yoyang@redhat.com> > Cc: Jianhong Yin <jiyin@redhat.com> > --- > net/core/filter.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/net/core/filter.c b/net/core/filter.c > index fc92ebc4e200..7d8ef239b9af 100644 > --- a/net/core/filter.c > +++ b/net/core/filter.c > @@ -99,6 +99,8 @@ int sk_filter_trim_cap(struct sock *sk, struct sk_buff *skb, unsigned int cap) > * helping free memory > */ > if (skb_pfmemalloc(skb) && !sock_flag(sk, SOCK_MEMALLOC)) { > + printk_ratelimited(KERN_WARNING > + "dropped packet due to out-of-memory condition, please bump up vm.min_free_kbytes\n"); > NET_INC_STATS(sock_net(sk), LINUX_MIB_PFMEMALLOCDROP); > return -ENOMEM; > } > -- > 2.17.1 >
| |