[lkml]   [2020]   [Feb]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectPerformance impact in networking data path tests in Linux 5.5 Kernel
As part of VMware's performance regression testing for Linux Kernel upstream
releases, when comparing Linux 5.5 kernel against Linux 5.4 kernel, we noticed
20% improvement in networking throughput performance at the cost of a 30%
increase in the CPU utilization.

After performing the bisect between 5.4 and 5.5, we identified the root cause
of this behaviour to be a scheduling change from Vincent Guittot's
2ab4092fc82d ("sched/fair: Spread out tasks evenly when not overloaded").

The impacted testcases are TCP_STREAM SEND & RECV – on both small
(8K socket & 256B message) & large (64K socket & 16K message) packet sizes.

We backed out Vincent's commit & reran our networking tests and found that
the performance were similar to 5.4 kernel - improvements in networking tests
were no more.

In our current network performance testing, we use Intel 10G NIC to evaluate
all Linux Kernel releases. In order to confirm that the impact is also seen in
higher bandwidth NIC, we repeated the same test cases with Intel 40G and
we were able to reproduce the same behaviour - 25% improvements in
throughput with 10% more CPU consumption.

The overall results indicate that the new scheduler change has introduced
much better network throughput performance at the cost of incremental
CPU usage. This can be seen as expected behavior because now the
TCP streams are evenly spread across all the CPUs and eventually drives
more network packets, with additional CPU consumption.

We have also confirmed this theory by parsing the ESX stats for 5.4 and 5.5
kernels in a 4vCPU VM running 8 TCP streams - as shown below;

5.4 kernel:
  "2132149": {"id": 2132149, "used": 94.37, "ready": 0.01, "cstp": 0.00, "name": "vmx-vcpu-0:rhel7x64-0",
  "2132151": {"id": 2132151, "used": 0.13, "ready": 0.00, "cstp": 0.00, "name": "vmx-vcpu-1:rhel7x64-0",
  "2132152": {"id": 2132152, "used": 9.07, "ready": 0.03, "cstp": 0.00, "name": "vmx-vcpu-2:rhel7x64-0",
  "2132153": {"id": 2132153, "used": 34.77, "ready": 0.01, "cstp": 0.00, "name": "vmx-vcpu-3:rhel7x64-0",

5.5 kernel:
  "2132041": {"id": 2132041, "used": 55.70, "ready": 0.01, "cstp": 0.00, "name": "vmx-vcpu-0:rhel7x64-0",
  "2132043": {"id": 2132043, "used": 47.53, "ready": 0.01, "cstp": 0.00, "name": "vmx-vcpu-1:rhel7x64-0",
  "2132044": {"id": 2132044, "used": 77.81, "ready": 0.00, "cstp": 0.00, "name": "vmx-vcpu-2:rhel7x64-0",
  "2132045": {"id": 2132045, "used": 57.11, "ready": 0.02, "cstp": 0.00, "name": "vmx-vcpu-3:rhel7x64-0",

Note, "used %" in above stats for 5.5 kernel is evenly distributed across all vCPUs. 

On the whole, this change should be seen as a significant improvement for
most customers.

Rajender M
Performance Engineering
VMware, Inc.

 \ /
  Last update: 2020-02-25 06:47    [W:0.096 / U:0.156 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site