[lkml]   [2014]   [Apr]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subject[regression, 3.15-rc1] vdso_gettimeofday hogs all my CPU
Hi Peter,

I'm guessing that x86 vdso problems are in your area of expertise,
if not can you point me at the right person to bug?

I just spun up my XFS performance tests on 3.15-rc1, the first of
which runs a concurrent fsmark workload. fsmark uses gettimeofday to
do per-operation timing and performance is horrendous. 3.14.0 on this
workload does around 300,000 file creates per second. 3.15-rc1

# ./fs_mark -D 10000 -S0 -n 100000 -s 0 -L 32 -d /mnt/scratch/0 -d /mnt/scratch/1 -d /mnt/scratch/2 -d /mnt/scratch/3 -d /mnt/scratch/4 -d /mnt/scratch/5 -d /mnt/scratch/6 -d /mnt/scratch/7 -d /mnt/scratch/8 -d /mnt/scratch/9 -d /mnt/scratch/10 -d /mnt/scratch/11 -d /mnt/scratch/12 -d /mnt/scratch/13 -d /mnt/scratch/14 -d /mnt/scratch/15
# Version 3.3, 16 thread(s) starting at Tue Apr 15 17:37:20 2014
# Sync method: NO SYNC: Test does not issue sync() or fsync() calls.
# Directories: Time based hash between directories across 10000 subdirectories with 180 seconds per subdirectory.
# File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name)
# Files info: size 0 bytes, written with an IO size of 16384 bytes per write
# App overhead is time in microseconds spent in the test not doing file writing related system calls.

FSUse% Count Size Files/sec App Overhead
0 1600000 0 70950.5 155219877
0 3200000 0 71832.4 150138649
0 4800000 0 71195.5 150935842
0 6400000 0 71215.4 150906768
0 8000000 0 63707.7 166569461

about 70,000 files/sec, so it's way, way down on 3.14. The load has
pegged all 16 CPUs in the VM, and perf top tells me:

- 63.11% [vdso] [.] __vdso_gettimeofday
- __vdso_gettimeofday
44.24% stop
28.18% do_run
13.98% write_file
13.60% setup_file_name

that it's the gettimeofday calls that are causing the excessive CPU

I haven't changed anything in userspace for the past couple of
months, the only thing that has changed is the kernel. I haven't
looked at this any further and have no idea what to do to debug it
further, so let me know if you need more information....


Dave Chinner

 \ /
  Last update: 2014-04-15 10:41    [W:0.028 / U:0.688 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site