Messages in this thread | | | Date | Tue, 9 Dec 2008 16:38:21 -0800 | From | "Ray Lee" <> | Subject | Fwd: 2.6.26.3 kernel - progressive slowdown over NFS |
| |
[ adding netdev, linux-nfs to cc: -- rbl ]
Hi,
We have a simple python program which keeps running a C loop to lstat NFS mounted directories. We are seeing some weird behavior w.r.t. the run-time of this program on 2.6.26.3 kernel vs 2.6.24 kernel.
The run-time of the following code increases over time on the 2.6.26.3 kernel, whereas remains flat (as expected) on the 2.6.24 kernel. [See attached graphs - B1.jpg and B2.jpg] Once the 2.6.26.3 machine gets into this state, we need to restart the box to get back to reasonable run-times. Is this a known issue ?
Setup :
Machine A (2.6.26) : exports NFS directory - /a/baz contains 10,000 directories bar0 ... bar9999 /a/baz *(rw,sync,no_root_squash,no_all_squash,subtree_check)
Machine B1 (2.6.26.3) : mounts NFS dir RO from A - graph B1.jpg 10.x.x.x:/a/baz on /baz type nfs (ro,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nointr,nolock,proto=udp,timeo=11,retrans=2,sec=sys,mountproto=udp,addr=10.x.x.x)
Machine B2 (2.6.24) : mounts NFS dir RO from A - graph B2.jpg 10.x.x.x:/a/baz on /baz type nfs (ro,vers=3,rsize=4096,wsize=4096,hard,nointr,nolock,proto=udp,timeo=11,retrans=2,sec=sys,addr=10.x.x.x)
Repro :
B1 and B2, run the following python program : {{{ #!/usr/bin/env python
import os import sys import time
while True: t1 = time.time() rv = os.system("/a.out 10000 >& /dev/null") # lstat the 10000 directories mounted via NFS t2 = time.time() print >> sys.stderr, "%.3f" % (t2 - t1), rv }}}
where a.out is the following C code :
{{{ #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #include <sys/stat.h> #include <unistd.h>
int main(int argc, char **argv) { int i, n = atoi(argv[1]); char filename[64]; struct stat statbuf;
for (i = 0; i < n; i++) { sprintf(filename, "/baz/bar%d", i); lstat(filename, &statbuf); }
printf("done\n"); return 0; } }}}
Attachments :
B1.jpg and B2.jpg - runtimes on the 2 client machines - X-axis : iteration number Y-axis : run-time
Config files : - for the machines B1, B2 and A
b1.2.6.26.3.config b2.2.6.24.config a.2.6.26.config
- P [unhandled content-type:image/jpeg][unhandled content-type:image/jpeg][unhandled content-type:application/octet-stream][unhandled content-type:application/octet-stream][unhandled content-type:application/octet-stream] | |