Messages in this thread | | | Subject | Re: Bug: fio traps into kernel without exiting because futex has a deadloop | From | "Zhang, Yanmin" <> | Date | Thu, 11 Jun 2009 16:33:16 +0800 |
| |
On Thu, 2009-06-11 at 08:18 +0200, Peter Zijlstra wrote: > On Thu, 2009-06-11 at 07:55 +0200, Peter Zijlstra wrote: > > On Thu, 2009-06-11 at 11:08 +0800, Zhang, Yanmin wrote: > > > I investigate a fio hang issue. When I run fio multi-process > > > testing on many disks, fio traps into kernel and doesn't exit > > > (mostly hit once after runing sub test cases for hundreds of times). > > > > > > Oprofile data shows kernel consumes time with some futex functions. > > > Command kill couldn't kill the process and machine reboot also hangs. > > > > > > Eventually, I locate the root cause as a bug of futex. Kernel enters > > > a deadloop between 'retry' and 'goto retry' in function futex_wake_op. > > > By unknown reason (might be an issue of fio or glibc), parameter uaddr2 > > > points to an area which is READONLY. So futex_atomic_op_inuser returns > > > -EFAULT when trying to changing the data at uaddr2, but later get_user > > > still succeeds becasue the area is READONLY. Then go back to retry. > > > > > > I create a simple test case to trigger it, which just shmat an READONLY > > > area for address uaddr2. > > > > > > It could be used as a DOS attack. > > /me has morning juice and notices he sent the wrong commit... > > commit 64d1304a64477629cb16b75491a77bafe6f86963 > Author: Thomas Gleixner <tglx@linutronix.de> > Date: Mon May 18 21:20:10 2009 +0200 2.6.30 includes the new commit. I did a quick testing with my simple test case and it traps into kernel without exiting.
The reason is I use flag FUTEX_PRIVATE_FLAG. So the fshared part in function get_futex_key should be deleted. That might hurt performance.
Yanmin
#include <stdio.h> #include <stdlib.h> #include <linux/futex.h> #include <sys/time.h> #define _GNU_SOURCE /* or _BSD_SOURCE or _SVID_SOURCE */ #include <unistd.h> #include <sys/syscall.h> /* For SYS_xxx definitions */ #include <sys/types.h> #include <sys/shm.h> #include <sys/types.h> #include <sys/mman.h> #include <errno.h> #include <sys/stat.h> #include <fcntl.h> #include <sys/wait.h> #include <sys/utsname.h>
#define PAGE_SIZE (4096) int addr1=1;
int my_shmget(key_t key, int page_count, int *shmid, void **shmaddr) { int i, j, k; void *start_addr = NULL;
if ((*shmid =shmget(key, PAGE_SIZE*page_count, IPC_CREAT|0666 )) < 0) { perror("Failure:"); return -1; }
*shmaddr = shmat(*shmid, start_addr, SHM_RDONLY) ; if (*shmaddr == (void *) -1) { perror("shmget:Shared Memory Attach Failure:"); shmctl(*shmid, IPC_RMID, NULL); return -1; }
return 0; }
int my_shmput(int shmid, void *shmaddr) { if (shmdt((const void *)shmaddr) != 0) { perror("Detached Failure:"); return -1; } if(shmctl(shmid, IPC_RMID, NULL) != 0) { perror("Remove shm id of htlb page failure!\n"); return -1; }
return 0; }
int main() { int * uaddr = &addr1, *uaddr2; void * lp; int ret; int shmid; void *shmaddr;
if(my_shmget(10673861, 10, &shmid, &shmaddr)) exit(0);
uaddr2 = shmaddr;
//uaddr2 = 0;
ret = syscall(__NR_futex, uaddr, FUTEX_WAKE_OP|FUTEX_PRIVATE_FLAG, 1, NULL, uaddr2, 1);
printf("ret=%d\n", ret);
my_shmput(shmid, shmaddr);
return 0; }
| |