Messages in this thread | | | From | Αλέξανδρος Παπαδογιαννάκης <> | Subject | FW: sched_setaffinity not working with kernel 2.6.32.15 | Date | Sun, 20 Jun 2010 22:10:18 +0300 |
| |
From: psxlover@hotmail.com To: mingo@elte.hu; peterz@infradead.org Subject: sched_setaffinity not working with kernel 2.6.32.15 Date: Sun, 20 Jun 2010 17:14:12 +0300
.ExternalClass .ecxhmmessage P {padding:0px;} .ExternalClass body.ecxhmmessage {font-size:10pt;font-family:Verdana;}
On kernel 2.6.32.15 after I execute sched_setaffinity even though sched_getaffinity returns the right cpus the threads of my program are running on different cpus.
First time I came across the bug was on a PC on my unive- rsity which had just been updated to Debian 'squeeze'. I was using a library calles hwloc to get information about the to- pology of the system and bind threads to cpus. The binding while it wasn't returning any error wasn't working (I used top to check which cpus where being used). After contacting with the developers of hwloc they suggested that the pro- blem is either in the kernel or in glibc. I made a virtual machine on my pc using the latest Debian 'squeeze' and the problem was there. So I tried an older kernel I found in the Synaptic Package Manager and my pro- gram was working fine. So the problem was in the kernel. I e-mailed hwloc mailing list again and he suggested I tried the vanilla kernels and use git bisect to find where the problem started. With 2.6.34 and 2.6.33.5 I had no problem but with 2.6.32.15 sched_setaffinity wasn't working. I used git bisect and in the end I got this:
c6fc81afa2d7ef2f775e48672693d8a0a8a7325d is the first bad commit commit c6fc81afa2d7ef2f775e48672693d8a0a8a7325d Author: John Wright <john.wright@hp.com> Date: Tue Apr 13 16:55:37 2010 -0600
sched: Fix a race between ttwu() and migrate_task() Based on commit e2912009fb7b715728311b0d8fe327a1432b3f79 upstream, but done differently as this issue is not present in .33 or .34 kernels due to rework in this area. If a task is in the TASK_WAITING state, then try_to_wake_up() is working on it, and it will place it on the correct cpu. This commit ensures that neither migrate_task() nor __migrate_task() calls set_task_cpu(p) while p is in the TASK_WAKING state. Otherwise, there could be two concurrent calls to set_task_cpu(p), resulting in the task's cfs_rq being inconsistent with its cpu. Signed-off-by: John Wright <john.wright@hp.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
:040000 040000 a9d18950d8edddb761c0266706f671f0e9a006fe 2c3a7d7d5e616ecc276b4e93f4b6e5162a9382c8 M kernel
I've attached a simple program that binds a few threads on the first cpu. With 2.6.32.15 when I run it I can see with System Monitor that more than one cpu is being used by the threads.
-------------------------- Alexandros Papadogiannakis _________________________________________________________________ Hotmail: Trusted email with Microsoft's powerful SPAM protection. https://signup.live.com/signup.aspx?id=60969 #ifndef _GNU_SOURCE #define _GNU_SOURCE 1 #endif
#include <stdio.h> //#include <sched.h> //#include <hwloc.h> #include <stdlib.h> #include <pthread.h> #include <unistd.h> #include <sys/types.h> #include <sys/syscall.h>
#define NR_CPUS 8 #define NR_THREADS 5
pid_t tid[NR_THREADS];
pthread_barrier_t init_start, barrier2;
void print_affinity(pthread_t pid, int threadno) { int i; cpu_set_t set; CPU_ZERO(&set); sched_getaffinity( pid, sizeof(set), &set);
for( i = 0; i < NR_CPUS; i++) { if (CPU_ISSET(i, &set)) printf("Cpu #%d is in %d thread's affinity\n", i, threadno); } }
void set_schedaffinity(pthread_t pid, int cpu) { cpu_set_t set; CPU_ZERO(&set); CPU_SET(cpu, &set); sched_setaffinity(pid, sizeof(set), &set); }
void set_threadaffinity(pthread_t pid, int cpu) { cpu_set_t set; CPU_ZERO(&set); CPU_SET(cpu, &set); pthread_setaffinity_np(pid, sizeof(set), &set); }
void * threadCode(void * threadNo) { long int thread = (long int) threadNo; //Set current thread id //tid[thread] = (pid_t) syscall (SYS_gettid); //Wait for all the threads to set their thread id //pthread_barrier_wait(&init_start); //Wait for main to set the affinity pthread_barrier_wait(&barrier2); //sleep(1); //Print affinity print_affinity(0,thread); //Do something so that we can see cpu being used while (1);
pthread_exit(0); }
int main(int argc , char *argv[]) { long int i; pthread_t pid[NR_THREADS]; // Initialize barriers if(pthread_barrier_init(&init_start, NULL, NR_THREADS)) { printf("Could not create a barrier\n"); exit(1); } if(pthread_barrier_init(&barrier2, NULL, NR_THREADS)) { printf("Could not create a barrier\n"); exit(1); } //Create threads for (i = 0; i < NR_THREADS; i++) pthread_create(&pid[i], NULL, threadCode, (void *) i); //Wait for threads to set their thread id //pthread_barrier_wait(&init_start); //Set affinity for (i = 0; i < NR_THREADS; i++) //set_schedaffinity(tid[i],1); set_threadaffinity(pid[i],0); //Tell the threads to continue after setting affinity //pthread_barrier_wait(&barrier2); getchar(); return 0; }
| |