lkml.org 
[lkml]   [2005]   [Aug]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Where is the performance bottleneck?
On Wed, Aug 31 2005, Holger Kiehl wrote:
> On Wed, 31 Aug 2005, Jens Axboe wrote:
>
> >Nothing sticks out here either. There's plenty of idle time. It smells
> >like a driver issue. Can you try the same dd test, but read from the
> >drives instead? Use a bigger blocksize here, 128 or 256k.
> >
> I used the following command reading from all 8 disks in parallel:
>
> dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
>
> Here vmstat output (I just cut something out in the middle):
>
> procs -----------memory---------- ---swap-- -----io---- --system--
> ----cpu----^M
> r b swpd free buff cache si so bi bo in cs us sy id
> wa^M
> 3 7 4348 42640 7799984 9612 0 0 322816 0 3532 4987 0 22
> 0 78
> 1 7 4348 42136 7800624 9584 0 0 322176 0 3526 4987 0 23
> 4 74
> 0 8 4348 39912 7802648 9668 0 0 322176 0 3525 4955 0 22
> 12 66
> 1 7 4348 38912 7803700 9636 0 0 322432 0 3526 5078 0 23

Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.

> >You might want to try the same with direct io, just to eliminate the
> >costly user copy. I don't expect it to make much of a difference though,
> >feels like the problem is elsewhere (driver, most likely).
> >
> Sorry, I don't know how to do this. Do you mean using a C program
> that sets some flag to do direct io, or how can I do that?

I've attached a little sample for you, just run ala

# ./oread /dev/sdX

and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.

--
Jens Axboe

#include <stdio.h>
#include <stdlib.h>
#define __USE_GNU
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>

#define BS (131072)
#define ALIGN(buf) (char *) (((unsigned long) (buf) + 4095) & ~(4095))
#define BLOCKS (8192)

int main(int argc, char *argv[])
{
char *p;
int fd, i;

if (argc < 2) {
printf("%s: <dev>\n", argv[0]);
return 1;
}

fd = open(argv[1], O_RDONLY | O_DIRECT);
if (fd == -1) {
perror("open");
return 1;
}

p = ALIGN(malloc(BS + 4095));
for (i = 0; i < BLOCKS; i++) {
int r = read(fd, p, BS);

if (r == BS)
continue;
else {
if (r == -1)
perror("read");

break;
}
}

return 0;
}
\
 
 \ /
  Last update: 2005-08-31 18:25    [W:0.095 / U:1.988 seconds]
©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site