Messages in this thread | | | Date | Fri, 3 Nov 2006 03:23:35 -0500 (EST) | From | Brent Baccala <> | Subject | async I/O seems to be blocking on 2.6.15 |
| |
Hello -
I'm running 2.6.15 (Debian) on a Pentium M laptop, PCI attached ext3 filesystem.
I'm writing my first asynchronous I/O program, and for a while I thought I was really doing something wrong, but more and more I'm starting to conclude that the problem might be in the kernel.
Basically, I've narrowed things down to a test program which opens a large (700 MB) file in O_DIRECT mode and fires off 100 one MB async reads for the first 100 MB of data. The enqueues take about 5 seconds to complete, which is also about the amount of time this disk needs to read 100 MB, so I suspect that it's blocking.
I've gotten the POSIX AIO interface at least tolerably running using the GLIBC thread-based implementation, but I really want the native interface working.
I whittled the test program down to use system calls instead of the POSIX AIO library, and I'm attaching a copy. You put a big file at 'testfile' (it just reads it) and run the program:
baccala@debian ~/src/endgame$ time ./testaio Enqueues starting Enqueues complete
real 0m5.327s user 0m0.004s sys 0m0.740s baccala@debian ~/src/endgame$
Of that five seconds, it's almost all spent between the two "enqueues" messages.
If anybody can shed any light on this, I'd appreciate your feedback direct to cosine@freesoft.org (I don't read the list).
Thank you.
-bwb
Brent Baccala cosine@freesoft.org #define _GNU_SOURCE /* to get O_DIRECT */
#include <stdio.h> #include <stdlib.h> #include <string.h> /* for memset() */ #include <unistd.h> /* for _PC_REC_XFER_ALIGN */ #include <asm/unistd.h> #include <fcntl.h> #include <linux/aio_abi.h> #include <errno.h>
_syscall2(int, io_setup, int, maxevents, aio_context_t *, ctxp) _syscall3(int, io_submit, aio_context_t, ctx, long, nr, struct iocb **, iocbs)
#define NUMAIOS 100
#define BUFFER_BYTES (1<<20)
struct iocb iocb[NUMAIOS]; void *buffer[NUMAIOS];
aio_context_t aio_default_context;
main() { int fd; int i; int alignment; struct iocb * iocbp[1];
fd = open("testfile", O_RDONLY | O_DIRECT); alignment = fpathconf(fd, _PC_REC_XFER_ALIGN);
for (i=0; i<NUMAIOS; i++) { if (posix_memalign(&buffer[i], alignment, BUFFER_BYTES) != 0) { fprintf(stderr, "Can't posix_memalign\n"); } }
io_setup(1024, &aio_default_context);
fprintf(stderr, "Enqueues starting\n");
for (i=0; i<NUMAIOS; i++) {
memset(&iocb[i], 0, sizeof(struct iocb));
iocb[i].aio_lio_opcode = IOCB_CMD_PREAD; iocb[i].aio_fildes = fd; iocb[i].aio_buf = (unsigned long) buffer[i]; iocb[i].aio_nbytes = BUFFER_BYTES; iocb[i].aio_offset = BUFFER_BYTES * i; /* aiocb[i].aio_offset = 0; */
iocbp[0] = &iocb[i]; if (io_submit(aio_default_context, 1, iocbp) != 1) { perror(""); fprintf(stderr, "Can't enqueue aio_read %d\n", i); } }
fprintf(stderr, "Enqueues complete\n"); }
| |