lkml.org 
[lkml]   [2020]   [Feb]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectWriting to FUSE via mmap extremely slow (sometimes) on some machines?
Hey,

I’m running into an issue where writes via mmap are extremely slow.
The mmap program I’m using to test is attached.

The symptom is that the program usually completes in 0.x seconds, but
then sometimes takes minutes to complete! E.g.:

% dd if=/dev/urandom of=/tmp/was bs=1M count=99

% ./fusexmp_fh /tmp/mnt

% time ~/mmap /tmp/was /tmp/mnt/tmp/stapelberg.1
Mapped src: 0x10000 and dst: 0x21b8b000
memcpy done
~/mmap /tmp/was /tmp/mnt/tmp/stapelberg.1 0.06s user 0.20s system 48%
cpu 0.544 total

% time ~/mmap /tmp/was /tmp/mnt/tmp/stapelberg.1
Mapped src: 0x10000 and dst: 0x471fb000
memcpy done
~/mmap /tmp/was /tmp/mnt/tmp/stapelberg.1 0.05s user 0.22s system 0%
cpu 2:03.39 total

This affects both an in-house FUSE file system and also FUSE’s
fusexmp_fh from 2.9.7 (matching what our in-house FS uses).

While this is happening, the machine is otherwise idle. E.g. dstat shows:

--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read writ| recv send| in out | int csw
1 0 98 1 0| 0 0 | 19k 23k| 0 0 | 14k 27k
1 0 98 1 0| 0 0 | 33k 53k| 0 0 | 14k 29k
0 0 98 1 0| 0 176k| 27k 26k| 0 0 | 13k 25k
[…]

While this is happening, using cp(1) to copy the same file is fast (1
second). It’s only mmap-based writing that’s slow.

This is with Linux 5.2.17, but has been going on for years apparently.

I haven’t quite figured out what the pattern is with regards to the
machines that are affected. One wild guess I have is that it might be
related to RAM? The machine on which I can most frequently reproduce
the issue has 192GB of RAM, whereas I haven’t been able to reproduce
the issue on my workstation with 64GB of RAM.

Any ideas what I could check to further narrow down this issue?

Thanks,
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

/*
* An implementation of copy ("cp") that uses memory maps. Various
* error checking has been removed to promote readability
*/

// Where we want the source file's memory map to live in virtual memory
// The destination file resides immediately after the source file
#define MAP_LOCATION 0x6100

int main (int argc, char *argv[]) {
int fdin, fdout;
char *src, *dst;
struct stat statbuf;
off_t fileSize = 0;

if (argc != 3) {
printf ("usage: a.out <fromfile> <tofile>\n");
exit(0);
}

/* open the input file */
if ((fdin = open (argv[1], O_RDONLY)) < 0) {
printf ("can't open %s for reading\n", argv[1]);
exit(0);
}

/* open/create the output file */
if ((fdout = open (argv[2], O_RDWR | O_CREAT | O_TRUNC, 0600)) < 0) {
printf ("can't create %s for writing\n", argv[2]);
exit(0);
}

/* find size of input file */
fstat (fdin,&statbuf) ;
fileSize = statbuf.st_size;

/* go to the location corresponding to the last byte */
if (lseek (fdout, fileSize - 1, SEEK_SET) == -1) {
printf ("lseek error\n");
exit(0);
}

/* write a dummy byte at the last location */
write (fdout, "", 1);

/*
* memory map the input file. Only the first two arguments are
* interesting: 1) the location and 2) the size of the memory map
* in virtual memory space. Note that the location is only a "hint";
* the OS can choose to return a different virtual memory address.
* This is illustrated by the printf command below.
*/

src = mmap ((void*) MAP_LOCATION, fileSize,
PROT_READ, MAP_SHARED | MAP_POPULATE, fdin, 0);

/* memory map the output file after the input file */
dst = mmap ((void*) MAP_LOCATION + fileSize , fileSize ,
PROT_READ | PROT_WRITE, MAP_SHARED, fdout, 0);


printf("Mapped src: 0x%x and dst: 0x%x\n",src,dst);

/* Copy the input file to the output file */
memcpy (dst, src, fileSize);

printf("memcpy done\n");

// we should probably unmap memory and close the files
} /* main */
\
 
 \ /
  Last update: 2020-02-24 14:29    [W:0.057 / U:0.336 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site