lkml.org 
[lkml]   [2010]   [Mar]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Poor interactive performance with I/O loads with fsync()ing
    On Wed, 17 Mar 2010 15:53:50 +1100, Nick Piggin <npiggin@suse.de> wrote:
    > Where are the unrelated processes waiting? Can you get a sample of
    > several backtraces? (/proc/<pid>/stack should do it)
    >
    I wish. One of the incredibly frustrating characteristics of this issue is the
    difficulty in measuring it. By the time processes begin blocking, it's already
    far too late to open a terminal and cat to a file. By the time the terminal has
    opened, tens of seconds have passed and things have started to return to normal.
    >
    > > Moreover, the hit on unrelated processes is so bad
    > > that I would almost suspect that swap I/O is being serialized by fsync() as
    > > well, despite being on a separate swap partition beyond the control of the
    > > filesystem.
    >
    > It shouldn't be, until it reaches the bio layer. If it is on the same
    > block device, it will still fight for access. It could also be blocking
    > on dirty data thresholds, or page reclaim though -- writeback and
    > reclaim could easily be getting slowed down by the fsync activity.
    >
    Hmm, this sounds interesting. Is there a way to monitor writeback throughput.

    > Swapping tends to cause fairly nasty disk access patterns, combined with
    > fsync it could be pretty unavoidable.
    >
    This is definitely a possibility. However, it seems to me like swapping should
    be at least mildly favored over other I/O by the I/O scheduler. That being
    said, I can certainly see how it would be difficult to implement such a
    heuristic in a fair way so as not to block out standard filesystem access
    during a thrashing spree.

    > >
    > > Xapian, however, is far from the first time I have seen this sort of
    > > performance cliff. Rsync, which also uses fsync(), can also trigger this sort
    > > of thrashing during system backups, as can rdiff. slocate's updatedb
    > > absolutely kills interactive performance as well.
    > >
    > > Issues similar to this have been widely reported[1-5] in the past, and despite
    > > many attempts[5-8] within both I/O and memory managements subsystems to fix
    > > it, the problem certainly remains. I have tried reducing swappiness from 60 to
    > > 40, with some small improvement and it has been reported[20] that these sorts
    > > of symptoms can be negated through use of memory control groups to prevent
    > > interactive process pages from being evicted.
    >
    > So the workload is causing quite a lot of swapping as well? How much
    > pagecache do you have? It could be that you have too much pagecache and
    > it is pushing out anonymous memory too easily, or you might have too
    > little pagecache causing suboptimal writeout patterns (possibly writeout
    > from page reclaim rather than asynchronous dirty page cleaner threads,
    > which can really hurt).
    >
    As far as I can tell, the workload should fit in memory without a problem. This
    machine has 4 gigabytes of memory, of which currently 2.8GB is page cache.
    Seems high perhaps? I've included meminfo below. I can completely see how
    overly-aggressive page-cache would result in this sort of behavior.

    - Ben


    MemTotal: 4048068 kB
    MemFree: 47232 kB
    Buffers: 48 kB
    Cached: 2774648 kB
    SwapCached: 1148 kB
    Active: 2353572 kB
    Inactive: 1355980 kB
    Active(anon): 1343176 kB
    Inactive(anon): 342644 kB
    Active(file): 1010396 kB
    Inactive(file): 1013336 kB
    Unevictable: 0 kB
    Mlocked: 0 kB
    SwapTotal: 4883756 kB
    SwapFree: 4882532 kB
    Dirty: 24736 kB
    Writeback: 0 kB
    AnonPages: 933820 kB
    Mapped: 88840 kB
    Shmem: 750948 kB
    Slab: 150752 kB
    SReclaimable: 121404 kB
    SUnreclaim: 29348 kB
    KernelStack: 2672 kB
    PageTables: 31312 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    WritebackTmp: 0 kB
    CommitLimit: 6907788 kB
    Committed_AS: 2773672 kB
    VmallocTotal: 34359738367 kB
    VmallocUsed: 364080 kB
    VmallocChunk: 34359299100 kB
    HugePages_Total: 0
    HugePages_Free: 0
    HugePages_Rsvd: 0
    HugePages_Surp: 0
    Hugepagesize: 2048 kB
    DirectMap4k: 8552 kB
    DirectMap2M: 4175872 kB



    \
     
     \ /
      Last update: 2010-03-26 04:31    [from the cache]
    ©2003-2011 Jasper Spaans