lkml.org 
[lkml]   [2007]   [Nov]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    SubjectRe: Major mke2fs slowdown (reproducable, bisected)
    From
    Hi Alexey,

    On (12/11/07 21:25), Alexey Dobriyan didst pronounce:
    > Cross-compile farm here migrated to .ccache and build dir on separate
    > disks and now I have a way to blow up .ccache without waiting half an
    > hour for rm(1) to finish. It's called mke2fs(8).
    >
    > However, in e.g 2.6.24-rc2 mke2fs is amazingly slow if done right after
    > several fat cross-compile builds. Normally it takes ~11 seconds to
    > finish. After commit 5adc5be7cd1bcef6bb64f5255d2a33f20a3cf5be aka
    > "Bias the placement of kernel pages at lower PFNs" it takes several
    > minutes. 2.6.24-rc2 without this patch also gives normal mkfs speeds.
    > I'm pretty sure bisection wasn't screwed up.
    >
    >
    > Details:
    >
    > Core 2 Duo on x86_64, no debugging
    > 4G RAM
    > 30G ext2 .ccache partition with noatime
    > 100G ext2 partition with source tree and build dirs with noatime
    >
    > I build alpha-allnoconfig, alpha-defconfig and 4 allmodconfigs
    > (SMP=y/n x DEBUG_KERNEL=y/n).
    >
    > Right after compilation finishes, free(1) reports more or less the same
    > picture (VM hackers, please, tell me which info you need):
    >
    > total used free shared buffers cached
    > Mem: 4032320 2802604 1229716 0 97160 2424816
    > -/+ buffers/cache: 280628 3751692
    > Swap: 7823644 0 7823644
    >
    > Last steps of build script are:
    >
    > umount /home/ad/.ccache
    > sudo mkfs.ext2 -m 0 <=== this is slow
    >

    Thanks very much for the report and the bisect. I spent the day trying
    to reproduce it but I'm having trouble seeing the same problem using just
    mke2fs. I've tried

    Pentium III x86 machine with 1GB of RAM, 9GB partition
    4-way Opteron with 8GB RAM, 10GB partition
    2-way Opteron with 2GB RAM, 10GB partition
    Pentium D (duel core) with 2GB RAM, 128GB partition

    In all cases, the comparison between 2.6.23, latest git and latest git
    with patch reverted were the same. For example, on the Pentium D, I got

    2.6.23: 95.672 real, 0.068 user, 10.334 sys
    2.6.24-rc2-git: 96.112 real, 0.08 user, 10.664 sys
    2.6.24-rc2-revert: 96.182 real, 0.072 user, 10.602 sys

    This is an average of 5 runs on a 128GB partition. Somewhat unexpectedly,
    the revert was fractionally slower. The deviation between runs was around
    the 0.4 second mark so the differences appear to be in the noise.

    On the other machines, the reverted version was slightly faster but I was
    seeing about 0.5% of overall running time, not the massive differences you
    were seeing. Clearly there is still a problem because reverting the patch
    fixes your problem.... As I write this, it occurs to me that it might be
    because your compile-job has created very long free-lists and searching them
    is causing problems.

    Can you post the contents of /proc/buddyinfo before and after you run
    mke2fs? It will give an indication of how long the linked lists are being
    searched. After I push send here, I'll be trying the tests after running
    compile-tests similar to yours to see if that reproduces the problem.

    Here are some other questions I hope you can answer just to eliminate them
    as possibilities. Can you tell me what sort of disk driver you are using
    (results here are for sata_nv)? Are you using RAID or MD? Is anything running
    in the background while mke2fs is running? What is the output of mke2fs -V,
    gcc --version and ld -v? Finally, can you mail me your .config and I'll try
    it on my machine here.

    In the meantime, it is safe to revert this patch. Andy Whitcroft tested
    the behaviour of anti-fragmentation on a number of machines and while the
    results are adversely affected in terms of hugepage allocation success rates,
    they are still pretty decent. We will investigate a less expensive way of
    achieving the same effect of the patch without the potentially long searches.

    --
    Mel Gorman
    Part-time Phd Student Linux Technology Center
    University of Limerick IBM Dublin Software Lab
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2007-11-13 17:57    [W:3.057 / U:0.008 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site