lkml.org 
[lkml]   [2005]   [Mar]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    SubjectRE: Direct io on block device has performance regression on 2.6.x kernel
    Date
    Andrew Morton wrote on Wednesday, March 09, 2005 12:05 PM
    > "Chen, Kenneth W" <kenneth.w.chen@intel.com> wrote:
    > > Let me answer the questions in reverse order. We started with running
    > > industry standard transaction processing database benchmark on 2.6 kernel,
    > > on real hardware (4P smp, 64 GB memory, 450 disks) running industry
    > > standard db application. What we measured is that with best tuning done
    > > to the system, 2.6 kernel has a huge performance regression relative to
    > > its predecessor 2.4 kernel (a kernel from RHEL3, 2.4.21 based).
    >
    > That's news to me. I thought we were doing OK with big database stuff.
    > Surely lots of people have been testing such things.

    There are different level of "big" stuff. We used to work on 32-way numa
    box, but other show stopper issues popping up before we get to the I/O stack.
    The good thing came out of that work is the removal of global unplug lock.


    > > And yes, it is all worth pursuing, the two patches on raw device recuperate
    > > 1/3 of the total benchmark performance regression.
    >
    > On a real disk driver? hm, I'm wrong then.
    >

    Yes, on real disk driver (qlogic fiber channel) and with real 15K rpm disks.


    > Did you generate a kernel profile?

    Top 40 kernel hot functions, percentage is normalized to kernel utilization.

    _spin_unlock_irqrestore 23.54%
    _spin_unlock_irq 19.27%
    __blockdev_direct_IO 3.57%
    follow_hugetlb_page 1.84%
    e1000_clean 1.38%
    kmem_cache_alloc 1.31%
    put_page 1.29%
    __generic_file_aio_read 1.18%
    e1000_intr 1.07%
    schedule 1.01%
    dio_bio_complete 0.97%
    mempool_alloc 0.96%
    kmem_cache_free 0.90%
    __end_that_request_first 0.88%
    __copy_user 0.82%
    kfree 0.77%
    generic_make_request 0.73%
    _spin_lock 0.73%
    kref_put 0.73%
    vfs_read 0.68%
    update_atime 0.68%
    scsi_dispatch_cmd 0.67%
    fget_light 0.66%
    put_io_context 0.60%
    _spin_lock_irqsave 0.58%
    scsi_finish_command 0.58%
    generic_file_aio_write_nolock 0.57%
    inode_times_differ 0.55%
    break_fault 0.53%
    __do_softirq 0.48%
    aio_read_evt 0.48%
    try_atomic_semop 0.44%
    sys_pread64 0.43%
    __bio_add_page 0.43%
    __mod_timer 0.42%
    bio_alloc 0.41%
    scsi_decide_disposition 0.40%
    e1000_clean_rx_irq 0.39%
    find_vma 0.38%
    dnotify_parent 0.38%


    Profile with spin lock inlined, so that it is easier to see functions
    that has the lock contention, again top 40 hot functions:

    scsi_request_fn 7.54%
    finish_task_switch 6.25%
    __blockdev_direct_IO 4.97%
    __make_request 3.87%
    scsi_end_request 3.54%
    dio_bio_end_io 2.70%
    follow_hugetlb_page 2.39%
    __wake_up 2.37%
    aio_complete 1.82%
    kmem_cache_alloc 1.68%
    __mod_timer 1.63%
    e1000_clean 1.57%
    __generic_file_aio_read 1.42%
    mempool_alloc 1.37%
    put_page 1.35%
    e1000_intr 1.31%
    schedule 1.25%
    dio_bio_complete 1.20%
    scsi_device_unbusy 1.07%
    kmem_cache_free 1.06%
    __copy_user 1.04%
    scsi_dispatch_cmd 1.04%
    __end_that_request_first1.04%
    generic_make_request 1.02%
    kfree 0.94%
    __aio_get_req 0.93%
    sys_pread64 0.83%
    get_request 0.79%
    put_io_context 0.76%
    dnotify_parent 0.73%
    vfs_read 0.73%
    update_atime 0.73%
    finished_one_bio 0.63%
    generic_file_aio_write_nolock 0.63%
    scsi_put_command 0.62%
    break_fault 0.62%
    e1000_xmit_frame 0.62%
    aio_read_evt 0.59%
    scsi_io_completion 0.59%
    inode_times_differ 0.58%



    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2005-03-22 14:11    [W:2.721 / U:0.252 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site