lkml.org 
[lkml]   [2005]   [Sep]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Where is the performance bottleneck?
On Wed, 31 Aug 2005, Holger Kiehl wrote:

> On Thu, 1 Sep 2005, Nick Piggin wrote:
>
>> Holger Kiehl wrote:
>>
>>> meminfo.dump:
>>>
>>> MemTotal: 8124172 kB
>>> MemFree: 23564 kB
>>> Buffers: 7825944 kB
>>> Cached: 19216 kB
>>> SwapCached: 0 kB
>>> Active: 25708 kB
>>> Inactive: 7835548 kB
>>> HighTotal: 0 kB
>>> HighFree: 0 kB
>>> LowTotal: 8124172 kB
>>> LowFree: 23564 kB
>>> SwapTotal: 15631160 kB
>>> SwapFree: 15631160 kB
>>> Dirty: 3145604 kB
>>
>> Hmm OK, dirty memory is pinned pretty much exactly on dirty_ratio
>> so maybe I've just led you on a goose chase.
>>
>> You could
>> echo 5 > /proc/sys/vm/dirty_background_ratio
>> echo 10 > /proc/sys/vm/dirty_ratio
>>
>> To further reduce dirty memory in the system, however this is
>> a long shot, so please continue your interaction with the
>> other people in the thread first.
>>
> Yes, this does make a difference, here the results of running
>
> dd if=/dev/full of=/dev/sd?1 bs=4M count=4883
>
> on 8 disks at the same time:
>
> 34.273340
> 33.938829
> 33.598469
> 32.970575
> 32.841351
> 32.723988
> 31.559880
> 29.778112
>
> That's 32.710568 MB/s on average per disk with your change and without
> it it was 24.958557 MB/s on average per disk.
>
> I will do more tests tomorrow.
>
Just rechecked those numbers. Did a fresh boot and run the test several
times. With defaults (dirty_background_ratio=10, dirty_ratio=40) I get
for the dd write tests an average of 24.559491 MB/s (8 disks in parallel)
per disk. With the suggested values (dirty_background_ratio=5, dirty_ratio=10)
32.390659 MB/s per disk.

I then did a SW raid0 over all disks with the following command:

mdadm -C /dev/md3 -l0 -n8 /dev/sd[cdefghij]1

(dirty_background_ratio=10, dirty_ratio=40) 223.955995 MB/s
(dirty_background_ratio=5, dirty_ratio=10) 234.318936 MB/s

So the differnece is not so big anymore.

Something else I notice while doing the dd over 8 disks is the following
(top just before they are finished):

top - 08:39:11 up 2:03, 2 users, load average: 23.01, 21.48, 15.64
Tasks: 102 total, 2 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 17.7% sy, 0.0% ni, 0.0% id, 78.9% wa, 0.2% hi, 3.1% si
Mem: 8124184k total, 8093068k used, 31116k free, 7831348k buffers
Swap: 15631160k total, 13352k used, 15617808k free, 5524k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3423 root 18 0 55204 460 392 R 12.0 0.0 1:15.55 dd
3421 root 18 0 55204 464 392 D 11.3 0.0 1:17.36 dd
3418 root 18 0 55204 464 392 D 10.3 0.0 1:10.92 dd
3416 root 18 0 55200 464 392 D 10.0 0.0 1:09.20 dd
3420 root 18 0 55204 464 392 D 10.0 0.0 1:10.49 dd
3422 root 18 0 55200 460 392 D 9.3 0.0 1:13.58 dd
3417 root 18 0 55204 460 392 D 7.6 0.0 1:13.11 dd
158 root 15 0 0 0 0 D 1.3 0.0 1:12.61 kswapd3
159 root 15 0 0 0 0 D 1.3 0.0 1:08.75 kswapd2
160 root 15 0 0 0 0 D 1.0 0.0 1:07.11 kswapd1
3419 root 18 0 51096 552 476 D 1.0 0.0 1:17.15 dd
161 root 15 0 0 0 0 D 0.7 0.0 0:54.46 kswapd0
1 root 16 0 4876 372 332 S 0.0 0.0 0:01.15 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
7 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/3
9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3

A loadaverage of 23 for 8 dd's seems a bit high. Also why is kswapd working
so hard? Is that correct.

Please just tell me if there is anything else I can test or dumps that
could be useful.

Thanks,
Holger

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-09-01 11:15    [W:0.052 / U:0.304 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site