lkml.org 
[lkml]   [2002]   [Oct]   [17]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: I/O performance test
Date
On torsdagen den 17 oktober 2002 11.58, G.de-With wrote:
> Hello
>
> Since a month we have a LINUX BEOWULF cluster, the clusters contains 7
> P4 dual processor 2GHz computers, with 8Gb of RAM per machine. For our
> network we have used Gigabit ethernet. On our cluster we are running RH
> 7.2, with Linux 2.4.19.
>
> These are some of the hardware specs:
>
> - Dual Intel Xeon 2.GHz, 512 cache 400 MHz FSB
> - 8 Gb ECC DDR
> - Fujitsu 72Gb 10.000 rpm Ultra 160 SCA HDD SCSI HARDDISK

Do you have a 72 GB HD per dual processors? Or only on one?
Or do the data have to pass through the ethernet for your tests?

>
> The problem we have with our cluster is as follows. When running large
> scientific simulations the computer performance gradually goes down as
> the I/O activity is increasing. At some point the response of the
> computer is so poor that we have to kill the simulation. In a worst case
> when the simulation was running overnight the computer hangs and a reset
> of the computer is necessary. Nevertheless, even when we manage to kill
> the simulation in time the computer remains very slow and a reboot is
> necessary to regain full computer power.
>
> My first suspicion was that the computer simply started swapping, but
> there is no swap space being used, instead free RAM memory is still
> apparent
> (between 5-10%) and only 90% of the RAM is in use whereby 50% is cached
> and another 50% is in usage. In addition the cpu usage is very low as
> well.

Could still be that all DMA or normal zone memory is used up.
Use the magic sysrq to get zone info.
Alt+Sys Rq+M
check in vt10 (Alt+F10 or Ctrl+Alt+F10) or with dmesg

>
> To investigate the I/O performance I installed an I/O performance
> benchmark program called bonnie++. The first test I performed was a
> single bonnie++ -s 16096 instance.
>
> # bonnie++ -s 16096
>
> Unfortunately, as a result of running bonnie++ the computer started to
> slow down till it finally hang. All the symptoms I discover with
> bonnie++ are identical to what I experience when running our scientific
> calculations.

Fortunately!

You may have found a way for others to reproduce your problem :-)

>
> In order to improve the I/O performance I have add some patches to the
> kernel, including the patch for 00_block-highmem-all-19-1, to avoid
> bounce buffers. Unfortunately none of the patches let to any improvement
> in the I/O performance.
>
> I don't think the machine should be behaving like this. I certainly
> expect some slowdowns with that much IO, but the computer should still
> be reasonably responsive, particularly because no system or user files
> that need to be accessed are on that channel of the SCSI controller.
>
> As a sort of a desperate move I did two other test in addition which
> could be of use to the understanding of the problem:
>
> - I removed 6Gb from the server and run the test bonnie++ -s 16096
> succesfully with 2Gb of RAM.

This points towards a highmem issue...

> - I placed an IDE disk 40Gb and run the test bonnie++ -s 16096 with 8Gb
> of RAM succesfully.

This is why I asked the first question.

My guess: Highmem issue with Gb Ethernet?

Brand and model of the Ethernet boards?


--
Roger Larsson
Skellefteå
Sweden

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:30    [W:0.069 / U:23.432 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site