lkml.org 
[lkml]   [2002]   [Oct]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: I/O performance test
Hello Roger

> > Since a month we have a LINUX BEOWULF cluster, the clusters contains 7
> > P4 dual processor 2GHz computers, with 8Gb of RAM per machine. For our
> > network we have used Gigabit ethernet. On our cluster we are running RH
> > 7.2, with Linux 2.4.19.
> >
> > These are some of the hardware specs:
> >
> > - Dual Intel Xeon 2.GHz, 512 cache 400 MHz FSB
> > - 8 Gb ECC DDR
> > - Fujitsu 72Gb 10.000 rpm Ultra 160 SCA HDD SCSI HARDDISK
>
> Do you have a 72 GB HD per dual processors? Or only on one?
> Or do the data have to pass through the ethernet for your tests?


I have 3*72Gb HD per computer. The test is carried out both on the server,
whereby data was written directly to the harddisk, as well the test was
carried out from one of the nodes whereby data had to pass through the
ethernet.
When carrying out the test of the ethernet no problems were discovered.

> > The problem we have with our cluster is as follows. When running large
> > scientific simulations the computer performance gradually goes down as
> > the I/O activity is increasing. At some point the response of the
> > computer is so poor that we have to kill the simulation. In a worst case
> > when the simulation was running overnight the computer hangs and a reset
> > of the computer is necessary. Nevertheless, even when we manage to kill
> > the simulation in time the computer remains very slow and a reboot is
> > necessary to regain full computer power.
> >
> > My first suspicion was that the computer simply started swapping, but
> > there is no swap space being used, instead free RAM memory is still
> > apparent
> > (between 5-10%) and only 90% of the RAM is in use whereby 50% is cached
> > and another 50% is in usage. In addition the cpu usage is very low as
> > well.
>
> Could still be that all DMA or normal zone memory is used up.
> Use the magic sysrq to get zone info.
> Alt+Sys Rq+M
> check in vt10 (Alt+F10 or Ctrl+Alt+F10) or with dmesg.

OK, good idea, didn't knew this.

> > In order to improve the I/O performance I have add some patches to the
> > kernel, including the patch for 00_block-highmem-all-19-1, to avoid
> > bounce buffers. Unfortunately none of the patches let to any improvement
> > in the I/O performance.
> >
> > I don't think the machine should be behaving like this. I certainly
> > expect some slowdowns with that much IO, but the computer should still
> > be reasonably responsive, particularly because no system or user files
> > that need to be accessed are on that channel of the SCSI controller.
> >
> > As a sort of a desperate move I did two other test in addition which
> > could be of use to the understanding of the problem:
> >
> > - I removed 6Gb from the server and run the test bonnie++ -s 16096
> > succesfully with 2Gb of RAM.
>
> This points towards a highmem issue...
>
> > - I placed an IDE disk 40Gb and run the test bonnie++ -s 16096 with 8Gb
> > of RAM succesfully.
>
> This is why I asked the first question.
>
> My guess: Highmem issue with Gb Ethernet?
>
> Brand and model of the Ethernet boards?

Someone suggested me to install the -aa patch 2.4.19rc5aa1. When running the
modified kernel all test were performed well.

Hereby I send you the output from Bonnie++. In this test I run bonnie++ on
the server, whereby it was writting directly to the 72Gb HD.

[govert@galaxy govert]$ bonnie++ -s 16096
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.02c ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
galaxy 16096M 16268 72 38745 21 17456 10 18672 82 37469 14 318.1
1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 3665 99 +++++ +++ +++++ +++ 3762 99 +++++ +++ 9147
99
galaxy,16096M,16268,72,38745,21,17456,10,18672,82,37469,14,318.1,1,16,3665,99,+++++,+++,+++++,+++,3762,99,+++++,+++,9147,99

[govert@galaxy govert]$ emacs bonnie++
bonnie++16096.txt bonnie++4084.txt
[govert@galaxy govert]$


Govert


--
------------------------------------------------------------
| Dr. Govert de With Research Fellow |
| Fluid Mechanics Research Group |
| University of Hertfordshire |
| Tel: 01707 284124 Fax: 01707 285086 |
------------------------------------------------------------
| Der Horizont vieler Menschen ist ein Kreis mit Radius Null |
| und das nennen sie ihren Standpunkt. |
------------------------------------------------------------



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:30    [W:2.775 / U:0.076 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site