lkml.org 
[lkml]   [1998]   [Jul]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: ext2fs enhacement/fix [re: shrinking directories]
Date
From
> Only for some unlinks. Assuming 4k block and a news server where filenames
> are about 5 chars long, we can get 200 or do files per block, so if we have
> to remove all files from a directory that once contained 40,000 files, the
> shrink operation only happens 8 times or so.

Just to give some real figures rather than vague estimates at what might
happen on a news server (note - this is innd-1.7.2 - well something based
on innd-1.7.2 at least; 2.0 I believe can be configured to not do the
traditional inn storage structure);

I have now split /news/control/cancel into 256 parts some time ago.
each one of them is now over 64k in size...

[...]
drwxrwxr-x 2 news news 64512 Jul 16 06:41 pt100/
drwxrwxr-x 2 news news 64512 Jul 16 06:42 pt101/
drwxrwxr-x 2 news news 64512 Jul 16 06:42 pt102/
drwxrwxr-x 2 news news 64512 Jul 16 06:43 pt103/
drwxrwxr-x 2 news news 64512 Jul 16 06:43 pt104/
drwxrwxr-x 2 news news 64512 Jul 16 06:44 pt105/
[...]

This is using file names in the directory 2-3 characters shorter than
they would be in an un-tiered directory, and ultra-short expire times on
that directory. (why are the filenames shorter? because I calculate a
unique number for each file in each slot, but slots can all contain
a file of each number)

And the total space of 256 * 64k - well, #%#$% huge. When this
is a real group, like a seasonal group (eg, football groups),
it swells at one time of the year then shrinks and if the directory
isn't shrunk then it's a real waste on CPU, disk space, memory, etc.
(and the benefit from shrinking it manually by making a new one
is quite noticeable)

Anyway the article numbers are now around
control.cancel 0032272953 0032186164 y
(I can see that hitting the field boundary soon, if growth continues as
it tends to on the internet... and some of the jobs groups, which we no
longer receive due to policy, used to get more articles than
/news/control/cancel and tiering can't be used there without teaching things
like the nnrpd's about tiering).

Anyway if control.cancel was in one directory, that's
8-digit file name 8
terminating \0 9
pad to 4 bytes 12
4 byte inode number 16
Number of articles at once can be around 512,000 even on ultra-frequent
expiry (I've got around 2000 files in some of those ptXXX directories above
and the expire time on control.cancel is 1-2 days so they wait around just
enough to hopefully get on to the downstream hosts).

512,000 * 16 = 8Mb
The sum of the individual directories is 16Mb, which means the current
number of articles stored in control.cancel is half the peak it's seen
since I last recreated them.

The biggest directory sizes I usually see on news servers are around a
couple of Mb, since past this the machine just gets too slow and we have
to drop the group (or for control.cancel, tier it).

Of course, this is just a news server in Australia. American ones could
see way more articles.

David.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.altern.org/andrebalsa/doc/lkml-faq.html

\
 
 \ /
  Last update: 2005-03-22 13:43    [W:0.097 / U:0.028 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site