lkml.org 
[lkml]   [1997]   [Dec]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: Filesystem optimization.. - why not optimise squid?
>> ...  INN is moving towards a database, so if Squid does the same then
>> what do you gain from a new FS?

From the point of view of the respective application developers, they gain
>portability. People want to run INN and Squid on a lot of platforms.
>OTOH, if Linux had an ideal filesystem on which to run these apps, a lot
>of people might switch over to our camp.

I'm suddenly thinking of Sun's tempfs (file system which resides in RAM
and swap and is mounted on /tmp). I've heard many reports of it being
buggy (doesn't get the testing that other FSs get) and it's got it's own
special drawbacks (you really don't want to run out of space). This is the
only case I can think of where someone has created a FS for a single
purpose, and it hasn't been that successful IMHO.
Also we can expect the authors of INN and Squid to run OSs other than
Linux (I believe that DEC UNIX is very popular at NLANR and I think that
Solaris is used in the development of INN). Therefore even if we develop a
file system that gives maximum speed for these applications (faster than
any available application-level database) then we can still expect future
versions (such as INN 1.8 or 2.0) to use their own databases so they work
well on other platforms. When you're writing portable code and you have a
choice between making it really fast on one platform and dead slow on the
rest or making it perform reasonably well everywhere then you'll likely go
for the latter option.
At an AUUG dinner I talked with one of the Squid developers, when
someone asked him about the possibility of using the hack to open inodes
directly which apparently some hacked version of INN uses he seemed very
adament about not using such hacks. Apparently other options for Squid
data storage are being considered.

>A filesystem has a huge advantage in that all the traditional Unix tools
>can be used; how are you going to back up your current news or cache
>partition, convert it to a raw DBMS partition, and restore the backup?

You don't necessaryily have to have a raw DBMS partition, large database
files will do. Raw partitions are easy to backup anyway "dd if=/dev/hda1 |
bzip2 -9 > database.bz2" does it well. In fact backing up a set of large
database files or database partitions would be a lot easier than backing up
a current INN or Squid database (last time I copied an INN database between
IDE hard drives it averaged less than 50K of data transferred per second -
copying large files on the same hard drives gave well over 1 meg per
second).
If the database was active when you wanted to copy it then it could be
more difficult. For INN you can just pause incoming news most INN access
is just reading and you can backup while users are reading news - news that
they post should be sent immidiately and probably shouldn't be backed up.
Assuming that INN 1.8 has seperate databases for incoming unprocessed news,
unprocessed news posted by users, and the main news database, you could
just pause the processing of incoming news and then have a nice quiet
database to backup. As for Squid, you could have a way of telling Squid to
run in proxy-only mode for the duration of the backup (should be ~100 lines
of code changed to make it take SIGUSR2 or something as the signal to run
proxy-only and stop writing to the database). But why would you want to
backup a Squid database? Most people don't even backup an INN spool...

>For a special filesystem to be mounted at /usr/local/squid/cache or
>/var/spool/news, we'd need to optimize to create a file, open and read it,
>unlink it, and (for news only) hardlink it. Rarely if ever will a file be
>modified, appended, or seeked (history and logs belong elsewhere).

While we're at it we might as well skip the ATIME and CTIME fields and
the owner/group/permissions values for the files (have one
owner/group/permission setting for the whole partition). We could skip the
hierarchical directory thing as we would only need to have files in the
root directory.

>The big problem with the collapsed directory structure is opendir().
>Neither app uses this functionality, but it is useful for maintenance. It
>might still be necessary to chain files together by directory so that a

Squid does use opendir() in storeDirClean(). I think that this gets
called at Squid startup as well as other configured times.


--
-----------------------------------------------------------
In return for "mailbag contention" errors from buggy Exchange
servers I'll set my mail server to refuse mail from your domain.
The same response applies when a message to a postmaster
account bounces.
"Russell Coker - mailing lists account" <bofh@snoopy.virtual.net.au>
-----------------------------------------------------------


\
 
 \ /
  Last update: 2005-03-22 13:40    [W:0.044 / U:1.096 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site