Messages in this thread | | | From | "Russell Coker - mailing lists account" <> | Date | Wed, 31 Dec 97 17:49:17 +1000 | Subject | Re: Filesystem optimization.. - why not optimise squid? |
| |
>> ... INN is moving towards a database, so if Squid does the same then >> what do you gain from a new FS?
From the point of view of the respective application developers, they gain >portability. People want to run INN and Squid on a lot of platforms. >OTOH, if Linux had an ideal filesystem on which to run these apps, a lot >of people might switch over to our camp.
I'm suddenly thinking of Sun's tempfs (file system which resides in RAM and swap and is mounted on /tmp). I've heard many reports of it being buggy (doesn't get the testing that other FSs get) and it's got it's own special drawbacks (you really don't want to run out of space). This is the only case I can think of where someone has created a FS for a single purpose, and it hasn't been that successful IMHO. Also we can expect the authors of INN and Squid to run OSs other than Linux (I believe that DEC UNIX is very popular at NLANR and I think that Solaris is used in the development of INN). Therefore even if we develop a file system that gives maximum speed for these applications (faster than any available application-level database) then we can still expect future versions (such as INN 1.8 or 2.0) to use their own databases so they work well on other platforms. When you're writing portable code and you have a choice between making it really fast on one platform and dead slow on the rest or making it perform reasonably well everywhere then you'll likely go for the latter option. At an AUUG dinner I talked with one of the Squid developers, when someone asked him about the possibility of using the hack to open inodes directly which apparently some hacked version of INN uses he seemed very adament about not using such hacks. Apparently other options for Squid data storage are being considered.
>A filesystem has a huge advantage in that all the traditional Unix tools >can be used; how are you going to back up your current news or cache >partition, convert it to a raw DBMS partition, and restore the backup?
You don't necessaryily have to have a raw DBMS partition, large database files will do. Raw partitions are easy to backup anyway "dd if=/dev/hda1 | bzip2 -9 > database.bz2" does it well. In fact backing up a set of large database files or database partitions would be a lot easier than backing up a current INN or Squid database (last time I copied an INN database between IDE hard drives it averaged less than 50K of data transferred per second - copying large files on the same hard drives gave well over 1 meg per second). If the database was active when you wanted to copy it then it could be more difficult. For INN you can just pause incoming news most INN access is just reading and you can backup while users are reading news - news that they post should be sent immidiately and probably shouldn't be backed up. Assuming that INN 1.8 has seperate databases for incoming unprocessed news, unprocessed news posted by users, and the main news database, you could just pause the processing of incoming news and then have a nice quiet database to backup. As for Squid, you could have a way of telling Squid to run in proxy-only mode for the duration of the backup (should be ~100 lines of code changed to make it take SIGUSR2 or something as the signal to run proxy-only and stop writing to the database). But why would you want to backup a Squid database? Most people don't even backup an INN spool...
>For a special filesystem to be mounted at /usr/local/squid/cache or >/var/spool/news, we'd need to optimize to create a file, open and read it, >unlink it, and (for news only) hardlink it. Rarely if ever will a file be >modified, appended, or seeked (history and logs belong elsewhere).
While we're at it we might as well skip the ATIME and CTIME fields and the owner/group/permissions values for the files (have one owner/group/permission setting for the whole partition). We could skip the hierarchical directory thing as we would only need to have files in the root directory.
>The big problem with the collapsed directory structure is opendir(). >Neither app uses this functionality, but it is useful for maintenance. It >might still be necessary to chain files together by directory so that a
Squid does use opendir() in storeDirClean(). I think that this gets called at Squid startup as well as other configured times.
-- ----------------------------------------------------------- In return for "mailbag contention" errors from buggy Exchange servers I'll set my mail server to refuse mail from your domain. The same response applies when a message to a postmaster account bounces. "Russell Coker - mailing lists account" <bofh@snoopy.virtual.net.au> -----------------------------------------------------------
| |