Messages in this thread | | | Date | Thu, 30 Apr 1998 20:31:16 -0400 (EDT) | From | "Phil's Kernel Account" <> | Subject | Re: CDROM jukebox filesystem using autofs |
| |
On Thu, 30 Apr 1998, Rik van Riel wrote:
#On Thu, 30 Apr 1998, Dan A. Dickey wrote: #> Aaron Passey wrote: # #> > So far, I have made it move and mount disks on command and unmount and put away #> > disks in an LRU fashion. For example, if I do an ls of /autofs/1, it'll get #> > the disk out of slot 1 and put it in one of the drives, mount it and give me #> > the ls. #> > #> > There are several problems I am running into with this approach. First #> > of all, what do I do if I have 4 drives, all mounted and busy, and I get a #> > request for a fifth disk? #> #> return EBUSY. # #Or you could: #- let the user of the first started request wait, and switch # drives again after XXX kilobytes (making sure that throughput # isn't hurt too much, latency will be bad anyway) #- have a caching HD partition and read the biggest file requested # for the next X MB, then quickly switch disks to the requested # one
Or, do it the way I've done it with a large quantity (12) of Pioneer 6 disc SCSI changers...
User1->Request File /cd3/images/boot.img (1 Second) User2->Request File /cd3/images/boot.img (3 Seconds) Server->Send File /cd3/images/boot.img /cdcache (1 Second) Server->Send File->User1 Server->Send File->User2
This is a very effecient way that I've found to deal with CD changers and jukeboxes when there are files that are frequently accessed.
User1->Request File /cd16/system/XFree86-All.tgz (1 Second) User2->Request File /cd17/misc/games/quake-tf.tgz (3 Seconds) Server->Send File /cd16/system/XFree86-All.tgz /cdcache (Wait for file to copy) Server->Send File /cdcache/XFree86-All.tgz->User1 Server->Change CD17 (3 Seconds) Server->Send File /cd17/misc/games/quake-tf.tgz->User2
This allows for multiple connection, with minimal slowdown and wait in a changer environment.
Now for the fun part, a jukebox environment. My example will use a file queueing system I used before with a machine that had 6 SCSI CD-ROMs, and nothing more. Boot disk was IDE. You know the routine. Ran Linux. :)
User1->Request File /cd5/thisfile.tgz User2->Request File /cd2/thatfile.txt User5->Request File /cd1/allfiles.txt User3->Request File /cd6/303wavs.gz Server->Process Requests, sort by filesize (File Sizes: thisfile.tgz 53K thatfile.txt 32K allfiles.txt 10K 303wavz.gz 23M;) Server->Send File->User5 (Wait for 10% of send before starting next) Server->Send File->User2 (Wait again) Server->Send File->User1 (Wait) Server->Send File->User3 User4->Request File /cd3/bob/thepipes.tgz (33M) Server->Send File->User4 User2->Request File /cd3/bob/thefrop.txt (60K) Server->Communicate->User2 "Server is busy, your request has been queued" (Wait till User3's file is 40% done) Server->Send File->User2
In a high use environment, the above principle would apply as well. With some modifications, of course. With the current technology, that kind of machine would have 6 CD-ROMs on an UltraWide PCI, and two 6.4G UW drives. One for the OS, the other for caching exclusively. It may sound a little big, but YMMV, as always. :)
#For truly high-latency apps this might not be the case. #Simply make sure that your readahead (on large files) is #big enough (and maybe HD cached) so that no more than #50% of the time/bandwidth is spent switching the disks.
The easiest way to catch this is user limits, and using the principle outlined above, where it waits for a certain percentage of the file to be sent. Another issue to consider is processor use for sorting files. In a Perfect Environment(tm), you wouldn't need to worry about it, and could be picky as all hell, down to a single bit of difference in size. In reality, you're probably running on a single P200 or dual PPro180, which means you do have limited processing power, and you need to keep it low. The easiest way to do this is to classify filesizes in blocks. 1bit to 4M, 4.001M to 10M, and so on. This lets the machine run through a small list of sizes, decide where the file fits, and throw it in there. (At this point, a coworker distracted me to fix his bloody AIX box.) (And again, to fix permissions. And again, to find X. And again, to fix something else) (Now I try to find train of thought again after about an hour.) By using filesize blocks, the machine can easily categorize. Utilize caching as well, and it speeds up a lot. Either way, jukebox will always be faster than changer, and I give up. I can't remember what I was going to say. :)
-Phil R. Jaenke (kernel@nls.net / prj@nls.net) TheGuyInCharge(tm), Ketyra Designs - We get paid to break stuff :) Linux pkrea.ketyra.INT 2.0.33 #15 Sat Apr 18 00:40:21 EDT 1998 i586 Linux eiterra.nls.net 2.0.33 #15 Fri Apr 17 00:22:13 EDT 1998 i586 - Linus says for 'brave people only.' I say 'keep a backup.' - :)
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu
| |