lkml.org 
[lkml]   [1999]   [Jan]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: us.kernel.org mirroring inconsistency
Date
	Been avoiding stepping into these waters...

Tigran Aivazian enscribed thusly:
> No, that is not foolish. Look not at 2M files but at 10M files, which
^^
I think the original author was talking about 2<G> as in the
entire archive... For the 2.2.0pre4 tarballs, the bz2 file is
10494542 while the gz file is 12969415. Ok... So that's a difference
of about 2.4 Meg. That's an increase of about 24% of the gzip over the
bzip2. That difference is twice the size of the entire 1.0 gziped tarball.
For some of us, that could be a significant difference.

> is the size of the kernel. I think it is more foolish to keep the old
> .gz files when it is so clear that .bz2 is much better. (And everyone
> running Red Hat Linux will have bzip2(1) pre-installed anyway).

=================================
Editorial note:

Based on rereading your comments above after writing my comments
below...

Your remark about "foolish to keep the old .gz files" has two
interpretations. I first read that as "gz" is the old format so let's
not keep any "gz" files around. A second reading made me realize that
you might have meant "old files which happen to be in gz format". That
implies something totally different. If you meant to imply that newer
gz files would be kept and only the older gz files would be dropped, then
we are not in such disagreement as my following comments would imply.

Under either interpretation, I'll let my suggestion for an attic
for older tarballs (gzip or bzip2), or an attic system/site, stand.
================================

Wrong'o Dan'o...

1) Not everyone is running RedHat.

2) Not all RedHat systems have bzip2 on them.

I've got some older RedHat 4.2 systems which do NOT have bzip2 on
them, so the statement "everyone running Red Hat" is inaccurate at best.
I really REALLY hate having to unpack kernel sources on one system and then
transfer them over to another system just because one has bzip2 and one
does not. Yes, I can install bzip2 on any of them and I can even install
the bzip2 version of tar on any of them. That has drawbacks in and of
itself, especially when you are in the middle of simply trying to update
the kernels. On one occasion, due to a variety of circumstances and problems
where I could only get a kernel bz2 tarball, I had to download the tarball
to one system, bunzip2 it and pipe that into gzip to create the gzip file
which I could then ship over to a system where I needed it. What's
that do to my bandwidth? Convoluted headstand? Yes. Routine? No.
Better ways? Absolutely, most of the time.

The arguements around tar understanding bzip2 not withstanding, we
are also sometimes in the position of needing to unpack these critters on
other systems where gzip is understood (hell - even WinZip understands
tar zip'ed file - amazed me!) but where bzip2 is not. As much as you
might LIKE it to be, bzip2 simply is not as widely recognized and supported
as gzip. When that day comes, fine. 'Till then, remarks about dropping
the gzip format (afternote - if that's what's implied) are out of place.

It might also help bzip2 adoption if bzip2 understood gzip (and
compress) formats the same way gzip understands compress (older standard
with .Z extensions) formats. Then we would only need ONE utility (gzip
is simply NOT going away) and we wouldn't NEED YATO (Yet Another Tar Option)
to get tar to unpack the files. As things stand, we have to support both
utilities, and that's a pain in the neck.

This irritates some of us who feel like this is being pushed by
people with an agenda other than saving network bandwidth. No, doing this
to spite RMS and his GNU/Linux crusade is NOT reasonable. Note: The remark
has been expressed that we should reduce our dependency on GNU utilities
to shut him up. I disagree with both him and that, but that's another story.

If lharc proved to have better compression (I don't know. Does it?
Probably not - too old.) would you start crusading to have us all convert
over to lharc?

Bzip2 has some size advantages over gzip but certainly not
processor advantages. Some of us have other tradeoffs regarding size,
bandwidth, expense, processor demand, and convenience.

I wouldn't want to see bz2 go away any more than I would want to see
gz go away. Each have their place.

I DO like the suggestion made by someone to eliminate most of the
older kernel tarballs (both bz2 and gz) on the mirrors to conserve space.
The patches are sufficient to reconstruct past kernels where necessary. I
was in the middle of helping diagnose the "flu" problem a couple of months
ago, so I know the need to get at past kernels, first hand.

File the older fat tarballs in an "attic" on the mail kernel site
and leave the last few (say three or four) kernels in each development line
in the main directories. You could also, maybe, trim out some of the older
patch files as well leaving the last dozen or so patchballs for the mirrors.
This can be automated pretty much when when posting new kernel tarballs. The
mirrors can then ignore the attics and just get the patches and recent
kernels. That would conserve their space and allow them to store both gz
and bz2 tarballs. Then the rest of us can pick and chose what compression
we want -or need- to use for whatever reasons we have. I don't dictate to
you to download gz files through your expensive or bandwidth-critical
connection and you don't dictate when I have to update are rearrange all
of my installations and utilities to suite the latest fad in compression
technology.

We could also provide some access to the older files in the attic
on one central server where the lower demand for those older tarballs would
not load it down so much. Something like ftp.attic.kernel.org for any
tarballs older than three revs back in a given revision line (1.0, 1.1, 1.2,
2.0, 2.1, 2.2, etc). Don't keep the recent ones on that site specifically
to avoid the temptation some would have to go to it for recent kernels.
That would preserve the historical trail (which is being painstakingly
reconstructed for some of the REALLY old 0.* line) without causing excessive
storage demands on the mirror sites (or bandwidth demands when a new mirror
comes on line - think about it!) and provides all of the benefits of having
up to date mirrors in the formats we use.

> Regards,
> Tigran A. Aivazian, http://www.aivazian.demon.co.uk/

> On Sun, 3 Jan 1999, Matti Aarnio wrote:

> > > I'm getting a bit disturbed by an inconsistency between the
> > > kernel.org mirrors. For example, I have just connected to
> > > katelyn.optilinkcomm.net, which apparently carries only bz2 files
> > > and not gz files.
> >
> > That is foolish, in fact I think the BZ2 at this point
> > is foolish thing to do as the only compression algorithm.
> > In case of ftp.kernel.org archive:
> > gz 2.4 GB
> > bz2 2.1 GB
> > That is, the 'bz2' version of compression is only about
> > 1/8:th denser, than gzip.
> >
> > Way back when the BSD compress got into trouble with
> > LZW patents, the rapid move to GNU-zip was well founded,
> > but now such a thing is not really warranted.
> >
> > Over the time there have been available other free high-
> > density compression algorithms (LHA, ARJ), which have not
> > gotten very wide adoptation. Mainly perhaps because while
> > they often produce denser results than GZIP, they always
> > do it with way more heavy CPU usage. Also the expansion-
> > algorithms of those seem to be heavier than gunzip.
> >
> > This thinking is basis on why ftp.funet.fi does not
> > mirror BZ2 files if the same files are also available
> > as GZ files.
> >
> > > It seems to defeat the purpose if kernel.org mirrors don't
> > > actually carry the full contents of kernel.org, even accounting
> > > for any lag in pulling down new files.
> >
> > Yeah, the http://www.kernel.org/mirrors/ COULD be further divided
> > down per what the sites have announced to mirror, e.g.:
> > ftp-gz.xx.kernel.org
> > ftp-bz2.xx.kernel.org
> > www-gz.xx.kernel.org
> > www-bz2.xx.kernel.org
> > if the site has both formats, then it should be aliased both ways.
> >
> > /Matti Aarnio <mea@nic.funet.fi> <matti.aarnio@sonera.fi>

Mike
--
Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com
(The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:49    [W:0.083 / U:0.260 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site