[lkml]   [2002]   [Jan]   [28]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectA modest proposal -- We need a patch penguin
    Patch Penguin Proposal.

    1) Executive summary.
    2) The problem.
    3) The solution.
    4) Ramifications.

    -- Executive summary.

    Okay everybody, this is getting rediculous. Patches FROM MAINTAINERS are
    getting dropped on the floor on a regular basis. This is burning out
    maintainers and is increasing the number of different kernel trees (not yet a
    major fork, but a lot of cracks and fragmentation are showing under the
    stress). Linus needs an integration lieutenant, and he needs one NOW.

    We need to create the office of "patch penguin", whose job would be to make
    Linus's life easier by doing what Alan Cox used to do, and what Dave Jones is
    unofficially doing right now. (In fact, I'd like to nominate Dave Jones for
    the office, although it's Linus's decision and it would be nice if Dave got
    Alan Cox's blessing as well.)

    And if we understand this position, and formalize it, we can make better use
    of it. It can solve a lot of problems in linux development.

    -- The problem.

    Linus doesn't scale, and his current way of coping is to silently drop the
    vast majority of patches submitted to him onto the floor. Most of the time
    there is no judgement involved when this code gets dropped. Patches that fix
    compile errors get dropped. Code from subsystem maintainers that Linus
    himself designated gets dropped. A build of the tree now spits out numerous
    easily fixable warnings, when at one time it was warning-free. Finished code
    regularly goes unintegrated for months at a time, being repeatedly resynced
    and re-diffed against new trees until the code's maintainer gets sick of it.
    This is extremely frustrating to developers, users, and vendors, and is
    burning out the maintainers. It is a huge source of unnecessary work. The
    situation needs to be resolved. Fast.

    If you think I'm preaching to the choir, skip to the next bit called "the
    solution". If not, read on...

    The Linux tree came very close to forking during 2.4. We went eleven months
    without a development tree. The founding of the Functionally Overloaded Linux
    Kernel project was a symptom of an enormous unintegrated patch backlog
    building up pressure until at least a small fork was necessary. Even with 2.5
    out, the current high number of seperate development trees accepting
    third-party is still alarmingly high. Linus and Marcelo have been joined by
    Dave Jones, Alan Cox, Andrea Arcangeli, Michael Cohen, and others, with
    distributors maintaining their own significantly forked kernel trees as a
    matter of course. Developers like Andrea, Robert Love and Rik van Riel either
    distribute others' relatively unrelated patches with their patch sets, or
    base their patches on other, as yet unintegrated patches for extended periods
    of time.

    Discussion of this problem was covered by kernel traffic and Linux Weekly
    News: (search for "patch management").

    During 2.4, the version skew between Alan Cox's tree and Linus's tree got as
    bad as it's ever been. Several of the bug fixes in Alan's tree (which he
    stopped maintaining months ago) still are not present in 2.4.17 or 2.5. Rik
    van Riel has publicly complained that Linus dropped his VM patches on the
    floor for several months, a contributing factor to the 2.4 VM's failure to
    stabilize for almost a -YEAR- after its release. (This is a bad sign. Whether
    Rik's or Andrea's VM is superior is a side issue. Alan Cox, and through him
    Red Hat, got Rik's VM to work acceptably by integrating patches from that
    VM's maintainer. The fact Linus didn't do as well is a symptom of this larger
    problem. The kind of subsystem swapping so major it requires a new maintainer
    should not be necessary during a stable series.)

    Speaking of Andrea Arcangeli, he's now integrating third-party patches into
    his own released development trees, because 2.5 isn't suitable to do
    development against and 2.4 doesn't have existing work (like low latency)
    he's trying to extend.

    Andre Hedrick just had to throw a temper tantrum to get any attention paid to
    his IDE patches, and he's the official IDE maintainer. Eric Raymond tells me
    his help file updates have now been ignored for 33 consecutive releases
    (again, he's the maintainer), and this combined with recent major changes
    Linus unilaterally made to 2.5's help files (still without syncing with the
    maintainer's version before doing so) has created a lot of extra and totally
    unnecessary work for Eric, and he tells me it's made the version skew between
    2.4 and 2.5 almost unmanageable.

    Andrew Morton's lock splitting low latency work has no forseeable schedule
    for inclusion, despite the fact it's needed whether or not Robert Love's
    preemption patch goes in. Ingo Molnar's O(1) scheduler did go in, but that
    was largely a case of good timing: it came right on the heels of a public
    argument about why Linus had not accepted patches to the scheduler for
    several years. The inclusion of code like JFS, XFS, and Daniel Phillips' ext2
    indexing patch are left up to distributions to integrate and ship long before
    they make it into Linus's tree. (Remember the software raid code in 2.2?)
    These are just the patches that have persisted, how much other good work
    doesn't last because its author loses interest after six months of the silent
    treatment? The mere existence of the "Functionally Overloaded Linux Kernel"
    (FOLK) project, to collect together as many unintegrated patches as possible,
    was a warning sign that things were not going smoothly on the patch
    integration front.

    The release of 2.5 has helped a bit, but by no means solved the problem. Dave
    Jones started his tree because 2.4 fixes Marcello had accepted were not
    finding their way into 2.5. Even code like Keith Owens' new build system and
    CML2, both of which Linus approved for inclusion at the Kernel summit almost
    a year ago and even tentatively scheduled for before 2.5.2, are still not
    integrated with no clear idea if they ever will be. (Yes Linus can change his
    mind about including them, but total silence isn't the best way to indicate
    this. Why leave Keith and Eric hanging, and wasting months or even years of
    their time still working on code Linus may not want?)

    The fact that 2.5 has "pre" releases seems suggestive of a change in mindset.
    A patch now has to be widely used, tested, and recommended even to get into
    2.5, yet how does the patch get such testing before it's integrated into a
    buildable kernel? Chicken and egg problem there, you want more testing but
    without integration each testers can test fewer patches at once.

    There has even been public talk of CRON jobs to resubmit patches to Linus
    periodically for as long as it takes him to get around to dealing with them.
    Linus has actually endorsed this approach (provided that re-testing of the
    patches against the newest release before they are remailed is also
    automated). This effort has spawned a mailing list. That's just nuts. The
    fact that Linus doesn't scale isn't going to be solved by increasing the
    amount of mail he gets. When desperate measures are being seriously
    considered, we must be in desperate times.

    -- The solution.

    The community needs to offload some work from Linus, so he can focus on what
    he does that nobody else can. This offloading and delegation has been done
    before, with the introduction of subsystem maintainers. We just need to
    extend the maintainer concept to include an official and recognized
    integration maintainer.

    During 2.1, when Linus burned out, responsibility for various subsystems were
    delegated to lieutenants to make Linus' job more manageable. Lieutenants are
    maintainers of various parts of the tree who collect and clean up patches,
    and make the first wave of obvious decisions ("this patch doesn't apply",
    "this bit doesn't compile", "my machine panicked", "look, it's not SMP safe")
    before sending tested code off to Linus. Linus still spends a lot of his time
    reading and auditing code, but by increasing the average quality of the code
    Linus looks at, the maintainers make his job easier. The more work
    maintainers can do the less Linus has to.

    So what tasks does Linus still personally do? He's an architect. He steers
    the project, vetoing patches he doesn't like and suggesting changes in
    direction to the developers. And he's the final integrator, pulling together
    dispirate patches into one big system.

    The job of architect is something only Linus can do. The job of integrator is
    something many people can do. Alan Cox did it for years. Dave Jones is doing
    it now, and Michael Cohen has yet another tree. Every Linux distributor has
    its own tree. Integrating patches so they don't conflict and porting them to
    new versions is hard work, but not brain surgery.

    Linus is acting as a central collection point for patches, and patches are
    getting lost on their way to that collection point. Patches as big as UML and
    EXT3 were going into Alan Cox's tree during 2.4, and now new patches are
    going into Dave Jones's tree to be tested out and queued for Linus.

    Integration is a task that can be delegated, and it has been. In Perl's
    model, Larry Wall is the benevolent dictator and architect, but integration
    work is done by the current holder of the "patch pumpkin". In Linux, Alan Cox
    used to be the de facto holder of the patch penguin, and now Dave Jones is
    maintaining a tree which can accept and integrate patches, and then feed them
    on to Linus when Linus is ready.

    This system should be formalized, we need the patch penguin to become
    official. The patch penguin seems to have passed from Alan Cox to Dave Jones.
    If we recognize this, we can make much better use of it.

    --- Ramifications.

    The holder of the patch penguin's job would be to accept patches from people
    other than linus, make them work together in a publicly compilable and
    testable tree, and feed good patches on to Linus. This may sound simple and
    obvious, but it's currenlty not happening and its noticeable by its absence.

    The purpose of the patch penguin tree is to make life easier, both for Linus
    and the developer community. With a designated patch collector and
    integrator, Linus's job becomes easier. Linus would still maintain and
    release his own kernel trees (the way he did when Alan Cox regularly fed him
    patches), and Linus could still veto any patch he doesn't like (whether it
    came from the patch penguin, directly from a subsystem maintainer, or
    elsewhere). But Linus wouldn't be asked to act as the kernel's public CVS
    tree anymore. He could focus on being the architect.

    The bulk of the patch penguin's work would be to accept, integrate, and
    update patches from designated subsystem maintainers, maintaining his own
    tree (seperate from linus's) containing the integrated collection of pending
    patches awaiting inclusion in Linus's tree. Patches submitted directly to the
    patch penguin could be redirected to subsystem maintainers where appropriate,
    or bounced with a message to that effect (at the patch penguin's option).
    This integration and patch tracking work is a fairly boring, thankless task,
    but it's work somebody other than Linus can do, which Linus has to do
    otherwise. (And which Linus is NOT doing a good job at right now.)

    The rest of the patch-penguin holder's job is to feed Linus patches. The
    patch penguin holder's tree would fundamentally be a delta against the most
    recent release from Linus (like the "-ac patches" used to be). If Linus takes
    several releases to get around to looking at a new patch, the patch penguin
    would resynchronize their tree with each new Linus release, doing the fixups
    on the pending patch set (or bullying the source of each patch into doing
    so). It would be the patch penguin's job to keep up with Linus, not the other
    way around. When a change happens in Linus's tree that didn't come from the
    patch penguin, the patch penguin integrates the change into their tree

    The holder of the patch penguin would feed Linus good patches, by Linus's
    standards. Not just tested ones, but small bite-sized patches, one per email
    as plain text includes, with an explanation of what each patch does at the
    top of the mail. (Just the way Linus likes them. :) Current pending patches
    from the patch penguin tree could even be kept at a public place (like so Linus could pull rather than push, and grab them when he has
    time. The patch penguin tree would make sure that when Linus is ready for a
    patch, the patch is ready for Linus.

    The patch penguin tree can act as a buffer between Linus and the flood of
    patches from the field. When Linus is not ready for a patch yet, he can hold
    off on taking it into his tree, and doesn't have to worry about the patch
    being lost or out of date by the time he's ready to accept it. When Linus is
    focusing on something like the new block I/O code, the backlog of other
    patches naturally feeds into the patch penguin tree until Linus is ready to
    look at them. People won't have to complain about dropped patches, and Linus
    doesn't have to worry that patches haven't been tested enough before being
    submitted to him. Users who want to live on the really bleeding edge have a
    place to go for a kernel that's likely to break. Testers can find bugs en
    masse without having to do integration work (which is in and of itself a
    source of potential bugs).

    Linus would still have veto power. He gets to reject any patch he doesn't
    like, and can ask for the integration lieutenant to back that patch out of
    the patch penguin tree. That's one big difference between the patch penguin
    tree and Linus's tree: the patch penguin tree is provisional. Stuff that goes
    in it can still get backed out in a version or two. Of course Linus would
    have to explicitly reject a patch to get it out of the patch penguin tree,
    meaning its developer stops fruitlessly re-submitting it to Linus, and maybe
    even gets a quick comment from Linus as to WHY it was unacceptable so they
    can fix it. (From the developer's point of view, this is a good thing. They
    either get feedback of closure.)

    Linus sometimes needs time off. Not just for vacations, but to focus on
    specific subsections, like integrating Jens Axobe's BIO patches at the start
    of 2.5. Currently, these periods hopelessly stall or tangling development.
    But in Linus's absence, the patch penguin could continue to maintain a delta
    against the last Linus tree, and generate a sequence of small individual
    patches like a trail of bread crumbs for Linus to follow when he gets back.
    Linus could take a month off, and catch back up in a fraction of that time
    when he returned. (And if Linus were to get hit by a bus, the same
    infrastructure would allow the community to select and support a new
    architect, which might help companies like IBM sleep better at night.) And if
    Linus rejected patches halfway through the bread crumb trail requiring a lot
    of shuffling in later patches, well, that's more work for the patch penguin,
    not more work for Linus.

    One reason Linus doesn't like CVS is he won't let other people check code
    into his tree. (This is not a capricious decision on Linus's part: no
    architect can function if he doesn't know what's in the system. Code review
    of EVERYTHING is a vital part of Linus's job.) With a patch penguin tree,
    there's no more pressure on Linus to use CVS. The patch penguin can use CVS
    if he wants to, and if he wants to give the subsystem maintainers commit
    access to the patch penguin tree, that's his business. The patch penguin's
    own internal housekeeping toolset shouldn't affect the patches he feeds on to
    Linus one way or the other.

    Again, Linus likes stuff tested by a wide audience before it's sent to him.
    With a growing list of multiple trees maintained by Dave Jones, Alan Cox,
    Michael Cohen, Andrea Arcangeli, development and testing become fragmented.
    With a single patch penguin tree, the patches drain into a common pool and
    the backlog of unintegrated patches can't build up dangerous amounts of
    pressure to interfere with development. A single shared "pending" tree means
    the largest possible group of potential testers.
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2005-03-22 13:15    [W:0.057 / U:3.136 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site