[lkml]   [2011]   [Sep]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectBackporting the Linux kernel, for good - was: Re: semantic patch inference
    Taking this to the lists for more eyeball review. Some background to
    folks who were not at Plumbers or at the 'Backporting the Linux
    kernel, for good' BoF:

    In short there are some metrics I have finally evaluated from the
    compat work we have. Each backported tarball provides metrics of how
    much lines of code we take from upstream, how much work is required
    through patches for backport work, and also how many lines of code go
    into the generic Linux kernel compatibility module. The talk dissected
    these metrics for the first time and provides some conclusions as to
    what we can do better, what the estimated cost is of backporting new
    subsystems, and some evolutionary practices that we can likely embrace
    to make backporting even more automatic which would not only use
    spatch but also Jesper's spdiff work. If you are interested in the
    metrics stuff please review the slides and if you have questions feel
    free to shoot here but below I review with Julia inquiries about
    assumptions on how we backport stuff in consideration of spdiff work.

    For those curious of spdiff here's a paper on 'Generic Patch
    Inference' which explains the logistics behind it:

    In short, spatch files can be used on target directories to generate
    patches. spdiff can read a patch file and generate an spatch file for
    you. What this means for the backporting world is if you backport one
    evolutionary change in the Linux kernel for one driver you can then
    backport the same change for *all* drivers. This is a quantum leap in
    terms of effort required to backport.

    Anyway, some details on review of assumptions below.

    On Fri, Sep 9, 2011 at 6:18 AM, Julia Lawall <> wrote:
    > Hi,
    > After some sleep it all seems more clear :)


    > First, the choice of patches from linux-next is not our problem.

    Exactly! We just

    cp -a $COMPAT/ .
    cp -a $LINUX_NEXT/subsystem/ .
    for i in foo bar
    cp -a drivers/$i .

    > Second, perhaps we could assume that the patches in compat are always of
    > the form
    > +#ifdef version <= n
    > +new code
    > +#else
    > old code
    > +#endif
    > (and variants - I see that sometimes there is >).  We are going to want to
    > treat that as
    > + new code
    > - old code
    > for the inference process, but I can see that it is very important to have
    > the version numbers associated with this information, so that can be
    > regenerated in the chosen patch.

    Yes, the variance of the #if's expands to other cases as well like:

    netdev->hw_features = NETIF_F_SG | NETIF_F_HW_VLAN_RX;
    +#if defined(NETIF_F_HW_VLAN_TX) || (LINUX_VERSION_CODE >=
    netdev->features |= (NETIF_F_HW_VLAN_TX | NETIF_F_HW_VLAN_RX);

    But I'm not even sure why I allowed this to go in, it makes more sense
    to remove the flag check on the second if defined. That sort of check
    though might be an alternative approach we can check for instead of
    kernel versions though, Johannes wanted to see if he can do
    backporting this way a while ago but I am not sure up to where he got
    up to. Reason for checking for features instead of kernel versions is
    some Linux distributions mangle their kernels with features from
    future kernels and still keep the same kernel release number. This
    makes some kernel backport assumptions incorrect for some of these
    distributions and this is why we likely have some of these feature
    checks in addition to kernel version checks.

    At this point though I feel as if adding a feature label for each
    backport work item might be a bit too much work, and sticking to
    kernel versions is the way to go. If the work to deal with other
    variants is a bit too much then I'd say we can keep those oddball
    variants in separate files.

    Additionally there is quite a bit of work on the patches/ directory
    that can be done to help improve the situation for inspection
    purposes. Although each file there does handle specific backport work
    items they can be split out even more atomically. Consider the
    01-netdev.patch. That has the netdev ops thingy, but also

    + /* can't we remove this? */
    + net->change_mtu = NULL;

    Note that I am not sure if we can just ignore this, but its just one
    example. This random set of bits that do not relate to netdev ops can
    be extracted into another file -- unless the work you have may be able
    to address multiple work items from one file. If its just targeted for
    one work item we can clean this stuff up further to help with that

    > Jesper's work takes a patch and the before and after versions of the
    > given files in the original code, and finds a semantic patch that takes
    > the before code safely to the after code.


    > But compat-wireless seems to apply to all versions.

    Nope, the patches in the patches/ directory are targeted to apply
    cleanly to work in linux-next.git. At times the patches may not apply
    cleanly with hunk offsets being off a bit but that is cleaned up
    easily through quilt refresh (./scripts/ refresh),
    after that the patches will apply cleanly onto linux-next.git.

    I should note though that the patches in the patches/ directory
    account for backporting effort for *all* supported kernel versions
    though. Today my goal is to try at the very least to support down to
    the last *supported* 2.6 stable kernel. Work for older kernels is
    welcomed though but I think supporting down to the latest stable is a
    decent goal.

    > So he should interpret the #ifs to figure out what versions a specific change applies to,

    Right, to what kernel version, lets call it, a 'backport work item' refers to.

    > and then apply the semantic patch inference process to all of those versions?

    Yes. But I should note that if the semantic patch inference process
    works atomically on only one type of change per reviewed file then
    some work is required to split up the patches a bit better. In the
    01-netdev.patch we have *most* netdev evolutionary changes in the
    kernel that we need to backport, not just one.

    > I guess he could take a compat patch and a list of versions, and generate a
    > patch with the appropriate
    > + new code
    > - old code
    > for each of those versions.

    Typically the patches address changes for *any* kernel release older
    than specific version, splitting the changes for individual kernels is
    possible but would be possible but would lead to a bit of redundancy
    in work required given that the same change is required for multiple
    older supported kernel versions. I will note though that at times the
    checks may make some exception to one kernel, but I do not think that
    the patches/ directory has any of those, instead I think we have that
    dealt with in the compat module -- that is not in patch files.

    > Then he could run his tool on those patches and the corresponding versions, and then
    > make a semantic patch that includes the appropriate #ifs that could apply to all versions.

    Well so in practice the patches/ files address a range of kernels for
    which a backport work item is required for can the tool can be made to
    infer a set of conditions under which the work item applies to ? In
    this case it would be a range of kernel versions.

    > If that all seems correct to you, perhaps it is not necessary for you to
    > come up from SFO today :)

    Ace! I got home at 11pm so it'd be great if I can stay but I do value
    this work ****a lot**** so will head out shortly if you think its best
    to review in person.

    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2011-09-09 22:01    [W:0.035 / U:1.780 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site