[lkml]   [2007]   [Jun]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: 2.6.22-rc5 regression
    On Mon, Jun 18, 2007 at 10:01:34AM -0700, Linus Torvalds wrote:
    > On Sun, 17 Jun 2007, Carlo Wood wrote:
    > >
    > > If you want my opinion on this: git bisect is broken :p
    > > I was very surprised that it printed this at this point.
    > Hmm. Possible. However, I *really* would need the git bisect log to see
    > what's up.

    I had already done a git bisect reset :/

    > So far, we have never seen a bug in "git bisect" that wasn't
    > either due to the user specifying path-names to limit the testing (and the
    > bug not being in that set of path-names), or the user not realizing that
    > with non-linear history the "git bisect" is actually a fairly complex op.

    I realize that it is non-linear - and I didn't do anything to "speed
    things up". Just build -> boot -> git bisect good/bad -> build etc.

    > That said, "git bisect" _can_ give the "wrong" results in the sense that
    > the commit it points to may not be the one you are actually looking for,
    > if:
    > - the bug is sporadic, and not entirely repeatable, and a kernel you
    > marked as good wasn't really good, your test just didn't happen to
    > catch it that time around.
    > - the bug comes and goes, and the commit that "git bisect" pinpoints may
    > well *show* the bug, but may not be the *cause* of the bug (ie there
    > might be two or more independent things that have to come together for
    > the bug to trigger, and as a result there is not a "single" commit that
    > acts as a clear boundary)

    I don't believe that either of these is the case. I have booted
    about 8 different kernel revisions several times (most three times),
    alternating between them (not the same three times on a row) and the
    ones that boot correctly always booted correctly, while the ones that
    hung, always hung (although in a totally reproducable way).

    Nevertheless, it is not 100% impossible. I have the feeling that it
    MIGHT be related to the fact that I have 4 CPU's - and that perhaps some
    race condition is involved. And if that is the case, then there might be
    some kernel version where boot/not-boot is less reliable then with the
    eight I tested - apart from that three times isn't very much (but doing
    it for four good kernels and two bad kernels, it still is a reasonable
    indication for reproducability).

    > But hey, a bug in "git bisect" is certainly _possible_. I just consider it
    > fairly unlikely by now.
    > > One bisect before, it said there were still 96 revision to check.
    > >
    > > Anyway - here are some facts of the kernels that I tested:
    > You seem to not actually have used "git bisect" to generate this list.

    I didn't -- what is the command to generate a list like this with git?

    > Quite frankly, the most likely cause for the bad bisection result is that
    > you have not used "git bisect" at all to let it pick the bisection points.

    Heh - now you are insulting me :p I said I did, and I did.
    The reason that I made that list manually is because wanted to have more
    overview. I use this alias to build the kernels:

    hikaru:/usr/src/kernel/git/linux-2.6>alias build
    alias build='cp /boot/config-2.6.22-rc4-hikaru-amd64 .config &&
    make-kpkg clean && VER=$(date +"%Y%m%d%H%M") && BRANCH=$(git branch |
    grep "^\*" | sed -e "s/\* //") && NAMEEXT="-$BRANCH-$(git rev-list
    --max-count=1 $BRANCH)-$(dpkg-architecture -qDEB_HOST_ARCH)" &&
    make-kpkg --revision=$VER --append-to-version=-$NAMEEXT --rootcmd
    fakeroot clean && make-kpkg --revision=$VER --append-to-version=$NAMEEXT
    --rootcmd fakeroot --initrd kernel_image modules_image'

    That results in debian kernel package names like:

    -rw-r--r-- 1 carlo carlo 18790180 2007-06-17 22:55 linux-image-2.6.22-rc4-bisect-d09c6b809432668371b5de9102f4f9aa6a7c79cc-amd64_200706172247_amd64.deb
    -rw-r--r-- 1 carlo carlo 18790446 2007-06-17 22:42 linux-image-2.6.22-rc4-bisect-ce9b2b0abbf019d5259eb089a1cc256852930f67-amd64_200706172234_amd64.deb
    -rw-r--r-- 1 carlo carlo 18789884 2007-06-17 22:24 linux-image-2.6.22-rc4-bisect-d1be0a8225f2cb1cdc356ebb0ae6800f023ce67d-amd64_200706172216_amd64.deb
    -rw-r--r-- 1 carlo carlo 18790420 2007-06-17 22:09 linux-image-2.6.22-rc4-bisect-de7f928ca460005086a8296be07c217aac4b625d-amd64_200706172201_amd64.deb
    -rw-r--r-- 1 carlo carlo 18790430 2007-06-17 21:04 linux-image-2.6.22-rc4-bisect-3e903e7b1605aff88d7f89a96fab5e43081b914f-amd64_200706172032_amd64.deb
    -rw-r--r-- 1 carlo carlo 18789698 2007-06-17 20:21 linux-image-2.6.22-rc4-bisect-cf68676222e54cd0a31efd968da00e65f9a0963f-amd64_200706172012_amd64.deb
    -rw-r--r-- 1 carlo carlo 18789796 2007-06-17 18:51 linux-image-2.6.22-rc5-master-188e1f81ba31af1b65a2f3611df4c670b092bbac-amd64_200706171843_amd64.deb
    -rw-r--r-- 1 carlo carlo 18814550 2007-06-17 17:53 linux-image-2.6.22-rc1-bisect-c4ca881796b7e14120851ddf6e04845ef94a314a-amd64_200706171745_amd64.deb
    -rw-r--r-- 1 carlo carlo 18814168 2007-06-17 17:14 linux-image-2.6.22-rc1-bisect-9614ece14f23f2ce54a076c471aec9c91e51e79c-amd64_200706171646_amd64.deb
    -rw-r--r-- 1 carlo carlo 18815254 2007-06-17 07:25 linux-image-2.6.22-rc1-bisect-df80b148869291621ddf51eb8716658d5bfba811-amd64_200706170717_amd64.deb
    -rw-r--r-- 1 carlo carlo 18826878 2007-06-17 06:36 linux-image-2.6.22-rc4-bisect-b44c0267b7571b449e05f390349c4e4d080f0f4c-amd64_200706170628_amd64.deb
    -rw-r--r-- 1 carlo carlo 18826402 2007-06-17 04:20 linux-image-2.6.22-rc4-bisect-e141d999b682cda9907179e3b843acb64c34a1d8-amd64_200706170412_amd64.deb
    -rw-r--r-- 1 carlo carlo 18830334 2007-06-17 03:30 linux-image-2.6.22-rc4-bisect-3334500b460a5eede2e3466ca97a90fe3b91ceb5-amd64_200706170323_amd64.deb
    -rw-r--r-- 1 carlo carlo 18827134 2007-06-17 02:50 linux-image-2.6.22-rc4-bisect-bb3d2dd72302ea3eefcc6738cdd39ed5864b62f8-amd64_200706170243_amd64.deb
    -rw-r--r-- 1 carlo carlo 18824900 2007-06-17 02:37 linux-image-2.6.22-rc4-bisect-1a539a87280b3032fd12bc93a4a82f1d8aa97ca8-amd64_200706170230_amd64.deb
    -rw-r--r-- 1 carlo carlo 18829904 2007-06-16 16:24 linux-image-2.6.22-rc4-master-99f9f3d49cbc7d944476f6fde53a77ec789ab2aa-amd64_200706161616_amd64.deb
    -rw-r--r-- 1 carlo carlo 18818812 2007-06-16 05:59 linux-image-2.6.22-rc4-master-5ecd3100e695228ac5e0ce0e325e252c0f11806f-amd64_200706160551_amd64.deb
    -rw-r--r-- 1 carlo carlo 18818284 2007-06-15 21:39 linux-image-2.6.22-rc3-bisect-c420bc9f09a0926b708c3edb27eacba434a4f4ba-amd64_200706152131_amd64.deb
    -rw-r--r-- 1 carlo carlo 18823918 2007-06-15 21:26 linux-image-2.6.22-rc4-bisect-5ecd3100e695228ac5e0ce0e325e252c0f11806f-amd64_200706152119_amd64.deb
    -rw-r--r-- 1 carlo carlo 18817072 2007-06-15 18:13 linux-image-2.6.22-rc3-bisect-0e9871df2389560e94ba01e40959140ee56def4b-amd64_200706151805_amd64.deb
    -rw-r--r-- 1 carlo carlo 18815324 2007-06-15 17:41 linux-image-2.6.22-rc2-bisect-55b637c6a003a8c4850b41a2c2fd6942d8a7f530-amd64_200706151733_amd64.deb

    or after installation:

    ||/ Name Version Description
    ii linux-image-2.6.22-rc4-bisect-3e903e7b1605aff88d7f89a96fab5e43081b914f-amd64 200706172032 Linux kernel binary image for version 2.6.22

    and kernel versions like 2.6.22-rc4-bisect-3e903e7b1605aff88d7f89a96fab5e43081b914f-amd64 etc.
    Heh - hopefully you have 20" monitors like met :p

    However - it didn't give me an overview of the git bisect good/bad
    process. Therefore I used 'gitk -all' and it's search function to
    find all the kernels that I tested and put them in a file, such
    creating that list I posted here. If there is a way to generate
    this list with a command, please tell me :)

    > That really doesn't work. If you start giving "git bisect" points to test
    > that aren't "within" the space of points you had already told git bisect
    > about, you're no longer bisecting, you're giving it random points.

    Some online documention said you can use git reset --hard gitId to
    choose a different point nearby what git bisect 'suggests' to test next.
    I didn't do that in this case however.

    > > I'd appreciate any suggestions or questions at this point, as I have no
    > > idea what to do next to find this problem.
    > If you do a real "git bisect", and don't just give it random points that
    > you want to check (you obviously _do_ need to give it one "good" and one
    > "bad" initially, but after that you *have* to pick a point that is within
    > the query space), you'll not get any sensible values out of git bisect.

    I suppose you mean: ... then you WILL get sensible values out of git
    bisect. But, since I already did a real "git bisect" without giving it
    random points, I am afraid you jumped conclusions.

    > "git bisect" will also give you a log in ".git/BISECT_LOG", which others
    > can use to follow your bisection. That might be useful to see.

    No such file exists (anymore).

    However, I can easily reproduce it. From my history file I can see that I started with:

    git bisect start
    git bisect bad v2.6.22-rc5
    git bisect good 99f9f3d49cbc7d944476f6fde53a77ec789ab2aa

    I wrote everything down on paper (the git id's and whether they
    were good or bad), so I can reproduce it with:

    hikaru:/usr/src/kernel/git/linux-2.6>git bisect start
    hikaru:/usr/src/kernel/git/linux-2.6>git bisect bad v2.6.22-rc5
    hikaru:/usr/src/kernel/git/linux-2.6>git bisect good 99f9f3d49cbc7d944476f6fde53a77ec789ab2aa
    Bisecting: 128 revisions left to test after this
    D include/asm-blackfin/macros.h
    M scripts/package/Makefile
    D scripts/package/builddeb
    [cf68676222e54cd0a31efd968da00e65f9a0963f] Blackfin serial driver: actually implement the break_ctl() function
    hikaru:/usr/src/kernel/git/linux-2.6>git bisect good
    Bisecting: 111 revisions left to test after this
    D include/asm-blackfin/macros.h
    M scripts/package/Makefile
    D scripts/package/builddeb
    [3e903e7b1605aff88d7f89a96fab5e43081b914f] cpuset: zero malloc - fix for old cpusets
    hikaru:/usr/src/kernel/git/linux-2.6>git bisect bad
    Bisecting: 103 revisions left to test after this
    D include/asm-blackfin/macros.h
    M scripts/package/Makefile
    D scripts/package/builddeb
    [de7f928ca460005086a8296be07c217aac4b625d] Merge
    hikaru:/usr/src/kernel/git/linux-2.6>git bisect bad
    Bisecting: 98 revisions left to test after this
    D include/asm-blackfin/macros.h
    M scripts/package/Makefile
    D scripts/package/builddeb
    [d1be0a8225f2cb1cdc356ebb0ae6800f023ce67d] ide-scsi: fix OOPS in idescsi_expiry()
    hikaru:/usr/src/kernel/git/linux-2.6>git bisect bad
    Bisecting: 97 revisions left to test after this
    D include/asm-blackfin/macros.h
    M scripts/package/Makefile
    D scripts/package/builddeb
    [ce9b2b0abbf019d5259eb089a1cc256852930f67] Resume from RAM on HPC nx6325 broken
    hikaru:/usr/src/kernel/git/linux-2.6>git bisect bad
    Bisecting: 96 revisions left to test after this
    D include/asm-blackfin/macros.h
    M scripts/package/Makefile
    D scripts/package/builddeb
    [d09c6b809432668371b5de9102f4f9aa6a7c79cc] mm: Fix memory/cpu hotplug section mismatch and oops.
    hikaru:/usr/src/kernel/git/linux-2.6>git bisect bad
    d09c6b809432668371b5de9102f4f9aa6a7c79cc is first bad commit
    commit d09c6b809432668371b5de9102f4f9aa6a7c79cc
    Author: Paul Mundt <>
    Date: Thu Jun 14 15:13:16 2007 +0900

    mm: Fix memory/cpu hotplug section mismatch and oops.

    When building with memory hotplug enabled and cpu hotplug disabled, we
    end up with the following section mismatch:

    WARNING: mm/built-in.o(.text+0x4e58): Section mismatch: reference to
    .init.text: (between 'free_area_init_node' and '__build_all_zonelists')

    This happens as a result of:

    -> free_area_init_node()
    -> free_area_init_core()
    -> zone_pcp_init() <-- all __meminit up to this point
    -> zone_batchsize() <-- marked as __cpuinit fo

    This happens because CONFIG_HOTPLUG_CPU=n sets __cpuinit to __init, but
    CONFIG_MEMORY_HOTPLUG=y unsets __meminit.

    Changing zone_batchsize() to __devinit fixes this.

    __devinit is the only thing that is common between CONFIG_HOTPLUG_CPU=y and
    CONFIG_MEMORY_HOTPLUG=y. In the long run, perhaps this should be moved to
    another section identifier completely. Without this, memory hot-add
    of offline nodes (via hotadd_new_pgdat()) will oops if CPU hotplug is
    not also enabled.

    Signed-off-by: Paul Mundt <>
    Acked-by: Yasunori Goto <>
    Signed-off-by: Linus Torvalds <>


    mm/page_alloc.c | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)

    :040000 040000 230b105fa4d9eb2ed873cca8e9ec1b5502ffce79 37636618f4eb88827ec1e524ebb3ac37e44e90f1 M mm

    and (useless, on top of the above, now I see it):

    hikaru:/usr/src/kernel/git/linux-2.6>git bisect log
    git-bisect start
    # bad: [aec07c7abc280bd5d0ca33b7cda3eb7b9b6e89c1] Linux 2.6.22-rc5
    git-bisect bad aec07c7abc280bd5d0ca33b7cda3eb7b9b6e89c1
    # good: [99f9f3d49cbc7d944476f6fde53a77ec789ab2aa] Merge branch 'for-linus' of git://
    git-bisect good 99f9f3d49cbc7d944476f6fde53a77ec789ab2aa
    # good: [cf68676222e54cd0a31efd968da00e65f9a0963f] Blackfin serial driver: actually implement the break_ctl() function
    git-bisect good cf68676222e54cd0a31efd968da00e65f9a0963f
    # bad: [3e903e7b1605aff88d7f89a96fab5e43081b914f] cpuset: zero malloc - fix for old cpusets
    git-bisect bad 3e903e7b1605aff88d7f89a96fab5e43081b914f
    # bad: [de7f928ca460005086a8296be07c217aac4b625d] Merge
    git-bisect bad de7f928ca460005086a8296be07c217aac4b625d
    # bad: [d1be0a8225f2cb1cdc356ebb0ae6800f023ce67d] ide-scsi: fix OOPS in idescsi_expiry()
    git-bisect bad d1be0a8225f2cb1cdc356ebb0ae6800f023ce67d
    # bad: [ce9b2b0abbf019d5259eb089a1cc256852930f67] Resume from RAM on HPC nx6325 broken
    git-bisect bad ce9b2b0abbf019d5259eb089a1cc256852930f67
    # bad: [d09c6b809432668371b5de9102f4f9aa6a7c79cc] mm: Fix memory/cpu hotplug section mismatch and oops.
    git-bisect bad d09c6b809432668371b5de9102f4f9aa6a7c79cc

    Carlo Wood <>
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2007-06-18 20:15    [W:0.044 / U:0.064 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site