oneliner / finding fixed kernel bugs

Recently we were bitten by an old kernel bug on Ubuntu that only rarely triggers. Finding out where the problem was is easier if you know where to look.

We had no kernel logs to go on. Only a hanging machine with no output. And a hunch that the running kernel version linux-image-5.4.0-122 was the likely culprit. Was there another way than meticulously reading all changelogs and changes to find out which bug we're dealing with?

For issues that are already fixed in a newer kernel, the following recipe seems to work. (Note that we would like to use commit ids instead of string matching, but this won't work because Ubuntu cherry-picks the commits, altering the commit ids.)

Step 1: get the Ubuntu kernel repo:

$ git clone git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal \
    linux-ubuntu-focal

Step 2: check which versions have been released:

$ git -c tag.sort=-v:refname tag | grep Ubuntu-5.4.0
Ubuntu-5.4.0-144.161
Ubuntu-5.4.0-143.160
...
Ubuntu-5.4.0-123.139
Ubuntu-5.4.0-122.138
Ubuntu-5.4.0-121.137
Ubuntu-5.4.0-120.136
...

Step 3: pick which versions we want to check. In this case, we're assuming that the bug was introduced in Ubuntu-5.4.0-122.138, so we'll plug in Ubuntu-5.4.0-121.137..Ubuntu-5.4.0-122.138. And we'll plug in Ubuntu-5.4.0-122.138..Ubuntu-5.4.0-144.161 so we get all versions to the end.

$ git log Ubuntu-5.4.0-121.137..Ubuntu-5.4.0-122.138 --format=%s |
    grep -v ^UBUNTU: |
    while read -r l; do res=$(
      git log Ubuntu-5.4.0-122.138..Ubuntu-5.4.0-144.161 \
        --grep "Fixes:.*$(printf '%s' "$l" | sed -e 's/[^A-Za-z0-9 ]/./g')" \
        --oneline)
    test -n "$res" && echo "$res (fixes: $l)"
  done

97842ea930e0 tcp: make sure treq->af_specific is initialized (fixes: tcp: md5: incorrect tcp_header_len for incoming connections)

$ git show 97842ea930e0
commit 97842ea930e0eb94bfdb87beaf87d56224c1e8ad
Author: Eric Dumazet <edumazet@google.com>
Date:   Sun Apr 24 13:35:09 2022 -0700

    tcp: make sure treq->af_specific is initialized
...
    Fixes: 5b0b9e4c2c89 ("tcp: md5: incorrect tcp_header_len for incoming connections")

Nice. One result. Seems to be the one we're looking for.

$ git -c tag.sort=-v:refname tag --contains 97842ea930e0 | tail -n1
Ubuntu-5.4.0-123.139

And it was included in the next release already.