dirmngr / keyserver / disable-ipv6 / bionic

This morning a build pipeline failed. dirmngr called by apt-key tried to use IPv6, even though it was disabled.

The build logs had this to say:

21.40 + apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
21.50 Warning: apt-key output should not be parsed (stdout is not a terminal)
21.57 Executing: /tmp/apt-key-gpghome.KzTTOZjgZP/gpg.1.sh --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
21.65 gpg: keyserver receive failed: Cannot assign requested address

This was strange for a number of reasons:

1. The same Docker build scripts succeeded for other Ubuntu releases. The problem only seemed to affect the bionic based build: on focal it was not broken.

2. I could reproduce in a similar bionic Docker environment. But behaviour appeared to be random — maybe based on the order of the DNS responses:

   # apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
   Executing: /tmp/apt-key-gpghome.2XtcT1xWZb/gpg.1.sh --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
   gpg: keyserver receive failed: Cannot assign requested address
   

   # apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
   Executing: /tmp/apt-key-gpghome.TyIbtADNXD/gpg.1.sh --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
   gpg: key F1656F24C74CD1D8: "MariaDB Signing Key <signing-key@mariadb.org>" not changed
   gpg: Total number processed: 1
   gpg:              unchanged: 1
   

3. Calling it with strace showed that it tried to connect to an IPv6 address:

   # strace -e 'signal=!all' -e trace=clone,connect,execve -s 32 -f \
       apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 \
         0xF1656F24C74CD1D8 2>&1 |
       grep -vE 'attached$|exited with|resumed>'
   ...
   [pid  9749] execve("/usr/bin/dirmngr", ["dirmngr", "--daemon", "--homedir", "/tmp/apt-key-gpghome.0zrNQAlNLG"], 0x7fff73d35cb8 /* 9 vars */ <unfinished ...>
   ...
   [pid  9749] clone(child_stack=NULL, flags=..., child_tidptr=...) = 9750
   ...
   [pid  9750] clone(child_stack=..., flags=..., parent_tidptr=..., tls=..., child_tidptr=...) = 9751
   ...
   [pid  9751] connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.1.1.1")}, 16) = 0
   [pid  9751] connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.1.1.1")}, 16) = 0
   [pid  9751] connect(7, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2620:2d:4000:1007::70c", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address)
   gpg: keyserver receive failed: Cannot assign requested address
   ...
   

   That log reads as: apt-key spawns (something that spawns) dirmngr, which forks twice and then does two DNS lookups (via 1.1.1.1) and finally tries to connect to 2620:2d:4000:1007::70c.
   
   That is unexpected because...

4. In this Docker environment there wasn't even any IPv6 at all:

   # ip -br a
   lo               UNKNOWN        127.0.0.1/8
   eth0@if153       UP             172.17.0.2/16
   

   No assigned IP address.

   # sysctl -a 2>/dev/null | grep disable_ipv6
   net.ipv6.conf.all.disable_ipv6 = 1
   net.ipv6.conf.default.disable_ipv6 = 1
   net.ipv6.conf.eth0.disable_ipv6 = 1
   net.ipv6.conf.lo.disable_ipv6 = 1
   

   And in fact IPv6 is disabled on all interfaces through sysctl.

5. If getaddrinfo() with AF_UNSPEC is used for the DNS lookup, it shouldn't even try to get an IPv6 address, because dns4only is enabled by default in this environment. With dns4only enabled requesting an unspecified address family (IPv4 or IPv6), only results in IPv4 responses.
   
   Without dns4only, you'd see this Python transcript:

   >>> from socket import *
   >>> set([i[4][0] for i in getaddrinfo('keyserver.ubuntu.com', 80)])
   {'2620:2d:4000:1007::d43', '185.125.188.27', '185.125.188.26', '2620:2d:4000:1007::70c'}
   

   But here, Python correctly yields only these:

   >>> from socket import *
   >>> set([i[4][0] for i in getaddrinfo('keyserver.ubuntu.com', 80)])
   {'185.125.188.27', '185.125.188.26'}
   

   This meant that gnupg dirmngr was doing something custom with DNS resolving, like manually looking up both IPv6 and IPv4.

6. The offending dirmngr has version 2.2.4-1ubuntu1.6 and according to the changelog (1.3..1.5 diff) there was a fix for IPv6 in 2.2.4-1ubuntu1.4; but we were already using that.

7. Interestingly, that bugfix from LP#1910432 does point to the means of checking connectivity. It checks whether socket(AF_INET6) succeeds instead of whether connect(ipv6_address) succeeds:

   $ cat debian/patches/dirmngr-handle-EAFNOSUPPORT-at-connect_server.patch
   ...
   @@ -2940,6 +2942,15 @@ connect_server (const char *server, unsi
              sock = my_sock_new_for_addr (ai->addr, ai->socktype, ai->protocol);
              if (sock == ASSUAN_INVALID_FD)
                {
   +              if (errno == EAFNOSUPPORT)
   +                {
   +                  if (ai->family == AF_INET)
   +                    ignore_v4 = 1;
   +                  if (ai->family == AF_INET6)
   +                    ignore_v6 = 1;
   +                  continue;
   ...
   

   But the exposed Linux kernel interface (version 5.x) has no problem with creating an AF_INET6 socket:

   >>> from socket import *
   >>> s = socket(AF_INET6)  # no failure on this line
   >>> s.connect(('keyserver.ubuntu.com', 80))
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   OSError: [Errno 99] Cannot assign requested address
   

   It's first the connect() that fails — with a EADDRNOTAVAIL: Cannot assign requested address.

8. However, the dirmngr code in the Ubuntu focal version doesn't seem that different. That one also does two DNS lookups too, but that one correctly selects AF_INET when connecting.
   
   The lookups as seen with strace -e trace=sendto:

   >>> print(':'.join('%02x' % (i,) for i in (
   ...     b"\317/\1\0\0\1\0\0\0\0\0\0\tkeyserver\6ubuntu\3com\0\0\1\0\1")))
   cf:2f:01:00:00:01:00:00:00:00:00:00:09:6b:65:79:73:65:72:76:65:72:06:75:62:75:6e:74:75:03:63:6f:6d:00:00:01:00:01
   >>> print(':'.join('%02x' % (i,) for i in (
   ...     b"\307\275\1\0\0\1\0\0\0\0\0\0\tkeyserver\6ubuntu\3com\0\0\34\0\1")))
   c7:bd:01:00:00:01:00:00:00:00:00:00:09:6b:65:79:73:65:72:76:65:72:06:75:62:75:6e:74:75:03:63:6f:6d:00:00:1c:00:01
   

   (Here the lookup with 00:1c is the the AAAA query.)

Really unexpected. And another example of old software sinking development time. I hope this writeup saves someone else a little time.

Workarounds?

For a while, I thought the dirmngr code might respect /etc/gai.conf and tried to add/enable precedence ::ffff:0:0/96 100 there. But that just proved to be intermittent (random) success.

A shittier, but working, workaround is hacking the /etc/hosts file with this oneliner in the Dockerfile:

{ test "$UBUNTU_RELEASE" = "bionic" && \
  getent ahosts keyserver.ubuntu.com | \
  sed -e '/STREAM/!d;s/[[:blank:]].*/ keyserver.ubuntu.com/' | \
   shuf >> /etc/hosts || true; }

Not pretty, but it works. And it only affects the (soon to be obsolete) bionic build.

An alternative solution is provided at usbarmory-debian-base_image#9:

{ mkdir -p ~/.gnupg && \
  echo "disable-ipv6" >> ~/.gnupg/dirmngr.conf && \
  apt-key adv --homedir ~/.gnupg --keyserver ...; }