viewing unencrypted traffic / ltrace / bpftrace

Written by
Walter Doekes
Published on 2023-07-03

Can we view TLS-encrypted traffic on the originating or terminating host, without having to decode the data from the wire?

This is a question that comes up every now and then when trying to debug a service by looking at how it communicates.

For the most insight, we should capture the encrypted traffic and use the (logged!) pre-master secret keys. See example details in make-master-secret-log discussing how to have HAProxy log them. Oftentimes this involves a lot of work: capturing the traffic, logging the keys, collecting both and running them through Wireshark on a local machine.

If we have access to the encryption libraries, there can be easier ways.

Using ltrace?

We should be able to use ltrace which traces the invocation of library calls. ltrace is the library counterpart of strace which traces system calls. Looking at the network system calls (recvfrom or sendto) would be useless: we'd see the encrypted traffic. Encryption is done in userland, so we need to be one step closer to the application.

By using ltrace, we can trace function calls before data gets encrypted and after data gets decrypted. In this case we're interested in SSL_read and SSL_write from the OpenSSL library — assuming that that is the TLS library our application is using.

Let's try it on cURL. This cURL build is compiled against OpenSSL, as is common:

$ curl --version | grep -i ssl

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 ...

Invoke it from ltrace:

$ ltrace curl --http1.1 -I https://www.google.com -sSfLo/dev/null

+++ exited (status 0) +++

Nothing! No library calls whatsoever?

It turns out that this version of ltrace (0.7.3-6.1ubuntu6 on Ubuntu/Jammy) is too old and suffers from ltrace produces no debug output at all.

The problem is fixed by ec56370 (Add Intel CET support). After fetching and building that, we can try the newer ltrace:

$ ltrace-0.8.x -e SSL_read -e SSL_write \
    curl --http1.1 -I https://www.google.com -sSfLo/dev/null

libcurl.so.4->SSL_write(0x55ad69380010, 0x55ad693a0f30, 79, 1) = 79
libcurl.so.4->SSL_read(0x55ad69380010, 0x55ad691eaa80, 0x19000, 1) = 1215
libcurl.so.4->SSL_read(0x55ad69380010, 0x7ffd2590e4e0, 32, 0x55ad691e5828) = 0xffffffff
+++ exited (status 0) +++

Good! We can see the calls to SSL_write and SSL_read.

(Ignore the mention of libcurl.so.4 here. We are in fact seeing the libssl calls.)

Unfortunately ltrace doesn't know that the second argument to both functions is a character buffer that often contains readable strings. We see the memory locations of the buffers but we do not get to see their contents. So, using plain ltrace does not help us.

Using latrace?

There exists a similar application that we can use: latrace. We can convince latrace to print arguments (see -A and -a). But there we'll get the arguments before the call only, not after: we can see the write data going into SSL_write but we will only see uninitialized data going into SSL_read, not the read data coming out of it.

Using bpftrace?

Maybe eBPF can come to the rescue?

First we need to know which library our application is using exactly:

$ ldd `which curl` | grep -E 'ssl|tls'
  libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007feb84d20000)
  libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007feb843a6000)

We can use bpftrace and attach BPF userland probes on SSL_read and SSL_write in libssl.so.3:

$ sudo bpftrace -e '
    uprobe:/lib/x86_64-linux-gnu/libssl.so.3:SSL_read,
    uprobe:/lib/x86_64-linux-gnu/libssl.so.3:SSL_write {
        @ctx[pid] = arg0; @buf[pid] = arg1; @len[pid] = arg2;
    }
    uretprobe:/lib/x86_64-linux-gnu/libssl.so.3:SSL_read,
    uretprobe:/lib/x86_64-linux-gnu/libssl.so.3:SSL_write {
        printf("[%d/%s] %s(%p, %p, %d) = %d", pid, comm, probe, @ctx[pid], @buf[pid], @len[pid], retval);
        if ((int32)retval > 0) {
            @slen = retval;
            if (@slen >= 64) {
                printf(" [[\n%s\n]] (truncated)", str(@buf[pid], @slen));
            } else {
                printf(" [[\n%s\n]]", str(@buf[pid], @slen));
            }
        }
        printf("\n");
        delete(@ctx[pid]); delete(@buf[pid]); delete(@len[pid]);
    }
'

(Our cURL uses OpenSSL. Adapting this for GnuTLS encrypted reads/writes is left as an exercise for the reader.)

eBPF does not support spawning the process itself. We'll have to ensure the sniffed program is started elsewhere:

$ curl --http1.1 -I https://www.google.com -sSfLo/dev/null

The bpftrace probe displays this:

Attaching 4 probes...

[2556865/curl] uretprobe:/lib/x86_64-linux-gnu/libssl.so.3:SSL_write(0x55ad2054d010, 0x55ad2056df30, 79) = 79 [[
HEAD / HTTP/1.1
Host: www.google.com
User-Agent: curl/7.81.0
]] (truncated)

[2556865/curl] uretprobe:/lib/x86_64-linux-gnu/libssl.so.3:SSL_read(0x55ad2054d010, 0x55ad203b7a80, 102400) = 1214 [[
HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
C
]] (truncated)

[2556865/curl] uretprobe:/lib/x86_64-linux-gnu/libssl.so.3:SSL_read(0x55ad2054d010, 0x7fff2f584060, 32) = -1

^C

Looking better! Some contents. Not very conveniently formatted, but it's a start.

The biggest problem here is that we're limited by BPFTRACE_STRLEN this time. We can get at most 64 characters, or maybe 200 if you push it.

Alternatives?

Are there alternatives?

We could attach a full fledged debugger (gdb), but that might feel a bit heavy. We could LD_PRELOAD a library that wraps the SSL_read and SSL_write calls, but this only works if we start the application with this wrapper. We could use ptrace(2) and create a minimal debugger that does what ltrace does, but then ensure that the buffers are printed in full.

Options, options.

And then we still haven't tackled the issue of statically compiled programs: many services in today's ecosystem are written in golang. They use no (or very few) dynamic libraries. Trying to trap OpenSSL or GnuTLS function calls there is meaningless. I'm afraid we'll have to dig into golang debugging at some point...

viewing unencrypted traffic / ltrace / bpftrace