openvpn dco / mtu / mss clamping / packet fragmentation

Since we switched OpenVPN to the in-kernel data channel (DCO, ovpn module), traffic through the tunnel produced oversized outer UDP datagrams that got IP-fragmented.

Fixed in two steps:

1. MSS clamping in mangle on the vpn-server interface (TCP: MSS 1460 → 1400).
2. tun-mtu 1440 in the OpenVPN config, for everything MSS clamping can't touch (QUIC, VPN-in-VPN).

Verified with a bpftrace script that counts IPv4 fragments and oversized GRO segments. Both counters stay at zero.

The problem

OpenVPN carries tunnel traffic over UDP. Every inner packet gets an outer header and crypto overhead:

With the default tun MTU of 1500, a full-size inner packet becomes a 1552-byte outer datagram. That doesn't fit a 1500 link, let alone a 1492 PPPoE link, so the outer datagram is IP-fragmented: one large fragment plus a tiny trailing one.

Why that hurts:

• Efficiency. Two packets per datagram (one big and one small), reassembly at the receiver, and loss of either fragment loses the whole datagram.
• Packet scrubbing. Fragments carry no L4 header, so they can't be matched against port-based rules, and fragment floods are a classic attack. When enabled, our DDoS scrubbing drops them. The datagram never gets reassembled; TCP retransmits at the same size and the flow stalls. From the user's side this looks like a PMTU blackhole: small requests work, large transfers hang. Some (WiFi-)routers or other middleboxes also drop these fragments, causing the same kinds of behaviour.

Why this didn't happen before DCO

Userspace OpenVPN saw every data packet and applied mssfix (default mssfix 1492 mtu since 2.6: the outer IP packet is limited to 1492). It rewrote the MSS option in TCP SYN/SYN-ACK packets passing through the tunnel, so TCP endpoints never produced segments too large for the encapsulated path.

With DCO the data channel lives in the kernel. Userspace only handles control traffic and no longer sees the SYNs, so mssfix silently does nothing. Kernel-side support (OVPN_A_PEER_MSSFIX) has been proposed but is not merged into mainline:

• patchwork.openvpn.net/project/ovpn/list/?q=TCP+MSS
• lore.kernel.org/openvpn-devel: ovpn: implement TCP MSS clamping

Even once merged, OVPN_A_PEER_MSSFIX only fixes TCP. It does nothing for UDP payloads (QUIC) or anything else that doesn't negotiate an MSS.

Fix 1: MSS clamping

Do in netfilter what mssfix used to do in userspace: rewrite the MSS in SYN and SYN-ACK packets entering or leaving the tunnel.

# Needed because openvpn-DCO (ovpn module) does not apply mssfix when both
# sides have DCO. This results in (too large) tcp segments and fragments.
# Setting 1400 forces it down from 1460, per this calculation:
# // 1492  outer IP packet (PPPoE)
# //  -52  outer IPv4 20 + UDP 8 + opcode/peer-id 4 + packet-id 4 + AEAD tag 16
# // 1440  inner IP packet  (= effective tunnel MTU)
# //  -40  inner IPv4 20 + TCP 20 -> MSS 1400
# //  -60  inner IPv6 40 + TCP 20 -> MSS 1380 (or 1360 if we're doing v6 in v6)
# REMEMBER: Add mangle for v6 too once we route v6 over the VPN.
# UPSTREAM: https://patchwork.openvpn.net/project/ovpn/list/?series=&submitter=&state=*&q=TCP+MSS&archive=both&delegate=
# UPSTREAM: https://lore.kernel.org/openvpn-devel/20260515075941.102225-2-ralf@mandelbit.com/
iptables -t mangle -A INPUT -i vpn-server -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1401:65535 -j TCPMSS --set-mss 1400
iptables -t mangle -A FORWARD -i vpn-server -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1401:65535 -j TCPMSS --set-mss 1400
iptables -t mangle -A FORWARD -o vpn-server -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1401:65535 -j TCPMSS --set-mss 1400
iptables -t mangle -A OUTPUT -o vpn-server -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1401:65535 -j TCPMSS --set-mss 1400

Notes on the rules:

• --tcp-flags SYN,RST SYN matches both SYN and SYN-ACK. Each side announces its own MSS, so both directions must be clamped.
• --mss 1401:65535 only lowers the MSS, never raises a smaller value one side already chose.
• The four chains cover all paths: forwarded in/out of the tunnel, and connections terminating on or originating from the VPN server itself.
• MSS is per-direction and per-address-family. IPv6 needs its own ip6tables rules with MSS 1380 once v6 is routed over the VPN.

Why not rely on PMTUD instead?

Forwarded TCP connections negotiate MSS from the endpoints' own interfaces (typically 1500 → MSS 1460). They only learn the tunnel is smaller via ICMP "fragmentation needed", which is often filtered and costs a round trip per connection anyway. Clamping the MSS in the SYN avoids both.

Verifying the clamp

Capture SYNs on both sides of the VPN server. tcpdump sees packets before mangle on ingress and after it on egress, so the rewrite is visible across interfaces:

tcpdump -ni vpn-server 'tcp[tcpflags] & tcp-syn != 0'
tcpdump -ni ens18      'tcp[tcpflags] & tcp-syn != 0'

A SYN coming in from a client on vpn-server shows options [mss 1460,...]; the same SYN leaving via ens18 shows mss 1400. Same for the SYN-ACK in the other direction.

For connections terminating on the VPN server (INPUT chain), check the socket instead: ss -ti shows mss:1400 (or lower) for clients over the tunnel.

The rule counters (iptables -t mangle -L -v -n) should increase with new connections.

Fix 2: tun-mtu 1440

MSS clamping fixed most of it, but fragments remained. Two traffic classes don't go through the clamp:

• QUIC / UDP. No MSS to clamp. QUIC sizes datagrams by its own PMTU discovery, bounded by the interface MTU. With a 1500 tun, a 1500-byte inner datagram goes through, the outer one is fragmented and reassembled (when not scrubbed), and QUIC concludes that 1500 works. Fragmentation hides the real limit.
• VPN-in-VPN. A client running another VPN inside ours sizes its packets to our tun MTU. Its inner TCP SYNs are encrypted, so our clamp can't see them.

Lowering the tunnel interface MTU fixes both: inner packets with DF that won't fit are rejected (EMSGSIZE for local senders, ICMP frag-needed when forwarding), so applications and inner VPNs size down accordingly. Packets without DF are fragmented before encapsulation, so the outer datagram still fits.

tun-mtu 1440

1440 + 52 = 1492: a full-size inner packet now exactly fits a 1492-byte outer packet. DCO picks this up as the MTU of the ovpn netdev.

We deliberately chose 1492 as maximum packet size, for consistency with the OpenVPN default mssfix.

Verification: counting fragments

count-fragged-ip.bt (full source below) prints two maps every 60 seconds:

• @mf — IPv4 first fragments (MF=1, offset 0) on every interface except the VPN device, keyed by direction, device, protocol, and src:sport → dst:dport. Only the first fragment carries the L4 header, so each count is one fragmented datagram. Outer OpenVPN traffic shows up here as UDP to/from the OpenVPN port.
• @gso — GRO aggregates received on the VPN device whose individual segment size exceeds the given inner MTU, i.e. the peer is sending inner packets larger than our tunnel MTU (misconfigured peer side).

Run:

./count-fragged-ip.bt vpn-server 1440

Implementation notes:

• Probes net:netif_receive_skb rather than ip_rcv: the latter misses packets delivered in batches via ip_list_rcv.
• On rx, skb->data points at the L3 header; on tx (net:net_dev_queue) it doesn't yet, so network_header is used.
• GRO aggregates without gso_size carry no segment size and are skipped.
• IPv4 only. IPv6 fragments (extension header) are not counted.
• The commented-out @lg block reports oversized inner packets on the VPN device per flow; it's kept for when per-flow detail is needed.

Result

Before the fixes @mf filled up with OpenVPN UDP fragments. After MSS clamping most were gone; the remainder were QUIC and VPN-in-VPN. After tun-mtu 1440, both @mf and @gso stay empty.

If entries reappear, check which flows and from which peers, and revisit the MSS/MTU values above (mobile network peers → lower tun-mtu and MSS clamp further; IPv6 over the tunnel → add ip6tables clamping at MSS 1380).

Notes on configuring DCO

If the module is loaded, a recent enough OpenVPN will use it and log it as DCO device <DEV> opened.

The speed benefits of DCO are apparent on the side that is using it. But first once both sides start using it do we experience the problems discussed here.

Make loading of the module(s) explicit by writing this to /etc/modules-load.d/ovpn.conf:

# DKMS (old style)
#ovpn-dco-v2.ko.zst
# Vanilla (new style, requires openvpn 2.7+)
ovpn
# see: ip -d link show | grep -B2 ovpn
xt_TCPMSS
xt_tcpmss
# TCPMSS is needed for the DCO-has-no-mssfix-workaround using iptables.

Script

#!/usr/bin/env bpftrace
/* vim: set syn=c:
 *
 * usage: count-fragged-ip.bt <vpn-dev> <inner-mtu>
 *
 * @mf: IPv4 first-fragments (MF=1, offset=0) on every device except
 *      <vpn-dev>, per direction, dev, proto, src:sport, dst:dport.
 *      Later fragments carry no L4 header, so each count is one datagram.
 *      rx: skb->data is at L3; tx: use network_header.
 *      netif_receive_skb, not ip_rcv: the latter misses ip_list_rcv batches.
 *
 * @gso: rx GRO aggregates on <vpn-dev> whose segments exceed <inner-mtu>.
 *      Aggregates without gso_size carry no segment size and are skipped.
 */
BEGIN
{
    if ($# != 2) {
        printf("usage: count-fragged-ip.bt <vpn-dev> <inner-mtu>\n");
        exit();
    }
}

// The probe is on netif_receive_skb rather than ip_rcv because ip_rcv misses
// traffic from drivers that deliver packets in batches through ip_list_rcv.
tracepoint:net:netif_receive_skb
/str(args.name) != str($1)/
{
    $skb = (struct sk_buff *)args.skbaddr;
    if ($skb->protocol != 0x0008) { // htons(ETH_P_IP), LE host
        return;
    }
    $iph = (struct iphdr *)$skb->data;
    if ((bswap($iph->frag_off) & 0x3fff) != 0x2000) {
        return;
    }
    $p = $iph->protocol;
    $sport = (uint16)0;
    $dport = (uint16)0;
    if ($p == 6 || $p == 17 || $p == 132) {
        $uh = (struct udphdr *)((uint8 *)$iph + $iph->ihl * 4);
        $sport = bswap($uh->source);
        $dport = bswap($uh->dest);
    }
    @mf["rx", $skb->dev->name, $p, ntop($iph->saddr), $sport, ntop($iph->daddr), $dport] = count();
}

tracepoint:net:net_dev_queue
/str(args.name) != str($1)/
{
    $skb = (struct sk_buff *)args.skbaddr;
    if ($skb->protocol != 0x0008) {
        return;
    }
    $iph = (struct iphdr *)($skb->head + $skb->network_header);
    if ((bswap($iph->frag_off) & 0x3fff) != 0x2000) {
        return;
    }
    $p = $iph->protocol;
    $sport = (uint16)0;
    $dport = (uint16)0;
    if ($p == 6 || $p == 17 || $p == 132) {
        $uh = (struct udphdr *)((uint8 *)$iph + $iph->ihl * 4);
        $sport = bswap($uh->source);
        $dport = bswap($uh->dest);
    }
    @mf["tx", $skb->dev->name, $p, ntop($iph->saddr), $sport, ntop($iph->daddr), $dport] = count();
}

tracepoint:net:netif_receive_skb
/str(args.name) == str($1) && args.len > $2/
{
    $skb = (struct sk_buff *)args.skbaddr;
    if ($skb->protocol != 0x0008) {
        return;
    }
    $sh = (struct skb_shared_info *)($skb->head + $skb->end);
    if ($sh->gso_size == 0) {
        return;
    }
    $iph = (struct iphdr *)$skb->data;
    $l4len = (uint32)8;
    if ($iph->protocol == 6) {
        $l4len = ((struct tcphdr *)((uint8 *)$iph + $iph->ihl * 4))->doff * 4;
    }
    $seg = $iph->ihl * 4 + $l4len + $sh->gso_size;
    if ($seg > $2) {
        @gso[$iph->protocol, ntop($iph->saddr), $seg] = count();
    }
}

/*
// Inner IPv4 packets on the ovpn-dco netdev whose on-wire segment size
// exceeds what fits in one outer datagram. ovpn does GSO on tx and GRO on
// rx, so skb->len is not the segment size: use gso_size when set.
tracepoint:net:net_dev_start_xmit
/str(args.name) == str($1) && args.protocol == 0x0800 && args.len > $2/
{
    $skb = (struct sk_buff *)args.skbaddr;
    $iph = (struct iphdr *)($skb->head + $skb->network_header);
    $p = $iph->protocol;
    $l4 = (uint8 *)$iph + $iph->ihl * 4;
    $len = (uint32)args.len;
    if (args.gso_size) {
        $l4len = (uint32)8;
        if ($p == 6) {
            $l4len = ((struct tcphdr *)$l4)->doff * 4;
        }
        $len = $iph->ihl * 4 + $l4len + args.gso_size;
        if ($len <= $2) {
            return;
        }
    }
    $sport = (uint16)0;
    $dport = (uint16)0;
    if ($p == 6 || $p == 17 || $p == 132) {
        $sport = bswap(((struct udphdr *)$l4)->source);
        $dport = bswap(((struct udphdr *)$l4)->dest);
    }
    @lg["tx", $p, ntop($iph->saddr), $sport, ntop($iph->daddr), $dport] = max($len);
}

tracepoint:net:netif_receive_skb
/str(args.name) == str($1) && args.len > $2/
{
    $skb = (struct sk_buff *)args.skbaddr;
    if ($skb->protocol != 0x0008) {
        return;
    }
    $iph = (struct iphdr *)$skb->data;
    $p = $iph->protocol;
    $l4 = (uint8 *)$iph + $iph->ihl * 4;
    $len = (uint32)args.len;
    $gso = ((struct skb_shared_info *)($skb->head + $skb->end))->gso_size;
    if ($gso) {
        $l4len = (uint32)8;
        if ($p == 6) {
            $l4len = ((struct tcphdr *)$l4)->doff * 4;
        }
        $len = $iph->ihl * 4 + $l4len + $gso;
        if ($len <= $2) {
            return;
        }
    }
    $sport = (uint16)0;
    $dport = (uint16)0;
    if ($p == 6 || $p == 17 || $p == 132) {
        $sport = bswap(((struct udphdr *)$l4)->source);
        $dport = bswap(((struct udphdr *)$l4)->dest);
    }
    @lg["rx", $p, ntop($iph->saddr), $sport, ntop($iph->daddr), $dport] = max($len);
}
*/

interval:s:60
{
    // Haha. For loop exists in 0.21.0 -- we're using 0.20.2:
    // https://github.com/bpftrace/bpftrace/issues/2955
    //for ($kv : @mf) {
    //    printf("dir=%s proto=%d %s:%d -> %s:%d count=%d\n",
    //        $kv.0.0, $kv.0.1, $kv.0.2, $kv.0.3, $kv.0.4, $kv.0.5, $kv.1);
    //}

    print(@mf);
    clear(@mf);
    print(@gso);
    clear(@gso);
    //print(@lg);
    //clear(@lg);
}

END
{
    clear(@mf);
    clear(@gso);
    //clear(@lg);
}