certificate verify fail / crt / bundle
So. SSL certificates are still black magic to me. Especially when they cause trouble.
Like when one of the sysadmins has forgotten to add the certificate
bundle to the apache2 config.
Then you get stuff like this:
$ hg pull -u
abort: error: _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Most web browsers do not notice this as they already have the
intermediate CA files, but /etc/ssl/certs/ca-certificates.crt
seemingly doesn’t.
The problem in this case was not that I was missing any certificates locally. The problem was that the web server was not publishing the child certificates of which the server certificate was a grandchild.
openssl can tell you all this, if you know how:
$ echo quit | openssl s_client -connect code.osso.nl:443 | sed -e '1,/^Server certificate/!d'
depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.osso.nl
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.osso.nl
verify error:num=27:certificate not trusted
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.osso.nl
verify error:num=21:unable to verify the first certificate
verify return:1
CONNECTED(00000003)
DONE
---
Certificate chain
0 s:/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.osso.nl
i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
---
Server certificate
I had to add the bundle of CA files that reference each other: issuer is the next signee in the list.
For apache2 this bundle is supposed to be ordered bottom-up. So,
next to your SSLCertificateFile you should always have a
SSLCertificateChainFile pointing to the bundle of intermediates.
SSLCertificateChainFile /etc/ssl/certs/code_osso_nl_CA.pem
Now, the openssl connect output looks a lot better:
$ echo quit | openssl s_client -connect code.osso.nl:443 | sed -e '1,/^Server certificate/!d'
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
verify return:0
CONNECTED(00000003)
DONE
---
Certificate chain
0 s:/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.osso.nl
i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
There’s still one error left, but that’s correct. We need to trust the root certificate.
Now the hg pull works again, like it should.
P.S. Some applications want the certificate bundle in a different order.
dovecot (ssl_cert_file) wants it bottom-up, with the server
certificate first in the same file. postfix (smtpd_tls_CAfile), on
the other hand, likes it top-down.