Funny Curly things

Introduction

The curl tool is sometimes very handy to query grid services. For example, you can use curl to query the status of grid jobs, and in theory you could even submit a grid job using curl.

Starting with Fedora 12, RedHat has decided to change the way curl is built by default. As Fedora 12 is the 'baseline' for RedHat Enterprise Linux 6, this also affects the curl command in RHEL6, CentOS6 and Scientific Linux 6. In the past, curl was built and linked using the OpenSSL libraries. With Fedora 12+/RHEL6+, curl is now built and linked using the NSS library. This has an impact on how you can use curl to query different grid services.

The problem does not apply to grid services only, every site that requires a client-side certificate chain, that is, the client needs to present both a certificate and an intermediary key, is affected.

This page is the result of a comparison of the different SSL libraries that curl can be built against. The following SSL libraries were tested, using curl-7.22.0, which was the latest and greatest version of curl at the time of writing:

• OpenSSL 1.0.0d
• NSS 3.12.10
• GnuTLS 2.8.6
• PolarSSL 0.14.3

CA & sub-CA setup

The CA and sub-CA and certificate setup can be depicted using ASCII art:

Nikhef CA
  + 127.0.0.1 (server)
  + subca 
      + client

First we create a Certificate Authority (CA) using the easy-rsa scripts as found in OpenVPN. The resulting ca.pem has the following subject:

$ openssl x509 -subject -issuer -noout -in ca.pem
subject= /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=Nikhef CA
issuer=  /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=Nikhef CA

Next we create a sub-CA and sign it using the CA from the previous step:

$ openssl x509 -subject -issuer -noout -in subca.pem
subject= /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=subca
issuer=  /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=Nikhef CA

The server certificate for host 127.0.0.1 is signed by the CA itself:

$ openssl x509 -subject -issuer -noout -in server.pem 
subject= /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=127.0.0.1
issuer=  /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=Nikhef CA

But the client certificate is signed by the subCA:

$ openssl x509 -subject -issuer -noout -in client-cert.pem 
subject= /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=client
issuer=  /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=subca

We also create some stacked PEM files, which are simply concatenations of the certificates listed above:

• ca+subca.pem = ca.pem + subca.pem
• client.pem = client-cert.pem + client-key.pem
• client-certchain.pem = client-cert.pem + subca.pem

And for the --capath test a directory containing the hashed certificate names is created.

A tarball containing the entire certificate setup used for this investigation is available: curl-ca.tar.gz

Notes

• the hashed certificate names are created using openssl 1.0
• the files 'client.pem' and 'server.pem' contain both the public certificate and the private key.

The idea behind this CA and sub-CA setup is that we will configure the server to trust all certificates signed by the CA only. If a client certificate is signed by the sub-CA then the client must present both its certificate and the sub-CA certificate to the server. This is more or less the reverse to what happens with most SSL-enabled websites: the browser trusts only a set of root CAs so the HTTPS webserver has to present all intermediary certificates to the browser, otherwise the browser will not trust the server.

Testing setup

All tests, except for the GSI proxy certificate tests, can be performed on any (Linux) machine using the curl-ca tarball from above.

As the number of tests is quite large, a testcurl script was created to automate testing. The script and all needed certificate files, as well as a helper script to generate RFC3820 proxies, are included in the curltest.tar.gz tarball.

The testcurl script can perform three tests:

• regular certificate test
• RFC proxy (RFC3820) certificate test. An RFC3820 proxy certificate is generated on-the-fly
• GSI proxy certificate test; for this a valid GSI proxy needs to be present on the system.

The testcurl script launches a server process based on 'openssl s_server':

$ openssl s_server -Verify 0 -CAfile ca.pem -cert server.pem -key server.pem -dhparam dh2048.pem

after which curl is called with many different combinations of command-line parameters. The testcurl script accepts several command-line options

Curl certificate/ca+subca/proxy certificate test script
testcurl v1.0 - JJK - Nikhef
Usage:
    ./testcurl -d|--debug --(test-build)|--no-(test-build) --(test-type) --no-(test-type) --all -h|--help
Options
    -d|--debug    increase debugging verbosity
    --all         run all tests 
    -h|--help     display this help message
Where (test-build) can be
    system     for the system version of curl (default, use --no-system to override)
    openssl    for curl built and linked against OpenSSL
    nss        for curl built and linked against NSS
    gnutls     for curl built and linked against GnuTLS
    polarssl   for curl built and linked against PolarSSL
and where (test-type) can be
    cert-test  to test regular certificates using a CA and sub-CA
    gsi-test   to test GSI proxy certificates (file ./gsiproxy)
    rfc-test   to test RFC3820 compliant proxy certificates (file ./rfcproxy)

By default the testcurl script will test only regular certificates using the system version of curl. For the {openssl|nss|gnutls|polarssl} builds of curl it is assumed that all files are present in

/home/janjust/src/curl-7.22.0

but this can easily be changed in the testcurl script.

Test results

The testcurl script was tested on several (64bit) platforms:

• CentOS 5
• Scientific Linux 6
• Fedora 14

Only on Fedora 14 multiple builds of curl were tested.

CentOS 5

This version of curl is linked against OpenSSL; all tests performed by the testcurl script pass ("ok").

Curl version:

curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

Test output: testcurl-centos5.log

Scientific Linux 6

This version of curl is linked against NSS.

Curl version:

curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2

Test output: testcurl-sl6.log

We can observe several things in the log output:

• the line

Error: curl -s -S --cert client.pem --key client.pem --cacert ca+subca.pem <URL>

shows us that we can no longer "just" specify a certificate on the command-line. It is now required to specify a pathname, either using ./... or using an absolute path

Appendix: building curl

For completeness' sake this appendix lists how the different versions of curl were actually built.

All builds were done on a Fedora 14 x86_64 machine.

curl/OpenSSL

$ tar xzf curl-7.22.0.tar.gz
$ mv curl-7.22.0 curl-7.22.0-openssl
$ cd curl-7.22.0-openssl
$ ./configure --with-ssl=/usr/include --without-nss
$ make

To verify that the right curl, and more importantly, libcurl.so were created I ran

$ LD_LIBRARY_PATH=lib/.libs src/.libs/curl -V
curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0 zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz

curl/NSS

$ tar xzf curl-7.22.0.tar.gz
$ mv curl-7.22.0 curl-7.22.0-nss
$ cd curl-7.22.0-nss
$ ./configure --with-nss --without-ssl
$ make

To verify that the right curl, and more importantly, libcurl.so were created I ran

$ LD_LIBRARY_PATH=lib/.libs src/.libs/curl -V
curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 NSS/3.12.10.0 zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz 

curl/GnuTLS

$ tar xzf curl-7.22.0.tar.gz
$ mv curl-7.22.0 curl-7.22.0-gnutls
$ cd curl-7.22.0-gnutls
$ ./configure --with-gnutls=/usr/include --without-nss --without-ssl
$ make

To verify that the right curl, and more importantly, libcurl.so were created I ran

$ LD_LIBRARY_PATH=lib/.libs src/.libs/curl -V
curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 GnuTLS/2.8.6 zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz 

curl/PolarSSL

$ tar xzf curl-7.22.0.tar.gz
$ mv curl-7.22.0 curl-7.22.0-polarssl
$ cd curl-7.22.0-polarssl
$ ./configure --with-polarssl=/usr/include --without-nss --without-ssl --disable-ntlm-wb
$ make

Note: if I did not add --disable-ntlm-wb then the build failed in the linking phase

To verify that the right curl, and more importantly, libcurl.so were created I ran

$ LD_LIBRARY_PATH=lib/.libs src/.libs/curl -V
curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 PolarSSL zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IPv6 Largefile SSL libz