Funny Curly things

Introduction

The curl tool is sometimes very handy to query grid services. For example, you can use curl to query the status of grid jobs, and in theory you could even submit a grid job using curl.

Starting with Fedora 12, RedHat has decided to change the way curl is built by default. As Fedora 12 is the 'baseline' for RedHat Enterprise Linux 6, this also affects the curl command in RHEL6, CentOS6 and Scientific Linux 6. In the past, curl was built and linked using the OpenSSL libraries. With Fedora 12+/RHEL6+, curl is now built and linked using the NSS library. This has an impact on how you can use curl to query different grid services.

The problem does not apply to grid services only, every site that requires a client-side certificate chain, that is, the client needs to present both a certificate and an intermediary key, is affected.

This page is the result of a comparison of the different SSL libraries that curl can be built against. The following SSL libraries were tested, using curl-7.22.0, which was the latest and greatest version of curl at the time of writing:

• OpenSSL 1.0.0d
• NSS 3.12.10
• GnuTLS 2.8.6
• PolarSSL 0.14.3

This wiki page has also been entered as a comment to https://bugzilla.redhat.com/show_bug.cgi?id=733794

CA & sub-CA setup

The CA and sub-CA and certificate setup can be depicted using ASCII art:

Nikhef CA
  + 127.0.0.1 (server)
  + subca 
      + client

First we create a Certificate Authority (CA) using the easy-rsa scripts as found in OpenVPN. The resulting ca.pem has the following subject:

$ openssl x509 -subject -issuer -noout -in ca.pem
subject= /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=Nikhef CA
issuer=  /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=Nikhef CA

Next we create a sub-CA and sign it using the CA from the previous step:

$ openssl x509 -subject -issuer -noout -in subca.pem
subject= /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=subca
issuer=  /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=Nikhef CA

The server certificate for host 127.0.0.1 is signed by the CA itself:

$ openssl x509 -subject -issuer -noout -in server.pem 
subject= /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=127.0.0.1
issuer=  /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=Nikhef CA

But the client certificate is signed by the subCA:

$ openssl x509 -subject -issuer -noout -in client-cert.pem 
subject= /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=client
issuer=  /C=NL/L=Amsterdam/O=Nikhef/OU=Curl-test/CN=subca

We also create some stacked PEM files, which are simply concatenations of the certificates listed above:

• ca+subca.pem = ca.pem + subca.pem
• client.pem = client-cert.pem + client-key.pem
• client-certchain.pem = client-cert.pem + subca.pem

And for the --capath test a directory containing the hashed certificate names is created.

A tarball containing the entire certificate setup used for this investigation is available: curl-ca.tar.gz

Notes

• the hashed certificate names are created using openssl 1.0
• the files 'client.pem' and 'server.pem' contain both the public certificate and the private key.

The idea behind this CA and sub-CA setup is that we will configure the server to trust all certificates signed by the CA only. If a client certificate is signed by the sub-CA then the client must present both its certificate and the sub-CA certificate to the server. This is more or less the reverse to what happens with most SSL-enabled websites: the browser trusts only a set of root CAs so the HTTPS webserver has to present all intermediary certificates to the browser, otherwise the browser will not trust the server.

Testing setup

All tests, except for the GSI proxy certificate tests, can be performed on any (Linux) machine using the curl-ca tarball from above.

As the number of tests is quite large, a testcurl script was created to automate testing. The script and all needed certificate files, as well as a helper script to generate RFC3820 proxies, are included in the curltest.tar.gz tarball.

The testcurl script can perform three tests:

• regular certificate test
• RFC proxy (RFC3820) certificate test. An RFC3820 proxy certificate is generated on-the-fly
• GSI proxy certificate test; for this a valid GSI proxy needs to be present on the system.

The testcurl script launches a server process based on 'openssl s_server':

$ openssl s_server -Verify 0 -CAfile ca.pem -cert server.pem -key server.pem -dhparam dh2048.pem

after which curl is called with many different combinations of command-line parameters. The testcurl script accepts several command-line options

Curl certificate/ca+subca/proxy certificate test script
testcurl v1.0 - JJK - Nikhef
Usage:
    ./testcurl -d|--debug --(test-build)|--no-(test-build) --(test-type) --no-(test-type) --all -h|--help
Options
    -d|--debug    increase debugging verbosity
    --all         run all tests 
    -h|--help     display this help message
Where (test-build) can be
    system     for the system version of curl (default, use --no-system to override)
    openssl    for curl built and linked against OpenSSL
    nss        for curl built and linked against NSS
    gnutls     for curl built and linked against GnuTLS
    polarssl   for curl built and linked against PolarSSL
and where (test-type) can be
    cert-test  to test regular certificates using a CA and sub-CA
    gsi-test   to test GSI proxy certificates (file ./gsiproxy)
    rfc-test   to test RFC3820 compliant proxy certificates (file ./rfcproxy)

By default the testcurl script will test only regular certificates using the system version of curl. For the {openssl|nss|gnutls|polarssl} builds of curl it is assumed that all files are present in

/home/janjust/src/curl-7.22.0

but this can easily be changed in the testcurl script.

Test results

The testcurl script was tested on several (64bit) platforms:

• CentOS 5
• Scientific Linux 6
• Fedora 14

Only on Fedora 14 multiple builds of curl were tested.

CentOS 5

This version of curl is linked against OpenSSL; all tests performed by the testcurl script pass ("ok").

Curl version:

curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

Test output: testcurl-centos5.log

Scientific Linux 6

This version of curl is linked against NSS.

Curl version:

curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2

Test output: testcurl-sl6.log

We can observe several things in the log output:

• the line

001 Error curl -s -S --cert client.pem --key client.pem --cacert ca+subca.pem https://127.0.0.1:4433

shows us that we can no longer "just" specify a certificate on the command-line. It is now required to specify a pathname, either using ./... or using an absolute path

• using both a stacked CA certificate (ca+subca.pem) or a directory containing certificate hashes work
• when running a proxy certificate test it is now MANDATORY to specify the proxy certificate itself as a CA certificate and to specify the actual CA certificates using the --cadir flag:

041 OK curl -s -S --cert ./rfcproxy --key ./rfcproxy --cacert ./rfcproxy --capath ./dir https://127.0.0.1:4433

This applies to all types of proxy certificates: GSI, delegated GSI, RFC and delegated RFC.

An alternative option for the proxy certificates is stack the proxy certificate with the CA certificates:

cat rfcproxy ca+subca.pem > stack.pem

and then use --cacert stack.pem. This test is not present in the testcurl script, as it is not considered scalable.

Fedora 14

This version of curl is linked against NSS.

Curl version:

curl 7.21.0 (x86_64-redhat-linux-gnu) libcurl/7.21.0 NSS/3.12.10.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.4

Test output: testcurl-fc14-system.log

Next to the observations made for the Scientific Linux 6 version of curl we see a few more oddities:

• the --capath ./dir option does not seem to work at all!
• None of the proxy ceritificate tests were successful. Even the test

041 Error curl -s -S --cert ./rfcproxy --key ./rfcproxy --cacert ./rfcproxy --capath ./dir https://127.0.0.1:4433

fails. The alternative option to stack the certificates

cat rfcproxy ca+subca.pem > stack.pem

does work.

Curl 7.22/OpenSSL

This version of curl is linked against OpenSSL.

Curl version:

curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0 zlib/1.2.5

Test output: testcurl-fc14-openssl.log

As expected, this verion of curl passes all tests.

Curl 7.22/NSS

This version of curl is linked against OpenSSL.

Curl version:

curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 NSS/3.12.10.0 zlib/1.2.5

Test output: testcurl-fc14-nss.log

This version of curl shows behaviour that is very similar to the SL6 version of curl:

• when running a proxy certificate test it is now MANDATORY to specify the proxy certificate itself as a CA certificate and to specify the actual CA certificates using the --capath flag:

041 OK curl -s -S --cert ./rfcproxy --key ./rfcproxy --cacert ./rfcproxy --capath ./dir https://127.0.0.1:4433

This applies to all types of proxy certificates: GSI, delegated GSI, RFC and delegated RFC.

Curl 7.22/GnuTLS

This version of curl is linked against GnuTLS.

Curl version:

curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 GnuTLS/2.8.6 zlib/1.2.5

Test output: testcurl-fc14-gnutls.log

When curl is linked against GnuTLS the behaviour changes yet again:

• the --capath option does not seem to work at all!
• proxy certificates can be used if the CA certificates are used in a stacked format
• when running curl against an openssl s_server server an error 56 is sometimes returned by curl:

001 Semi-OK: curl -s -S --cert client.pem --key client.pem --cacert ca+subca.pem https://127.0.0.1:4433
    (error 56 could be an s_server oddity)
032 Semi-OK: curl -s -S --cert rfcproxy --key rfcproxy --cacert ca+subca.pem https://127.0.0.1:4433
    (error 56 could be an s_server oddity)

This could be an oddity in the 's_server' code itself, but the OpenSSL and NSS linked versions of curl do not show this behaviour

Curl 7.22/PolarSSL

This version of curl is linked against PolarSSL.

Curl version:

 curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 PolarSSL zlib/1.2.5

Test output: testcurl-fc14-polarssl.log

curl/PolarSSL behaves rather similar to curl/GnuTLS:

• the --capath option does not seem to work at all!
• when running curl against an openssl s_server server an error 56 is sometimes returned by curl:

005 Semi-OK: curl -s -S --cert client-cert.pem --key client-key.pem --cacert ca+subca.pem https://127.0.0.1:4433
    (error 56 could be an s_server oddity)
032 Semi-OK: curl -s -S --cert rfcproxy --key rfcproxy --cacert ca+subca.pem https://127.0.0.1:4433
    (error 56 could be an s_server oddity)

This could be an oddity in the 's_server' code itself, but the OpenSSL and NSS linked versions of curl do not show this behaviour.

Note that there are also alarming new errors:

011       curl: (77) Error reading ca cert file /etc/pki/tls/certs/ca-bundle.crt: -0x01A0
Error    curl -s -S --cert client.pem --key client.pem --capath dir https://127.0.0.1:4433

041       curl: (58) Error reading private key ./rfcproxy2: -0x024A
Error    curl -s -S --cert ./rfcproxy2 --key ./rfcproxy2 --cacert ./rfcproxy2 --capath ./dir https://127.0.0.1:4433

It seems that curl/PolarSSL has trouble reading some certificate files. In fact, it looks like curl/PolarSSL accepts only certificate files contain the block

-----BEGIN CERTIFICATE-----
[BLOB]
-----END CERTIFICATE-----

Anything else (such as the certificate text, which is often present) is considered an error. The same applies to private keys.

Note the Fedora 14 version of PolarSSL is quite old; however, linking curl against PolarSSL 1.0.0-gpl does not improve anything.

Conclusions

The overal conclusions of this investigation are

• the behaviour of curl greatly depends on the underlying SSL library
• it IS possible to query grid services using curl/OpenSSL and curl/NSS in a uniform way, by providing both the --cacert and --capath options:

curl -s -S --cert $X509_USER_PROXY --key $X509_USER_PROXY --cacert $X509_USER_PROXY --capath $X509_CERT_DIR <URL>

The question is whether this behaviour is guaranteed : a future update of either curl or nss might break this "hack". It would be great if RedHat or the Curl developers could comment on this.

• the base-OS version of curl on Fedora 14 (and Fedora 13) does not accept the above solution
• the base-OS version of curl on Scientific Linux 6 (and derivatives) does accept the above solution
• when linking curl against GnuTLS or PolarSSL a different commandline is required.

A test script is available for stand-alone testing. Please let me know your results and I will try to fit them into this report.

Appendix: building curl

For completeness' sake this appendix lists how the different versions of curl were actually built.

All builds were done on a Fedora 14 x86_64 machine.

curl/OpenSSL

$ tar xzf curl-7.22.0.tar.gz
$ mv curl-7.22.0 curl-7.22.0-openssl
$ cd curl-7.22.0-openssl
$ ./configure --with-ssl=/usr/include --without-nss
$ make

To verify that the right curl, and more importantly, libcurl.so were created I ran

$ LD_LIBRARY_PATH=lib/.libs src/.libs/curl -V
curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0 zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz

curl/NSS

$ tar xzf curl-7.22.0.tar.gz
$ mv curl-7.22.0 curl-7.22.0-nss
$ cd curl-7.22.0-nss
$ ./configure --with-nss --without-ssl
$ make

To verify that the right curl, and more importantly, libcurl.so were created I ran

$ LD_LIBRARY_PATH=lib/.libs src/.libs/curl -V
curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 NSS/3.12.10.0 zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz 

curl/GnuTLS

$ tar xzf curl-7.22.0.tar.gz
$ mv curl-7.22.0 curl-7.22.0-gnutls
$ cd curl-7.22.0-gnutls
$ ./configure --with-gnutls=/usr/include --without-nss --without-ssl
$ make

To verify that the right curl, and more importantly, libcurl.so were created I ran

$ LD_LIBRARY_PATH=lib/.libs src/.libs/curl -V
curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 GnuTLS/2.8.6 zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz 

curl/PolarSSL

$ tar xzf curl-7.22.0.tar.gz
$ mv curl-7.22.0 curl-7.22.0-polarssl
$ cd curl-7.22.0-polarssl
$ ./configure --with-polarssl=/usr/include --without-nss --without-ssl --disable-ntlm-wb
$ make

Note: if I did not add --disable-ntlm-wb then the build failed in the linking phase

To verify that the right curl, and more importantly, libcurl.so were created I ran

$ LD_LIBRARY_PATH=lib/.libs src/.libs/curl -V
curl 7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 PolarSSL zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IPv6 Largefile SSL libz