FetchCRL3

The fetch-crl utility will retrieve certificate revocation lists (CRLs) for a set of installed trust anchors, based on crl_url files or IGTF-style info files. It will install these for use with OpenSSL, NSS or third-party tools.

latest version package

3.0.23 (at https://dl.igtf.net/distribution/util/fetch-crl/)

source code repository

https://github.com/dlgroep/fetch-crl

most important changes in 3.0.23

* make the http_proxy setting also affect https retrievals, while adding support for explicitly setting https_proxy in the trust anchor configuration for retrieval.

most important changes in 3.0.22

* fix race condition in CRL file re-writing for cases where the CRL directory itself is writable (thanks to Arjen Nienhuis)

most important changes in 3.0.21

* Add option to override the UserAgent string sent in the LWP web requests (both HEAD and GET)

* Add option for postexec hook to execute a command after running fetch-crl (or after each CRL installation)

* fix superfluous newline in DER formatted CRL files

previous significant changes

* Added cache state freshness constraints (default maxcachetime set to 96hrs)

* Re-set cache expiry of state data if CRL nextUpdate is within or beyond 7 hrs (config "expirestolerance") claimed URL Expiry or Cache-control max-age

* Add rcmode config option (added differentiated reporting and success-on-solely-retrieval-errors)

* Add --define key=val command line argument to augment configuration data

* Added option "--inet6glue" and "inet6glue" config setting to load the Net::INET6Glue perl module (if it is available) to use IPv6 connections in LWP to download CRLs

New features in version 3.0

• support for multiple output formats: OpenSSL 1 in dual-hash mode, specific DER and PEM outputs, and NSS databases
• support for multiple CRLs for a single CA, allowing more than one CA with the same subject name but different CLRs. Review your client software to see if and how these CRLs are used.
• stateful retrieval helps reduce bandwidth usage by caching the CRLs locally and respecting the Cache Control headers sent by the web server hosting the CRL. This can reduce the number of downloads
• support for HEAD-only requests when state preservation is used (initially only retrieve HTTP headers, and only if the CRL actually changed to a full download)
• support for more CRL retrieval protocols (file:// and ftp://)
• ability to try site-local URLs first, before relying on the URLs shipped with the trust anchor. This allows building an explicit local caching (web) server.
• ability to specify additional URLs to try in case the URLs shipped with the trust anchor were not responsive. This allows for automatic fall-back to (local or global) mirror services for CRL downloads
• warnings and errors can be suppressed on a per-trust anchor basis, to allow silencing for particularly unstable trust anchors
• aging tolerance (the delay time before errors are generated in case downloads consistently fail) can be configured on a per-trust anchor basis
• parallel downloading for multiple trust anchors
• minimized use of temporary files in the file system (now limited to the invocation of OpenSSL only, and only for brief periods of time)
• dependencies on wget, lynx and other unix utilities have been removed
• explicit web proxy support (using LWP http proxies)
• completely re-written in perl, with some (hopefully minimal) dependencies: LWP, Sys::Syslog, POSIX. And Data::Dumper (when debugging is enabled), and IO::Select (if parallel downloads are enabled).

Differences with respect to the previous versions

• when downloading CRLs via https, the server certificate is not checked, neither for the correct DNS name nor for being issued by a valid CA. Since the CRL in itself is signed, this is not a security vulnerability. If stricter checking is anyway desired, and the Crypt::SSLeay perl module has been installed, set the HTTPS_CA_FILE environment variable before invoking fetch-crl -- but keep in mind that the DNS name verification is limited and will (incorrectly) reject DNS names if these are listed only in the subjectAlternativeName of the server certificate
• Existing files with a name that matches a CRL target name are overwritten, even if they did not originally contain CRL data. In v2 this was configurable via the FORCE_OVERWRITE configuration setting. In version 3, files are overwritten by default, and this can no longer be configured.
• fetch-crl3 will no longer check CA certificates for consistency or validity by themselves, only retrieved CRLs are verified

The new version

• requires perl5 to be installed (tested with perl 5.8.0 and higher) with libwww-perl, whereas version 2 only required a traditional Bourne shell
• requires a version of OpenSSL (0.9.5a or better) to be installed. Needs OpenSSL 1.0.0 (at least beta5) for dual-hash support, and needs OpenSSL even when only the NSS database output format has been selected
• when using parallel downloads, it can only run on pure-POSIX systems (parallelism in combination with the NSS database output format has not been tested)

Download

You can download Fetch-CRL3 from the IGTF Distribution web sites:

• https://dist.eugridpma.info/distribution/util/fetch-crl3/

Configuration file

Fetch CRL comes with some built-in default, but virtually all of its behaviour can be controlled through configuration files and command-line arguments. The command-line arguments are a subset of the directives, but override those directives when specified at run-time. The list of valid configuration options is given below - but note that directives that are not recognised will be silently ignored (this does not apply to command-line arguments).

By default, the configuration file "/etc/fetch-crl.conf" is read on start up. If this does not exist, but "/etc/fetch-crl.cnf" does exist, the latter is read. If an explicit configuration file is specified on the command-line ("-c file"), this file is used.

Additional configuration has be given through separate configuration files in a configuration directory (by default: "/etc/fetch-crl.d"). All files in this directory are read (in arbitrary order) and interpreted as configuration files. The configuration is added to the existing configuration, and entries are merged with the existing configuration. The config directory can be specified (in the main config file) using the "cfgdir=" directive.

The configuration directory is especially useful to override per-trust-anchor download settings and to add non-packaged (local) configuration.

Configuration options

General section

infodir

directory containing the meta-data (".info") or crl_url files

Default: /etc/grid-security/certificates

cadir

directory containing the trust anchors against which retrieved CLRs will be verified

Default: set to infodir

output

directory where the resulting CRLs will be written, unless a format-specific destination is specified

Default: set to infodir

output_(pem,der,openssl,nss)

format-specific output directory

Default: set to output

statedir

directory where the state-and-cache file for each CRL is kept. If this directory is not set or does not exist, no state is kept. See the section on Stateful retrieval for more information

Default: set to /var/cache/fetch-crl if this directory exists, undefined otherwise

logmode

distination(s) for log messages generated: direct - print message immediately, qualified - print message immediately prefixed with the type of message, cache - save messages and print them all when the log is flushed, and syslog - send messages to syslog at the syslogfacility facility with a severity based on the type of message (NOTICE for warnings, ERR for errors, and DEBUG for verbose and debug messages)

Default: qualified

syslogfacility

facility used by the syslog output driver.

Default: daemon

formats

output formats in which the CRLs will be installed. Comma-separated list of one or more of openssl, der, pem, and/or nss

Default: openssl

openssl

location of the OpenSSL binary to use

Default: openssl (uses path to resolve)

opensslmode

use only the default hash (single) or also the pre-1.0 hash (dual) of the subject name for CRLs written with for use with OpenSSL 1.0 and up. Is only used in base the OpenSSL binary is version 1.0.0 or later.

Default: dual

path

executable search path ($PATH)

Default: unchanged from invocation

randomwait

wait up to x seconds before commencing the retrieval process

Default: unset

parallelism

number of parallel processes used to retrieve trust anchor CRLs. The parallelism is on a per-trust anchor basis (so multiple CRLs for the same trust ancnhor will be retrieved by the same sub-process

Default: unset

httptimeout

maximum time spent in a single HTTP request (HEAD or GET) in seconds

Default: 120

http_proxy

HTTP (and HTTPS) proxy URL, or the token "ENV" - in which case the environment variable settings for the perl LWP library are used

Default: unset

https_proxy

HTTPS-specific proxy URL, in case the generic http_proxy is not appropriate

Default: unset

catemplate

list of filename templates to 'guess' the name of the trust anchor (certificate) file in the cadir belonging to a particular CRL

Default: @ALIAS@.pem; @ALIAS@.@R@; @ANCHORNAME@.@R@

nametemplate_der

template of the name of the CRL file for the DER output writer

Default: @ANCHORNAME@.@R@.crl

nametemplate_pem

template of the name of the CRL file for the PEM output writer

Default: @ANCHORNAME@.@R@.crl.pem

nssdbprefix

prefix for the NSS database

Default: ""

prepend_url

list of URL templates to be tried first, before the crl_url list in the trust anchor metadata (or the lines in the crl_url file) are attempted. The "@ALIAS@", "@ANCHORNAME@", and "@R@" tokens are expanded for each trust anchor CRL.

Default: unset

postpend_url

list of URL templates to be tried in all other URLs failed,after the crl_url list in the trust anchor metadata (or the lines in the crl_url file) have been attempted. The "@ALIAS@", "@ANCHORNAME@", and "@R@" tokens are expanded for each trust anchor CRL.

Default: unset

(no)warnings

disable or enable warning messages

Default: warnings enabled

(no)errors

disable or enable error messages

Default: error messages enabled

nocache

do not use cached data in the state file (actually, set its best-before date of the cache to the epoch). This way a retrieval of the data is always done for each run, but it is different than disabling the state directory in that HEAD requests are still done for http(s) requests and, if the last-modified time is unchanged the cached data will still be used (as the web server claims it was not modified).

To disable also this use of the cache, disable the state directory completely (using the stateless configuration option, or on a per-trust anchor basis by setting nostatedir

Default: unset (uses cache)

(no)backups

(do not) make backups of the trust anchor files when written (but does not apply to NSS databases).

Default: no backups

stateless

do not use from nor write in a state directory to manage historic data, even if a state directory is present

Default: will use state directory if it exists

version

set the fetch-crl version number, e.g., used in the User-Agent header for HTTP requests.

Default: 3.0

packager

override the packager of this distribution

Default: EUGridPMA

cfgdir

directory to read additional configuration data. All files in this directory are read and their contents are effectively consolidated into a single configuration. It can be used to augment the configuration (e.g. with per-trust anchor URl overrides), although it cannot unset pre-existing config and it does not result in multiple invocations of fetch-crl (with different settings).

Default: /etc/fetch-crl.d/

Availability: 3.0.10+

rcmode

rcmode determines if the return code of fetch-crl will be influenced by CRL retrieval errors. If rcmode is "normal" (default), any reported errors will cause the return exit status to be "1".

• normal - both retrieval and other errors set exit code 1
• differentiated - retrieval errors result in exit code 2, presence of any other reported errors result in exit 1
• noretrievalerrors - retrieval errors only results in exit code 0, presence of any other reported errors result in exit 1

Note that setting "noerrors" will suppress retrieval errors entirely!

Default: normal

Availability: 3.0.13+

maxcachetime

maximum period in seconds for which the local state/cache data is to be considered valid before a request is done to either check Last-Modified, or download new CRL data

Default: 8294400 (96hrs)

Availability: 3.0.16+

expirestolerance

maximum period in seconds by which the Expires (or equivalent cache-control:max-age) is permitted to be before the nextUpdate time of the CRL itself. I.e.: if the CRL will expire expirestolerance seconds before the cache expiration, or later, a CRL request will always be performed

Default: 25200 (7hrs)

Availability: 3.0.16+

cache_control_request

sends a cache-control max-age hint towards the server in the HTTP request, that suggests to intermediate caches and reverse proxies to cache CRL replies no longer than the specified time This control is a hint towards caching servers and CDNs and cannot be enforced. It does NOT affect the cache local to fetch-crl

Default: unset (recommended: 3600, unit: seconds)

Availability: 3.0.17+

postexec

run a script after the completion of every fetch-crl run, set this path to point to an executable. The named program will be invoked with the following arguments

"v1" "global" infodir-path cadir-path output-path

- the return code of the program will influence return status of fetch-crl

- this must be a program path - no arguments are allowed here. Use wrapping in a script if you must pass your own arguments as well

Default: unset

Availability: 3.0.18+

user_agent

override the UserAgent string sent to the server for all downloads. This may be temporarily needed if you hit an over-active firewall or proxy in your network path that blocks apparent libwww-perl user agents. This setting can also be specified per trust anchor.

Default: fetch-crl/3+ (libwww-perl/version; EUGridPMA)

Availability: 3.0.21+

Trust Anchor sections

A trust anchor section must be named after the 'alias' or 'anchorname' (filename) of the trust anchor. In case a .info meta-data file is used, the alias is defined in this meta-data file. In case a crl_url file is used to load the URLs, the alias will be set to the basename of the crl_url file, without the ".crl_url" suffix. In case the alias is different from the anchorname, both will be used for section lookup, with a section for the alias taking precedence over the section for the anchorname.

crl_url.i

override the list of CRL download URLs for this trust anchor by the URL(s) given in this list. The list of URLs is on a single line, with URLs separated by semi-colons (";").

agingtolerance, httptimeout, nametemplate_der, nametemplate_pem, cadir, catemplate, statedir, user_agent

override the global defaults for this trust anchor. This override takes precendence overglobal configuration settings as well as over command-line supplied settings!

(no)prepend_url, (no)postpend_url, (no)warnings, (no)errors, (no)cache, (no)http_proxy

override global default, with options that are revertible, so "nopostpend_url" will prevent the default postpend_url from being used for this trust anchor. "nohttp_proxy" will prevents the HTTP proxy from being used to download CRLs for this trust anchor.

postexec

path of the script after the successful completion of each CRL retrieval (default: unset). The program is invoked with the following arguments: "v1" "ta" ta-alias infofilename cadir-path output-path

As always, the "@R@", "@ALIAS", and "@ANCHORNAME@" tokens are expended in URLs.

In catemplate also "@HASH@" can be used. When @HASH@ (c_hash from default OpenSSL version as based on the retrieved CRL) is used in this template list, a CRL will always be retrieved first, even if no corresponding trust anchor is found later. Use of @HASH@ is only recommended in case the name of the crl_url or info file is different from the name of the trust anchor.

Example

# configuration file fetch-crl3
# use SEMICOLON (;) or \001 (^A) as list separators in values
#
infodir = /etc/grid-security/certificates
statedir = /var/cache/fetch-crl
formats = openssl, pem
output_pem = /etc/pki/tls/certs
nametemplate_pem = @ALIAS@.r@R@.pem
opensslmode = single
randomwait = 30
httptimeout = 30
nowarnings
prepend_url = file:///share/grid-security/certificates/@ANCHORNAME@.r@R@
postpend_url = http://dist.eugridpma.info/certificates/@ANCHORNAME@.r@R@
openssl = /global/ices/toolset/arch/i686-pc-linux-gnu/openssl-1.0.0a/bin/openssl
path = /bin:/usr/bin:/usr/ucb
nobackups

[EDG-Tutorial-CA]
agingtolerance = 168
noerrors
nowarnings
nopostpend_url
crl_url.1 = file:///usr/local/etc/extracrl.pem

which will

• actually write two (2!) copies of each downloaded CRL: one in /etc/grid-security/certificates/hash.ri, and another copy in /etc/pki/tls/certs/alias.ri.pem
• try to retrieve the CRL first from a local file (e.g. NFS or AFS mounted), then from the URLs contained in the trust anchor store, and if all else fails goes to the EUGridPMA web page to get the last copy of the CRL cached there (assuming the ANCHORNAME is the OpenSSL0 hash)
• for the EDG Tutorial CA, the postpend_url emergency is removed, since this CA is not with the IGTF anyway and there is no cache at the EUGridPMA site for it.

An even more extensive example and explanation van be found File:Fetch-crl.cnf.example

CRL URL and INFO formats

The CRL URL(s) come from a per-trustanchor ".crl_url" or ".info" file. Where both are available, the ".info" file is preferred and the crl_url file ignored. The URLs must be resolvable by libwww-perl, so http, ftp, file, or gopher URLs are allowed, as well as any other LWP::Protocols that have been explicitly added (e.g. LWP::Protocols::ldap). Support for https (and ldaps) requires perl's IO::Socket::SSL or Net::SSLeay.

CRL_URL file format

The crl_url file contains a list of URLs, one per line, that point to alternative but equivalent locations to find the CRL for the trust anchor whose name matches the basename. Comments, introduced by a number sign ("#", U+0023) preceded by zero or more spaces, at the end of each line are ignored (a number sign in the URL must be escaped by URL encoding). The simplest use is a single line. For a trust anchor with the name "87250dfa.0" (the DutchGrid Root CA G1), the file would be "87250dfa.crl_url" and contain the single line

http://ca.dutchgrid.nl/dcaroot/g1/crl/crl.crl

A more complex example, for a trust anchor in file "75304a28.0", the CRL file "/etc/grid-security/certificates/75304a28.crl_url" could contain:

http://certificate.nikhef.nl/dutchdemo/cacrl.pem
http://ca.dutchgrid.nl/demo/crl/cacrl.pem # alternate

INFO file format

The IGTF ".info" file format is documented on the EUGridPMA pages, yet from this file only a few elements are used by fetch-crl: the attributes crl_url, crl_url.i (when present), status (when present), and alias (for the name, required). Other fields are not actively used. Alternate CRL URL locations for the same trust anchor CA can be specified on the same line, separated by a semi-colon (";"). Comments (introduced by a number sign "#") at the end of a line are allowed. For example, for the DigitalTrustIGTFCA trust anchor, the info file could contain:

# example IGTF info file sufficient for fetch-crl
alias = DigitalTrustIGTFCA
crl_url = http://crl1.digitaltrust.ae/dtigtfca.crl;http://crl2.digitaltrust.ae/dtigtfca.crl

Having both a series of crl_url.i attributes as well as a crl_url attribute is considered an error, and the CRL for that trust anchor will not be retrieved or installed.

Known issues

• although fetch-crl3 will install multiple CRLs in the CRl stores (called ".r0", ".r1", or labelled appropriately in an NSS store), if the number of CRLs decreases the left-overs are not automatically removed. So if the number of CRLs for a particular CA does down from n to n-1, the file ".rn" must be removed manually.
• to remove ".rn" files that have no corresponding CA file anymore, use the clean-crl script. See clean-crl(8) for details.

Source code management and issue tracking

The code is managed in Github, where also issues may be reported:

• https://github.com/dlgroep/fetch-crl

but please be advised that this product is supported on a best-effort basis. Supplying patches that fix the issue is encouraged and appropriate patches will be reviewed and included with attribution (unless you want to remain anonymous). Submitted patches and code shall be governed by the Apache 2.0 license as well. Expedited issue processing can be obtained by sending the appropriate amount of apple pie (preferably fresh so no sea freight shipping please).

Previous versions of the sources were managed in the Nikhef NDPF Subversion repository (https://ndpfsvn.nikhef.nl/repos/pdpsoft/nl.nikhef.pdp.fetchcrl/). On porting, all relevant tags were migrated to Github, and all outstanding issues resolved. Youcan file new issues at

• https://github.com/dlgroep/fetch-crl/issues

License, copyrights and acknowledgments

Fetch-crl3 is a complete re-write of the utility, inspired by the previous versions but no code has been re-used from there.

Copyright 2010-2021 David Groep, Nationaal instituut voor subatomaire fysica Nikhef

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

This work is part of the research programme of the Dutch Foundation for Fundamental Research on Matter (FOM), which is financially supported by the Netherlands Organisation for Scientific Research (NWO).

This work is part of the programme of BiG Grid, the Dutch e-Science Grid, which is financially supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek (Netherlands Organisation for Scientific Research, NWO).