AARC Pilot - Ansible

From GridWiki
Jump to: navigation, search

Contents

Introduction

After experimenting with Jenkins as a method of deployment for the CILogon Pilot we quickly had to come to the conclusion that Jenkins was not meant to be used for such deployments. Although it can spin up machines and run any kind of configuration scripts on them, it was still lacking the kind of configuration control and templating capabilities that we were looking for. We decided to move away from it and use one of the more widespread tools for deployments and configuration.

Out pick fell on Ansible mainly because of the following points:

  • support for configuration templating
  • easy to use modules to interact with systems
  • no software required on target machine (runs through ssh)
  • good documentation
  • widely used

Note! Deployments done through these Ansible scripts are expecting a clean system with only the OS (CentOS 6 or CentOS 7) installed. No packages have to be installed manually before executing the deployment scripts, since Ansible will do that for you.

General Structure

The ansible deployment scripts are structured into, what ansible calls plays. In our deployment scripts we used a single play to configure a single host of a specific kind. Ansible plays are made up of roles. We are using roles as a means of grouping tasks that logically belong together.

Giving input to the deployment scripts is done through overwriting existing variables. Each ansible role contains two sets of variables:

  • roles/x/vars/main.yml holds OS specific variables (like package names, repositories, service name, etc...). These are only used within a role (x), and they cannot be overwritten by variables defined inside a play (see variable precedence). However, in order to support different OS versions, we have separate OS specific variable files under roles/x/vars/, like rh7.yml or rh6.yml. Depending on the OS version that you are using for your deployment, you should define the roles/x/vars/main.yml as a symlink to one of the existing OS specific variable files.
  • roles/x/defaults/main.yml holds application specific variables (like configuration options, paths, etc...). These are only used within a role (x), and they can be overwritten by variables defined inside a play. For each defined role 'x' you will find a corresponding variables file name 'x_env.yml' in the top level directory. This variables file is used together with the 'x' role inside a play. Use this variables file to overwrite defaults! Note that not ALL variables from the defaults are meant to be overwritten (for examples paths can stay the same). You will find a 'x_env.yml' template for each role that will tell you which variables you SHOULD overwrite (like passwords).

Generating Secrets

Both Master Portal and Delegation Server setup requires several different passwords for different components (database, rsync, keystore). To ease the pain of making up new passwords we provide an extra play that can be executed to generate a set of required passwords. The passwords are created with the openssl 'rand' module with special characters omitted. The play generating new passwords runs on your local machine and it creates a file called secrets_env.yml in the current directory. This file is then used as input for the Master Portal play and the Delegation Server play respectively. In case you don't want to use our password generator make sure to fill secrets_env.yml manually with all the required passwords!

To generate a new set of passwords, execute:

ansible-playbook secrets.yml

Note! Re-executing the play will NOT override the previously generated passwords from secrets_env.yml. In case you want a password regenerated, you should remove it from the file before re-executing the play.

Note! The play will only generate a fixed set of required passwords for the setup. In case you want to tweak the list of passwords that the play has to generate, look into the secrets.yml play!

Master Portal

The ansible scripts for deploying a Master Portal can be found on github as part of the rcauth-eu organization. You can use these scripts to deploy a Master Portal and Credential Store on two separate hosts, or into one single host. Check them out via:

git clone https://github.com/rcauth-eu/aarc-ansible-master-portal.git

Before you begin executing plays make sure to decide whether you're deployment will use two separate hosts for Master Portal and Credential Store, or a single host. Fill in you machine hostname[s] into the hosts.inventory file accordingly. The hostname[s] set in the inventory files will be set on the target machine[s].

These scripts expect you to have a basic CentOS 6 or 7 installation ready. Since the two OS installations have slight differences between them (like repository addresses) we created two sets of variables for each OS version. Once you decided which OS version you want to use, make sure to check the symlinks called roles/*/vars/main.yml. These symlinks can either point to rh6.yml or to rh7.yml under the same directory in which they are. Set these according to the OS of your choice.

Roles

basic

The basic role tries to cover the general environment setup that is needed for both Master Portal and Credential Store. The tasks belonging to the basic role cover the following configurations:

  • install / update required packages and repositories
  • configure access to the machine (iptables, ssh)
  • deploy host credentials (see input 1. below)
  • enable required services, disable not required services

Before applying the basic role make sure to provide the following inputs for the ansible scripts:

  1. You need to have host certificates ready to apply this role. Place your PEM formatted certificate and key file under the 'roles/basic/files/' directory. Note that the private key of the host certificate should be left unencrypted, so that services running on the host can use it as is. You will need to call these credentials 'hostname'.crt and 'hostname'.key, where 'hostname' is the same name that you provided in the hosts.inventory file of your play.
    • By default, the 'TERENA eScience SSL CA 3' trust anchor is installed for the provided host certificate. If your host certificate is issued by a different CA, you can install the CA's trust anchor by changing the required package name in hostcred_trust_anchor_package in the basic_env.yml file.
    • In case your host certificate is issued by a custom CA which cannot be installed via the package manager, you can install this by putting the PEM formatted CA certificate file and the CA's signing policy file into roles/basic/files. Now fill in the hostcred_trust_anchor_pem and the hostcred_trust_anchor_signing_policy variables from the basic_env.yml file with the corresponding filenames.
  2. Overwrite the recommended default variables from basic_env.yml to match your environment.

credstore

The credstore role takes care of deploying and configuring any Credential Store specific software. The tasks belonging to the credstore role cover the following:

  • install and configure MyProxy Server as a Credential Store
  • deploy trust anchors for Online CA (see input 1. below)
  • configure iptables rules for Credential Store (open myproxy port 7512)
  • add additional services (myproxy_purge for expired/revoked certificates)
  • start/restart relevant services (myproxy-server)

Before applying the credstore role make sure to provide the following inputs for the ansible scripts:

  1. MyProxy Server only stores credentials that it can verify, therefore it's very important to have the Online CA (which will issue user certificates) as a trusted certificate on the Credential Store machine. In case you are planning to connect to RCAuth.eu, you can find the appropriate trust anchors at the IGTF accredited repository (configure both RCAuth and its dependency DCAROOT as well). You can configure any alternative trust root you would like to use in credstore_env.yml.
  2. Overwrite the recommended default variables from credstore_env.yml to match your environment.

masterportal

The masterportal role takes care of deploying and configuring the Master Portal and its dependencies. The tasks belonging to the masterportal role cover the following:

  • install and configure dependencies : java, tomcat, apache-httpd
  • add database support through either mysql or mariadb (postgres is not yet supported). Note: If you're on RH6 use mysql, if you're on RH7 stick to mariadb.
  • install and configure Master Portal Server and Master Portal Client
  • install and configure demo VO Portal (optional, useful for testing). You can enable/disable the execution of this task in role/masterportal/tasks/main.yml
  • configure iptables rules for Master Portal (open https port 443)
  • start/restart relevant services (tomcat, apache-httpd)

Before applying the masterportal role make sure to provide the following inputs for the ansible scripts:

  1. Add the compiled Master Portal wars to the deployment. If you don't have a compiled Master Portal you can build it yourself. Add the compiled war files (mp-oa2-client.war and mp-oa2-server.war) into the 'role/masterportal/files' directory.
  2. Add the compiled Master Portal CLI to the deployment. You will need this CLI tool to approve and manage your Master Portal clients! If you don't have a compiled CLI you can build it yourself. Place the compiled jar (oa2-cli.jar) into the 'role/masterportal/files' directory.
  3. In case you are also deploying the VO Portal, make sure to also put the compiled VO Portal war file (vo-portal.war) into the 'role/masterportal/files' directory.
  4. Overwrite the recommended default variables from masterportal_env.yml to match your environment.
  5. Generate secrets by executing the password generator.
  6. Fill in required MP Client ID and Secret into secrets_env.yml.
  7. For new installations you will also need to create a set of JSON web keys, this can be done e.g. using the oa2-cli tool with the use signing command. The resulting jwk file is expected to be called mp.jwk and should be put in roles/masterportal/files/. You should specify the default Key ID as mp_server_jwk_key_id in the masterportal_env.yml file. Note: you can also run the oa2-cli directly using
    java -jar oa2-cli.jar -cfg barecfg.xml -log oa2-cli.log
    where barecfg.xml contains
    <config><service/></config>

Plays

These ansible scripts include two plays:

  • credstore.yml to deploy a Credential Store. This will apply the basic and the credstore roles.
  • masterportal.yml to deploy a Master Portal. This will apply the basic and the masterportal roles.

You can execute these plays, after you've give the required deployment input for the roles, by calling:

ansible-playbook -i hosts.inventory credstore.yml 
ansible-playbook -i hosts.inventory masterportal.yml

It is safe to re-execute these plays multiple times, therefore you can also use these scripts to do an update on a Master Portal with a new release!

Note! If you're doing a single host deployment (Master Portal and Credential Store on one host) make sure to always execute the credstore play before the masterportal play. In case you re-execute credstore make sure to re-execute the masterporal play as well. If you fail to do so you will end up with a broken firewall setup!

Delegation Server

The ansible scripts for deploying a Delegation Server can be found under Nikhef subversion, here. Note that these ansible scripts will only deploy the Delegation Server and not the Online CA itself. If you want to deploy an Online CA as well, you should take a look at these deployment scripts. The Online CA is expected to be up and running before the deployment of the Delegation Server. See the integration section for more details. Check out the Delegation Server ansible scripts from:

svn co https://ndpfsvn.nikhef.nl/repos/pdpsoft/trunk/eu.rcauth.pilot-ica/DS/

These scripts are not as general as the Master Portal scripts, because they contain more Nikhef infrastructure specific configuration. This guide will tell you where these specific configurations are, so you will know what to tweak in case you are deploying a Delegation Server somewhere else. See the Nikhef specific section for this.

As a first step, make sure to configure your own inventory file with the hostname of the target machine on which you want a Delegation Server deployed. Look into the nikhef inventory file for inspiration.

Note! Currently only CentOS 7 is supported as an OS for the Delegation Server deployment scripts.

Integrating with Online CA

It is recommended to dedicate a full physical machine for the Delegation Server with two separate network interfaces. One of the network interfaces, what we call public_interface will be the outwards facing interface for incoming requests. The second interface, what we call private_interface is a dedicated connection to the Online CA. It is assumed that both Delegation Server and Online CA are kept in a safe and secure environment with nothing between the Delegation Server and Online CA other than a simple network connection. Consult the RCauth.eu policy for more details on recommended safety measures.

After configuring your inventory file with the right 'hostname', you will have to configure some additional host-specific variables. Create a new file under 'host_vars/' with the name 'hostname'. Enter the following host-specific variables into the created file:

private_network: 192.168.1.0/24
private_ds_address: 192.168.1.254
private_ca_address: 192.168.1.1

private_interface: enp8s0
public_interface: enp4s0

private_domain: canet
private_ds_hostname: frontend
private_ca_hostname: ca

Tweak the variables according to your own setup environment, especially the two interface names. The 'ds' and 'ca' are abbreviations for 'Delegation Server' and 'Online CA' respectively.

The communication between the Delegation Server and the Online CA through the private_network is secured via host certificates. If you followed the suggested Online CA setup you will see that the Online CA creates a dedicated root certificate and two host certificates: one for himself and one for the Delegation Server. The Online CA setup scripts will package the host certificate belonging to the Delegation Server and it will copy it into the Delegation Server machine under /root/frontend.canet.tgz

If you're setting up the Delegation Server for the first time (not updating it) check that the /root/frontend.canet.tgz exists and contains host certificates created by the Online CA. The ansible scripts will process these credentials from PEM into PKCS12, understood by Delegation Server. It will also take care of removing the original /root/frontend.canet.tgz in order to avoid having unprotected duplicate credentials lying around.

If you're updating the Delegation Server with some new configuration or version, without any update to the private communication link, you can simply ignore this file. In its absence the ansible scripts will simply ignore this part of the setup and will assume that the credential has already been translated to PKCS12 and copied to the desired location.

Nikhef specific configurations

The cafrontend role contains all of the Nikhef specific configuration that you might want to check and tweak in case you are using these scripts somewhere else. You should take a look at the following :

  • iptables and ip6tables rules from template will reflect the Nikhef intranet. Make sure to check these, otherwise you risk closing yourself out of your own machine!
  • access.conf template and access.yml will apply access control rules on the machine which are again specific to the Nikhef intranet. Make sure to check these, otherwise you risk closing yourself out of your own machine!
  • the nikhef local repository is installed (but no enabled) by default, together with some software (mkgroup-sshlpk) for provisioning ssh access to the machine.

Roles

cafrontend

The cafrontend role takes care of deploying a basic environment for the Delegation Server in the context of it being a frontend for the Online CA. The tasks belonging to the cafrontend role cover the following:

  • install / update required packages and repositories
  • configure access to the machine (iptables, ssh, access.conf)
  • deploy host credentials (see input 1. below)
  • postfix configuration for mail sending and forwarding from the Online CA
  • proxy configuration for the Online CA. The Online CA needs to apply updates through the Delegation Server.
  • backup configuration via rsync.
  • enable required services, disable not required services

Before applying the cafrontend role make sure to provide the following inputs for the ansible scripts:

  1. You need to have host certificates ready to apply this role. Place your PEM formatted certificate and key file under the 'roles/cafrontend/files/' directory. Note that the private key of the host certificate should be left unencrypted, so that services running on the host can use it as is. You will need to call these credentials 'hostname'.crt and 'hostname'.key, where 'hostname' is the same name that you provided in your inventory file of your play. In case your host certificate is signed by an intermediate CA you might want to also set the 'hostcert_intermediate' variable in the delegserver role, so that apache-httpd will send along the intermediate CA together with the host certificate.
  2. Overwrite the recommended default variables from cafrontend_env.yml to match your environment.

delegserver

The delegserver role takes care of deploying and configuring the Delegation Server and its dependencies. The tasks belonging to the delegserver role cover the following:

  • install and configure dependencies : java, tomcat, apache-httpd
  • add database support through either mysql or mariadb (postgres is not yet supported).
  • install and configure Shibboleth service provider
  • install and configure the Delegation Server
  • start/restart relevant services (tomcat, apache-httpd, shibd)

Before applying the delegserver role make sure to provide the following inputs for the ansible scripts:

  1. Add the compiled Delegation Server war to the deployment. If you don't have a compiled Delegation Server you can build it yourself. Add the compiled war file (oauth2.war) into the 'role/delegserver/files' directory.
  2. Add the compiled Delegation Server CLI to the deployment. You will need this CLI tool to approve and manage your Master Portals (Delegation Server clients)! If you don't have a compiled CLI you can build it yourself. Place the compiled jar (oa2-cli.jar) into the 'role/delegserver/files' directory.
  3. Overwrite the recommended default variables from delegserver_env.yml to match your environment.
  4. Generate secrets by executing the password generator.
  5. For new installations you will also need to create a set of JSON web keys, this can be done e.g. using the oa2-cli tool with the use signing command. The resulting jwk file is expected to be called mp.jwk and should be put in roles/delegserver/files/. You should specify the default Key ID as oa4mp_server_jwk_key_id in the delegserver_env.yml file. Note: you can also run the oa2-cli directly using
    java -jar oa2-cli.jar -cfg barecfg.xml -log oa2-cli.log
    where barecfg.xml contains
    <config><service/></config>

Plays

There is one main play called delegserver.yml which takes care of starting the Delegation Server deployment. This play includes both cafrontend and delegserver roles. Before executing the play make sure to have the secrets_env.yml file ready with all the required passwords for the setup! You can generate the secrets.yml file using the secret generator. You can execute the play, after you've given the required deployment input for the both roles, by calling:

ansible-playbook -i nikhef delegserver.yml

Replace the 'nikhef' inventory file with your own inventory file. It is safe to re-execute the plays multiple times, therefore you can also use this script to do an update on a Delegation Server with a new release!

Views
Personal tools