Difference between revisions of "RCauth Delegation Server & MasterPortal - Ansible scripts"

From PDP/Grid Wiki
Jump to navigationJump to search
Line 100: Line 100:
 
  git clone https://github.com/rcauth-eu/aarc-delegation-server.git
 
  git clone https://github.com/rcauth-eu/aarc-delegation-server.git
  
Refer to the README there for instructions on how to use the set of scripts. The remainder of this page contains background information on the layout of the scripts.
+
Refer to the README there for instructions on how to use the set of scripts. The remainder of this section contains background information on the layout of the scripts.
  
 
Note that these scripts are less general than those for the [[#Master_Portal | Master Portal ]], because they need to be more infrastructure specific and don't handle a 'real' online CA.
 
Note that these scripts are less general than those for the [[#Master_Portal | Master Portal ]], because they need to be more infrastructure specific and don't handle a 'real' online CA.

Revision as of 10:36, 3 September 2019

Introduction

Background

After experimenting with Jenkins as a method of deployment for the CILogon Pilot we quickly had to come to the conclusion that Jenkins was not meant to be used for such deployments. Although it can spin up machines and run any kind of configuration scripts on them, it was still lacking the kind of configuration control and templating capabilities that we were looking for. We decided to move away from it and use one of the more widespread tools for deployments and configuration.

Our pick fell on Ansible mainly because of the following points:

  • support for configuration templating
  • easy to use modules to interact with systems
  • no software required on target machine (runs through ssh)
  • good documentation
  • widely used

Note! Deployments done through these Ansible scripts are expecting a clean system with only the OS (CentOS 6 or CentOS 7) installed. No packages have to be installed manually before executing the deployment scripts, since Ansible will do that for you.

General Structure

The ansible deployment scripts are structured into, what ansible calls plays. In our deployment scripts we use a single play to configure a single host of a specific kind. Ansible plays are made up of roles. We are using roles as a means of grouping tasks that logically belong together.

Giving input to the deployment scripts is done through overwriting existing variables. Each ansible role contains two sets of default variables:

  • roles/x/vars/main.yml holds OS specific variables (like package names, repositories, service name, etc...). These are only used within a role (x), and they should not be overridden by variables defined inside a play (see variable precedence). However, in order to support different OS versions, we have separate OS specific variable files under roles/x/vars/, like rh7.yml or rh6.yml. Depending on the OS version that you are using for your deployment, you should define the roles/x/vars/main.yml as a symlink to one of the existing OS specific variable files.
  • roles/x/defaults/main.yml holds application specific variables (like configuration options, paths, etc...). These are only used within a role (x), and they may be overridden by variables defined inside a play.

The variables set in roles/x/defaults/main.yml are overridden by files in a subdirectory of the top-level config/ directory. See config/PLACEHOLDER/x_env.yml for example files for each role (x). For a specific deployment a new subdirectory needs to be created and defined as configdir in the inventory file, see hosts.inventory.PLACEHOLDER for an example.

Generating Secrets

Both Master Portal and Delegation Server setup requires several different passwords for different components (database, rsync, keystore). To ease the pain of making up new passwords we provide an extra play that can be executed to generate a set of required passwords. The passwords are created with the openssl 'rand' module with special characters omitted. The play generating new passwords runs on your local machine and it creates a file called secrets_env.yml in the current directory. This file is then used as input for the Master Portal play and the Delegation Server play respectively. In case you don't want to use our password generator make sure to fill secrets_env.yml manually with all the required passwords!

For further details on how to use this play, see the README files.

Master Portal

The ansible scripts for deploying a Master Portal can be found on github as part of the rcauth-eu organization. You can use these scripts to deploy a Master Portal, Credential Store and SSH host on separate hosts, or on a single host. They assume a basic CentOS 6 or 7 installation.

git clone https://github.com/rcauth-eu/aarc-ansible-master-portal.git

Refer to the README there for instructions on how to use the set of scripts. The remainder of this section contains background information on the layout of the scripts.

Roles

common

This 'role' only loads variables common to all roles, it does not contain actual tasks.

basic

The basic role tries to cover the general environment setup that is needed for all components. The tasks belonging to the basic role cover the following configurations:

  • basic host setup (hostname, selinux, sshd)
  • install / update required packages and repositories (repos, yum)
  • configure access to the machine and ntp (network, iptables, access, ntpd)
  • deploy host credentials and grid related services (fetchcrl, hostcreds, vomses)
  • enable/disable services

credstore

The credstore role takes care of deploying and configuring any Credential Store specific software. The tasks belonging to the credstore role cover the following:

  • install and configure MyProxy Server as a Credential Store (myproxy-credstore)
  • install the relevant IGTF CA distributions (deploy-igtf-distrib)
  • add myproxy_purge for expired/revoked certificates (purger)
  • start/restart myproxy-server

masterportal

The masterportal role takes care of deploying and configuring the Master Portal, VO-portal, SSH-portal and their dependencies. The tasks belonging to the masterportal role cover the following:

  • install and configure java, tomcat, apache-httpd (java, tomcat, httpd)
  • install and configure mysql or mariadb (mysql). Note postgres is not yet supported but might work. Note: on RH6 mysql is used while on RH7 mariadb is used instead.
  • install mysql backup script and cronjob (mysql-backup)
  • install keyutil for creating key- and truststores (keyutil)
  • install and configure Master Portal Client and Server parts (oa4mp-client, oa4mp-server)
  • install and configure SSH portal for uploading SSH keys (optional, can be disabled by undefining sshkey_portal)
  • install and configure VO Portal for testing proxy retrieval (optional, can be disabled by undefining vo_portal)
  • start/restart relevant services tomcat, httpd and mariadb/mysql

sshhost

This role takes care of installing the host that will run the ssh server for getting proxy certificates on the command line.

  • create the proxy user account (proxyuser)
  • configure the hostcert/key for the proxy user (proxycreds)
  • copy the AuthorizedKeysCommand (authz_cmd) and myproxy_cmd scripts and their dependencies (scripts)
  • update the sshd config file and restart sshd (sshd)
  • update the access.conf file (access)

Plays

The ansible scripts include plays for each host type.
Each play will first apply the common and basic roles followed by either the credstore, masterportal or sshhost role.

Delegation Server

The ansible scripts for deploying a Delegation Server can be found on github as part of the rcauth-eu organization. Note that these ansible scripts will only deploy the Delegation Server and optionally a demo online CA. If you want to deploy proper Online CA as well, you can take a look at these deployment scripts.

git clone https://github.com/rcauth-eu/aarc-delegation-server.git

Refer to the README there for instructions on how to use the set of scripts. The remainder of this section contains background information on the layout of the scripts.

Note that these scripts are less general than those for the Master Portal , because they need to be more infrastructure specific and don't handle a 'real' online CA.

Integrating with Online CA

It is recommended to dedicate a full physical machine for the Delegation Server with two separate network interfaces. One of the network interfaces, what we call public_interface will be the outwards facing interface for incoming requests. The second interface, what we call private_interface is a dedicated connection to the Online CA. It is assumed that both Delegation Server and Online CA are kept in a safe and secure environment with nothing between the Delegation Server and Online CA other than a direct network connection. Consult the RCauth.eu CP/CPS for more details on recommended safety measures.

Roles

common

This 'role' only loads variables common to all roles, it does not contain actual tasks.

basic

The basic role tries to cover the general environment setup that is needed for all components. The tasks belonging to the basic role cover the following configurations:

  • basic host setup (hostname, sshd)
  • install / update required packages and repositories (epelrepo, yum)
  • configure access to the machine and ntp (network, iptables, access, ntpd)
  • enable/disable services

cafrontend

The cafrontend role takes care of deploying a basic environment for the Delegation Server in the context of it being a frontend for the Online CA. The tasks belonging to the cafrontend role cover the following:

  • configure the internal network (ca-network)
  • configure services related to the internal network (dnsmasq, postfix, squid, ca-sshd)
  • enable services

democa

This role sets up a fully-functional but demonstration-only online CA. In case of a 'real' CA, this is the place to configure it (i.e. after the cafrontend role). The tasks belonging to the democa role cover the following:

  • configure basic services (hosts, nameserver, yum_squid, postfix, rsyslog)
  • setup private CA for TLS on private net (canet_ca)
  • setup backup with rsync over ssh via cronjob (rsync_sshkey, backup)
  • configure myproxy-server as CA, export relevant parts to frontend (myproxy, hostcreds, democa_tarball)

delegserver

The delegserver role takes care of deploying and configuring the Delegation Server and its dependencies once the rest is setup. The tasks belonging to the delegserver role cover the following:

  • finalize setup for communication with Online CA (ca-sshd, ca-backup, ca_checker)
  • install and configure main services making up the Delegation Server (java, tomcat, httpd)
  • install and configure mariadb (mysql). Note postgres is not yet supported but might work.
  • install and configure Shibboleth service provider (shib)
  • install and configure backups (rsyncd, mysql-backup)
  • install and configure the Delegation Server tomcat servlet and cli (oa4mp-server)
  • configure the web page, e.g. for CRL production (webroot)
  • configure the credentials, certificate directories etc. (hostcreds-tomcat, hostcreds-apache)
  • start/restart relevant services

Plays

There are three main ansible scripts: two for setting up the delegation server, one for setting up a demo CA. The delegation server is set up in two stages, one part before the Online CA is setup, one part after.

  • The cafrontend play, being the first part of the Delegation Server setup, will apply the common, basic and cafrontend roles.
  • The demoCA play will apply the common, basic (unless co-hosted with the Delegation Server) and democa roles.
  • The delegserver play, being the second part the Delegation Server setup, will apply only the of the common and delegserver roles since the basic role has already been applied in the cafrontend play.

In case of a full backend CA, you would not run the second playbook, but instead run an installation of that CA at that point in time.