Difference between revisions of "RCauth Delegation Server & MasterPortal - Building from Source"

From PDP/Grid Wiki
Jump to navigationJump to search
(→‎Delegation Server - CLI: build cli without modifying POM)
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
= Introduction =
 
= Introduction =
  
Both Master Portal and Delegation Server are built on top of the [http://grid.ncsa.illinois.edu/myproxy/oauth/ OA4MP] upstream project written in Java. As also stated in the [http://grid.ncsa.illinois.edu/myproxy/oauth/server/manuals/getting-started.xhtml Prerequisites] only '''Java 1.8''' from '''Oracle/Sun''' is supported. Older java version and OpenJDK are not supported! The tested java version is '''1.8u45''', but the [https://java.com/en/download/linux_manual.jsp latest release] of java 1.8 should also work.  
+
Both Master Portal and Delegation Server are built on top of the [http://grid.ncsa.illinois.edu/myproxy/oauth/ OA4MP] upstream project written in Java. As also stated in the [http://grid.ncsa.illinois.edu/myproxy/oauth/server/manuals/getting-started.xhtml Prerequisites] '''run time''' only '''Java 1.8''' from '''Oracle/Sun''' is supported. Older java version and OpenJDK are not supported!<br>
 +
For building, (Open)JDK 8 and 11 are both tested.
  
Both projects are built using [https://www.apache.org/ Maven]. Versions 2.x and 3.x of Maven were both tested and found to be working with the source code. If you are using Maven from the command line, make sure to set it up such that it would recognize the java version you downloaded for the build. You can easily do this by setting JAVA_HOME in your '''~/.mavenrc''' as such:
+
All projects are built using [https://www.apache.org/ Maven]. The currently tested versions are 3.5 and 3.6. If you are using Maven from the command line, make sure to set it up such that it would recognize the correct java version. You can easily do this by setting JAVA_HOME in your '''~/.mavenrc''' as such:
  
 
  export JAVA_HOME=<path to jdk 1.8>
 
  export JAVA_HOME=<path to jdk 1.8>
  
== Working with Eclipse ==
+
== Working with Eclipse and IntelliJ ==
  
In case you are doing some development work on the project, you might want to use an IDE for convenience. If you're only building the project the command line maven might be the easier way to go. When working with Eclipse you will need a couple of plugins in order to build the code:
+
In case you are doing some development work on the project, you might want to use an IDE for convenience. If you're only building the project the command line maven will be the easier way to go.<br>
 +
Good IDEs to use are Eclipse or IntelliJ. IntelliJ works pretty much without further adaptations. Just make sure to use an appropriate OpenJDK.
 +
 
 +
When working with Eclipse you will need a couple of plugins in order to build the code:
  
 
* Maven SCM Handler for EGit  
 
* Maven SCM Handler for EGit  
Line 15: Line 19:
 
* m2e connector for build-helper-maven-plugin
 
* m2e connector for build-helper-maven-plugin
  
Make sure that Eclipse is working with the right java version! You can change the default java version by navigating to ''Window -> Preferences -> Installed JREs''. Change the default java version '''before''' adding the project to the workspace, otherwise the project will be configured to use the old default. After doing the first time configuration from above you can check out the project into your workspace with ''File -> Import -> Maven -> Check out Maven Projects from SCM'' and following the instruction on the screen. Happy coding!  
+
Make sure that Eclipse is working with the right java version! You can change the default java version by navigating to ''Window -> Preferences -> Installed JREs''. Change the default java version '''before''' adding the project to the workspace, otherwise the project will be configured to use the old default. After doing the first time configuration from above you can check out the project into your workspace with ''File -> Import -> Maven -> Check out Maven Projects from SCM'' and following the instruction on the screen. Happy coding!
  
= Building the Master Portal =
+
= Rebuilding from source =
  
The Master Portal source code is currently hosted on [https://github.com/ttomttom/aarc-master-portal github]. The main (maven) project is called 'aarc-master-portal', which is just a parent project containing the following (maven) modules:
+
== Dependencies ==
 
 
* '''master-portal-client''' : contains the [[Master_Portal_Internals | MP Client]] component which makes requests to the Delegation Server.
 
* '''master-portal-server''' : contains the [[Master_Portal_Internals | MP Server]] component which takes requests from registered Portals.
 
* '''master-portal-common''' : contains a set of common constants and utility classes shared between the MP Client and MP server. Both master-portal-client and master-portal-server include this module as a dependency
 
  
Make sure to have the required [[#Dependencies | dependencies]] ready before compiling the Master Portal.
+
Both the MasterPortal and Delegation Server require two dependencies to be built first: security-lib and OA4MP. You need to build and install the RCauth flavour of both before building the MasterPortal or Delegation Server.
  
== Dependencies ==
+
=== background ===
  
The '''master-portal-client''' module is based on a modified version of the [http://grid.ncsa.illinois.edu/myproxy/oauth/client/index.xhtml OA4MP Client], while the the '''master-portal-server''' module is based on a modified version of the [http://grid.ncsa.illinois.edu/myproxy/oauth/server/index.xhtml OA4MP Server]. The upstream OA4MP implementation is under a project called [https://sourceforge.net/p/cilogon/code/HEAD/tree/trunk/edu.uiuc.ncsa/myproxy/ myproxy] with a parent project called [https://sourceforge.net/p/cilogon/code/HEAD/tree/trunk/edu.uiuc.ncsa/ncsa-security-all/ ncsa-security-all]. These two project also stand at basis of the [http://www.cilogon.org/ CILogon] project. We forked both the '''myproxy''' and the '''ncsa-security-all''' projects to complete them with the following features:
+
The upstream OA4MP implementation can be found as two github repositories on the [https://github.com/ncsa National Center for Supercomputing Applications] organization, in particular [https://github.com/ncsa/OA4MP Open Authorization for MyProxy (OA4MP)] with a parent project [https://github.com/ncsa/security-lib security-lib]. These two project also stand at basis of the [http://www.cilogon.org/ CILogon] project. We forked both projects as '''[https://github.com/rcauth-eu/OA4MP OA4MP] ''' and '''[https://github.com/rcauth-eu/security-lib security-lib]''' to complete them with (among others) the following features:
  
* the [[OAuth_for_MyProxy_GetProxy_Endpoint | /getproxy]] endpoint
+
* [[OAuth_for_MyProxy_GetProxy_Endpoint_-_Implementation | implementing a /getproxy]] endpoint
 
* an updated set of executable commands from the [http://grid.ncsa.illinois.edu/myproxy/protocol/ MyProxy Protocol] (GET with VONAME and VOMSES, INFO, PUT, STORE)
 
* an updated set of executable commands from the [http://grid.ncsa.illinois.edu/myproxy/protocol/ MyProxy Protocol] (GET with VONAME and VOMSES, INFO, PUT, STORE)
  
 
The [[OAuth_for_MyProxy_GetProxy_Endpoint | /getproxy]] endpoint is a new endpoint we are using on the Master Portal Server to serve [https://tools.ietf.org/html/rfc3820 RFC3820] proxy certificates, while the updated MyProxy commands set is used by the Master Portal to save (PUT, STORE) user proxies in a MyProxy Credential Store, and retrieve (INFO, GET) proxies with VOMS extensions.  
 
The [[OAuth_for_MyProxy_GetProxy_Endpoint | /getproxy]] endpoint is a new endpoint we are using on the Master Portal Server to serve [https://tools.ietf.org/html/rfc3820 RFC3820] proxy certificates, while the updated MyProxy commands set is used by the Master Portal to save (PUT, STORE) user proxies in a MyProxy Credential Store, and retrieve (INFO, GET) proxies with VOMS extensions.  
  
We also considered implementing these modifications into the Master Portal project directly, but we abandoned this approach because it would have meant extensive code duplication from the upstream OA4MP project. The implemented features are generic enough so that it might benefit the upstream OA4MP project as well, and might, some day end up being merged into it.  
+
We considered implementing these modifications into the Master Portal project directly, but abandoned such an approach as it would have meant an extensive code duplication from the upstream OA4MP project. The implemented features are generic enough so that it might benefit the upstream OA4MP project as well, and might, some day end up being merged into it.
  
=== ncsa-security-all-fork ===
+
=== security-lib ===
  
The '''ncsa-security-all''' forked project can be found on [https://github.com/ttomttom/ncsa-security-all-fork github]. In order to keep in sync with the upstream code we have two branches. The 'master' branch holds a clean copy of the upstream ncsa-security-all project, while the 'devel' branch holds the actual forked code. Whenever there's an update on the master branch we can merge it into the 'devel' branch to keep our fork up to date.
+
The '''security-lib''' forked project can be found on [https://github.com/rcauth-eu/security-lib github]. The 'master' branch holds a clean copy of the upstream security-lib repository, while the RCauth labelled branches and tags hold our customized code. Security-lib is a necessary dependency for [[#OA4MP]].
  
In order to build the Master Portal you will need to clone the '''devel''' branch from ncsa-security-all-fork, compile it, and install it in your local maven repository. You do this by executing the following:
+
The README.md file in the top level directory contains all the steps for building and installing the security-lib component.
  
git clone -b devel https://github.com/ttomttom/ncsa-security-all-fork.git
+
=== OA4MP ===
cd ncsa-security-all-fork/ncsa-security-all
 
mvn package install
 
 
 
'''Note!''' Make sure to do this only BEFORE you compile [[#myproxy-fork | myproxy-fork]]!
 
 
 
=== myproxy-fork ===
 
 
   
 
   
The '''myproxy''' forked project can be found on [https://github.com/ttomttom/myproxy-fork github]. Just as with [[#myproxy-fork | myproxy-fork]], in order to keep in sync with the upstream code we have two branches. The 'master' branch holds a clean copy of the upstream myproxy project, while the 'devel' branch holds the actual forked code. Whenever there's an update on the master branch we can merge it into the 'devel' branch to keep our fork up to date.
+
The '''OA4MP''' forked project can be found on [https://github.com/rcauth-eu/OA4MP github]. Just as with [[#security-lib | security-lib]], the 'master' branch holds a clean copy of the upstream OA4MP repository, while the RCauth labelled branches and tags hold our customized code. OA4MP relies on [[#security-lib]].
  
In order to build the Master Portal you will need to clone the '''devel''' branch from myproxy-fork, compile it, and install it in your local maven repository. You do this by executing the following:
+
The README.md file in the top level directory contains all the steps for building and installing the OA4MP component.
  
git clone -b devel https://github.com/ttomttom/myproxy-fork.git
+
== Building the Master Portal ==
cd myproxy-fork/myproxy
 
mvn package install
 
  
'''Note!''' Make sure to do this only AFTER you complied and installed [[#ncsa-security-all-fork | ncsa-security-all-fork]] otherwise it will fail to find the right dependencies!
+
The '''master-portal-client''' module is based on a modified version of the [http://grid.ncsa.illinois.edu/myproxy/oauth/client/index.xhtml OA4MP Client], while the the '''master-portal-server''' module is based on a modified version of the [http://grid.ncsa.illinois.edu/myproxy/oauth/server/index.xhtml OA4MP Server].
  
== Master Portal ==
+
The Master Portal source code is hosted on [https://github.com/rcauth-eu/aarc-master-portal github]. The main (maven) project is called 'aarc-master-portal', which is just a parent project containing the following (maven) modules:
  
After you build the required [[#Dependencies | dependencies]] you can go ahead and build the Master Portal itself. Check out the Master Portal source code from [https://github.com/ttomttom/aarc-master-portal github], and build it as such:
+
* '''master-portal-client''' : contains the [[Master_Portal_Internals | MP Client]] component which makes requests to the Delegation Server.
 
+
* '''master-portal-server''' : contains the [[Master_Portal_Internals | MP Server]] component which takes requests from registered Portals.
git clone https://github.com/ttomttom/aarc-master-portal.git
+
* '''master-portal-common''' : contains a set of common constants and utility classes shared between the MP Client and MP server. Both master-portal-client and master-portal-server include this module as a dependency
cd aarc-master-portal/master-portal
 
mvn package
 
 
 
After maven has finished you should end up with two separate .war files in your target directory, one for the MP Server and one for the MP Client:
 
 
 
master-portal-server/target/mp-oa2-server.war
 
master-portal-client/target/mp-oa2-client.war
 
 
 
'''Note!''' If you're using [[CILogon_Pre-Pilot_Work_-_Ansible#masterportal | Ansible]] to deploy the Master Portal, copy these war files over to the appropriate location in the Ansible scripts!
 
 
 
== Master Portal - CLI ==
 
 
 
Next to the Master Portal you will also want to build the Master Portal CLI for managing and approving client (Portal) registrations. Without this you won't be able to use any Portals with your Master Portal (unless you approve them manually). The CLI is originally a [http://grid.ncsa.illinois.edu/myproxy/oauth/server/manuals/cli.xhtml OA4MP tool], but it has to be recompiled together with the Master Portal, because the original CLI does not include all the relevant additional classes implemented and used by the Master Portal. Only the CLI compiled from the Master Portal will work together with a Master Portal!
 
 
 
The Master Portal CLI is built by applying the '''cli''' profile while building the '''master-portal-server''' module. Before you execute the maven command you will have to change the '''master-portal-server''' pom.xml to produce a .jar file instead of a .war file, since the '''cli''' target is also going to be a jar file which has to include the '''master-portal-server''' code.
 
 
# replace 'war' with 'jar' in the server pom.xml file
 
cd aarc-master-portal/master-portal
 
sed -i 's#^\(\s*\)<packaging>war</packaging>\(\s*\)$#\1<packaging>jar</packaging>\2#' master-portal-server/pom.xml
 
 
# build the master portal and install it to the local maven repository
 
# without this the 'cli' maven profile cannot find its dependency
 
mvn clean package install
 
 
# build the cli
 
cd master-portal-server
 
mvn clean package -P cli
 
 
 
After it has finished you should en up with the resulting cli .jar file inside the target directory:
 
 
 
master-portal-server/target/oa2-cli.jar
 
 
 
Don't forget to change the pom.xml file back to it's original state, producing .war files!
 
 
 
'''Note!''' If you're using [[CILogon_Pre-Pilot_Work_-_Ansible#masterportal | Ansible]] to deploy the Master Portal, copy this cli jar file over to the appropriate location in the Ansible scripts!
 
 
 
= Building the Delegation Server =
 
 
 
The Delegation Server is currently hosted on [https://github.com/ttomttom/aarc-delegation-server github]. It's made up of a single module which is built on top of the [http://grid.ncsa.illinois.edu/myproxy/oauth/server/index.xhtml OA4MP Server], just like the [[#Building_the_Master_Portal | master-portal-server]] module. But unlike the [[#Building_the_Master_Portal | master-portal-server]] module, the Delegation Server is built on top of the plain OA4MP Server and not the forked one. The reason being that the Delegation Server doesn't need any of the added features for it to function. This saves us some effort since we don't need to compile the forked OA4MP Server, we can simply rely on the plain OA4MP Server to be pulled in by maven.
 
 
 
== Delegation Server ==
 
 
 
Check out the Delegation Server source code from [https://github.com/ttomttom/aarc-delegation-server github] and compile it by executing:
 
 
 
git clone https://github.com/ttomttom/aarc-delegation-server.git
 
cd aarc-delegation-server/delegation-server
 
mvn package
 
 
 
After the building has finished you should end up with a single .war file in
 
 
 
aarc-delegation-server/delegation-server/target/oauth2.war
 
 
 
'''Note!''' If you're using [[CILogon_Pre-Pilot_Work_-_Ansible#Delegation_Server | Ansible]] to deploy the Delegation Server, copy this .war file over to the appropriate location in the Ansible scripts!
 
 
 
== Delegation Server - CLI ==
 
 
 
Next to the Delegation Server you will also want to build the Delegation Server CLI for managing and approving client (Master Portal) registrations. Without this you won't be able to use any Master Portals with your Delegation Server (unless you approve them manually). The CLI is originally a [http://grid.ncsa.illinois.edu/myproxy/oauth/server/manuals/cli.xhtml OA4MP tool], but it has to be recompiled together with the Delegation Server because the original CLI does not include all the relevant additional classes implemented and used by the Delegation Server. Moreover, the Delegation Server CLI has been extended to support an addition 'description' field for it's registered clients. Only the CLI compiled from the Delegation Server will work together with a Delegation Server!
 
 
 
Just as with the [[#Master_Portal_-_CLI | Master Portal CLI]], you have to apply the '''cli''' profile and build the Delegation Server as a .jar file:
 
  
cd aarc-delegation-server/delegation-server
+
The README.md file in the top level directory contains all the steps for building the MasterPortal components.
mvn clean package -P cli
 
  
After it has finished you should en up with the resulting cli .jar file inside the target directory:
+
In case you want to deploy also a VO portal and/or SSH key portal, you can find their sources at
 +
* [https://github.com/rcauth-eu/aarc-vo-portal VO-portal]
 +
* [https://github.com/rcauth-eu/aarc-ssh-portal SSH key portal]
 +
Like for the MasterPortal, the README.md file in the top level directory contains all the steps for building.
  
aarc-delegation-server/delegation-server/target/oa2-cli.jar
+
== Building the Delegation Server ==
  
Don't forget to change the pom.xml file back to it's original state, producing .war files!
+
The Delegation Server is hosted on [https://github.com/rcauth-eu/aarc-delegation-server github]. It's made up of a single module which is built on top of the [http://grid.ncsa.illinois.edu/myproxy/oauth/server/index.xhtml OA4MP Server], just like the [[#Building_the_Master_Portal | master-portal-server]] module.<br>
 +
The code could in principle compile against the plain upstream code (since it doesn't need any of the added features for it to function), but it is strongly advised to use also for this the same RCauth forks as described above under [[#Dependencies | dependencies]].
  
'''Note!''' If you're using [[CILogon_Pre-Pilot_Work_-_Ansible#delegserver | Ansible]] to deploy the Delegation Server, copy this cli jar file over to the appropriate location in the Ansible scripts!
+
The README.md file in the top level directory contains all the steps for building the Delegation Server components.

Latest revision as of 09:19, 5 September 2019

Introduction

Both Master Portal and Delegation Server are built on top of the OA4MP upstream project written in Java. As also stated in the Prerequisites run time only Java 1.8 from Oracle/Sun is supported. Older java version and OpenJDK are not supported!
For building, (Open)JDK 8 and 11 are both tested.

All projects are built using Maven. The currently tested versions are 3.5 and 3.6. If you are using Maven from the command line, make sure to set it up such that it would recognize the correct java version. You can easily do this by setting JAVA_HOME in your ~/.mavenrc as such:

export JAVA_HOME=<path to jdk 1.8>

Working with Eclipse and IntelliJ

In case you are doing some development work on the project, you might want to use an IDE for convenience. If you're only building the project the command line maven will be the easier way to go.
Good IDEs to use are Eclipse or IntelliJ. IntelliJ works pretty much without further adaptations. Just make sure to use an appropriate OpenJDK.

When working with Eclipse you will need a couple of plugins in order to build the code:

  • Maven SCM Handler for EGit
  • m2e - Maven Integration for Eclipse
  • m2e connector for build-helper-maven-plugin

Make sure that Eclipse is working with the right java version! You can change the default java version by navigating to Window -> Preferences -> Installed JREs. Change the default java version before adding the project to the workspace, otherwise the project will be configured to use the old default. After doing the first time configuration from above you can check out the project into your workspace with File -> Import -> Maven -> Check out Maven Projects from SCM and following the instruction on the screen. Happy coding!

Rebuilding from source

Dependencies

Both the MasterPortal and Delegation Server require two dependencies to be built first: security-lib and OA4MP. You need to build and install the RCauth flavour of both before building the MasterPortal or Delegation Server.

background

The upstream OA4MP implementation can be found as two github repositories on the National Center for Supercomputing Applications organization, in particular Open Authorization for MyProxy (OA4MP) with a parent project security-lib. These two project also stand at basis of the CILogon project. We forked both projects as OA4MP and security-lib to complete them with (among others) the following features:

The /getproxy endpoint is a new endpoint we are using on the Master Portal Server to serve RFC3820 proxy certificates, while the updated MyProxy commands set is used by the Master Portal to save (PUT, STORE) user proxies in a MyProxy Credential Store, and retrieve (INFO, GET) proxies with VOMS extensions.

We considered implementing these modifications into the Master Portal project directly, but abandoned such an approach as it would have meant an extensive code duplication from the upstream OA4MP project. The implemented features are generic enough so that it might benefit the upstream OA4MP project as well, and might, some day end up being merged into it.

security-lib

The security-lib forked project can be found on github. The 'master' branch holds a clean copy of the upstream security-lib repository, while the RCauth labelled branches and tags hold our customized code. Security-lib is a necessary dependency for #OA4MP.

The README.md file in the top level directory contains all the steps for building and installing the security-lib component.

OA4MP

The OA4MP forked project can be found on github. Just as with security-lib, the 'master' branch holds a clean copy of the upstream OA4MP repository, while the RCauth labelled branches and tags hold our customized code. OA4MP relies on #security-lib.

The README.md file in the top level directory contains all the steps for building and installing the OA4MP component.

Building the Master Portal

The master-portal-client module is based on a modified version of the OA4MP Client, while the the master-portal-server module is based on a modified version of the OA4MP Server.

The Master Portal source code is hosted on github. The main (maven) project is called 'aarc-master-portal', which is just a parent project containing the following (maven) modules:

  • master-portal-client : contains the MP Client component which makes requests to the Delegation Server.
  • master-portal-server : contains the MP Server component which takes requests from registered Portals.
  • master-portal-common : contains a set of common constants and utility classes shared between the MP Client and MP server. Both master-portal-client and master-portal-server include this module as a dependency

The README.md file in the top level directory contains all the steps for building the MasterPortal components.

In case you want to deploy also a VO portal and/or SSH key portal, you can find their sources at

Like for the MasterPortal, the README.md file in the top level directory contains all the steps for building.

Building the Delegation Server

The Delegation Server is hosted on github. It's made up of a single module which is built on top of the OA4MP Server, just like the master-portal-server module.
The code could in principle compile against the plain upstream code (since it doesn't need any of the added features for it to function), but it is strongly advised to use also for this the same RCauth forks as described above under dependencies.

The README.md file in the top level directory contains all the steps for building the Delegation Server components.