LGI Pilotjob Framework

The Leiden Grid Infrastructure (LGI) is a framework for executing high-performance applications on different computer systems. It consists of one or more project servers that keep track of all jobs and resources, and a collection of resources that regularly contact the project server for work.

This project makes a connection to gLite grid infrastructure (EGI) by introducing an efficient way to run jobs on the grid that are managed by an LGI setup. This would have the following benefits:

• Improved latency with respect to the grid.
• Users do not have to port an application to the grid, as the LGI administrator makes sure applications are running properly.
• Better scalability, since the WMS can give problems with a large number of jobs.
• Possibility for username/password authentication instead of certificates (with the aid of a robot certificate).
• Possibility to mix grid and non-grid systems.

With a little luck, this will make the grid more accessible to less technically minded people.

Architecture of the LGI pilotjob framework.

Architecture

An existing LGI setup is the base, and that's all that's visible to the user. This is centered around the project server. The project server communicates with resources that execute the work. A resource consists of a resource daemon that runs the application. Now on the grid there are pilotjobs, which do nothing but running a single application by means of a LGI resource daemon. They are the work-horses. The lifecycle of these pilotjobs is managed by the pilotjob manager (which makes sure enough pilotjobs are running all the time).

With everything properly tuned, this means in practice that users can submit jobs using one of the LGI interfaces, while they are executed on the grid.

Status

The current prototype shows good results. We're currently shaping up the code for a release. The current code is in beta-shape and available from github.

R

One idea is to use this framework to provide easy access to R on the grid. A prototype Rlgi package is available at github.

• netWorkSpaces with R, seems to have moved to the commercial scene at Revolution Analytics: a special edition of R for large-scale computing
• Biocep-R. Example: Elastic-R, a google docs-like portal for data analysis in the cloud
• State-of-the-art in Parallel Computing with R
• High-Performance and Parallel Computing with R
• Running R on HPC
• Other R portals
  • GridSphere R portal work plan
  • D4Science obtained integrated R support in 2011
  • Opani
• Other seamless submission portals
  • picloud
• Ubalo

Links

• EGEE Pilotjobs explained
• Pilotjob frameworks
  • DIANE & Ganga
  • Panda
  • Wisdom?
• Similar projects
  • A suite for distributed execution of general purpose applications on the Grid for small- and mid-size VOs
  • Job Submission Tool, web interface and WebDAV data management
• Pilotjob articles
  • SAGA BigJob: An Extensible and Interoperable Pilot-Job Abstraction for Distributed Applications and Systems (2010)
  • Modelling Pilot-Job Applications on Production Grids (2009) (pdf)