LGI Pilotjob Framework

The Leiden Grid Infrastructure (LGI) is a framework for executing high-performance applications on different computer systems. It consists of one or more project servers that keep track of all jobs and resources, and a collection of resources that regularly contact the project server for work.

This project makes a connection to gLite grid infrastructure (EGI) by introducing an efficient way to run jobs on the grid that are managed by an LGI setup. This would have the following benefits:

• Improved latency with respect to the grid.
• Users do not have to port an application to the grid, as the LGI administrator makes sure applications are running properly.
• Better scalability, since the WMS can give problems with a large number of jobs.
• Possibility for username/password authentication instead of certificates (with the aid of a robot certificate).
• Possibility to mix grid and non-grid systems.

With a little luck, this will make the grid more accessible to less technically minded people.

Architecture of the LGI pilotjob framework.

Architecture

An existing LGI setup is the base, and that's all that's visible to the user. This is centered around the project server. The project server communicates with resources that execute the work. A resource consists of a resource daemon that runs the application. Now on the grid there are pilotjobs, which do nothing but running a single application by means of a LGI resource daemon. They are the work-horses. The lifecycle of these pilotjobs is managed by the pilotjob manager (which makes sure enough pilotjobs are running all the time).

With everything properly tuned, this means in practice that users can submit jobs using one of the LGI interfaces, while they are executed on the grid.

Status

The current prototype shows good results. When we have some experience with real-world applications, we plan to release a public version.

R

One idea is to use this framework to provide easy access to R on the grid.

• netWorkSpaces with R, seems to have moved to the commercial scene at Revolution Analytics: a special edition of R for large-scale computing
• Biocep-R. Example: Elastic-R, a google docs-like portal for data analysis in the cloud
• State-of-the-art in Parallel Computing with R
• High-Performance and Parallel Computing with R
• Running R on HPC
• Other R portals
  • GridSphere R portal work plan

Links

• Pilotjob frameworks and related
  • DIANE
  • Ganga
  • Panda