Difference between revisions of "LGI Pilotjob Framework"
m (→Status) |
m (→R: add links) |
||
| Line 25: | Line 25: | ||
* [https://spaces.umbc.edu/display/hpc/Running+R+on+HPC Running R on HPC] | * [https://spaces.umbc.edu/display/hpc/Running+R+on+HPC Running R on HPC] | ||
* [http://www.revolutionanalytics.com/ Revolution Analytics]: special edition of R for large-scale computing | * [http://www.revolutionanalytics.com/ Revolution Analytics]: special edition of R for large-scale computing | ||
| + | * Other R portals | ||
| + | ** [http://technical.bestgrid.org/index.php/GridSphere_R_Portal_Initial_Work_Plan GridSphere R portal work plan] | ||
| + | ** [http://user2010.org/tutorials/Chine.html Elastic-R], a google docs-like portal for data analysis in the cloud | ||
Revision as of 10:09, 25 October 2010
The Leiden Grid Infrastructure (LGI) is a framework for executing high-performance applications on different computer systems. It consists of one or more project servers that keep track of all jobs and resources, and a collection of resources that regularly contact the project server for work.
This project makes a connection to gLite grid infrastructure (EGI) by introducing an efficient way to run jobs on the grid that are managed by an LGI setup. This would have the following benefits:
- Improved latency with respect to the grid.
- Users do not have to port an application to the grid, as the LGI administrator makes sure applications are running properly.
- Better scalability, since the WMS can give problems with a large number of jobs.
- Possibility for username/password authentication instead of certificates (with the aid of a robot certificate).
- Possibility to mix grid and non-grid systems.
With a little luck, this will make the grid more accessible to less technically minded people.
Architecture
An existing LGI setup is the base, and that's all that's visible to the user. This is centered around the project server. The project server communicates with resources that execute the work. A resource consists of a resource daemon that runs the application. Now on the grid there are pilotjobs, which do nothing but running a single application by means of a LGI resource daemon. They are the work-horses. The lifecycle of these pilotjobs is managed by the pilotjob manager (which makes sure enough pilotjobs are running all the time).
With everything properly tuned, this means in practice that users can submit jobs using one of the LGI interfaces, while they are executed on the grid.
Status
The current prototype shows good results. When we have some experience with real-world applications, we plan to release a public version.
R
One idea is to use this framework to provide easy access to R on the grid.
- State-of-the-art in Parallel Computing with R
- High-Performance and Parallel Computing with R
- Running R on HPC
- Revolution Analytics: special edition of R for large-scale computing
- Other R portals
- GridSphere R portal work plan
- Elastic-R, a google docs-like portal for data analysis in the cloud
