UNICORE Notes and an Adventure!

Today's sessions were all about UNICORE. The first part of the day was an introduction to UNICORE and then there was a more in-depth and technical discussion of how everything works. In the afternoon, we were able to have a hands-on experience with the technology in the lab.

UNICORE Part 1 - Mathilde Romberg

The lectures were led by people from Jülich Forschungszentrum. One presenter said that UNICORE got started in about '97, and that she was around for pretty much all of its development. As far as the material goes...I took some notes that I would be happy to share with anyone that is interested, but I'm not sure how much of it I should put in the blog. Basically, the first presenter went through the concept of job submission and job scheduling. She said that there were two main approaches to resource management: time sharing (as in round robin) and space sharing (exclusively assigned resources). UNICORE uses space sharing, so it requires a separate batch scheduler (like Torque or PBS). Then she went through the format for submitting a job using LoadLeveler and then for Maui. Next she showed us an example XML file example of Job Submission Description Language (JSDL), which is used to provide interoperability between middleware. It only supports "simple" descriptions of resource requirements. Reflecting on it now, I'm not exactly sure what makes a description simple or complex. Lastly, she said that the job of the middleware is to handle execution and job management: initiating, monitoring, managing, coordinating/scheduling jobs and workflows.

UNICORE Part Two - Rebecca Breu

The next presenter focused on security, similar to Emidio's presentation yesterday, but more specific to UNICORE. She went through the different components that make the software work. My understanding is this: There is a gateway, and all user to server to server-component communication goes through it. There is a registry, the services register with it as they become available, and then the client contacts it (through the gateway) to find out what they can use. There is also the UNICOREx which is responsible for translating JSDL jobs to concrete jobs that can be understood by specific machines, and for authoring requests using XUUDB, which is a database that provides a mapping from X.509 certificates to logins. There is also the Target System Interface (TSI) which communicates with the batch system so that it can manage the working directory and actually start the process for the job. There was also something called the Incarnation Database (IDB), which translates abstract jobs into executable scripts. But, I'm pretty sure that the job descriptions written in JSDL are abstract jobs, and I thought that UNICOREx was responsible for the translation, so I'm not sure which part of my interpretation of the presentation is wrong.

UNICORE Part 3 - Bastian Demuth

The last presentation of the morning was a very specific description of how the system works, and for that I would suggest that you take a look at the slides. There were very good diagrams that clearly showed how everything worked together that explain it better than I can here. In short, there are a variety of web services (using the gateway, registry, and other terms in the previous paragraph) that provide the functionality that is necessary to run jobs.

UNICORE Practical

In the afternoon sessions, we did a hands-on exercise involving submitting real jobs to UNICORE. There were a few problems with the configuration at first, it would work for some people...but not for others, and everyone wasn't getting the same errors...but eventually it all worked out. We switched to the graphical UNICORE Rich client while the problems with the configuration for the command line version were being worked out. I was very impressed with the software, because it was very easy to understand and had a lot of extra features that I didn't think were available. For example, I'm moderately familiar with the idea of using a DAG with Condor to run jobs. However, with Rich, it's very clear and you can construct a diagram representing the workflow using pieces that represent the scripts. There are also ways to include while loops and if statements within the workflow diagram, and it's extremely easy to customize it to suit your needs. Another thing I was impressed with was the ability to trace through as it executes, literally watching it progress through the DAG. There was even the ability to see which path of the diagram was followed (based on an if statement, for example). From my understanding though, UNICORE is a European technology...so hopefully I will be as impressed by Globus.

------------------------------------------

After the talks, we were given an assignment. A scavenger hunt! With our newly assigned team-mates. They hid clues all around the campus, and our job was to find them using the hints that we were given, do some searching on the Internet, and eventually come up with a Gaelic phrase that we were supposed to report as our answer as soon as possible. It was a lot of fun, and did serve as a very good bonding exercise for our team...but the clues were so well hidden! There was one inside a trashcan...one WAY in the bushes...one under a bridge that everyone had trouble finding. It was a good time though. I am very glad that I got to explore a bit yesterday, because I already had a vague idea of where some of the places were. Next time, I will skip dinner before starting so that my team can have a headstart. :)