Today's sessions were all about UNICORE. The first part of the day was an introduction to UNICORE and then there was a more in-depth and technical discussion of how everything works. In the afternoon, we were able to have a hands-on experience with the technology in the lab.
UNICORE Part 1 - Mathilde Romberg
The lectures were led by people from Jülich Forschungszentrum. One presenter said that UNICORE got started in about '97, and that she was around for pretty much all of its development. As far as the material goes...I took some notes that I would be happy to share with anyone that is interested, but I'm not sure how much of it I should put in the blog. Basically, the first presenter went through the concept of job submission and job scheduling. She said that there were two main approaches to resource management: time sharing (as in round robin) and space sharing (exclusively assigned resources). UNICORE uses space sharing, so it requires a separate batch scheduler (like Torque or PBS). Then she went through the format for submitting a job using LoadLeveler and then for Maui. Next she showed us an example XML file example of Job Submission Description Language (JSDL), which is used to provide interoperability between middleware. It only supports "simple" descriptions of resource requirements. Reflecting on it now, I'm not exactly sure what makes a description simple or complex. Lastly, she said that the job of the middleware is to handle execution and job management: initiating, monitoring, managing, coordinating/scheduling jobs and workflows.
UNICORE Part Two - Rebecca Breu
The next presenter focused on security, similar to Emidio's presentation yesterday, but more specific to UNICORE. She went through the different components that make the software work. My understanding is this: There is a gateway, and all user to server to server-component communication goes through it. There is a registry, the services register with it as they become available, and then the client contacts it (through the gateway) to find out what they can use. There is also the UNICOREx which is responsible for translating JSDL jobs to concrete jobs that can be understood by specific machines, and for authoring requests using XUUDB, which is a database that provides a mapping from X.509 certificates to logins. There is also the Target System Interface (TSI) which communicates with the batch system so that it can manage the working directory and actually start the process for the job. There was also something called the Incarnation Database (IDB), which translates abstract jobs into executable scripts. But, I'm pretty sure that the job descriptions written in JSDL are abstract jobs, and I thought that UNICOREx was responsible for the translation, so I'm not sure which part of my interpretation of the presentation is wrong.
UNICORE Part 3 - Bastian Demuth
The last presentation of the morning was a very specific description of how the system works, and for that I would suggest that you take a look at the slides. There were very good diagrams that clearly showed how everything worked together that explain it better than I can here. In short, there are a variety of web services (using the gateway, registry, and other terms in the previous paragraph) that provide the functionality that is necessary to run jobs.
UNICORE Practical
In the afternoon sessions, we did a hands-on exercise involving submitting real jobs to UNICORE. There were a few problems with the configuration at first, it would work for some people...but not for others, and everyone wasn't getting the same errors...but eventually it all worked out. We switched to the graphical UNICORE Rich client while the problems with the configuration for the command line version were being worked out. I was very impressed with the software, because it was very easy to understand and had a lot of extra features that I didn't think were available. For example, I'm moderately familiar with the idea of using a DAG with Condor to run jobs. However, with Rich, it's very clear and you can construct a diagram representing the workflow using pieces that represent the scripts. There are also ways to include while loops and if statements within the workflow diagram, and it's extremely easy to customize it to suit your needs. Another thing I was impressed with was the ability to trace through as it executes, literally watching it progress through the DAG. There was even the ability to see which path of the diagram was followed (based on an if statement, for example). From my understanding though, UNICORE is a European technology...so hopefully I will be as impressed by Globus.
------------------------------------------
After the talks, we were given an assignment. A scavenger hunt! With our newly assigned team-mates. They hid clues all around the campus, and our job was to find them using the hints that we were given, do some searching on the Internet, and eventually come up with a Gaelic phrase that we were supposed to report as our answer as soon as possible. It was a lot of fun, and did serve as a very good bonding exercise for our team...but the clues were so well hidden! There was one inside a trashcan...one WAY in the bushes...one under a bridge that everyone had trouble finding. It was a good time though. I am very glad that I got to explore a bit yesterday, because I already had a vague idea of where some of the places were. Next time, I will skip dinner before starting so that my team can have a headstart. :)
Tuesday, July 7, 2009
Day two from Cole
This morning's lectures were nice, fairly informative, and had a bit of background on both schedulers and Unicore, with the emphasis on Unicore. The presentations were given by a group from Jülich-FZ. They shared the beginnings of Unicore and the present work being done in their group, and in several derivative groups. The ISSGC09 Unicore info was given to us for the practicals in the afternoon. After a few difficulties, we finally got up and running, and I am mightily impressed by the Client UI features.
This is definitely a tech to watch, and I am curious to play with it myself and run some tests.
As I'm writing this, we've only really created a few workflows yet, but so far, I'm liking it!
So, onto what is becoming my "SOP", how about a few pictures?
From the lectures in the AM, in order, we have Mathilde Rothemburg, Rebecca Breu, Bastian Demuth all from the same group.
For the practicals, it was J. M. Daivandy with Rebecca Breu, but for all my tries, I could not get him when he was facing me and nobody was in my way with a camera, so candid-camera-activate FAIL.
This is definitely a tech to watch, and I am curious to play with it myself and run some tests.
As I'm writing this, we've only really created a few workflows yet, but so far, I'm liking it!
So, onto what is becoming my "SOP", how about a few pictures?
From the lectures in the AM, in order, we have Mathilde Rothemburg, Rebecca Breu, Bastian Demuth all from the same group.
For the practicals, it was J. M. Daivandy with Rebecca Breu, but for all my tries, I could not get him when he was facing me and nobody was in my way with a camera, so candid-camera-activate FAIL.
An improptu trip to Antibes
Ok, so first I think I am obligated to say that Antibes has a silent "s", since that was said to me repeatedly ;) - on top of that, the gang doesn't like my game of "loud obnoxious american who speaks Spanish poorly while in France" but we had fun with it, and the locals laughed, so that was fun. And yes, I did it, but I made it obvious I was doing it (this is for the ones who know me stateside and are holding their heads saying ... "Cole, what were you thinking, no you didn't"). Seriously, I barely speak English well, so ... do what you do, I'll just be me. :-D
Now, on with the trip. It started as we had an early release for the day. We decided to goto Antibes, and have dinner! We started by the student housing at Isaac Newton House, walked over the hills and got confused (well, I kept saying follow me, but they wanted to look around) and finally got to the bus stop, and waited for like 25 minutes. Here's some photos :D
Now, on with the trip. It started as we had an early release for the day. We decided to goto Antibes, and have dinner! We started by the student housing at Isaac Newton House, walked over the hills and got confused (well, I kept saying follow me, but they wanted to look around) and finally got to the bus stop, and waited for like 25 minutes. Here's some photos :D
This was NOT our bus! :-(
So we waited (and this was where I played my "game") - hey, if you can't have fun, what can you have?
This was the bus ride -
Loving the Pink Andrew!
LET US OFF!!!
I loved this candid I snapped
Then we finally made it to the town, which was nice, and I didn't take near enough photos...
Apparently Malina can't count, she thought this was over 50Euro, and we just had to do a doubletake...
Here's what the street in the old-town we ended up going down looked like:
So, on a side alley we found a little cafe (and we didn't take a picture, oh no!) and we grabbed a little booth, very nice. So here's our dinner photos (with select captions from yours truly):
What to eat?
A special?
Well, we ordered, so let's munch on some bread...
Andrew wants to know how it tastes
Oh no, Cole's already making jokes ... this is terrible, stop Cole! sudo stop Cole!
Kristen was quite tired of the sudo jokes early on, so we gave her a hard time about not being part of the "in-crowd", which ensured more hilarity. Geeks, discuss in the comments, sudo jokes funny all the time, or do they eventually stop being funny?
Yay, the food arrives! (hey, we were there for the experience, who minds waiting for excellent food?) Somebody's gonna need a wheelbarrow after this!
Kristen got this awesome kebab plate:
Dinner was GREAT! Now for desert!
And to wrap up the evening, we went to the beach, and saw the old city walls. Unfortunately, our cameras weren't liking taking pictures at night, reference this photo of the fort for what we were doing wrong. We'll figure it out, we have two weeks to visit again! :D
Loving the Pink Andrew!
LET US OFF!!!
I loved this candid I snapped
Then we finally made it to the town, which was nice, and I didn't take near enough photos...
Apparently Malina can't count, she thought this was over 50Euro, and we just had to do a doubletake...
Here's what the street in the old-town we ended up going down looked like:
So, on a side alley we found a little cafe (and we didn't take a picture, oh no!) and we grabbed a little booth, very nice. So here's our dinner photos (with select captions from yours truly):
What to eat?
A special?
Well, we ordered, so let's munch on some bread...
Andrew wants to know how it tastes
Oh no, Cole's already making jokes ... this is terrible, stop Cole! sudo stop Cole!
Kristen was quite tired of the sudo jokes early on, so we gave her a hard time about not being part of the "in-crowd", which ensured more hilarity. Geeks, discuss in the comments, sudo jokes funny all the time, or do they eventually stop being funny?
Yay, the food arrives! (hey, we were there for the experience, who minds waiting for excellent food?) Somebody's gonna need a wheelbarrow after this!
Kristen got this awesome kebab plate:
Cole and Malina managed to order the same thing ;)
Yes, it was VERY good. Yum!
Andrew ordered French fries - doh!
Dinner was GREAT! Now for desert!
And to wrap up the evening, we went to the beach, and saw the old city walls. Unfortunately, our cameras weren't liking taking pictures at night, reference this photo of the fort for what we were doing wrong. We'll figure it out, we have two weeks to visit again! :D
Monday's Notes
Monday was the first day of lectures. For the most part, a general introduction to the summer school, the concepts behind grid computing, and the large project on the last day of the school. My favorite presenter was Miron Livny, though all were very interesting. I took notes during the presentations, and I'll try to summarize the main points below. As a disclaimer, just because I took notes does not mean that I fully understand what was discussed. If anyone sees any errors or would like to discuss something, please feel free in the comments or if you see me around the school.
Introduction - Per Oster:
This presentation was a brief overview about European grid efforts and their progress and general structure. He explained the grid "ecosystem" that exists in Europe, where the different pieces that make up the grid community is organized into pyramid-like layers organized by the amount of use (with personal and local grids on the lowest and largest layers, and national grids on the highest/smallest portion).
He also spoke about the European Grid Initiative (EGI) that is trying to unite Europe's many different grids (Omii and Egee are just two of the many large ones). They have decided to start their effort under Dutch law, and they're having a meeting sometime this week to discuss the plan.
Distributed System Principles - Miron Livny:
I said earlier that Dr. Livny was my favorite presenter, and I hope that I can make my reasons clear in this short description.
He started out with a smart remark about how the US had already figured out what Europe was hard at work on (regarding the EGI). Dr. Oster didn't look too happy after hearing the comment, but everyone realized that it was...mostly...a joke.
His main point was that although there have been great technological advancements over the years, the principles behind their development and the general problems that they are designed to solve have not really changed. He talked a bit about the cycle that computer science has followed since its development: single user terminals to batch jobs to PCs with a single processor...but now with grids and PCs with multiple cores, we are back to batch jobs.
There was an interesting sentence on one of his slides:
He also went through the "benefits for distributed systems" outlined by a paper written by Enslow in 1978 (and pointed out how we are still having trouble with fully realizing them). He also discussed a list of the requirements that a grid must meet to actually be a grid: multiplicity of resources, component interaction (loose coupling), unity of control (desire to achieve a common goal), system transparency (the appearance of a single virtual machine to the user), and component autonomy (ability to refuse jobs based on its own policies). To tie these two points together, he said to be wary of a grid system that claims to have all of the benefits that Enslow proposed, because it probably doesn't meet the requirements.
Then there was just some random stuff in his presentation that I thought was interesting. Dr. Livny said that he is more concerned with High Throughput Computing (HTC) than High Performance Computing (HPC) because researchers usually express their needs for the system by asking "How many simulations can I run in X amount of time", instead of "How fast can I run this particular piece of code". So, optimizing your grid to provide good throughput should not be done as an afterthought, and that FLOPY != (60*60*24*7*52) * FLOPS.
Anyway, very interesting talk.
Distributed System Architectures and Scenarios - Miron Livny:
This talk was more specifically about Condor. Mentioned that one researcher solved the NUG30 problem in 6 days, 22 hours using Condor...a task that amounted to 11 years of work.
Then he went through the general steps for a Condor submission:
t-Infrastructure and Security(Practical) - Emidio Giorgio:
This discussion was mostly getting us set up with our user names and explaining about how to request an X.509 certificate. To request one, first contact the Registration Authority for your country become authorized, and then contact the certificate authority to actually receive the certificate. He also discussed the main security issues involving grids: DDOS attacks, illegal data distribution, and viruses.
General Introduction to Technologies - Per Oster
One thing that had nothing to do with the presentation that I found interesting was a word on one of the slides. I don't remember if it was the town that Dr. Oster was from, or what. The word is Otaniemi. I had a professor at Clemson named Dr. Hedetniemi, and on the first day of class he said that Hedet meant...something in Finnish, and that Niemi meant...river, I think. So the name of the town must mean something-river. Not a very good story since I don't remember the details, but it was interesting to me and I thought I would share.
Anyway, this discussion was an outline of the different technologies that we are going to cover during the school. Here were the important connections that he made:
Introduction to Integrating Practicals - David Fergusson
The last event of the day was a short discussion of the practical, the final project at the end of the school. It was a summary, and doesn't make very much sense at the moment because it was just supposed to give us some idea so we can think about it over the next two weeks. Here is my understanding of the problem. we are going to be given a large coordinate plane split into sections (one for each technology that we discuss) with lots of background noise. Our job is to find pillars, rectangular objects hidden in the noise. On these pillars are blocks, and on these blocks are RDF triples. These triples can be used to search through an ontology to find text. The text is our answer. I don't know how many pillars there are, but our job is to find all of them. Reminds me of some sort of spy mission.
-----------------------
OKAY. That was a long recap. I hope it wasn't very boring...maybe these whole day summaries aren't such a good idea. We also went to Antibes, but Cole did a nice job of summarizing that, so I won't say any more about it. I took some pictures too, but mine were also dark. I didn't want to be the "stupid American" taking pictures of everything. No offense, Cole. :)
Introduction - Per Oster:
This presentation was a brief overview about European grid efforts and their progress and general structure. He explained the grid "ecosystem" that exists in Europe, where the different pieces that make up the grid community is organized into pyramid-like layers organized by the amount of use (with personal and local grids on the lowest and largest layers, and national grids on the highest/smallest portion).
He also spoke about the European Grid Initiative (EGI) that is trying to unite Europe's many different grids (Omii and Egee are just two of the many large ones). They have decided to start their effort under Dutch law, and they're having a meeting sometime this week to discuss the plan.
Distributed System Principles - Miron Livny:
I said earlier that Dr. Livny was my favorite presenter, and I hope that I can make my reasons clear in this short description.
He started out with a smart remark about how the US had already figured out what Europe was hard at work on (regarding the EGI). Dr. Oster didn't look too happy after hearing the comment, but everyone realized that it was...mostly...a joke.
His main point was that although there have been great technological advancements over the years, the principles behind their development and the general problems that they are designed to solve have not really changed. He talked a bit about the cycle that computer science has followed since its development: single user terminals to batch jobs to PCs with a single processor...but now with grids and PCs with multiple cores, we are back to batch jobs.
There was an interesting sentence on one of his slides:
Re-labeling, buzz words, or hype do not deliver as the problems we are facing are fundamental and silver bullet technologies are pipe dreams.He discussed some of the main problems with grid computing. Problems as in...things that should be considered when designing a grid, or things that having a grid is supposed to solve, or things that might be made more difficult because of the grid. These were: data movement, identity management, and debugging.
He also went through the "benefits for distributed systems" outlined by a paper written by Enslow in 1978 (and pointed out how we are still having trouble with fully realizing them). He also discussed a list of the requirements that a grid must meet to actually be a grid: multiplicity of resources, component interaction (loose coupling), unity of control (desire to achieve a common goal), system transparency (the appearance of a single virtual machine to the user), and component autonomy (ability to refuse jobs based on its own policies). To tie these two points together, he said to be wary of a grid system that claims to have all of the benefits that Enslow proposed, because it probably doesn't meet the requirements.
Then there was just some random stuff in his presentation that I thought was interesting. Dr. Livny said that he is more concerned with High Throughput Computing (HTC) than High Performance Computing (HPC) because researchers usually express their needs for the system by asking "How many simulations can I run in X amount of time", instead of "How fast can I run this particular piece of code". So, optimizing your grid to provide good throughput should not be done as an afterthought, and that FLOPY != (60*60*24*7*52) * FLOPS.
Anyway, very interesting talk.
Distributed System Architectures and Scenarios - Miron Livny:
This talk was more specifically about Condor. Mentioned that one researcher solved the NUG30 problem in 6 days, 22 hours using Condor...a task that amounted to 11 years of work.
Then he went through the general steps for a Condor submission:
- user submits a job to schedd
- startd assigns a resource to schedd
- shadow is delegated a job and resource
- shadow sends both to the starter
t-Infrastructure and Security(Practical) - Emidio Giorgio:
This discussion was mostly getting us set up with our user names and explaining about how to request an X.509 certificate. To request one, first contact the Registration Authority for your country become authorized, and then contact the certificate authority to actually receive the certificate. He also discussed the main security issues involving grids: DDOS attacks, illegal data distribution, and viruses.
General Introduction to Technologies - Per Oster
One thing that had nothing to do with the presentation that I found interesting was a word on one of the slides. I don't remember if it was the town that Dr. Oster was from, or what. The word is Otaniemi. I had a professor at Clemson named Dr. Hedetniemi, and on the first day of class he said that Hedet meant...something in Finnish, and that Niemi meant...river, I think. So the name of the town must mean something-river. Not a very good story since I don't remember the details, but it was interesting to me and I thought I would share.
Anyway, this discussion was an outline of the different technologies that we are going to cover during the school. Here were the important connections that he made:
- UNICORE - job submission and execution management
- Condor - HTC
- Globus - Service-oriented architecture
- gLite - Distributed Data Management
- Nordugrid/ARC - HPC
- OGSA-DAI/SAGA - Higher level APIs and metadata management
- Workflows - organizing work, P-GRADE portal
Introduction to Integrating Practicals - David Fergusson
The last event of the day was a short discussion of the practical, the final project at the end of the school. It was a summary, and doesn't make very much sense at the moment because it was just supposed to give us some idea so we can think about it over the next two weeks. Here is my understanding of the problem. we are going to be given a large coordinate plane split into sections (one for each technology that we discuss) with lots of background noise. Our job is to find pillars, rectangular objects hidden in the noise. On these pillars are blocks, and on these blocks are RDF triples. These triples can be used to search through an ontology to find text. The text is our answer. I don't know how many pillars there are, but our job is to find all of them. Reminds me of some sort of spy mission.
-----------------------
OKAY. That was a long recap. I hope it wasn't very boring...maybe these whole day summaries aren't such a good idea. We also went to Antibes, but Cole did a nice job of summarizing that, so I won't say any more about it. I took some pictures too, but mine were also dark. I didn't want to be the "stupid American" taking pictures of everything. No offense, Cole. :)
Monday, July 6, 2009
Afternoon Practicals - Monday's session...
Today's practicals were very nice introductory material, designed to bring everyone up on speed on login, credentials, and what are practical facilities are.
The only suggestion I would make to the staff for the ISSGC is to permit students to remote VNC in while on campus, which would be nice. (I actually did broach this with one staffer, and I have some other names to discuss it with) - the caveat is that not everyone groks VNC, so it can be a loaded proposition. Perhaps we can set it up on a per-user request. We shall see.
For the non-technical folks, what this means is this. Normally to use a "linux" or "unix" machine, system administrators use the command line (that black screen with the white text, yay fun!). However, sometimes we need to use a mouse and windows (think "desktop") so there is a way that we can use the desktop over the network. But it's not normally turned on by default, so we have to ask permission. That's the nutshell version, and I hope it clarifies why I would suggest that they do something they didn't already do. The physicists and bio-med students wouldn't know VNC any more than most, so it's not something that's inherently beneficial to everyone. But a lot of the students here use VNC back home, so it's an option that's reasonable. Just not standard.
Otherwise, I thought that the practicals were nice from the standpoint of bringing people online and getting them up to speed.
The only suggestion I would make to the staff for the ISSGC is to permit students to remote VNC in while on campus, which would be nice. (I actually did broach this with one staffer, and I have some other names to discuss it with) - the caveat is that not everyone groks VNC, so it can be a loaded proposition. Perhaps we can set it up on a per-user request. We shall see.
For the non-technical folks, what this means is this. Normally to use a "linux" or "unix" machine, system administrators use the command line (that black screen with the white text, yay fun!). However, sometimes we need to use a mouse and windows (think "desktop") so there is a way that we can use the desktop over the network. But it's not normally turned on by default, so we have to ask permission. That's the nutshell version, and I hope it clarifies why I would suggest that they do something they didn't already do. The physicists and bio-med students wouldn't know VNC any more than most, so it's not something that's inherently beneficial to everyone. But a lot of the students here use VNC back home, so it's an option that's reasonable. Just not standard.
Otherwise, I thought that the practicals were nice from the standpoint of bringing people online and getting them up to speed.
Monday, July 6, 2007 Sessions Notes
So, here's some pics of some of the folks who have been presenting today:
Local school guy, Dean of the Engineering school, no clue on the name
Per Öster
Miron Livny from the Condor group
Local school guy, Dean of the Engineering school, no clue on the name
Per Öster
Miron Livny from the Condor group
For students at the 09 conference...
To visit the Casino Supermarket (like a Kroger's but smaller) - Take a left at the foot of the drive for the Technical Building, go "straight" past the unnamed carrefour, go "straight" at carrefour Golf and turn 90 degrees left on carrefour St. Phillipe
Once you see the building, go straight on the street past the two cafe's and climb the steps.
BEWARE: It smells like FISH
Once you see the building, go straight on the street past the two cafe's and climb the steps.
BEWARE: It smells like FISH
Subscribe to:
Posts (Atom)