OSG EOT

Friday's session, a finale -- "Clouds and Grids"

2009-07-17T06:59:00.003-05:00

So the Friday session was an interesting one, to say the least. The topic varied sharply from the previous set of topics to a future-looking commercialization topic, that of virtualization of hardware and commoditization of the grid. Now, if you participated and thought I brought away the wrong concept from this, feel free to let me know.

Here's what I boiled down the gist of Friday's single session to: virtualization is a key component in rapid deployment of technologies on the grid, and the use of virtualization technologies can allow cluster administrators and grid manipulators to move resources to where they can be most effective in a logical fashion, all while avoiding the underlying failure of physical machines as much as possible. The focus was on technologies like KVM, Xen, and VMWare and organizations like Amazon (EC2) and Google (AppEngine).

It is not fair of me to say that these two types of cloud or grid computing are disparate entities, because they're not. They're closely interwoven. However, it's also not fair of anyone else to say that they are directly tied together, except as all things seem to be directly tied together. Grid as it exists now is a means to solve large problems, and Cloud as it exists is a means to provide the solvers for those large problems. They're complimentary. I just don't personally see where it had anything to do with the rest of the school, except both are about huge numbers of computers under the control of a few operators. This is a subjective opinion, and if you have your own opinion, feel free to voice it below, or write yer own post ;)

At the moment I'm posting this as complete, but expect a followup post before too long for me to redress part of this. The total composition of Friday's presentation obviously cannot be compiled into three paragraphs, but neither can any other day's session be compiled as short as I've made those. It's just that Friday's session was the most pertinent to a "moving forward" group of students.

Thursday - Group Collaboration

2009-07-16T02:56:00.003-05:00

Thursday's events were primarily focused on a team-exercise that was intended to showcase our ability to use the technologies we had learned about during the previous two weeks. This opportunity was not so much given for ourselves as for the staff to see where the individuals had learned from the presentations, and perhaps to offer feedback for the next cycle of the ISSGC.

Having said that, I'll mention briefly that the students had been organized as one team for the scavenger hunt, but that due to some unintentional "weighting" of my original team, the teams were restructured. Our original team was solely OSG participants, and as a team we thought this was unfair to the spirit of the competition, so we made sure to mention it to the staff. Therefore, the new teams were given to us almost just-in-time for this competition. Nevertheless, we each as appropriate bandied together as a team and set off to work when the assignment was given to us.

So let me set up the "what happened here" so everyone can get a glimpse. The morning of the competition we gathered together in the main auditorium and David set the stage for what we were about to be given. We had approximately 24 hours to complete the task, which was to find each of six "pillars" focused around a given technology. Each pillar could only be found with the technology in question, so for instance, the gLite pillar could not be found with the Condor toolkit.

Each pillar in question was arranged in a 2D space on a Cartesian grid of -10,000 to 10,000 on each axis. Each pillar then contained a plaque that would contain a word, and once the word was found, it was to be keyed into the scoring system along with the coordinates where it was found. The executable to be used to find each word was a pre-compiled jar given to us by the project leads and listed on the appropriate technologies page. Some pretense was made that the experiment was in a 3D space, but I've done the exercise, it was 2D only.

Initially the students were given a database to write an app to retrieve values from, and those values were to be clues to help students find some initial pillars. Most of the teams got those clues pretty quickly, and were off to the next part; find the pillars themselves.

Ok, so now I'm going to drop out of narrative and I'm going to get into "Cole's point of view" so this will be totally subjective and might even upset people. Guess what, it's a blog. If you don't like it, I don't know what to tell you. But consider yourself warned. I was not impressed, so this is likely to be somewhat inflammatory.

The point of the project is to find the pillars by searching a region of "space", and once the pillar is found, dig down till you find the plaque. The way it should work was like this:

java jar x1 y1 x2 y2 scale

This generates a region (think piece of graph paper) with the number of cells equal to (x2-x1)*(y2-y1)*(1/scale) so if you chose

-10 -10 10 10 1 you would get a region with 400 cells

and then you get back one of the three results:
These are not the droids you are looking for, move along (ok, humor aside: no pillar here)
Found something interesting, can't see it
Hey, writing! (and then some one or more letters)

However, the results were stored in one of 5 files (for each pillar) with the location of each pillar (top left) and the height and width of each pillar. Lastly was the text.

To examine the pillar, the jar would read that data (every time, and as I'll get to in a moment, that's a lot of reads, so a lot of disc-IO) and then compute some random noise to fill a grid the size that was specified in the inputs. The problem is that it would also sometimes fill the cell in question with noise, so given the same inputs to the same technology, it was possible to get different results (trust me, I repeated it, I got different results for the same inputs). Now, do I expect every program to always give me perfect results? No. But if I'm doing statistical analysis against a rowset returned from a database or if I'm running a grep for a string, I don't expect to get different results everytime, provided the inputs are consistent and provided there's no room for network errors. I'll accept that those may be flawed, and that that could be the problem. But for something this simple (I decompiled the JAR, I know what the code did), I would've expected consistent results, which I didn't get.

Given a perfect world, a person could take an example like the above, make the sized chunks reasonably sized, and then look to see if a pillar was found in that region. If not, expand the region slightly, and search again. If so, then the difference region was where a pillar could be found. If not, repeat and test again This is basic geometry.

However, and this is where I got disillusioned, the students would form one large region with one really small scale and they would then send a batch job with those inputs off to be farmed against the data. And they would do that with 800 batch jobs or more at one time. For each team on each technology.

Let me repeat that if I may. 5 teams submitted 5 technologies worth of jobs at 800+ runs per batch against a server, creating 100's of thousands of jobs, and each job had one output file for stdin, stdout and stderr. Therefore, they were creating 300's of thousands of output files, plus the inputs to run them in the first place, so 400's of thousands of very small files were created on systems that don't do well with lots of small files in one directory. So many jobs were submitted individually that the submission system effectively underwent a thread-fork, and the system also opened so many connections to the test server that the system had to be rebooted.

This was not what we were taught to do at the school, and this was not a reasonable way to use the system. No-one in the real world would submit 30,000 live job submissions into a queue creating over 100,000 files in a single directory, just to look for a bit of info in a database. Of course, the database's don't normally return garbage data on multiple runs of the same inputs. They may occasionally, and that's fine, but that's where you test three times, look for two consistent answers, and then accept that answer.

So, let's say that I'm the one out of line. Let's consider: The test systems were setup with no limitations (30k+ threads active at one time was not considered) even tho live systems would have those same limitations, the student's didn't consider what the problem was, they went for brute force, and then there's the matter of the queues.

The system was setup so that gLite, Globus and Unicore (IIRC) were all setup to use a single PBS scheduler. Three toolkits were competing for one hardware resource, with thousands of submissions against each queue. In a real-world environment, those queues wouldn't have been squashed with those numbers of requests, as gatekeeper software would've prevented those types of rapid fire submissions, and the meta-schedulers would've sent different requests off to different PBS queues in the first place.

So even if it were a valid real-world experience, nobody would submit a few hundred thousand jobs against two simple queues on two small clusters, that would've been submitted to the grid.

But also remember that each job had to read from the same set of data files, but there were five of them, so I'm sure either the file got cached into memory by the filesystem IO controller (plausible but unlikely) or the disks got thrashed. I'll never know for sure. But triggering a few hundred thousand reads against five files is not a sane activity. Triggering a few thousand reads against a few hundred files is sane. Does anyone see what I mean?

So, while I was quite pleased with the assignment, and while it would've been fun for group collaboration, most of the participants did not seem to understand what the goal was, nor how to attack it.

I personally did it the way it should've been done, and saw that it took just a few submissions to get results, and I watched Ben Clifford (one of the moderators) find his first pillar in about three minutes without brute force approaches, but when I spoke with many of the students, they didn't see where an elegant bounding approach was the best way to do it. They felt that submitting 100's of thousands of brute force attempts was sufficient.

Because of this, and because there weren't sufficient technological safeguards put into place, and because the code returned inconsistent results on the same inputs (and no, don't ask me now, it's been too many days since I did it, but I did bitch then and nobody asked me for those verification inputs) - because of those reasons, I was disillusioned with the experiment.

Now, having said ALL of that, let me finish my post with this. I learned a lot more that day than you might think. I am very very grateful to the organizers for setting that up. I had a lot of fun working with the advisers to solve the problems. And I'm glad I got the chance to play with real tech on a reasonable problem.

Ok, where do I need to clarify my points? Feedback people, feedback!

Wednesday's session, a placeholder

2009-07-15T07:00:00.004-05:00

Wednesday's AM sessions were on the PGRADE system, and then we talked about the semantic web.

PGRADE is a Hungarian based project that is similar to UNICORE. What PGRADE does is it helps to provide a more graphically oriented approach to monitoring an application on the grid. It also collects trace information as the programs run to help tweak future runs, and to understand what the project is doing on the grid at the time. The system also allows the user to access some pre-formed templates to assist in writing more coherent code, and code that should perform better in a parallel environment.

PGRADE is not a meta-scheduler or a resource handler, instead, it works "on top of" Globus or Condor, for example. It uses the same mechanisms for job submission and reconciliation as any of the command-line-only apps would use; PGRADE helps the user to use those utilities. In an environment where grid specialists can't help develop every application, this is a really handy utility, and the only room I can suggest for improvement is a standard reporting mechanism built into the Globus or Condor models (again, for instance) that would allow program tracing to be more effective. This is not a light endeavour - especially given as how neither the standard Linux nor Windows kernels offer such tracing by default - so I would have to say it appears they are doing the best they can to offer tracing.

As for the semantic web and it's relation to Grid. As a computer professional, I tend to get all choked up when people talk about "the semantic web" because it seems so self-pretentious. I just thought I might toss that out there, to kind of catch the attention of my readers. (And since some of the presenters and attendees felt the same way)

However, in this regard, I would have to say that I firmly believe that the OGSA ontological semantic model is doing exactly what semantic web was designed/intended to do. I'm not going to go into a huge number of specifics on what OGSA nor the semantic web are, because both are easily googlable. Instead I just wanted to mention that they were the topics of discussion.

I will say now that the concept of semantic data is best summed up as this: All data in modern systems is inherently binary, and as such, a binary blob has no distinct characteristics. Sure, a certain binary blob may correspond to ASCII text, and we may be able to read it, but the fact that it's ASCII does us no good, until we can associate, for instance, an author, or even a title for the file. That may sound too simple to the casual reader, but that's exactly what we're talking about. The trick is how does one go about attaching the metadata to the file? That's what the OGSA semantic ontological model is all about, giving us a defined format and model.

Further details are best left up to specific questions, in my mind, but perhaps that's because I also feel comfortable answering the question "what's the difference between a database and a filesystem". - Cheers!

Teusday - Were YOU paying attention?

2009-07-15T06:59:00.023-05:00

So Tuesday was the day we went to the beach for fireworks, but before that, we had lectures and lessons. We learned about GridSAM, and many of the other technologies coming from OMII, in the UK. I really like this tech, and am interested to deploy it on a test grid, so I can learn for myself.

Some of the highlights I took away for myself include (as I understand them):
Common-VFS hooks, so for the user it's seamless, and
Workflow support, and
Common support for all the major middlewares and submission engines.

I don't want to detract from GridSAM, it's just that by this point in the school, we were all a little overwhelmed in what we were learning.

So, look, everyone just wants to see pictures, these are the pictures! "The Bastille Day fireworks show", as I understand it, although I'm sure someone will correct me. These are what I think were fairly representative of the ones I took, but others took some much better shots, so you'll just have to check with them and around to see what there was. I'll also mention that this show was one of the shorter I've seen in a while.

Monday, July 13, 2009 Session Notes

2009-07-13T02:12:00.005-05:00

So today is a bit of a odd-schedule. First we started with a clip of the daily show from http://www.thedailyshow.com/watch/thu-april-30-2009/large-hadron-collider and let's see if I can embed it...

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Large Hadron Collider
www.thedailyshow.com

Yay, now you can see how we started our day. So otherwise, I would have to say, today is going pretty well. We've started with a discussion of grids and their common characteristics, talking about the infrastructure that's necessary to have an effective and efficient system. I firmly believe that we're seeing and discussing this today because we're the ones who are going to maintain and build this system; to improve it and to make it better. However, I'm not seeing any sort of non-political discussion, so once again I come back to something I've tried to keep off the blog, which is that the grid seems to have a nearly felonious association of productivity and politics. For now, I'll leave out this sort of discussion, as it can be very subjective.

So the afternoon was a set of Q&A on topics regarding the grid, and some of the topics were:
Filesystems on the grid, The Cloud vs the Grid, How do we do performance metrics on the Grid?, What are the key differences between HTC and HPC. The answers wavered between the technical and the requirements, but without the context of the school it's not really fair to examine the aspects of the Q&A. I'm just blogging this part in context of "this was part of the school".

Cheers, C

Sunday outing - Cole's perspective

2009-07-12T17:36:00.001-05:00

Well, today I finally passed 3GB of data collected with my camera. Yeah, at this point it's going to be difficult shy of hosting it on my own to get everyone a copy. Stay tuned, if I upload everything, and for those who want their own snapshot, bring me a laptop or a USB Drive of over 8 GB by the end of the show. Also, to all other participants, I'm collecting photos!

Saturday at the base camp

2009-07-11T16:44:00.008-05:00

Ok, so Saturday was a split day, where we went to lecture/practical in the AM, and we went to Nice in the evening. Let's start with the lecture/practical now, and I'll do the trip to Nice after a break. And yeah, I started with a pic of me, it's my blog post ;)

From my point of view, as a grid virgin, I thought that the presentation that Michael gave really coalesced everything into understanding, so I give kudos to him. Though it could be that I've just been listening to it for a week so it finally kicked in...

So anyways, Saturday was about ARC, which was previously NordicGrid, IIRC, and this was mostly a catch-up to the ones who are familiar with what has been happening with ARC, and how they are trying to get to a 1.0 release. This is an incomplete toolkit, but from what I've seen, of the middlewares, it seems to be the closest to my way of thinking of the EC toolkits. (I'm still partial to Condor for some reason, but I digress by mentioning Condor here)

And of course, after the lecture, nearly everyone was ready to goto Nice. - So let's talk about Nice. What's to say? We all went in different directions. The guys in my group went to the beach, the girls that I talked to went shopping. Later we all met at the restaurant for dinner. Now we have pics.

Haha, now you've got three men at the beach, so I'm not doing anymore beach pics here. C'mon, this is a family affair, no? Anyways, the beach was fun, dinner was better, and I'll be uploading those with the actual picture archives we're assembling later. Cheers.

Elmer

2009-07-11T05:11:00.002-05:00

http://www.csc.fi/english/pages/elmer

An open source finite element solver for multiphysical problems. Sounds groovy, downloadable GUI. Cheers

More pictures on Photobucket

2009-07-10T02:35:00.003-05:00

My pictures are too big, and I can't upload all of them on Flickr right now. So, until I get a chance to redo the Flickr ones I will put them here.

By the way, if you are in one of these pictures and you would like it taken down for whatever reason, just let me know. :)

So...I have pictures too

2009-07-09T09:50:00.005-05:00

I just hate the limits that blogger imposes. I'm posting mine here:

PICTURES!

I'm new to flickr, so it might take a while before the pictures show up in the search results, but they are all tagged with issgc09.

Globus day - Day four!

2009-07-09T06:36:00.003-05:00

So Thursday was our introduction to Globus. I'm going to admit fudging the numbers and say that I did insert some notes into this post after it was created. Shh, don't tell anyone.

So here's our morning's speaker. However, this was not as simple a show as one might imagine, there were several interactions between the presenters with plenty of discussion as to what is the correct way to do some thing, and what is not the correct way to do said thing. But the discussion seemed to me (the new kid) to be political, so I'm not going into it here.

Suffice to say, that I thought the info presented on Globus was pretty decent, even if I did leave scratching my head as to the purpose of the software stack. I think at this point most of the students are in information overload, as 12 hours of focus a day is a bit much. To those who would argue that we're not really doing 12 hours a day since we're not in lecture 12 hours a day, I say that sure we are, because we still discuss it amongst ourselves after the talks, and we talk to the faculty all throughout the day. The other primary presenter here was Ben Clifford.

I hear talks that Ben is someone of note in the Grid community, but he's been a little closed off as to his professional/academic career, so I'll go with he seems like a bright chap with a lot of drive, even if he comes across as an arse to some (and if you've spent time with him, you might think that). Life of the party is a phrase that comes to mind.

For the first time I am absolutely sure that this school needs to show students the same portfolio of tasks on each system, so one to setup proxies, one to submit jobs, etc, and to show how each system does these things, and what the output looks like, and to make note of special procedures or special features.

Ah well, what can I do, I'm just one kid. Lemme record it, and maybe somebody will do something about it, eh?

Photos from Lunch

2009-07-08T06:37:00.004-05:00

Just some pics from lunch, a lot of small thumbnails. Wanted to share our afternoon.

Day three from Cole

2009-07-08T03:31:00.008-05:00

So today is Condor day, and apparently it's going to be an auspicious day. I know this because I stayed up late talking with what started as six other students, and what dwindled eventually down to just two of us. We talked about host countries, and we discussed politics. Commerce, culture, and computers were all discussed. I thoroughly enjoyed the talks, and hoped that I wasn't interrupting others too often. I asked lots of questions, and definitely learned many new things. So "why", you ask, "do I care about this Cole?" And the answer to that is I overslept. See, totally makes sense now, eh? However, Andrew came and checked, and told me to get out of bed, "hurry so you don't miss the bus" - my thoughtful reply was "No, need a shower first". And yet, I still managed to be on time to the bus, with a shower, so I say the day has begun auspiciously.

Now, onto the lectures. David started the morning and introduced us back to Miron Livny again. Miron started with a background on Condor and its relation to HTC and HPC. There was some interesting discussion to questions asked by students, and because I feel the discussion became a little charged, I won't repeat it here. I appreciated both sides of the situation, and think that at the moment it is best for me to learn more about both sides before I can say anything else about it. Suffice to say I believe the act of a programmer looking for premature optimization is a "Bad Thing". No more on this.

So, Alain started a talk on Condor, and we keep getting interrupted. He got through about five slides, then it was time for break. Upon coming back, he asked someone to take a picture of him to prove to his wife that he actually does lecture*, so natch I volunteered. ;-) As a matter of fact, I believe I'll toss one of him on up in addition to the regular style picture I do for each lecturer. I took these pictures right after I wrote that, and as you can see, he's still getting interrupted. Poor Alain ;-(.

Well, on with the lectures...

I have to admit, today's questions are very direct, I think people are coming out of their shells, or, barring that, they're just now getting used to asking questions.

* he said something about being madly in love, etc etc ;-)

Scavenger Hunt

2009-07-08T03:07:00.009-05:00

Well, I think ya'll know I won't be outdone ;) - remarkably enough, I took like ZERO photos on the scavenger hunt yesterday. Perhaps I'll go back later and take pictures. What I do have pictures of are the group standing around after the hunt and having a good time, so several of those are grouped at the bottom in no particular order. HINT: These would make good facebook pictures for some of ya'll, very candid!

So I don't recall if the others said what the point of the scavenger hunt was, so I'll enumerate briefly. The School is about international collaboration and learning about cultures, so the people setting up the conference are trying to give us every opportunity to work and talk together, to build a team. At the end of the conference we will have one group project that will require us to work together, in less than 24 hours to solve a complex problem.

The scavenger hunt was fun, we worked together as a team, but we felt it was unfair, because they mistakenly put five Americans on a six person team at an international school. After we revealed this at the end, they agreed that it wasn't very fair, so we'll probably re-do groups before the weekend. Three of us had already walked the trail, and we all had a really good idea what to do to finish quickly.

Starting off though, we figured that we were beat already, that the other teams had had someone skip dinner, that others surely were ahead of us, etc. And then this team finished first. That was nice, but because of the way the other teams had done their planning, it was late before some of them returned, so we were very worried about them being out. Luckily there was a full-moon, and by 11ish they had returned, so all was well.

Scavenger Hunt Adventure

2009-07-08T00:53:00.008-05:00

Here are some pictures of Group 1 'hunting' for pictures of birds in the wild near the summer school. This proved to be a most enjoyable past time for at least our group. We got to know each other better and had some good laughs. The pictures were pretty hard to find after the first two. They were hanging under bridges, hidden in bushes and even placed in a garbage can. Our main 'hunter' in our group was amazing and could often find the picture within 5 minutes of starting. I was impressed.

It was fun seeing others from other groups scavenging around too.

We started walking from the summer school, so in the end, I think we might have walked a good 4-5 miles yesterday. We were all incredibly exhausted by the time we arrived at the houses.

Our group was only able to find 11 of the 12 pictures of birds (a miscommunication in our group interpreted one clue as 'near the basketball courts' not 'under the bridge near the basketball courts'. This mistake cost us about 20 minutes at the basketball courts with no bird. Also, we managed to mix up the order of 4 and 5. The answer was 'FAILTE ISSGC'.. which means 'a hundred thousand welcomes' ISSGC in Scottish.

UNICORE Notes and an Adventure!

2009-07-07T16:54:00.004-05:00

Today's sessions were all about UNICORE. The first part of the day was an introduction to UNICORE and then there was a more in-depth and technical discussion of how everything works. In the afternoon, we were able to have a hands-on experience with the technology in the lab.

UNICORE Part 1 - Mathilde Romberg

The lectures were led by people from Jülich Forschungszentrum. One presenter said that UNICORE got started in about '97, and that she was around for pretty much all of its development. As far as the material goes...I took some notes that I would be happy to share with anyone that is interested, but I'm not sure how much of it I should put in the blog. Basically, the first presenter went through the concept of job submission and job scheduling. She said that there were two main approaches to resource management: time sharing (as in round robin) and space sharing (exclusively assigned resources). UNICORE uses space sharing, so it requires a separate batch scheduler (like Torque or PBS). Then she went through the format for submitting a job using LoadLeveler and then for Maui. Next she showed us an example XML file example of Job Submission Description Language (JSDL), which is used to provide interoperability between middleware. It only supports "simple" descriptions of resource requirements. Reflecting on it now, I'm not exactly sure what makes a description simple or complex. Lastly, she said that the job of the middleware is to handle execution and job management: initiating, monitoring, managing, coordinating/scheduling jobs and workflows.

UNICORE Part Two - Rebecca Breu

The next presenter focused on security, similar to Emidio's presentation yesterday, but more specific to UNICORE. She went through the different components that make the software work. My understanding is this: There is a gateway, and all user to server to server-component communication goes through it. There is a registry, the services register with it as they become available, and then the client contacts it (through the gateway) to find out what they can use. There is also the UNICOREx which is responsible for translating JSDL jobs to concrete jobs that can be understood by specific machines, and for authoring requests using XUUDB, which is a database that provides a mapping from X.509 certificates to logins. There is also the Target System Interface (TSI) which communicates with the batch system so that it can manage the working directory and actually start the process for the job. There was also something called the Incarnation Database (IDB), which translates abstract jobs into executable scripts. But, I'm pretty sure that the job descriptions written in JSDL are abstract jobs, and I thought that UNICOREx was responsible for the translation, so I'm not sure which part of my interpretation of the presentation is wrong.

UNICORE Part 3 - Bastian Demuth

The last presentation of the morning was a very specific description of how the system works, and for that I would suggest that you take a look at the slides. There were very good diagrams that clearly showed how everything worked together that explain it better than I can here. In short, there are a variety of web services (using the gateway, registry, and other terms in the previous paragraph) that provide the functionality that is necessary to run jobs.

UNICORE Practical

In the afternoon sessions, we did a hands-on exercise involving submitting real jobs to UNICORE. There were a few problems with the configuration at first, it would work for some people...but not for others, and everyone wasn't getting the same errors...but eventually it all worked out. We switched to the graphical UNICORE Rich client while the problems with the configuration for the command line version were being worked out. I was very impressed with the software, because it was very easy to understand and had a lot of extra features that I didn't think were available. For example, I'm moderately familiar with the idea of using a DAG with Condor to run jobs. However, with Rich, it's very clear and you can construct a diagram representing the workflow using pieces that represent the scripts. There are also ways to include while loops and if statements within the workflow diagram, and it's extremely easy to customize it to suit your needs. Another thing I was impressed with was the ability to trace through as it executes, literally watching it progress through the DAG. There was even the ability to see which path of the diagram was followed (based on an if statement, for example). From my understanding though, UNICORE is a European technology...so hopefully I will be as impressed by Globus.

------------------------------------------

After the talks, we were given an assignment. A scavenger hunt! With our newly assigned team-mates. They hid clues all around the campus, and our job was to find them using the hints that we were given, do some searching on the Internet, and eventually come up with a Gaelic phrase that we were supposed to report as our answer as soon as possible. It was a lot of fun, and did serve as a very good bonding exercise for our team...but the clues were so well hidden! There was one inside a trashcan...one WAY in the bushes...one under a bridge that everyone had trouble finding. It was a good time though. I am very glad that I got to explore a bit yesterday, because I already had a vague idea of where some of the places were. Next time, I will skip dinner before starting so that my team can have a headstart. :)

Day two from Cole

2009-07-07T08:57:00.006-05:00

This morning's lectures were nice, fairly informative, and had a bit of background on both schedulers and Unicore, with the emphasis on Unicore. The presentations were given by a group from Jülich-FZ. They shared the beginnings of Unicore and the present work being done in their group, and in several derivative groups. The ISSGC09 Unicore info was given to us for the practicals in the afternoon. After a few difficulties, we finally got up and running, and I am mightily impressed by the Client UI features.

This is definitely a tech to watch, and I am curious to play with it myself and run some tests.

As I'm writing this, we've only really created a few workflows yet, but so far, I'm liking it!

So, onto what is becoming my "SOP", how about a few pictures?

From the lectures in the AM, in order, we have Mathilde Rothemburg, Rebecca Breu, Bastian Demuth all from the same group.

For the practicals, it was J. M. Daivandy with Rebecca Breu, but for all my tries, I could not get him when he was facing me and nobody was in my way with a camera, so candid-camera-activate FAIL.

An improptu trip to Antibes

2009-07-07T02:25:00.035-05:00

Ok, so first I think I am obligated to say that Antibes has a silent "s", since that was said to me repeatedly ;) - on top of that, the gang doesn't like my game of "loud obnoxious american who speaks Spanish poorly while in France" but we had fun with it, and the locals laughed, so that was fun. And yes, I did it, but I made it obvious I was doing it (this is for the ones who know me stateside and are holding their heads saying ... "Cole, what were you thinking, no you didn't"). Seriously, I barely speak English well, so ... do what you do, I'll just be me. :-D

Now, on with the trip. It started as we had an early release for the day. We decided to goto Antibes, and have dinner! We started by the student housing at Isaac Newton House, walked over the hills and got confused (well, I kept saying follow me, but they wanted to look around) and finally got to the bus stop, and waited for like 25 minutes. Here's some photos :D

This was NOT our bus! :-(

So we waited (and this was where I played my "game") - hey, if you can't have fun, what can you have?

This was the bus ride -
Loving the Pink Andrew!

LET US OFF!!!

I loved this candid I snapped

Then we finally made it to the town, which was nice, and I didn't take near enough photos...
Apparently Malina can't count, she thought this was over 50Euro, and we just had to do a doubletake...

Here's what the street in the old-town we ended up going down looked like:

So, on a side alley we found a little cafe (and we didn't take a picture, oh no!) and we grabbed a little booth, very nice. So here's our dinner photos (with select captions from yours truly):
What to eat?

A special?

Well, we ordered, so let's munch on some bread...

Andrew wants to know how it tastes

Oh no, Cole's already making jokes ... this is terrible, stop Cole! sudo stop Cole!

Kristen was quite tired of the sudo jokes early on, so we gave her a hard time about not being part of the "in-crowd", which ensured more hilarity. Geeks, discuss in the comments, sudo jokes funny all the time, or do they eventually stop being funny?

Yay, the food arrives! (hey, we were there for the experience, who minds waiting for excellent food?) Somebody's gonna need a wheelbarrow after this!

Kristen got this awesome kebab plate:

Cole and Malina managed to order the same thing ;)

Yes, it was VERY good. Yum!

Andrew ordered French fries - doh!

Dinner was GREAT! Now for desert!

And to wrap up the evening, we went to the beach, and saw the old city walls. Unfortunately, our cameras weren't liking taking pictures at night, reference this photo of the fort for what we were doing wrong. We'll figure it out, we have two weeks to visit again! :D

Pictures from Antibes

2009-07-07T02:24:00.000-05:00

Another group of 7 students from the summer school also made their way to Antibes on the second day. Part of the group went to a white sandy beach for a swim in the Mediterranean and the other part walked the streets of the old city of Antibes.

Monday's Notes

2009-07-07T01:52:00.016-05:00

Monday was the first day of lectures. For the most part, a general introduction to the summer school, the concepts behind grid computing, and the large project on the last day of the school. My favorite presenter was Miron Livny, though all were very interesting. I took notes during the presentations, and I'll try to summarize the main points below. As a disclaimer, just because I took notes does not mean that I fully understand what was discussed. If anyone sees any errors or would like to discuss something, please feel free in the comments or if you see me around the school.

Introduction - Per Oster:

This presentation was a brief overview about European grid efforts and their progress and general structure. He explained the grid "ecosystem" that exists in Europe, where the different pieces that make up the grid community is organized into pyramid-like layers organized by the amount of use (with personal and local grids on the lowest and largest layers, and national grids on the highest/smallest portion).

He also spoke about the European Grid Initiative (EGI) that is trying to unite Europe's many different grids (Omii and Egee are just two of the many large ones). They have decided to start their effort under Dutch law, and they're having a meeting sometime this week to discuss the plan.

Distributed System Principles - Miron Livny:

I said earlier that Dr. Livny was my favorite presenter, and I hope that I can make my reasons clear in this short description.

He started out with a smart remark about how the US had already figured out what Europe was hard at work on (regarding the EGI). Dr. Oster didn't look too happy after hearing the comment, but everyone realized that it was...mostly...a joke.

His main point was that although there have been great technological advancements over the years, the principles behind their development and the general problems that they are designed to solve have not really changed. He talked a bit about the cycle that computer science has followed since its development: single user terminals to batch jobs to PCs with a single processor...but now with grids and PCs with multiple cores, we are back to batch jobs.

There was an interesting sentence on one of his slides:

Re-labeling, buzz words, or hype do not deliver as the problems we are facing are fundamental and silver bullet technologies are pipe dreams.

He discussed some of the main problems with grid computing. Problems as in...things that should be considered when designing a grid, or things that having a grid is supposed to solve, or things that might be made more difficult because of the grid. These were: data movement, identity management, and debugging.

He also went through the "benefits for distributed systems" outlined by a paper written by Enslow in 1978 (and pointed out how we are still having trouble with fully realizing them). He also discussed a list of the requirements that a grid must meet to actually be a grid: multiplicity of resources, component interaction (loose coupling), unity of control (desire to achieve a common goal), system transparency (the appearance of a single virtual machine to the user), and component autonomy (ability to refuse jobs based on its own policies). To tie these two points together, he said to be wary of a grid system that claims to have all of the benefits that Enslow proposed, because it probably doesn't meet the requirements.

Then there was just some random stuff in his presentation that I thought was interesting. Dr. Livny said that he is more concerned with High Throughput Computing (HTC) than High Performance Computing (HPC) because researchers usually express their needs for the system by asking "How many simulations can I run in X amount of time", instead of "How fast can I run this particular piece of code". So, optimizing your grid to provide good throughput should not be done as an afterthought, and that FLOPY != (60*60*24*7*52) * FLOPS.

Anyway, very interesting talk.

Distributed System Architectures and Scenarios - Miron Livny:

This talk was more specifically about Condor. Mentioned that one researcher solved the NUG30 problem in 6 days, 22 hours using Condor...a task that amounted to 11 years of work.

Then he went through the general steps for a Condor submission:

user submits a job to schedd

startd assigns a resource to schedd

shadow is delegated a job and resource

shadow sends both to the starter

Some new information...new to me at least...was that if a workflow engine is submitted, it can submit jobs. Also, that the grid manager can submit jobs to Globus or another Condor pool.

t-Infrastructure and Security(Practical) - Emidio Giorgio:

This discussion was mostly getting us set up with our user names and explaining about how to request an X.509 certificate. To request one, first contact the Registration Authority for your country become authorized, and then contact the certificate authority to actually receive the certificate. He also discussed the main security issues involving grids: DDOS attacks, illegal data distribution, and viruses.

General Introduction to Technologies - Per Oster

One thing that had nothing to do with the presentation that I found interesting was a word on one of the slides. I don't remember if it was the town that Dr. Oster was from, or what. The word is Otaniemi. I had a professor at Clemson named Dr. Hedetniemi, and on the first day of class he said that Hedet meant...something in Finnish, and that Niemi meant...river, I think. So the name of the town must mean something-river. Not a very good story since I don't remember the details, but it was interesting to me and I thought I would share.

Anyway, this discussion was an outline of the different technologies that we are going to cover during the school. Here were the important connections that he made:

UNICORE - job submission and execution management

Condor - HTC

Globus - Service-oriented architecture

gLite - Distributed Data Management

Nordugrid/ARC - HPC

OGSA-DAI/SAGA - Higher level APIs and metadata management

Workflows - organizing work, P-GRADE portal

I'm not entirely sure of how much sense that makes to outside readers, but it's exactly what I have written in my notes. I'll be able to explain more in the coming days.

Introduction to Integrating Practicals - David Fergusson

The last event of the day was a short discussion of the practical, the final project at the end of the school. It was a summary, and doesn't make very much sense at the moment because it was just supposed to give us some idea so we can think about it over the next two weeks. Here is my understanding of the problem. we are going to be given a large coordinate plane split into sections (one for each technology that we discuss) with lots of background noise. Our job is to find pillars, rectangular objects hidden in the noise. On these pillars are blocks, and on these blocks are RDF triples. These triples can be used to search through an ontology to find text. The text is our answer. I don't know how many pillars there are, but our job is to find all of them. Reminds me of some sort of spy mission.

-----------------------

OKAY. That was a long recap. I hope it wasn't very boring...maybe these whole day summaries aren't such a good idea. We also went to Antibes, but Cole did a nice job of summarizing that, so I won't say any more about it. I took some pictures too, but mine were also dark. I didn't want to be the "stupid American" taking pictures of everything. No offense, Cole. :)

Afternoon Practicals - Monday's session...

2009-07-06T07:17:00.004-05:00

Today's practicals were very nice introductory material, designed to bring everyone up on speed on login, credentials, and what are practical facilities are.

The only suggestion I would make to the staff for the ISSGC is to permit students to remote VNC in while on campus, which would be nice. (I actually did broach this with one staffer, and I have some other names to discuss it with) - the caveat is that not everyone groks VNC, so it can be a loaded proposition. Perhaps we can set it up on a per-user request. We shall see.

For the non-technical folks, what this means is this. Normally to use a "linux" or "unix" machine, system administrators use the command line (that black screen with the white text, yay fun!). However, sometimes we need to use a mouse and windows (think "desktop") so there is a way that we can use the desktop over the network. But it's not normally turned on by default, so we have to ask permission. That's the nutshell version, and I hope it clarifies why I would suggest that they do something they didn't already do. The physicists and bio-med students wouldn't know VNC any more than most, so it's not something that's inherently beneficial to everyone. But a lot of the students here use VNC back home, so it's an option that's reasonable. Just not standard.

Otherwise, I thought that the practicals were nice from the standpoint of bringing people online and getting them up to speed.

Monday, July 6, 2007 Sessions Notes

2009-07-06T04:58:00.008-05:00

So, here's some pics of some of the folks who have been presenting today:

Local school guy, Dean of the Engineering school, no clue on the name

Per Öster

Miron Livny from the Condor group

For students at the 09 conference...

2009-07-06T04:25:00.005-05:00

To visit the Casino Supermarket (like a Kroger's but smaller) - Take a left at the foot of the drive for the Technical Building, go "straight" past the unnamed carrefour, go "straight" at carrefour Golf and turn 90 degrees left on carrefour St. Phillipe

Once you see the building, go straight on the street past the two cafe's and climb the steps.

BEWARE: It smells like FISH

Some pics of the room...

2009-07-06T03:05:00.024-05:00

Before I begin, a re-emphasis on what Kristen posted already ... NO A/C!!!
The Entrance

Some of the room

The Patio

Look, a Kitchenette

The Toilet

Day one from Cole, part Deux

2009-07-06T02:02:00.028-05:00

BTW: I found out I can post small images and they will auto-thumbnail from the larger images, with clickable links, so I'll be posting all small images now. Yeah, I could go back and change the others, but I won't.

You would not believe how BLUE the Med is around here from the air. This picture barely helps you to believe it.

Next is a mediatheque...

Next is a set of pics from a research park around here. Look whose name is on this door.And the parking spots, and ...

Notice the security cam below?

Day one from Cole

2009-07-05T23:59:00.007-05:00

Well, as Kristen mentioned, landing at Nice is an experience in and of itself. I'm going to attempt to post two videos of the landing, once in WMV, and once in OGG. If you don't know which of those to use, try the first one, it should work no matter what. If you know what OGG is, then you can try that one and let me know how it streams. But bear in mind that I am very bad at making movies on the fly, with no preparation, and I cannot hold the camera still, so that is something to consider. Also, the audio is really distorted, so I don't know if it's the camera or what, but it sounds bad all the way around... BTW, five minutes in is the money shot.
WMV
OGG

And to top it off, here are some photos from the Nice airport, showing the English and the French...

Ok, so it's only one picture, I was trying. I actually didn't take as many pictures at that airport, if only because it seeemed like there was so little to see there. It's a lovely airport, but it's tiny and that's about all there is to it.

Dinner was nice, here's a picture of it. It was a very nice chicken and (I say turkey, Andrew says pork) in a bed of vegetables (baby corn, asparagus and potatoes). It had some tomatoes with cheese (very good), some other cheeses (on the side), some fresh fruit, some bread and some oil/vinegar with herbs. All in all, very very yummy.

There appeared to be an alternate meal available, and there was a variety of deserts, but for me, this was more than enough.

Ok, so I'm going to jot a note now about what I could really use now, and what I left at home:
I left my neck pillow for the airplane, and my puzzle books. I could use a small fan. If you're going to be in the Mediterranean in the future, and you're from the South-ern United States, bring a fan. And I probably should have brought an alarm clock, but c'est la vie. (No, I already knew that one :-p )

While I have more photos and the like, I think this is enough for now. The server doesn't like my 50MB worth of videos, so I'll stop while I'm ahead. Look for more first day thoughts in my next post :D

I have arrived!

2009-07-05T11:28:00.006-05:00

Technically I was in France for my first post also...but we'll ignore that for now.

The city is so beautiful! Landing in Nice was an amazing experience because there was so much water surrounding the runway. That isn't my picture, but it's a good one.

When I arrived, Javier from ISSGC was waiting at the airport for me. He was kind enough to wait with my bags while I took a quick look around the airport. I did take a few pictures of some signs that I will upload later. The amount of English on everything was surprising. Javier let me know that it would not be the same outside of the airport. He was right, but it was not too overwhelming. After another ISSGC student arrived, we went outside to find a taxi.

Javier gave a nice taxi driver our directions, and then we were on our way to Sophia-Antipolis. The driver spoke (to himself while driving) in French, but he was able to understand us when we told him that we saw a sign with the building name on it. He even shouted "That's why I'm the best!" because he got us there in record time with ease. 140 km/hr...which is 87 mph, but he was a very safe driver. I've already noticed the first difference between France and the US. Apparently, we tipped him too much. Surprisingly though, he told us and gave back the excess money. Maybe that's just because he was a nice man, but I have never heard of that happening in the US.

After getting checked in, I went up to my room and realized that there is no air conditioner. The temperature is comfortable, although it is slightly hot. I've been here for a few hours though, and I'm already used to it.

One thing that I did have trouble with was buying the right adapter/converter for my US laptop charger. Luckily, Cole had an extra for me to borrow for a while. (Thanks, Cole!) My cell phone has no service at all, which was surprising. Oh well, I planned to use Skype anyway.

There is a dinner at 7 that I should be getting ready for, so I'll finish this here. Tomorrow, I'll discuss my experiences from the first day of school.

Another Introduction

2009-07-05T11:07:00.003-05:00

This is my first post here, so I'll briefly introduce myself. My name is Kristen, I'm in the MS program at Clemson University (in South Carolina, USA), and what I hope to do through this blog is to give you a little bit of insight about what's going on at ISSGC'09. There will be several authors contributing to this blog, so you'll be able to get a description of the same experience from different perspectives.

In the upcoming classes, there will be discussions about many different topics (listed here). Of these, the only one that I am familiar with is Condor. I am no Condor expert though, so there will be plenty to learn.

I want to keep this post as just an introduction, so to end it I'll just say that I'm looking forward to meeting new people and seeing a bit of the city, but I also hope to learn useful information that can help my research when I get back home.

Packing night tonight

2009-07-02T16:14:00.004-05:00

Well, seeing as how it's almost time to go, it's time to make sure I'm all packed up. I'll revisit at the end of the trip and tell you if I realize I forgot something major during the trip (of course, it'll be minor, the major stuff I'll pack, but it's what you don't bring that you miss the most).

The major stuff I've got covered is a variety of clothing and shoes, toiletries, and my computer accessories and plenty of computer accessory cables (hey, when you've got a phone, mobile hard drive, usb drives, camera, and on, and on - you tend to collect a lot of cables) but one cable that I am particularly happy to have when setting out is a power cable for my variable input laptop charger that I'm told is sure to work anywhere in France or Germany. Yay for being prepared. (Actually, I'm taking 2). We'll see just how forward-thinking that ended up being after the trip.

Ok, so if you can think of anything major I might have missed (and if it's not yet Saturday 8am, July 4, 2009) - leave a comment, I'll pack it extra.

And in closing, a travel thought for good luck:

The World is a book, and those who do not travel read only a page. ~St. Augustine

Preparing for ISSGC'09 -- only a week away

2009-06-29T20:53:00.000-05:00

Students will share their experiences at the International Summer School on Grid Computing ISSGC09 (www.issgc.org) -- July 2009. For more info, also see https://twiki.grid.iu.edu/bin/view/Education/OrientationISSG09.

ISSGC'09 draws near

2009-06-29T19:53:00.002-05:00

I guess I should start my first post on this blog with a tad introduction, as this should be for posterity as well as for the trip...

I'm just this guy, ya know? I've been to a half-a-dozen schools, doing this and that, and it wasn't till the last one that I got serious about getting a four-year degree (although I have an AAS as well and have traveled, so take the 6 figure with a grain of salt), but my education is nowhere near complete. As I've struggled with what I would like to do for a graduate degree in my upper-classman years at the University of Houston - Downtown, I got the opportunity to play with clusters and grids, even if that experience was lacking. To say I know anything substantial about clusters and grids would be comparable to saying a person was qualified to compete in the Le Mans tournament merely by having ridden in a motorcoach. But what can I say, I learn quickly.

So, having given you that convoluted back-story (a back-story that I shall try and avoid as much as possible in all my future OSG-EOT posts), let it be sufficient to say that I imagine myself to be one of the most junior participants in this journey. Perhaps calling myself one of the most junior is a bit romantic, but so be it, until this experience is over I should maintain a distinct naïveté.

So what, then, shall we discuss now that we've established that I'm young and inexperienced? What about my expectations of myself in this journey.

I intend to use this blog-space to comment on what:

We at the school have discussed throughout the day
I thought was exceptionally interesting
I still don't understand from the day
Any pictures that I take that are neat
I think another student in my position might need to know!

I think those would be good enough from my own part, and if I come up with anything else, well, what a wonderful surprise. So with that, I bid you adieu, and let's journey together, shall we?

And I'll part with a oft-quoted quip (seeing as how I'm excited and ready to travel!)

The journey of a thousand miles begins beneath one's feet. - Lao-tzu