Eug
Lifer
- Mar 11, 2000
- 24,158
- 1,806
- 126
Nope. That's the backup backbone. The primary is Mellanox Infiniband. 10 Gigabit/s x 2
Presentation on tech info.
pix
pix
To answer a few questions:
1) No, the migration is NOT complete. The PHYSICAL migration won't be complete until mid-March.... and the machine won't be operational until May. Testing is supposed to take the month of May. "Public" (meaning: research based) consumption starts in June, from what I hear. (its of great interest to me, since I'm doing a parameter study using my own FEA code.... and the computational facilities would be nice. You wouldn't believe how hard it is to locate a large number of computers to work on).
2) The "migration" was officially to get ECC RAM, consume less power, and take up less space. But the question to ask yourself--was this actually a "smart" move after spending $5+ million dollars and having it up and running with the G5 towers in November? Needless to say there were some problems. Some SERIOUS problems. I've heard hints and rumors that I can't confirm (VT and Apple are VERY VERY quiet about this).
3) They are selling off several hundred of the original G5s (with the 2GB of RAM but without a couple of the add-in cards, including the Infiniband. So yes, that means there will be another massive effort to put the cards in the Xserves). The rest are being disseminated through departments and the college itself.
I used the prior supercomputer to do work before. It wasn't that impressive to ME (as most mechanics based solutions are inherently non-parallelizable)... but it was nice to have a large number of nodes easily available. Also, it was nice that we could tap into a 100Mbps switch to download output (downloading 10 Gb of output can get REALLY boring). However... I had to stop using it prior to dismantling.... mainly because my run would be "cancelled" half the time when I got up in the morning. Stability, from what I understand, was an issue. It will be great when VT has it up--VT does some serious computational research (and is hiring more faculty to support it), and is in DIRE need of some computational facilities (we had very very poor ones prior).
Oh.. and as far as buying it.... a lot of people don't realize that a research grant that comes into VT (now) is cut in half--half goes to the university, and half goes to the PI to sponsor the research. So... if your prof gets a $100K grant.... $50K goes to the university immediately. The college of engineering covers a great deal of its own budget by doing this (think about the research dollar figures you've seen in the past.... and think of how much money HALF is). Some departments are "self-sufficient" in that this half can cover ALL of their operating expenses for the year.
Actually you'd have a hard time finding 10 engineering grad students who AREN'T paid to come here. I know I certainly am.... there are a lot of TA and RA (research-assistant positions) available. It can be hard to live on the meager salary... but you can afford to do it (I'm finishing my FIFTH year of grad school).
Eug
Lifer
- Mar 11, 2000
- 24,158
- 1,806
- 126
That's why I'm wondering about the ECC. Maybe much of the instability was related to lack of ECC. In fact, for VT's (and Apple's) sake I hope it's because of the lack of ECC, because if the G5 Xserve setup is still unstable, that would be a disaster. By the way, how many nodes were you running your code on?
I'm still surprised they went with a non-ECC machine from a company with zero history in supercomputing clusters, with interconnects never before tested on a Mac. It was a HUGE risk, and it paid off in terms of PR for Apple and VT because of the #3 supercomputer ranking, but the fact that the system is being dismantled just a few months later certainly doesn't look good.
They had 4 GB RAM, no? And the stores are selling them with 1 GB RAM.
- Feb 15, 2004
- 5,594
- 1
- 0
Engineering Fundamentals are a set of classes all entering engineers HAVE to take to transfer into a specialized department (EE, ME, ChE, etc.) I'm pretty sure most engineering schools have some form of Engineering Fundamentals. It involves learning to use MATLAB and Autodesk Inventor, sketching, interpreting drawings, etc. It's not very hard, but there's a LOT of work.... more than once, I was still busy with EF homework/projects while business majors were out partying on Thursday night.
In addition to EF, you still have to take calc, physics, chem, and all that good stuff.
My only question is one HokieESM can prolly answer too... Is the combined startup chime from that whole array as ear blowing as I can imagine? :beer:
Imagine if they played that chime on the 10 foot subwoofer from the beginning of Back to the Future
I was running my code on a single node (well, each instance of my code). A cluster is no better than its individual nodes if your code doesn't parallellize. And any mechanics solution--solid or fluid--has the unfortunate instance where every point affects every other point.... so it doesn't parallelize well. I only saw a 10% improvement going from one node to two (so why not run two runs at 100% rather than one run at 110%).
I hope ECC fixes some of it. I've heard that the BIG problems involved software incompatible with the G5, but compatible with Xserve (and the Xserve's version of Panther). But those are "rumors". I will just say this: the supercomputer was RARELY up for more than 24 continuous hours when it was under "public consumption".
The original nodes had 4GB of RAM (which was far, far more than I needed... my code is for more memory bandwidth dependent... it only used 100MB of RAM or so). The ones filtering into departments have 2GB. What VT/Apple is doing with the extra memory, I don't know. Because there is some serious "trading" going around--I don't think VT is footing the bill for this. ;-)
I don't think xServes have one.
Now here's an interesting bit of trivia - all the Macintosh startup 'chimes' have been guitar chords, sampled from a real, live guitar. If I recall, it's always been the same dude with the same guitar. Unless the G5 is different, I'm yet to see one in person (Have one on order, though, someone donated FinalCut Pro, and a $3000 computer is NOTHING when you just got a $2000+ piece of software for free, but don't have anything to run it on.)
I concure.Originally posted by: Nitemare
When they are not looking, install Seti for Team AT on it....
What cooling method is employed to cool it?
Is it possible to OC it?
Eug
Lifer
- Mar 11, 2000
- 24,158
- 1,806
- 126
Nobody has figured out how yet. One guy has tried by soldering resistors, but it didn't work. People have been successful with G4 Power Macs, but not G5s yet AFAIK.Originally posted by: OffTopic
I concure.Originally posted by: Nitemare
When they are not looking, install Seti for Team AT on it....
What cooling method is employed to cool it?
Is it possible to OC it?
I actually know some people that applied there and weren't offered anything and I think I know of some people that might be there without anything. They're MS-only though, and I think last year and the year before there was a flood of people applying for grad schools.
That's a negative... I guess you've never worked with an older, older mac... Mac Plus'es beep at you. Just to name one instance... whats funny tho is if you have a hardware error, you either hear the chimes of death or a car crash instead of mobo beep codes.
WINNAR. Was waiting for someone to pick that one up. The guitar stuff only applies to models that you could turn on by pressing the "reset" button on the keyboard. The rest of them, with standard PC-style power buttons or rocker switches on the back beeped.
Adul
Elite Member
They swicthed over to the Xerve dual G5 machines recently and all those machines are being sold off.
I seem to remember it being called power manager... Was introduced with the later revisions of the 68030 processor chipset and all powerbooks. I was a mac user for 10 years, and the most hard core of the hardcore. Switched to PC in two weeks flat december02-jan03 when I built a 300$ celeron 1.7 with a geforce 2 ultra that schooled my 3000$ mac in every bench I threw at it. Between that and how if you wanted mac parts, it was a black market... the switch was a no brainer.
Mac users everywhere, its not a matter of IF, but WHEN. :beer:
- Feb 15, 2004
- 5,594
- 1
- 0
Okay, got out of the presentation about an hour ago, here's what I've got for you guys:
The choice to use Mac hardware was not the original plan. The first arrangement was actually with Dell, using Itanium2 processors. However, less than 24 hours before the computers were shipped, Michael Dell himself cut the cord on the project. Apparently Dell already had several offers on the table for large-scale high-performance supercomputing projects, and Virginia Tech was working on a budget of about 5 million; relatively little payoff for a supercomputer. Dell went ahead and worked with NCSA on their supercomputer. Guess what? They're now number 4 on the top 500 list Ouch.
VT worked with a few other vendors, such as HP and IBM, but they kept coming back with higher numbers than VT wanted to spend; also, the Opteron processor could only do one floating point calculation per clock cycle, while the Itanium2 and G5 could do two per clock cycle. They researched doing 'white-box' custom builds as well, but the shipping dates on the components was way longer than VT wanted to spend.
And then came Apple. They offered a shipment of dual-proc G5s in September, and VT jumped on it.
The switch to Xserves is ENTIRELY about ECC RAM. (I asked specifically after the presentation.) Although it is freeing up a lot of space (the computers will take up about a third of the area as they once did), the presenter clearly said that if the towers had supported ECC RAM, they would have stayed with them. Because the computer is slated for simulations and scientific computations requiring high-precision, there didn't seem to be much question about it. I did try to ask about the cost of the upgrade, but they wouldn't give me any specifics (legally tongue-tied).
Backup power consists of a 1.5 MW upgrade to the existing power infrastructure of the building and a backup system capable of giving them about 30 minutes of uptime (yes, minutes, not seconds). They also installed LOT of additional cooling to the building. A plan to install a diesel generator to supplement the backup system was also mentioned.
Software-wise, large portions of Mac OS X were revised to take it from a single-user OS to a high-performance cluster OS, through collaboration between Apple and VT.
Don't doubt the power of a Mac either. A grad student was running a few simulators on the supercomputer as it was being torn down for the upgrade, and he got a performance boost in all of them over anything he'd run them on before (Itanium2 clusters, dual Xeon machines, etc.) One simulator ran 2.5x faster than he'd ever seen it go. Of course, Mac video hardware is lagging behind, as are the number of developers writing code to take advantage of G5 architecture as opposed to Pentium 4. But the potential is most definitely there.
I did note that they made no secret of the chances they took with this project. The fact that a project of this scale had never before been done with Mac hardware, combined with the fact the computers were some of the first G5s off the assembly line, made it a big risk. But, the payoff was huge, as you know.
The choice to use Mac hardware was not the original plan. The first arrangement was actually with Dell, using Itanium2 processors. However, less than 24 hours before the computers were shipped, Michael Dell himself cut the cord on the project. Apparently Dell already had several offers on the table for large-scale high-performance supercomputing projects, and Virginia Tech was working on a budget of about 5 million; relatively little payoff for a supercomputer. Dell went ahead and worked with NCSA on their supercomputer. Guess what? They're now number 4 on the top 500 list
Ouch.VT worked with a few other vendors, such as HP and IBM, but they kept coming back with higher numbers than VT wanted to spend; also, the Opteron processor could only do one floating point calculation per clock cycle, while the Itanium2 and G5 could do two per clock cycle. They researched doing 'white-box' custom builds as well, but the shipping dates on the components was way longer than VT wanted to spend.
And then came Apple.
They offered a shipment of dual-proc G5s in September, and VT jumped on it.The switch to Xserves is ENTIRELY about ECC RAM. (I asked specifically after the presentation.) Although it is freeing up a lot of space (the computers will take up about a third of the area as they once did), the presenter clearly said that if the towers had supported ECC RAM, they would have stayed with them. Because the computer is slated for simulations and scientific computations requiring high-precision, there didn't seem to be much question about it. I did try to ask about the cost of the upgrade, but they wouldn't give me any specifics (legally tongue-tied).
Backup power consists of a 1.5 MW upgrade to the existing power infrastructure of the building and a backup system capable of giving them about 30 minutes of uptime (yes, minutes, not seconds). They also installed LOT of additional cooling to the building. A plan to install a diesel generator to supplement the backup system was also mentioned.
Software-wise, large portions of Mac OS X were revised to take it from a single-user OS to a high-performance cluster OS, through collaboration between Apple and VT.
Don't doubt the power of a Mac either.
A grad student was running a few simulators on the supercomputer as it was being torn down for the upgrade, and he got a performance boost in all of them over anything he'd run them on before (Itanium2 clusters, dual Xeon machines, etc.) One simulator ran 2.5x faster than he'd ever seen it go. Of course, Mac video hardware is lagging behind, as are the number of developers writing code to take advantage of G5 architecture as opposed to Pentium 4. But the potential is most definitely there.I did note that they made no secret of the chances they took with this project. The fact that a project of this scale had never before been done with Mac hardware, combined with the fact the computers were some of the first G5s off the assembly line, made it a big risk. But, the payoff was huge, as you know.
Originally posted by: TheLonelyPhoenix
Okay, got out of the presentation about an hour ago, here's what I've got for you guys:
The choice to use Mac hardware was not the original plan. The first arrangement was actually with Dell, using Itanium2 processors. However, less than 24 hours before the computers were shipped, Michael Dell himself cut the cord on the project. Apparently Dell already had several offers on the table for large-scale high-performance supercomputing projects, and Virginia Tech was working on a budget of about 5 million; relatively little payoff for a supercomputer. Dell went ahead and worked with NCSA on their supercomputer. Guess what? They're now number 4 on the top 500 list
VT worked with a few other vendors, such as HP and IBM, but they kept coming back with higher numbers than VT wanted to spend; also, the Opteron processor could only do one floating point calculation per clock cycle, while the Itanium2 and G5 could do two per clock cycle. They researched doing 'white-box' custom builds as well, but the shipping dates on the components was way longer than VT wanted to spend.
And then came Apple.
The switch to Xserves is ENTIRELY about ECC RAM. (I asked specifically after the presentation.) Although it is freeing up a lot of space (the computers will take up about a third of the area as they once did), the presenter clearly said that if the towers had supported ECC RAM, they would have stayed with them. Because the computer is slated for simulations and scientific computations requiring high-precision, there didn't seem to be much question about it. I did try to ask about the cost of the upgrade, but they wouldn't give me any specifics (legally tongue-tied).
Backup power consists of a 1.5 MW upgrade to the existing power infrastructure of the building and a backup system capable of giving them about 30 minutes of uptime (yes, minutes, not seconds). They also installed LOT of additional cooling to the building. A plan to install a diesel generator to supplement the backup system was also mentioned.
Software-wise, large portions of Mac OS X were revised to take it from a single-user OS to a high-performance cluster OS, through collaboration between Apple and VT.
Don't doubt the power of a Mac either.
I did note that they made no secret of the chances they took with this project. The fact that a project of this scale had never before been done with Mac hardware, combined with the fact the computers were some of the first G5s off the assembly line, made it a big risk. But, the payoff was huge, as you know.
Awesome info man, welcome to AT :beer::beer::beer::beer::beer::beer:
I believe you are incorrect: http://www.folklore.org/StoryView.py?project=Macintosh&story=Boot_Beep.txt&sortOrder=Sort%20by%20Date&detail=medium
- Feb 15, 2004
- 5,594
- 1
- 0
The other clusters most likely ran on Unix-based systems as well, since Windows XP has awful clustering capabilities (I think there's only one Windows-based that has ever been made to work, at Cornell). I dont know the number of nodes he ran it on using the VT cluster, or the exact specs of what he'd run it on before - but I assume that if he was getting the best speeds he ever had before by a considerable margin, it definitely says something for the power of the PPC 970.
The speaker was Jason Lockhart. His job title is listed on the VT Terascale website as "director of the College of Engineering's High Performance Computing and Technology Innovation".
Originally posted by: TheLonelyPhoenix
Okay, got out of the presentation about an hour ago, here's what I've got for you guys:
The switch to Xserves is ENTIRELY about ECC RAM. (I asked specifically after the presentation.) Although it is freeing up a lot of space (the computers will take up about a third of the area as they once did), the presenter clearly said that if the towers had supported ECC RAM, they would have stayed with them. Because the computer is slated for simulations and scientific computations requiring high-precision, there didn't seem to be much question about it. I did try to ask about the cost of the upgrade, but they wouldn't give me any specifics (legally tongue-tied).
Software-wise, large portions of Mac OS X were revised to take it from a single-user OS to a high-performance cluster OS, through collaboration between Apple and VT.
Don't doubt the power of a Mac either.
I did note that they made no secret of the chances they took with this project. The fact that a project of this scale had never before been done with Mac hardware, combined with the fact the computers were some of the first G5s off the assembly line, made it a big risk. But, the payoff was huge, as you know.
A few notes:
Be CAREFUL about what you hear coming specifically from VT or Apple. VERY careful. There have been some very very very nasty legalities here of late. I know for a fact that the transition wasn't solely about the ECC RAM--it was definitely a part, however--BUT error-correction/node-management was the real reason. Let's just say that you won't hear from VT (and DEFINITELY not Apple) why--nor from me... I don't like lawsuits (and I don't know nuts-and-bolts specifics... although I do have some idea). If there was any doubt about this..... why would we build a $5 million dollar supercomputer, benchmark it for a month, let users "play" with it for a month, and then take it apart? Especially when we were specifically told that the Xserve would be ready in January? 1) They wanted it done in 2003... and 2) they thought the G5 would work as well as the Xserve. It didn't. And the "swap" isn't easy.... can you imagine just doing the card installation (the Infiniband cards) on 1100 nodes? And physically carrying them? Much less software installation. No sane person would do that... and although we ARE talking about a university (the home of insane people), they could have waited until January if the ECC RAM was really that big of an issue (and yes, they knew... the supercomputer team has some top-notch compsci people).
Oh, the grad student you're talking about... I know him.
The quote is right on... but somewhat misleading. The supercomputer IS fast.... IF your code parallelizes. He was using 600 nodes... as opposed to 100 nodes on the Itanium cluster here (its 200 nodes total). The available computation power on System X is tremendous--if you can use it all. Some people's work (particularly a lot of the quantum/atomic discrete work) parallelizes well--and he could use 600 processors for a single run. It was scary fast. He's very pissed that its not running right now... understandably. But to give a counterpoint... my thermoviscoplastic code ran 2.3 times faster on an Itanium 2 than System X. Of course, mine doesn't parallelize well.... and its VERY sensitive to the amount of cache on the chip. In any event... don't think I'm criticizing VT about the cluster. Its great. We needed it. I don't think it mattered whether it was Apple, Dell, whitebox, whatever... it was necessary. Apple cut us a great deal... and that was important. They dropped the ball in what they promised (cough cough).... and now they're having to "pay" for it. VT is very "complimentary" towards Apple because of their "service". Just don't expect to hear anyone say anything about it. Hopefully it will be up and running this summer without the bugs--I sure could use it.
Although, I'm still pissed that McDonald's wouldn't let us call it "Big Mac".Well, of course they might not be OFFERED something. VT is a top notch engineering research school in certain departments (well.... they have some top notch professors which is really what its all about), so their money only goes so far. I know most of the grad departments accept only about 30% of applicants... offering money to 80% of those. But almost ALL of the people who actually go here are funded--most can't rely on mommy and daddy to pay for it (i know i couldn't). There are a few, of course. But engineering is a well-funded field. I know the professors in my department "meet" with new grad students when they come in trying to "recruit" research assistants--because there are more positions than students at the moment.
Oh.... I did forget to mention: you will find that most professors and departments won't waste their time giving money to non-thesis master's students. So yes, there are several here that aren't getting funded.
Of course, frequently non-thesis masters students are being funded by their company, who doesn't want to "wait" for them to do research and write a thesis. TRENDING THREADS
-
Discussion Zen 5 Speculation (EPYC Turin and Strix Point/Granite Ridge - Ryzen 9000)- Started by DisEnchantment
- Replies: 25K
-
Discussion Intel Meteor, Arrow, Lunar & Panther Lakes + WCL Discussion Threads
- Started by Tigerick
- Replies: 23K
-
Discussion Intel current and future Lakes & Rapids thread- Started by TheF34RChannel
- Replies: 23K
-
-
Facebook
Twitter