Ok, I have the software for the Cluster computer

[WingZero94]

Here is what they want:

Software: MPICH installed on every PC. Upon completion of installation, one daemon, MPD.exe, will be running on each PC all the time.

The parallel code will be compiled on one computer. Then this computer will launch the com piled files on all the computers simultaneously using MPD. Each computer does the work computation independently, and exchange necessary data over the network. This data exchange is written in the code itself, and is controlled by MPD.


Your help is greatfully accepted. Thanks,

Paul

 

[Locutus4657]

Ok, so they want to do this with Windows then? This is fairly unusual, but certainly do able (and it does have some advantages). You will want to go to http://www-unix.mcs.anl.gov/mpi/mpich/download.html, and download MPICH for Windows NT. When you install MPICH, you will have to configure MPI run, and give every workstation a common run directory, and make sure every computer sees every other computer. I assume you will want to install a compiler on workstation 0, I would recommend Microsoft?s Visual Studio.net since Microsoft has become the chosen platform. The version of Windows I would go with for this would be WindowsXP as it has built in CPU Utilization monitors, Memory monitor, and network utilization monitors. These should prove useful in the development and optimization of parallel software.

 

[Armitage]

Wingzero:
You've just described a generic distributed program. That doesn't help much.
I would suggest they consider PVM in addition to MPI (MPICH is an implementation of the MPI standard). In my experience, PVM is much easier to use, and genrally more robust then MPI.
Also, MPICH doesn't have to have a daemon on each machine. It doesn't even have to be installed on each machine unless you use the chp4_mpd device. That is probably the better way to go though.

Locutus:
I don't think it's clear from the first post that they intend to run windows. MPD.exe seems to point that way, but does windows have daemons?
In any case, I hope they aren't considering using Windows for this ... I'd have to go back into rant-mode

 

[WingZero94]

Ok,

sorry it is so vague. He (Dr. Wang) isn't giving us a lot to grasp onto. Hopefully our meeting will shed some light. In the meantime, what do you think would be the best Athlon XP setup (Motherboard and RAM) for this? Is DDR333 going to really help us over DDR266? I know we'll need CAS 2.0 for the DDR333 to have a bigger impact. What do you think. Oh yea, It is windows.

Paul

TIA

 

[Armitage]

<< Ok,

sorry it is so vague. He (Dr. Wang) isn't giving us a lot to grasp onto. Hopefully our meeting will shed some light. In the meantime, what do you think would be the best Athlon XP setup (Motherboard and RAM) for this? Is DDR333 going to really help us over DDR266? I know we'll need CAS 2.0 for the DDR333 to have a bigger impact. What do you think. Oh yea, It is windows.

Paul

TIA >>



sigh ... ok, I warned you.
[rant] Why Windows!?!?!
The vast majority of the cluster computing community uses a unix-based OS, ussually Linux. What do you think you know, considering your obvious inexperience, that all the experts and practitioners in the field don't? There are tremendous resources for setting up, using, and managing cluster computers available for free, but it generally doesn't do windows.
First, and simplest, consider the cost? From your other post, you wanted $600/node. Let's assume for a moment that that price actually means something (see my post in that thread). Your going to spend bare minimum of $100/node for the OS (lowest price on pricewatch), probably much more till you get remote access and stuff set up. (I'm no expert on Windows licensing). So, you're effectively going to reduce the amount of hardware you get for your cluster by at least 17% by using windows.

Now, consider the ease of remote administration. A very complete & robust capability is built in on linux, and there are many tools built on top of that specifically for cluster computing. On Windows, you'll have to buy additional software or licenses. (again, I'm no windows expert, so correct me if I'm wrong.)

Now how about performance? Can you run windows without the gui sucking up ram? Can you netboot windows so you don't even need a harddrive on the nodes? Have you benchmarked your software on windows and linux?

What about stability. Sorry, but I've never seen a windows box come near the stability of a Linux box under high, sustained load. Remember, you need to divide your average uptime by the number of boxes in the cluster. If you average 2 months uptime on your windows boxes, that means that, on average, you'll be rebooting cluster nodes greater every other day for a 36 node cluster. Now, layer some hardware failures on top of that and you have a maintenance nightmare. To me, that's completely unacceptable. And before you shout "WindowsXP!", how long has XP been out? About 6 months I think. Even if you've been up continuously since day one, you're not close to the uptimes I've seen on clusters I've used & built.

As far as which AthlonXP setup to use ... nobody here can answer that for you. You need to go do the benchmarks yourself, that's the only way to tell. If you're serious about 36 nodes and can show some backing from your university, you should be able to interest some cluster vendors enough that they might make some systems available for you to test on. Barring that, find some local enthusiasts that would let you test on their machines ... the local Linux users group (lug) might be able to help ... oh wait, you're using windows.

Sorry, but I have a hard time taking your threads seriously at this point.[/rant]

 

[WingZero94]

Listen man,

WE ARE ONLY BUILDING THE HARDWARE FOR THE UNIVERSITY! I HAVE NO FRICKEN' SAY WHAT OS GOES ON THEM. THEY GET WINDOWS FOR FREE SO THAT MAY BE WHY THEY ARE USING IT. I HAVE NO FRICKEN CLUE!! If you don't want to take my threads seriously, then don't. You don't have to reply if you don't care. If you'd like to help speculate what hardware would work best with this setup, that'd be cool. I've already admitted I don't know much about clusters. This is not a battle of the wits dude, just trying to get the best information for my current job. MY JOB = BUILD BEST HARDWARE FOR JOB (and for less than 21000k), INSTALL WIN2K, DELIVER TO UNIVERSITY TO INSTALL SOFTWARE AND CONFIGURE.

 

[Evadman]

Alright then. This is what you build.

Tyan Tiger ( $129 FRom Newegg )
Dual Duron 1.0's ($98)
256 Megs of Registered DDR ($129 from Mushkin)
80 gig HD ($111 From Newegg )
Cheap Vid Card ($20 Compgeeks)
Floppy ($11 from Newegg)
CPU fans ($12 from newegg)
Nic card ( $15 each, almost everywhere, 10/100 )
80 MM case fans ($25 per system, anywhere )
Good case with 400+ watts PWS ($120 from Newegg )

And your at $680 and you get 2 processors per node instead of 1. To cut it further, you could use a K7S5A which has a onboard nic, and use a 1800+ and end up about $80 less. The dual board has more upgradability built in for the futrue tho, so if I were in charge that would be what I would get.

 

[RSMemphis]

Well, if you put it that way.

You want to eliminate as many bottlenecks as possible.
You won't need fast RAM, but more, so you are not bottlenecked by the harddrive. Many clusters live without a harddrive - which is a GOOD IDEA, since updating all 30 some odd computers with a servicepack or new software will suck.
And as I said in the other post, I think 100Mbit ethernet with the TCP/IP stack will kill most of the performance right there anyway, and UDP is too unreliable.

Drop the costs wherever you can and see if you can do something on the network side. I see a major bottleneck there, especially with 30 some odd computers. The line to the master of the cluster will be totally cluttered.

 

[figgypower]

I think most people can't take this thread seriously, because it seems like your university or whoever is in charge is filled with "Cluster Idiots".
I'm not an expert on a cluster-based super computer, but that's exactly why I'm not trying to build one. If you're actually making money on
this, I'd try putting together something like Evadman said. Otherwise it's a waster of time. My recommendation would be to tell them that
they're fantasizing about computer technology and they're STUPID.

 

[Armitage]

<< Well, if you put it that way.

You want to eliminate as many bottlenecks as possible.
You won't need fast RAM, but more, so you are not bottlenecked by the harddrive. Many clusters live without a harddrive - which is a GOOD IDEA, since updating all 30 some odd computers with a servicepack or new software will suck.
And as I said in the other post, I think 100Mbit ethernet with the TCP/IP stack will kill most of the performance right there anyway, and UDP is too unreliable.

Drop the costs wherever you can and see if you can do something on the network side. I see a major bottleneck there, especially with 30 some odd computers. The line to the master of the cluster will be totally cluttered. >>



Whether the network will be a bottleneck @ 100baseT for 30 nodes depends completely on the application. I've saturated a 100baseT network with only 16 nodes, but I have an app now that's only using about 10% of the available 100baseT link with the same 16 nodes.
Living without a harddrive, as you put it, will put alot of additional strain on your network, I'd want want dual 100baseT nics out of the server at least. If you netboot, you also want to have a ton of ram on the nodes.
But he (or his customer) wants to run windows on it, so netboot isn't really an option anyway.

 

[bubba]

<< Can you netboot windows so you don't even need a harddrive on the nodes? >>



I want to netboot (off of a floppy) some of the nodes of my Linux lab, but I have only been able to find projects like the LTSP, which run the process on the server, not the client. For a parallel cluster you would (obviously) need to run processes on the individual nodes. How would one do this?

I have been asking this question for a long time, but haven't got a good answer. We have a linux lab and a Windows lab. I would like to put together a boot floppy so that I could boot some of the windows boxes into linux off of the server or perhaps even turn the entire linux lab into a netboot system. However, all applications would need to run on the individual clients. Any help?

 

[Armitage]

<< For a parallel cluster you would (obviously) need to run processes on the individual nodes. How would one do this? >>



After you boot the remote nodes, NFS mount a filesystem off the server. Actually, if you're really netbooting, and not just running a micro distro, you already have an NFS mount on the server. Now you can execute programs from that volume on each of the floppy-booting nodes. You'll load the program from the server, but it will run on the remote nodes.

Or you could just log into the server and use rsh to launch the jobs onto the remote nodes. Or you use PVM or MPI, both odf which use rsh to actually start the remote jobs.

 

[bubba]

I already use MPICH on the Linux nodes I manage two parallel processing clusters and soon to be more. However, this floppy boot system would only be for fallover use from an over-crowded linux lab. Do you have any links on how to do this. I have not been able to figure out what to do with the /etc/ tree, the swap areas, etc...

Thanks

 

[Armitage]

I think I would try using a micro distribution, and not bother with netbooting. You could keep some of the stuff you need in a FAT partition on the windows boxen your usurping. You could maybe even put a swap file under FAT, but that may be iffy.
I have to add here, that the only micro distro I've used is freesco, which isn't really what you're looking for. And I've never netbooted (yet), just done some reading on it. So, I'm about at the end of my usefullness here

In any case, have a look at the netbooting howtos and maybe look at some of the micro distros.

elladen seems to have done some homework here, you might ask him.

 

[bubba]

I have looked at the netbooting howtos. I have asked here. That is why I asked in this thread. I would like to do it, but few know how. I would appreciate any help I could get.