64-bit x86 computing questions

[joecool]

i'm in charge of charting my groups course into 64-bit x86 computing and i've got some questions i'm hoping you folks can answer. here's the long & short of it:

- we need to move to 64-bit machines because we need the extra memory. currently we are using several sun boxes for all of our large jobs but the cost of these boxes is prohibitive and the speed is discouraging. we use x86 based linux machines for all of our other work and want to get the speed and cost benefits for our big jobs as well.

- as i understand it, there are currently four different 64-bit x86 platforms available: itanium, xeon-64,, opteron, and amd-64. i'm frankly very unclear on what the differences are between these platforms, and i have yet to find a straightforward comparison of them all.

- i've had a hard time finding any systems available that offer more than 16GB of RAM. the only one i've found so far is a 32GB system from Dell, which costs $40k. at that price it's not much of a deal.

so, here's my questions:

1) per above, what exactly are the differences between the various 64-bit x86 processors?

2) is anyone aware of off-the-shelf systems offering more than 16GB of RAM, and preferrably expandable up to 64GB or higher? at the rate our needs are growing we will probably need more than 64GB within the next year.

3) with our sun systems i know that a single job has access to ALL of the memory in the box. is the same true of pc systems? all of the large RAM pc systems/motherboards i have seen advertise RAM/CPU, and the boards seem to be configured that way - say, two CPU sockets and two sets of four dimm slots located next to each CPU socket. does this mean each CPU only has access to a specific set of dimms?

i'm relatively pc and tech savvy so don't be shy about posting technical details, sorry if these questions seem dumb but the info on 64-bit computing available on the web is scattered and largely uninformative - mostly what i've found is marketing speak, not facts and details.

thanks in advance,

joe

 

[Matthias99]

Originally posted by: joecool
i'm in charge of charting my groups course into 64-bit x86 computing and i've got some questions i'm hoping you folks can answer. here's the long & short of it:

- we need to move to 64-bit machines because we need the extra memory. currently we are using several sun boxes for all of our large jobs but the cost of these boxes is prohibitive and the speed is discouraging. we use x86 based linux machines for all of our other work and want to get the speed and cost benefits for our big jobs as well.

Define "large jobs". Can you provide any details on what you're doing, exactly?

- as i understand it, there are currently four different 64-bit x86 platforms available: itanium, xeon-64,, opteron, and amd-64. i'm frankly very unclear on what the differences are between these platforms, and i have yet to find a straightforward comparison of them all.

1) per above, what exactly are the differences between the various 64-bit x86 processors?

Itanium is Intel's 64-bit-only server platform. Thus far, it hasn't done much, largely due to a very high entry price and the fact that few applications are available on it. It uses a very different architecture than most PCs.

Xeon-64 (not out yet) is Intel's implementation of the x86-64 standard. These chips should, in theory, support 64-bit and 32-bit applications, and should bring the Xeon chips most of the advantages of the more recent P4s (800Mhz FSB, etc.)

Opteron and Athlon64 are AMD's x86-64 chips. The Opteron chips (and the newer Socket939 Athlon64s) have dual-channel memory controllers. Only the Opteron chips can be used in multiprocessor configs (the 1XX chips are for 1-way systems, 2XX for 2-way, and 8XX for 4-way and 8-way boxes). The "1XX" Opterons are basically identical to the Socket939 Athlon64 chips (they just use a different socket). These also run 64-bit and 32-bit applications.

- i've had a hard time finding any systems available that offer more than 16GB of RAM. the only one i've found so far is a 32GB system from Dell, which costs $40k. at that price it's not much of a deal.

2) is anyone aware of off-the-shelf systems offering more than 16GB of RAM, and preferrably expandable up to 64GB or higher? at the rate our needs are growing we will probably need more than 64GB within the next year.

I don't know the details of Itanium and the Xeon-64, but Opteron systems support 4 DIMM slots per processor (so, right now you'd have a max of 8GB/processor using 2GB DIMMs, and in theory that would double to 16GB/processor once 4GB DIMMs become available). A 4-way Opteron system could load up 32GB of RAM right now, and 64GB with 4GB DIMMs (assuming it has the proper memory config; see below). An 8-way system would double those amounts, although I don't know about their availability.

3) with our sun systems i know that a single job has access to ALL of the memory in the box. is the same true of pc systems? all of the large RAM pc systems/motherboards i have seen advertise RAM/CPU, and the boards seem to be configured that way - say, two CPU sockets and two sets of four dimm slots located next to each CPU socket. does this mean each CPU only has access to a specific set of dimms?

Again, I'm unsure about the details on the Itanium (because I haven't used them) and the Xeon-64 (because they're not out yet). The Opteron uses a NUMA (Non-Uniform Memory Architecture) design that allows any processor to access the RAM attached to the others over a high-speed HyperTransport bridge (although it's slower than the direct-attached RAM, since the other processor must process the request). I can't imagine anyone producing a multiprocessor system where this isn't the case; the whole point of multiprocessing is that you can have multiple CPUs accessing and working on the same data. I've also seen Opteron motherboards with two CPU sockets but only a single set of RAM sockets -- both CPUs can still access the RAM, but the second one is going to be slower, and it halves the total amount of RAM you could have.

As far as a single process being able to access that much RAM, it's up to the OS -- I assume a 64-bit OS would allow a 64-bit process to access the entire memory range if need be. A 32-bit process could only map 4GB of RAM (barring hacks like Windows' PAE), obviously.

 

[Vee]

Itanium is not a x86-64 platform. It's a 64-bit cpu from HP and Intel. EPIC aka IA64 ISA.

AMD64 is a collective name for all AMD x86-64 processors. It includes Opteron.
Considering your RAM requirements, I would think your choices are Opteron and Xeon64.

HP offers a 4-cpu Opteron server, Proliant DL585 with up to 64GB RAM.

IBM have their Opteron e325 series, but you won't get more than 16GB there. They also offer Clusters.

Besides IBM and HP, you're limited to rather modest systems sofar, for the Opteron. 16GB as you say. Intel have a good grip on both Dell and HP, so you're not really going to see any serious competition for their Itanium and Xeon from them. There is one big player that is currently working on very serious Opteron support hardware (chip sets etc), but basically Intel is probably farther down the road of chipsets and hardware support for Xeon64, since they intend to converge the hardware standards for Itanium and Xeon64. So my guess is that you will see bigger solutions for the Xeon64, sooner than for the Opteron.

Xeon64 will not perform as well in multiway systems as Opteron. Opteron scales excellently with number of CPUs. Xeon64 scales poorly. I also strongly suspect 64 performance will not be better than 32-bit performance from the Xeon. Perhaps even worse. There are rumours, Xeon64 performance is very poor with more than 4GB RAM. For a couple of years, at least, I think you can expect the Opteron to be a considerably better CPU than Xeon64.
Xeon64 is also just released, but I suppose we'll see hardware soon enough from Intel's partners, Dell and HP.

The choices are hard here. Opteron is a better cpu than Xeon64. But I expect hardware soon to be more abundant for Xeon64. And solutions matching your needs seem more likely to come there. On the other hand, I don't think Intel will ever allow Xeon64 to become a threat to their Itanium, as long as that is still alive. Opteron will OTOH.

I think you're a little bit early for large X86-64 systems, unless you consider Linux clusters.
On the other hand, one advantage of the more mainstream and cheaper x86-64 solutions is that you can and should replace your hardware more often. Here, the performance is not defined so much by cost, as by age. The major attraction of x86-64, is not hardware for future needs now, but that the future hardware for your needs then, will be more affordable.

Ideally, you're not moving into x86-64 from Sun, but from PCs and Intel 32-bit servers.

 

[joecool]

thanks, matthias, this all makes sense. i was able to learn some more after i posted my questions and everything you say backs up what i think i now know. basically, there are only two competing 64-bit platforms - the itanium (based on ia-64?) and amd-64, which is used on opterons, athlon 64's, and the future xeon 64's - like you said. there are some machines available now that support 16gb/processor so either the 4gb dimms are available or they're putting more than 4 slots/cpu on the boards.

our particular application are running various tools in asic development, our asics are getting so big that some of the tools need up to 16gb to run. naturally we anticipate this will get even bigger on our upcoming projects. right now we use a couple of sun boxes to run the big jobs buts suns are EXPENSIVE and SLOW compared to pc's; since we do everything else on linux x86 boxes we'd like to migrate these jobs as well. seems like opterons are the better bet for us given the backwards compatibility with 32-bit apps; since some of our apps don't yet support 64-bit x86 we could still run them on these big boxes, but just in 32-bit mode.

thanks again for the info!

joe

 

[CTho9305]

Originally posted by: joecool
our particular application are running various tools in asic development, our asics are getting so big that some of the tools need up to 16gb to run. naturally we anticipate this will get even bigger on our upcoming projects. right now we use a couple of sun boxes to run the big jobs buts suns are EXPENSIVE and SLOW compared to pc's; since we do everything else on linux x86 boxes we'd like to migrate these jobs as well. seems like opterons are the better bet for us given the backwards compatibility with 32-bit apps; since some of our apps don't yet support 64-bit x86 we could still run them on these big boxes, but just in 32-bit mode.

thanks again for the info!

joe

For what it's worth, we run our design tools on various AMD64 boxes (lots of quad opterons w/8GB ram, som slower machines too), and they perform great. We also migrated from Sun boxes.

edit: I should probably mention that "we" is AMD, so there was a large cost advantage to going with AMD processors. Of course, that doesn't mean AMD64 wasn't the best choice anyway. It's convenient running the same OS and software setup on the desktops as the cluster machines - something you couldn't do as easily with Itanium.

 

[jhu]

does that mean intel could also have been using sun also?

 

[CTho9305]

Originally posted by: jhu
does that mean intel could also have been using sun also?

I think they write a lot more of their own tools. AMD probably used Sun because all the good spice engines / design tools ran on Solaris.