Are Intel's CPU memory frequency limitations "hard"?

[eLiu]

Hi all,
Say intel lists memory compatibility for a given processor (e.g., Xeon E5-2620) as DDR3-800, 1066, and 1333. Is this a hard limit (like the memory controller can't go any faster)? That is, if I put in DDR3-1600, I will see no gain in bandwidth?

I'm pretty sure this is untrue--no hard limit. If that is correct, then could there be other limitations that prevent me from running DDR3 at 1600mhz with E5-2620? Like maybe these expensive server-grade motherboards (e.g., those by supermicro) won't allow me to 'overclock' the memory controller?

I'm asking b/c someone I know needs to purchase a workstation, but it *must* be a dual-socket machine assembled by Penguin Computing (go go red tape). His application is memory bandwidth-limited, so DDR3-1600 (or higher) would be a good boost for him. But I don't want to advise him to buy fast memory if the CPU, motherboard, BIOS, or whatever will prevent the faster RAM from running above 1333mhz.

Thanks!

 

[IntelEnthusiast]

On the Intel® Xeon® E5-2620 in order to use faster memory (DDR 3 1600) you have to overclock the memory in the Bios if the board supported overclocking of memory. Do you really think you will come anywhere close to filling the bandwidth on quad channel DDR 3 1333 per processor?

 

[T_Yamamoto]

On the Intel® Xeon® E5-2620 in order to use faster memory (DDR 3 1600) you have to overclock the memory in the Bios if the board supported overclocking of memory. Do you really think you will come anywhere close to filling the bandwidth on quad channel DDR 3 1333 per processor?

+1

 

[eLiu]

On the Intel® Xeon® E5-2620 in order to use faster memory (DDR 3 1600) you have to overclock the memory in the Bios if the board supported overclocking of memory. Do you really think you will come anywhere close to filling the bandwidth on quad channel DDR 3 1333 per processor?

It's not really memory overclocking if he purchases DDR3-1600 memory in the first place; more like memory controller overclocking. Should we expect there to be motherboard limitations (remember, these are server boards since this will be a 2 socket machine) that will prevent us from running at 1600 speeds? Or is support for DDR3-1600 pretty widespread nowadays?

And yes, this application is bandwidth-limited and it will *easily* use all of the bandwidth available in quad-channel DDR3-1600; in fact we could consume something like 5x-10x that bandwidth before floating point throughput becomes the limiting factor. I wouldn't have asked this question if I didn't have a need. The application is a finite element solver for computational fluid dynamics applications; it involves manipulating large (=once parallelized, several gigabytes per core), sparse matrices, where the cost of floating point computation is very minor compared to the cost of reading matrix entries out of memory.

edit: whoa, I didn't realize your title is "intel representative." Cool. I had no idea Intel guys read these forums.

 

[Charles Kozierok]

Basically, Intel says "we certify the memory controller to run at X". And then it is up to the motherboard maker to decide if they also want to certify at "X" or use a higher number.

So really, you need to check the specs on the motherboard.

Maybe consider a dual CPU setup?

 

[eLiu]

Basically, Intel says "we certify the memory controller to run at X". And then it is up to the motherboard maker to decide if they also want to certify at "X" or use a higher number.

So really, you need to check the specs on the motherboard.

Maybe consider a dual CPU setup?

Maybe I should've been asking this in the motherboard forum then?

Yes, it is already going to be a dual-cpu setup; that's what I meant in the OP when I said it must be a dual-socket machine (assembled by penguin). I can't tell you which motherboard is used since the site:
http://www.penguincomputing.com/
doesn't specify. Looking at dual-socket boards on newegg, it looks like DDR3-1600 support definitely exists on dual socket LGA-2011 boards (Asus, EVGA, Tyan, Supermicro), but it isn't everywhere (e.g., some intel-branded boards list 1333mhz, particularly cheaper ones).

But considering how much these penguin guys are charging, I should be able to have my friend get them to specify a model number.k

 

[Charles Kozierok]

Sorry, should have noticed that you already specified dual CPU.

Ya, find out what the motherboard is exactly. That will help a great deal. But to be perfectly honest, my guess is that even if the application is memory-limited, the difference in overall performance between DDR3-1333 and DDR3-1600 is not going to be as much as you'd think.

 

[DominionSeraph]

The application is a finite element solver for computational fluid dynamics applications

Isn't this something that ports rather well to GPUs?

With Big Kepler on the horizon...

 

[exar333]

Did a bit of research based on a similar Intel board, the S2600GZ. Here is an excerpt from the manual and a few links:

http://www.legitreviews.com/article/1871/2/

http://download.intel.com/support/motherboards/server/sb/g24881004_s2600gzgl_tps_r1_1.pdf

BIOS section Page 132:
Memory Operating Speed Selection
Option Values: Auto
800
1066
1333
1600
Help Text: Force specific Memory Operating Speed or use Auto setting.
Comments: Allows the user to select a specific speed at which memory will
operate. Only speeds that are legitimate are available, that is, the user can only specify
speeds less that or equal to the auto-selected Memory Operating Speed. The default
Auto setting will select the highest achievable Memory Operating Speed consistent with
the DIMMs and processors installed.
1600 MT/s memory speed is available only on certain models

This leads me to believe that ONLY speeds up to the rated amount of the CPU would be selectable. I would suggest getting the faster quad-core rated at 1600 if it is an absolute requirement. Penguin should be able to confirm this 100%, but it looks like you are limited to the rated memory speeds from what I can see.

Hope that helps!

 

[eLiu]

Sorry, should have noticed that you already specified dual CPU.

Ya, find out what the motherboard is exactly. That will help a great deal. But to be perfectly honest, my guess is that even if the application is memory-limited, the difference in overall performance between DDR3-1333 and DDR3-1600 is not going to be as much as you'd think.

I've tested this on a slightly older processor--i7 950. The memory limited part of the program, which can compose as much as 90% of the runtime, scales linearly with memory frequency at least over tested values of 800mhz, 1066mhz, and 1333mhz.

The simplified 'thought'-model that's relevant here is the following... say I divide 1GB of doubles into a bunch of 100x100 blocks. I randomly select blocks (without replacement) and operate on them (say I sum all the numbers; it doesn't matter). The time needed to traverse the entire 1GB of data closely models the time we need to do one round of matrix operations.

Isn't this something that ports rather well to GPUs?

With Big Kepler on the horizon...

Yes and no. Finite element simulations usually have 2 major time-costs: residual evaluation and linear solve. Not every finite element scheme will need the linear solve, but ours does. Residual evaluation parallelizes readily (each element is nearly independent and even within an element there's additional parallelism) so this part will run super fast on a GPU.

The linear solve part does not do so well on GPUs. "Exact" solves (even on sparse systems) are too expensive to be practical, leaving us with iterative methods. The existing literature on ILU-preconditioned GMRES (method we use) shows that you get speedups of like 5x, tops. So that's a ton of coding effort/headache for not really that much gain. And having worked with GPUs before, I'm of the opinion that they're still a bit too bleeding edge for us anyway.

Did a bit of research based on a similar Intel board, the S2600GZ. Here is an excerpt from the manual and a few links:

http://www.legitreviews.com/article/1871/2/

http://download.intel.com/support/motherboards/server/sb/g24881004_s2600gzgl_tps_r1_1.pdf

This leads me to believe that ONLY speeds up to the rated amount of the CPU would be selectable. I would suggest getting the faster quad-core rated at 1600 if it is an absolute requirement. Penguin should be able to confirm this 100%, but it looks like you are limited to the rated memory speeds from what I can see.

Hope that helps!

Interesting, thanks for providing that information. Unfortunately, getting a pair of CPUs with 1600mhz as a 'feature' adds like 1000-1500 to the cost (or something like that), which puts him over budget.

I was asking about this b/c his budget can fit DDR3-1600 RAM, so the thought process was like:
1) if DDR3-1600 will run with the E5-2620, then there's no reason not to do it
2) if it won't, then we shouldn't waste money buying 1600 when it'll be running at 1333.

But yes, I will look into the motherboard model and see if I can glean specific information. My experience with "enthusiast" boards leads me to believe that it is at least possible that the motherboard will allow you to violate the 'default' settings of ram/mem controller.

 

[exar333]

I've tested this on a slightly older processor--i7 950. The memory limited part of the program, which can compose as much as 90% of the runtime, scales linearly with memory frequency at least over tested values of 800mhz, 1066mhz, and 1333mhz.

The simplified 'thought'-model that's relevant here is the following... say I divide 1GB of doubles into a bunch of 100x100 blocks. I randomly select blocks (without replacement) and operate on them (say I sum all the numbers; it doesn't matter). The time needed to traverse the entire 1GB of data closely models the time we need to do one round of matrix operations.



Yes and no. Finite element simulations usually have 2 major time-costs: residual evaluation and linear solve. Not every finite element scheme will need the linear solve, but ours does. Residual evaluation parallelizes readily (each element is nearly independent and even within an element there's additional parallelism) so this part will run super fast on a GPU.

The linear solve part does not do so well on GPUs. "Exact" solves (even on sparse systems) are too expensive to be practical, leaving us with iterative methods. The existing literature on ILU-preconditioned GMRES (method we use) shows that you get speedups of like 5x, tops. So that's a ton of coding effort/headache for not really that much gain. And having worked with GPUs before, I'm of the opinion that they're still a bit too bleeding edge for us anyway.



Interesting, thanks for providing that information. Unfortunately, getting a pair of CPUs with 1600mhz as a 'feature' adds like 1000-1500 to the cost (or something like that), which puts him over budget.

I was asking about this b/c his budget can fit DDR3-1600 RAM, so the thought process was like:
1) if DDR3-1600 will run with the E5-2620, then there's no reason not to do it
2) if it won't, then we shouldn't waste money buying 1600 when it'll be running at 1333.

But yes, I will look into the motherboard model and see if I can glean specific information. My experience with "enthusiast" boards leads me to believe that it is at least possible that the motherboard will allow you to violate the 'default' settings of ram/mem controller.

The Xeon's make it much more difficult vs. an i7/i5/is solution. Almost any board on the consumer side will definitely allow you to OC the RAM independantly, and often beyond the rated limits for the board even.

The Xeon SKUs differ so much, and they are locked-down much more, that it becomes a lot more difficult. If you were able to build a workstation from something like the EVGA SR-2 or SR-X (dual socket enthusiast MBs that use Xeons) it might be a different story. Wish you the best of luck, and I would be interested to see what you find out!

 

[jeremiahbarrar]

Hi all,
Say intel lists memory compatibility for a given processor (e.g., Xeon E5-2620) as DDR3-800, 1066, and 1333. Is this a hard limit (like the memory controller can't go any faster)? That is, if I put in DDR3-1600, I will see no gain in bandwidth?

I'm pretty sure this is untrue--no hard limit. If that is correct, then could there be other limitations that prevent me from running DDR3 at 1600mhz with E5-2620? Like maybe these expensive server-grade motherboards (e.g., those by supermicro) won't allow me to 'overclock' the memory controller?

I'm asking b/c someone I know needs to purchase a workstation, but it *must* be a dual-socket machine assembled by Penguin Computing (go go red tape). His application is memory bandwidth-limited, so DDR3-1600 (or higher) would be a good boost for him. But I don't want to advise him to buy fast memory if the CPU, motherboard, BIOS, or whatever will prevent the faster RAM from running above 1333mhz.

Thanks!

What you really need to look at is RAM timings. Your trying to obtain the maximum bandwidth (GB/second) if I understand correctly. 1333 can be faster than 1600 depending on the timings. 1600 times per second the ram is queried. Timings determine the queries it takes for a given data read or write. 1600 is typically around 10 queries. 1333 is lower, around 8. If you find ram with 6 or 7 timing ratings, it will be as good or better than average 1600 RAM. This will also ensure stable processor operation. This is an old thread but I found it with google while reaserching the E5-2620. I thought this reply would be helpful to others.

 

[pantsaregood]

Timings have a very minimal effect on RAM at this point.

The ASUS Z9PE-D8 WS should handle what you want, though. You may have to overvolt the memory controllers a bit, but this board will likely allow it. It is a server board sold as an enthusiast board, basically.

EDIT: The EVGA Classified SR-X will also work for this purpose.