Help me step off the ledge (future proofing, ECC, narrow ILM and sketchy retailers)

[tynopik]

I really really really hate moving systems.

I really hate corrupted data and unstable systems.

I hate noise

Given those constraints, I wanted to construct a system that would last me a decade (or as close as possible)

Fortunately I'm not a gamer and don't even hit the CPU that hard (currently running a stock Q6600 that I'm OK with speed wise). What does always force me to upgrade is memory. Currently I have 8GB and that's not enough when running several VMs (development work).

For ECC memory, the E3 Xeons look interesting, but only support 32GB, which just doesn't seem very future proof. Also, there's still enough complaints about people not being able to actually run 4 x 8GB DIMMs in systems that I'm somewhat leery of it.

Which brings me to the E5 Xeon series. I like 8 memory slots, I like that registered DIMMs go above 8GB and I like the concept of registered DIMMs reducing the load on the memory controller, making it less likely to have a problem fully populating the slots.

I'm not interested in dual CPUs and only mildly interested in >4 cores, thus there's only one CPU for me: The E5-1620 (4c, 3.6/3.8)

Unfortunately, the E5-1620 is very hard to find in stock at reputable dealers at reasonable prices.

Also, I want a motherboard that supports 8 slots of registered memory and has at least one physical PCIe x16 slot. The only company that makes single-processor boards that meet this criteria is Supermicro. Unfortunately Supermicro has chosen to give all their boards in that category a Narrow ILM CPU socket.

Nobody makes a Narrow ILM cpu cooler.

Well, if you look on Newegg, Dynatron has some absolute garbage available. Frankly I have no idea how some of those passive coolers are even supposed to work.

Actually, Supermicro does make a cooler, the SNK-P0050AP4. But as you can see from the review, it's fairly noisy. Fortunately he has a solution: replacing the fan with a Noctua 92mm unit. Combined that's $66 for the HSF, which is quite a lot to swallow for someone who has always used the stock fan before.

And that doesn't even mention the non-standard mounting points on the board. Here's some comments from around the internet:

"The mounting points are unusual, and my consumer case needed some modding"

"The issue with DIMM slots noted in other review is most likely due to bent pins or grounding because of contact with mounting offsets in case. Putting memory in the slots 4 A/B lead to a non-post with one of my two new systems. I was ready to RMA, but decided it was too much of a coincidence having exactly the same problem, so I pounded flat an unaligned mounting support in my case, and things have been working great for the past 24 hours"

"Non standard ATX holes, although this seems to be common to Supermicro. Check your case throughly."

I have the original CM Stacker and this makes me somewhat nervous.

Putting it all together, here is my projected build:

$317 E5-1620 3.6/3.8
http://www.wiredzone.com/Intel-CM80...eon-E5-1620-4-Core-Socket-2011~32027900~0.htm

$290 Supermicro X9SRA
http://www.newegg.com/Product/Produc...82E16813182336

$44.24 Supermicro SNK-P0050AP4 Heatsink
http://www.wiredzone.com/Intel-CM80...eon-E5-1620-4-Core-Socket-2011~32027900~0.htm

$22 - Noctua NF-B9-1600 92mm Case Fan
http://www.newegg.com/Product/Produc...82E16835608005

$60 x 2 - Kingston 8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1333 Server Memory DR x4 1.35V Hynix C Model KVR13LR9D4/8HC
http://www.newegg.com/Product/Produc...82E16820239160

$793


Just for comparison, I specced out a similar E3 build:

$245 - E3-1230 V2 (Ivy Bridge) 3.3/3.7
http://www.newegg.com/Product/Produc...82E16819117286

$190 - Intel S1200BTSR
http://www.newegg.com/Product/Produc...82E16813121619

$73 X 2 - Kingston 8GB 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 Server Memory Intel Model KVR13E9/8I
http://www.newegg.com/Product/Produc...82E16820239116

$30 - Cooler Master Hyper 212 Plus RR-B10-212P-G1
http://www.newegg.com/Product/Produc...82E16835103065

$611


and if I was willing to do away with ECC altogether

$300 - i7-3770 3.4/3.9
http://www.newegg.com/Product/Produc...82E16819116502

$67 - Foxconn B75M
http://www.newegg.com/Product/Produc...82E16813186228

$65 - Corsair 16GB (2 x 8GB) DDR3 1333 CMV16GX3M2A1333C9
http://www.newegg.com/Product/Produc...82E16820233270

$30 - Cooler Master Hyper 212 Plus RR-B10-212P-G1
http://www.newegg.com/Product/Produc...82E16835103065

$462

Basically I'm paying $150 more just for ECC and then ANOTHER $182 for confidence in being able to upgrade to 32GB without problem plus the ability to go even higher.

Which altogether is a $332 premium.

Which I would really rather not pay.

Maybe I should just get the E3 and stuff it full right now so any problems become immediately apparent and just assume 32GB will last me long enough?

I just really do love the idea of registered DIMMs and I can picture a system built around that Supermicro board lasting me forever.

 

[riversend]

I'm probably not a lot of help here, but given how important the work is to you that would seem to override your vehemence toward moving systems.

The price scenarios seem to be a pretty good incentive to save the cash for upgrades later on down the road. If the Q6600 lasted you 6+years then I would think the 3770 would do the same. By that time you could likely upgrade the mobo/CPU/RAM again.

You have definitely put some difficult constraints on your build.

 

[tynopik]

If the Q6600 lasted you 6+years then I would think the 3770 would do the same.

I'm not going to get another system without ECC, so unfortunately the 3770 is not an option. I just priced it out for a comparison.

Perhaps someone can convince me AMD is the way to go . . .

 

[crazymonkeyzero]

I think I can try to be of some help since I just made 2 ECC based worskstations for our lab (since data integrity is kinda important here) in the past few months. One was based on the Intel Xeon E5 2630 and the other was the Intel Xeon E3 1275V2. Personally I think the E3 is much better for the price(it performs about 25% faster for the software we run), due to higher clock speeds, and improved instruction sets and ipc of ivy bridge, However if you run a lot of high memory bandwidth intensive tasks such as photoshop,and video editing software, go with the E5, lga 2011 route.

Here is the basic E3 setup I have. I have included the relevant parts.

Intel Xeon E3 1275 V2. (this is essentially a i7 3770 with ecc support) I am using integrated graphics which are good enough for me, but you can easily instead buy the 1270 v2 and add a dedicated gpu as well.
http://www.newegg.com/Product/Produc...e3%201275%20v2

Asus P8C WS (despite the review on newegg, it's a very good board that supports unbuffered ECC. Don't listen to trolls)

http://www.newegg.com/Product/Produc...sus%20p8c%20ws

Kingston DDR3 1600 Memory (4gb) I choose to buy 4 sticks, totaling 16gb
(which is future proof for most people as current baseline standard is still 4gb, while 8gb is premium for most.)
http://www.newegg.com/Product/Produc...82E16820239223

Case: Fractal Design Define R4 (very silent, filtered, and looks awesome)
http://www.newegg.com/Product/Produc...ine+r4&x=0&y=0

I can 100% guarantee, the parts above will be compatible, b/c I'm using them.

Sidenote:

The only downside of Xeons in general is that they are clock locked; in exchange for the ability to use ecc, you can't OC for beans unlike the i5 and i7 which can have up to 20-30% more juice squeezed out of them for additional performance~so I suggest getting as high a clock speed as possible, within reasonable price.

 

[imagoon]

What is the reason you need ECC? Mostly curious.
I built a Z77 Ivy Bridge running 32GB of RAM without issue so I personally don't know why so many people have issues with 32GB other than not knowing how to set the BIOS. On mine all I had to do is go in there and set it to "Use RAM SPD Profile 2" and it set the voltage / bus speeds / CAS etc etc for me. (Default was an always start profile of 800mhz etc, Profile 2 was the rated speeds.)

This specific test rig has been running ESXi5 with 22-28GB of RAM in use typically. I pushed it to 31GB during testing and it was stable while slow because I had caused ESXi to start swapping.

 

[Insert_Nickname]

Here is the basic E3 setup I have. I have included the relevant parts.

Intel Xeon E3 1275 V2. (this is essentially a i7 3770 with ecc support) I am using integrated graphics which are good enough for me, but you can easily instead buy the 1270 v2 and add a dedicated gpu as well.
http://www.newegg.com/Product/Produc...e3%201275%20v2

Asus P8C WS (despite the review on newegg, it's a very good board that supports unbuffered ECC. Don't listen to trolls)

http://www.newegg.com/Product/Produc...sus%20p8c%20ws

Kingston DDR3 1600 Memory (4gb) I choose to buy 4 sticks, totaling 16gb
(which is future proof for most people as current baseline standard is still 4gb, while 8gb is premium for most.)
http://www.newegg.com/Product/Produc...82E16820239223

Case: Fractal Design Define R4 (very silent, filtered, and looks awesome)
http://www.newegg.com/Product/Produc...ine+r4&x=0&y=0

I can 100% guarantee, the parts above will be compatible, b/c I'm using them.

About that would be my recommendation too... 😉

Remember a RAID5/6 array and a robust backup system...

 

[tynopik]

What is the reason you need ECC?

I'm sick of dealing with corruption issues and I just don't trust regular memory

I built a Z77 Ivy Bridge running 32GB of RAM without issue so I personally don't know why so many people have issues with 32GB other than not knowing how to set the BIOS.

It could just be that such a build really stretches the limits of the memory controller, so some combinations of cpu/board/memory have nary a problem while other combinations just never work quite right

 

[tynopik]

Intel Xeon E3 1275 V2.

The main problem I have with the E3 series is that I'm just not sure 32GB will be enough

My Q6600 system started with 2GB, upgraded to 4GB, upgraded again to 8GB and now that's not enough.

If I need 16GB now, how long will 32GB last?

One was based on the Intel Xeon E5 2630 and the other was the Intel Xeon E3 1275V2. Personally I think the E3 is much better for the price(it performs about 25% faster for the software we run), due to higher clock speeds, and improved instruction sets and ipc of ivy bridge

E5-2630 - 6c 2.3/2.8
E3-1275V2 - 4c 3.5/3.9

yeah, unless you fully utilize all 6 cores, the E3 is going to be quite a bit faster

should have gotten an E5-1620 😉

 

[imagoon]

I'm sick of dealing with corruption issues and I just don't trust regular memory



It could just be that such a build really stretches the limits of the memory controller, so some combinations of cpu/board/memory have nary a problem while other combinations just never work quite rite

I doubt it is the memory controller. It is on the CPU die. I would lean towards crap ram or system boards.

To each their own. ECC just detects errors, it doesn't fix the underlying cause.

 

[crazymonkeyzero]

The main problem I have with the E3 series is that I'm just not sure 32GB will be enoughMy Q6600 system started with 2GB, upgraded to 4GB, upgraded again to 8GB and now that's not enough.
If I need 16GB now, how long will 32GB last?

If you want to run a ridiculous number such as 20 vms, of coarse, you need more ram, but for most people, 16gb is still a lot of headroom. You can probably get by with 12gb. (about 2gb ram per vm to run smoothly...)

TBH, your cpu will be more of a bottleneck in that time than the memory is lol. (even if you bought the best darn chip intel has to offer right now)

E5-2630 - 6c 2.3/2.8
E3-1275V2 - 4c 3.5/3.9

yeah, unless you fully utilize all 6 cores, the E3 is going to be quite a bit faster

should have gotten an E5-1620 😉

Actually, our software does utilize all the cores...the hex core is still slower. Wish 1620 was in stock then :/.

 

[tynopik]

You can probably get by with 12gb.

I'm currently 'getting by' with 8, so no doubt i could get by with 12

But back to my first point, I really really really hate to switch system, so I'm looking for something that can last 10 years.

So it's not so much if that's enough RAM now, it's will it be enough 6, 8 years down the road?

 

[crazymonkeyzero]

So it's not so much if that's enough RAM now, it's will it be enough 6, 8 years down the road?

That really depends...will you be exponentially increasing the number of vms you run down that road? If so, then go for 32gb. If not 16 is enough.


here, I found 8gb ecc unbufferd sticks. Cant guarantee they'll work as I haven't tested them. But they probably will as asus has that mem-OK button on their boards which ensure like 99% modules chosen (within reason), work. You can ask them to verify.

http://www.newegg.com/Product/Produc...=1&srchInDesc=

 

[Zap]

Which I would really rather not pay.

That could be a problem. You have very specific and expensive desires in your component choices. Life is full of compromises. Luckily you can choose yours.

 

[tynopik]

That could be a problem. You have very specific and expensive desires in your component choices. Life is full of compromises. Luckily you can choose yours.

Yes, well I was hoping someone could offer a better/cheaper solution (or convince me that my desires are silly 😉 )

the 'very specific and expensive desires' basically comes down to supporting >32GB of ECC RAM

 

[imagoon]

Yes, well I was hoping someone could offer a better/cheaper solution (or convince me that my desires are silly 😉 )

the 'very specific and expensive desires' basically comes down to supporting >32GB of ECC RAM

You could look at something like this that is "more standard." You are seeing the same issue I did when building my test rig. I landed on z77 + 32GB of RAM because the boards that had more than 4 RAM slots typically had a very large markup over 4.

http://www.newegg.com/Product/Produc...79%20Extreme11

That board will do 64GB unbuffered ECC via 8GB sticks for example. It also requires the Sandybridge-E which can give you 8 cores + 8 HT's if you really wanted it. The price reflects it though. That system board costs more than my entire i5-3750k + 32GB RAM + PSU + Case. At that price point I was just planning on building 2 systems and pairing them via vcenter.

 

[tynopik]

You could look at something like this that is "more standard."

yeah, I think I'll stick with the Supermicro X9SRA for less than half the price

See, I really did try to find the most affordable solution

At that price point I was just planning on building 2 systems and pairing them via vcenter.

This does bring up an interesting point. Currently I just use VMWare Workstation (and an old one at that). On and off I've tried to look at the different products that VMWare offers, but the descriptions are either so filled with marketing speak as to be meaningless or so filled with VMWare technojargon that it's impossible to decipher.

So, can anyone offer a brief overview of what VMWare offers for remote VMs that is affordable for mere mortals? I've heard a lot about ESXi but I also thought I saw a mention that the new regular VMWorkstation allows VMs to be run remotely.

In the past I've used the VNC option in VMWare to run machines remotely, but I really need a more integrated solution with drag and drop and sound that basically acts as if it was running locally.

 

[Zap]

Yes, well I was hoping someone could offer a better/cheaper solution (or convince me that my desires are silly 😉 )

the 'very specific and expensive desires' basically comes down to supporting >32GB of ECC RAM

I don't think there are any cheaper or "better" solutions, so maybe we can convince you that your desires are silly? 😛

But first how about you answer these questions?

Other than that, let's summarize and support/debunk what your "desires" are.

I really really really hate moving systems.

I really hate corrupted data and unstable systems.

I hate noise

Given those constraints, I wanted to construct a system that would last me a decade

So, you want an appliance that you can use for 10 years without having to change anything, and have it just work?

Even the Titan Supercomputer is on a 4 year upgrade cycle.

I think you need to drop the 10 years down to 5 years.

For stability, just avoid the hardcore "enthusiast" stuff such as high speed RAM.

Low noise is easy to do by avoiding noisy parts and not overclocking. Extremely low noise to the point of not hearing it if you aren't actively concentrating on hearing the noise is not too difficult with some care in choosing parts. Truly "silent" as in physically making zero noise is very difficult to do, and you'll never get rid of electrical noise.

Here's the deal. You can spend tens of thousands of dollars and still not get a computer which lasts 10 years without data corruption or parts failures. There is no amount of money in the world that can guarantee what you want for a decade.

Focus on your achievable and affordable needs.

 

[tynopik]

So, you want an appliance that you can use for 10 years without having to change anything, and have it just work?

Even the Titan Supercomputer is on a 4 year upgrade cycle.

I don't have the needs of a supercomputer

I think you need to drop the 10 years down to 5 years.

I'm using a system that is 5 years old and is 'OK' except for memory. Considering how the pace of improvement is slowing down, especially at the high-end as more effort is spent on power efficiency and mobile, 10 doesn't seem like an unattainable goal.

For stability, just avoid the hardcore "enthusiast" stuff such as high speed RAM.

The Google study showed one-third of their machines experienced memory errors every year, and they were using server-grade hardware. They also found that memory deteriorates with age. ECC doesn't seem an unreasonable precaution if you're concerned about corruption.

Low noise is easy to do by avoiding noisy parts and not overclocking.

check. no overclocking, replacing the heatsink fan with a quieter model, I'll also be using a passive video card and a reasonably quiet power supply.

Truly "silent" as in physically making zero noise is very difficult to do, and you'll never get rid of electrical noise.

Yeah, ok, I'm not aiming for that

Focus on your achievable and affordable needs.

Supprting >32GB of ECC ram doesn't seem like it is asking that much.

 

[mfenn]

Supprting >32GB of ECC ram doesn't seem like it is asking that much.

You're right about that. The problem isn't that any one of your requirenents is hard to meet, it's that the intersection of those requirements is hard meet.

 

[mfenn]

I really really really hate moving systems.

So here's a fundamental question we need to answer. What about moving systems do you hate? Installing hardware, reinstalling software, something else entirely?

Depending on the answer to this question, it may turn out that building a 10-year hardware platform is a solution to the wrong problem entirely.

Oh yeah, and keep in mind that the Q6600 is only 5 years old. 10 years is an original sub-2.0Ghz P4 or an Athlon XP.

Also, I want a motherboard that supports 8 slots of registered memory and has at least one physical PCIe x16 slot. The only company that makes single-processor boards that meet this criteria is Supermicro. Unfortunately Supermicro has chosen to give all their boards in that category a Narrow ILM CPU socket.

Nobody makes a Narrow ILM cpu cooler.

Well, if you look on Newegg, Dynatron has some absolute garbage available. Frankly I have no idea how some of those passive coolers are even supposed to work.

Actually, Supermicro does make a cooler, the SNK-P0050AP4. But as you can see from the review, it's fairly noisy. Fortunately he has a solution: replacing the fan with a Noctua 92mm unit. Combined that's $66 for the HSF, which is quite a lot to swallow for someone who has always used the stock fan before.

I can shed some light on this. These types of motherboards and coolers are primarily designed to go into 1U rackmount cases with a LARGE amount of linear airflow. The case provides the airflow for the whole system with a single bank of fans, everything else has passive heatsinks. Before you get excited about this, 1U systems are loud as hell.

 

[tynopik]

So here's a fundamental question we need to answer. What about moving systems do you hate? Installing hardware, reinstalling software, something else entirely?

I don't like change in general, so a new OS always takes a while to get adjusted to. Hence why now seems like a good time as Windows 8 just came out. Hopefully I can adjust once and then ride it for a good long while. Also some of the programs I use tend to not work quite right on newer versions so upgrading always brings a certain sense of trepidation.

But even if it was the same OS, getting everything configured the way I like on a system takes forever. For instance getting firefox setup correctly is going to be a pain.

Also moving systems screws with the activations in my VMWare operating systems, so I prefer to do that as little as possible.

Overall it just takes so freaking long to get settled into a new system, I prefer to avoid it.

What I do isn't really processor or video intensive, so there's no reason a system can't last me a good long while.

Oh yeah, and keep in mind that the Q6600 is only 5 years old. 10 years is an original sub-2.0Ghz P4 or an Athlon XP.

I can guarantee that the next 10 years won't bring even half as large a leap in general computing performance. Perhaps in special highly parallel code that can use GPUs, but that won't impact me anyways.

 

[imagoon]

I don't like change in general, so a new OS always takes a while to get adjusted to. Hence why now seems like a good time as Windows 8 just came out. Hopefully I can adjust once and then ride it for a good long while. Also some of the programs I use tend to not work quite right on newer versions so upgrading always brings a certain sense of trepidation.

But even if it was the same OS, getting everything configured the way I like on a system takes forever. For instance getting firefox setup correctly is going to be a pain.

Also moving systems screws with the activations in my VMWare operating systems, so I prefer to do that as little as possible.

Overall it just takes so freaking long to get settled into a new system, I prefer to avoid it.

What I do isn't really processor or video intensive, so there's no reason a system can't last me a good long while.



I can guarantee that the next 10 years won't bring even half as large a leap in general computing performance. Perhaps in special highly parallel code that can use GPUs, but that won't impact me anyways.

Well for one... use the profile mover tool that is built in to windows... Firefox has a built in method to move it's profile (it will move with the profile mover also). "Move" the VMware VM's don't "copy" them. Relocate them to the new machine, select "I moved it" and it leaves the VMX as is in workstation. Moving systems is mildly annoying but it sounds like you might be making it even harder on yourself.

 

[tynopik]

"Move" the VMware VM's don't "copy" them.

doesn't matter, after having been moved around as much as these have been, changing the cpu arch out from under them is enough to trigger it

 

[Zap]

I'm using a system that is 5 years old

So why not use it for another 5 years to make it your 10?

Oh right, not enough RAM. Did you not predict that 5 years ago?

Makes you think... what are you unable to predict when building a system today that you might decide you need in 5 years.

See my sig.

The Google study showed one-third of their machines experienced memory errors every year, and they were using server-grade hardware. They also found that memory deteriorates with age. ECC doesn't seem an unreasonable precaution if you're concerned about corruption.

This tells me two things.

First is that no matter how much money you spend on hardware, you may have a 1 in 3 chance of memory errors every year.

Second, since memory deteriorates with age, your hopes of keeping it unchanged for 10 years would be at odds with you not wanting memory issues.

So, according to what you said (as reported by Google on their own servers) you will probably not be able to avoid memory errors.

Supprting >32GB of ECC ram doesn't seem like it is asking that much.

If it isn't, then the asking PRICE shouldn't be too much either. You've already priced it, right? Then go for it!

tynopik, you have plenty of information already. If you get what you "want" then you will have to pay extra for it (compared to consumer level hardware). You can't get around that fact. If you can afford it and want it badly enough, then take a hard look at what crazymonkeyzero wrote, since he's one of the few here who seems to have actual experience in building computers with high availability. If you can't afford it or don't want it badly enough to pay the price, then you may want to consider curtailing your wish list.

 

[tynopik]

First is that no matter how much money you spend on hardware, you may have a 1 in 3 chance of memory errors every year.

Second, since memory deteriorates with age, your hopes of keeping it unchanged for 10 years would be at odds with you not wanting memory issues.

So, according to what you said (as reported by Google on their own servers) you will probably not be able to avoid memory errors.

But with ECC those errors are CORRECTABLE

while 1 in 3 machines had errors, only 1 in 100 had UNCORRECTABLE errors

so ECC can indeed save your bacon

tynopik, you have plenty of information already. If you get what you "want" then you will have to pay extra for it (compared to consumer level hardware).

Well as has been pointed out, there are other ways of attacking the problem. A second system to run VMs on is intriguing. An AMD evangelist might pop up with some options.

Just wanted to post to make sure I wasn't missing anything before making a decision