A lot of people dismiss the idea of having a home server setup because it's two expensive. While it certainly can be, it doesn't have to be. I've been building/buying home servers for 6 years now. You can build your own ESXI lab with two servers and shared storage for under $1,000 if you know how to bargain shop. While most of this info doesn't apply specifically to an ESXI lab, most of it is geared with that in mind. If you just want prices/parts, skip to the end.
There's two major choices when building a home lab. You either build your own new "servers" or you buy used. There's 4 major considerations in this decision. Hardware Costs, Environmental Costs, Compatibility, and Features.
Hardware Cost: You can buy used Enterprise level equipment for far cheaper than you can build equally performing new equipment. For example, you can get a new AMD 8-Core Piledriver CPU for about $150. You can get a complete used HP or Dell server with a pair of Quad Cores for about $200. Used equipment has the advantage here.
Environmental Costs: That may be a poor way to word it, but there's three things people often don't think about when buying a "real" server. Noise, Heat, and Electricity. "Real" servers are designed to run in a temperature controlled data center, not your garage or living room. Some servers are quieter than others but typically they are going to be at least loud enough to annoy anybody in the room. They also generate a LOT of heat and don't always like running in a room kept at 80+ degrees. You can somewhat minimize this by being selective with your equipment choices but it's still going to be more of an impact than a new desktop PC. Last but not least is electricity. Servers are power hungry. Again, the newer your equipment the better it will be but you're still going to be using more power than a desktop PC, probably by a lot and you're probably running it 24x7. Admittedly I've got a pretty sizable lab by most standards, but prior to my efficiency rebuild, I was pulling 1,300 watts CONSTANT from my outlet. That's a LOT of power. Building your own has a massive advantage here.
Compatibility: As I mentioned above, you can get an 8 Core AMD processor for about $150. Right now it's probably sounding pretty tempting to just build a cheap box around that (or something similar) and call it good. Unfortunately it's not quite that simple. VMWare (and other Linux/Unix builds) can be a bit picky about the hardware they will support. For example I installed ESXI onto my media center box which runs an AMD APU on a Gigabyte mobo. ESXI did not recognize the onboard NIC's. Then what if you want to expand? A lot of home boards won't boot off a true add-on RAID card. You can certainly make it work if you're selective about your parts but you can't just run down to Microcenter/Fry's and buy whatever is on sale. Buying used equipment has the advantage here.
Features: Servers have a lot of features that the average home user or new IT fledglings are unaware of. It's up to you to decide what, if anything, these features are worth to you.
First up is the Intelligent Platform Management Interface (IPMI). Pretty much all servers have one (or at least the option of one) but each manufacturer comes up with their own name. HP calls theirs Integrated Lights Out (iLo), Dell calls theirs Dell Remote Access Console (DRAC), and IBM calls theirs Remote System Administrator (RSA) on older servers and Integrated Management Module (IMM) on newer ones. Typically these have their own dedicated network port on the server (Dell is the exception, a dedicated port is an upgrade option). The purpose of IPMI is to allow you to have physical level access from a remote system. IPMI allows you to physically power on and power off a system as well as access it's health status remotely. Some systems support additional features such as a remote console, Virtual Media (allows you to give the server access to a USB or DVD plugged into your desktop), Firmware Updating, and Power Consumption info although these features often required a license purchase.
Second is Serial Attached SCSI (SAS) support. As a general rule SAS drives are faster and more reliable than SATA drives. Most servers have SAS controllers and if you're buying used they are probably including at least a pair of SAS drives. SAS and SATA use the same physical connector however so even if you have a SAS controller, you can still use SATA drives if cost is more important than performance. However, it only works one way. SAS controllers can run SATA drives, but SATA controllers cannot run SAS drives.
Third is the NIC's. Most servers have at least two NIC ports, newer ones have four or sometimes more. These are typically Intel NIC's or sometimes Broadcom. Both of which are supported by ESXI (and most other Linux/Unix builds) which is good.
It is worth mentioning that you CAN get some or all of these features on high end workstation/server boards from companies like Supermicro or Tyan. But your cost will go through the roof if doing that.
If at this point you've decided to build your own, here's the things you want to keep in mind. When running VM's, RAM is your friend unless you like waiting a lot. Personally I configure all my Windows VM's with at least 4Gb of RAM. Linux/Appliance VM's I may go less. CPU requirements is going to vary greatly depending on your usage but I would try to stick with at least 8 threads. For your NIC's stick with Intel. If your board doesn't have Intel NIC's onboard, plan on buying an add-on NIC. See end of this post for recommendations. You'll want at least 2 GbE ports. More if you are running network storage. Lastly is the disk requirements. If you're only running one or two VM's or don't care about performance, this isn't an issue. But if you're running half a dozen or more VM's or have expectations on disk performance keep in mind you'll either need several spindle drives or use SSD's.
If you've decided to buy used, here's some general recommendations.
1) Avoid going with anything older than a Xeon 5000 or Opteron 2300. They're slow, have a limited number of threads, and replacement parts are prohibitively expensive. Xeon 5500/5600 series or Opteron 6100 series would be greatly preferred for multiple reasons which I'll explain further below.
2) Avoid buying IBM. Their parts (as a general rule) are far more difficult to come by used and in my experience they are far less forgiving about being run in home environments. I could only run my memory banks half populated or the memory would overheat in my IBM x3650's.
3) Avoid being nickel and dime'd to death. When buying used servers pay very close attention to what you are and are not getting. Ask the seller if it's not clear. The used market is a bit odd and parts that were the big ticket items when new are now practically throw away. But parts that were almost an after thought can run you as much as you paid for the server. I can get a 6 core Hyper-threaded Xeon for a Poweredge R710 for $60 but a replacement/second power supply is $150. As previously mentioned Dell doesn't include a dedicated BMC NIC (DRAC Advanced) by default, that's another $30. Rail kits (if buying rackmount) are often almost $100 if bought separately. Drive blanks for empty bays (if you want them) can be $10/ea or more. All that adds up in a hurry. Memory will probably be your biggest component expense.
4) Avoid low density RAM. It's worth paying a small premium to get a server with larger DIMM's even if you're getting the same total capacity, especially if you're buying a server that uses DDR2. DDR2 FB-DIMMS run VERY hot and you may run into cooling issues if trying to run it with more than half the banks populated.
Now for specifics. I'm very partial to HP when it comes to used servers. I feel they give you the most bang for the buck. Newer Dell's are also decent however. I'm also partial to 2U for the additional expansion space and they are typically a bit quieter. Older servers are in a situation where their sum is greater than their parts.
Servers
Sub $300: HP DL360 G5 (1U) or DL380 G5 (2U). Don't get one with less than 16Gb of RAM, 32Gb would be preferred. Preferably using at least 4Gb DIMM's (less heat and electricity). Don't get one with only a single processor installed as you'll have to buy a VRM and Heatsink should you want to add a second. Expect the RAID controller battery to be due for replacement (although not strictly necessary) and possibly the CMOS battery. You can typically get one with a pair of Quad Cores and 16Gb of RAM for about $200 but it usually won't have rail kits and few or no drives. Be aware these servers use 2.5" drives (IE laptop sized drives) and have no place to mount 3.5" drives. Also be aware that the RAID controller included in these servers is a SAS 3.0Gb/s controller but SATA drives run at 1.5Gb/s speed so disk performance may not be what you expect. Lastly be aware that the noise, heat, and power consumption of this generation server is substantially higher than the next generation server we are looking at next. If you are going to be running this 24x7 and/or running it someplace where noise is a concern, I would STRONGLY recommend spending the extra cash for one generation newer.
Sub $600: HP DL380 G6 or Dell R710. Again, don't get one with less than 16Gb of RAM or only a single processor. If the price is within $80, get one that has Hex Cores rather than Quad Cores. Otherwise, spring for one with 32Gb RAM or more. Your usage plans will determine which of those two items is a higher priority. With a bit of shopping and maybe some patience you can get one with 32Gb of ram and a pair of hex cores for under $600. Your local Craigslist is going to be your best bet to find a REALLY good bargain. Most of the ones on eBay are recyclers. People that just want old stuff gone try Craigslist first. I just picked up a DL380G6 with 72Gb of RAM for $260. Some models of the Dell's were available with 3.5" drives but the HP's are 2.5" only. You can find low/no memory ones pretty consistently for $200-$250 these days. That combined with the substancialy lower heat/noise generation and lower power consumption makes it really hard to recommend the older G5's at any price.
Addons/Upgrades
CPU: It pays to shop around a bit here. Going one notch down on the processor will often reduce the price in half. For example, you can get a 6-Core Opteron for $25. You can get a lower power 6 Core Xeon (L5640) for $55. I strongly recommend going with the lower wattage processor. For a lab it's extremely unlikely you'll need the extra CPU horsepower, but the power and heat reduction will save you money over the long run.
Memory: Unfortunately hasn't really come down in price. This is why I recommend buying servers with sufficient memory to begin with. Expect to spend ~$200 for 32Gb of RAM for the above mentioned servers. More if you want higher density. For the servers running DDR2 you are limited to 8Gb modules. For the servers running DDR3 you can get 16Gb modules. If you buy memory, spend a bit extra for higher density modules unless you are matching modules already in the server.
NIC: Do not spend more than $100 on a NIC. You can get quad port Intel PRO/1000's for $100 or less on eBay. However, be aware if you bought a 1U server you may be limited to half height and/or half length cards (I'm looking at you Dell). Stick with Intel or Intel rebranded NIC's. Don't get Broadcom or off brand NIC's.
RAID/HBA: If you bought a complete server you probably already have a RAID card. But if you need a second one or are building your own you are hard pressed to beat the IBM ServeRAID M1015 for $95 which is a rebranded LSI 9220-8i. Be aware however that you will need a SAS breakout cable if you're connecting it to standard SATA drives instead of a SAS backplane. That will typically allow you to run up to 8 drives without an expander. If you are building a storage server, most people that run ZFS systems reflash the above mentioned card to IT (aka HBA mode). You will require a motherboard that does NOT have an onboard controller to compete this flash.
DRIVES: I strongly recommend booting off a USB thumbdrive whenever possible. It's cheap, fast, and doesn't generate extra noise/heat. If you want to stick with spindle drives, you can get 146Gb 10k SAS drives used for about $15/ea on ebay. With the decreasing cost of SSD's however, it's worth considering going that route. You can get a 128Gb Crucial MX100 SSD for $65 shipped on Amazon. It's going to be faster, quieter, cooler, and more efficient than a 10k or 15k SAS drive. If your server has 3.5" drive trays, be aware you may need a special tray to use a 2.5" drive. Most off the shelf adapters won't work as the drive wouldn't line up with the back plane. Also be aware that some SAS controllers don't run SATA drives at full speed.
SWITCHES: You don't have to break the bank here either. You can get a Cisco C2970G-24TS-E for under $200. That's a 24 port managed Gigabit switch. Be wary of running cheap switches if you're going to be building a larger lab. Many do not handle running at heavy load well. Be aware however that enterprise switches usually have a higher power consumption than home switches.
STORAGE: If you need more storage than the above servers will hold internally (or just want separate storage), you've got two options. Adding a storage enclosure or using networked storage. Unfortunately the only one of the "bargain" servers I've mentioned that includes an external SAS port is the IBM x-series and none of them have USB 3.0 or eSATA. However, if you can live with a 3Gb/s controller, you can get one with external ports for $50 and you can get an HP shelf (MSA) for $50-$60. Add another $30 for cables. That will give you room for 12x 3.5" or 10-25x 2.5" depending on model. Alternatively you can go network storage. For iSCSI you'll just need an OS that supports it and network connectivity. I would recommend a dedicated NIC for iSCSI however. Alternatively you can go with Fiber Channel. You can do a complete fiber channel setup including switch for under $200 that will give you 8Gb/s of connectivity between storage and host.
RACK: If you happen to be looking for a rack and live in a major city, hit up Craigslist. You'll often find people that just need the thing gone. I paid $100 for a complete HP 36U rack with doors, panels, wheels, and PDU's. $40 to rent a pickup from Uhaul. Don't cheap out and get a no-name 4 post rack or god forbid a 2 post rack.
There's two major choices when building a home lab. You either build your own new "servers" or you buy used. There's 4 major considerations in this decision. Hardware Costs, Environmental Costs, Compatibility, and Features.
Hardware Cost: You can buy used Enterprise level equipment for far cheaper than you can build equally performing new equipment. For example, you can get a new AMD 8-Core Piledriver CPU for about $150. You can get a complete used HP or Dell server with a pair of Quad Cores for about $200. Used equipment has the advantage here.
Environmental Costs: That may be a poor way to word it, but there's three things people often don't think about when buying a "real" server. Noise, Heat, and Electricity. "Real" servers are designed to run in a temperature controlled data center, not your garage or living room. Some servers are quieter than others but typically they are going to be at least loud enough to annoy anybody in the room. They also generate a LOT of heat and don't always like running in a room kept at 80+ degrees. You can somewhat minimize this by being selective with your equipment choices but it's still going to be more of an impact than a new desktop PC. Last but not least is electricity. Servers are power hungry. Again, the newer your equipment the better it will be but you're still going to be using more power than a desktop PC, probably by a lot and you're probably running it 24x7. Admittedly I've got a pretty sizable lab by most standards, but prior to my efficiency rebuild, I was pulling 1,300 watts CONSTANT from my outlet. That's a LOT of power. Building your own has a massive advantage here.
Compatibility: As I mentioned above, you can get an 8 Core AMD processor for about $150. Right now it's probably sounding pretty tempting to just build a cheap box around that (or something similar) and call it good. Unfortunately it's not quite that simple. VMWare (and other Linux/Unix builds) can be a bit picky about the hardware they will support. For example I installed ESXI onto my media center box which runs an AMD APU on a Gigabyte mobo. ESXI did not recognize the onboard NIC's. Then what if you want to expand? A lot of home boards won't boot off a true add-on RAID card. You can certainly make it work if you're selective about your parts but you can't just run down to Microcenter/Fry's and buy whatever is on sale. Buying used equipment has the advantage here.
Features: Servers have a lot of features that the average home user or new IT fledglings are unaware of. It's up to you to decide what, if anything, these features are worth to you.
First up is the Intelligent Platform Management Interface (IPMI). Pretty much all servers have one (or at least the option of one) but each manufacturer comes up with their own name. HP calls theirs Integrated Lights Out (iLo), Dell calls theirs Dell Remote Access Console (DRAC), and IBM calls theirs Remote System Administrator (RSA) on older servers and Integrated Management Module (IMM) on newer ones. Typically these have their own dedicated network port on the server (Dell is the exception, a dedicated port is an upgrade option). The purpose of IPMI is to allow you to have physical level access from a remote system. IPMI allows you to physically power on and power off a system as well as access it's health status remotely. Some systems support additional features such as a remote console, Virtual Media (allows you to give the server access to a USB or DVD plugged into your desktop), Firmware Updating, and Power Consumption info although these features often required a license purchase.
Second is Serial Attached SCSI (SAS) support. As a general rule SAS drives are faster and more reliable than SATA drives. Most servers have SAS controllers and if you're buying used they are probably including at least a pair of SAS drives. SAS and SATA use the same physical connector however so even if you have a SAS controller, you can still use SATA drives if cost is more important than performance. However, it only works one way. SAS controllers can run SATA drives, but SATA controllers cannot run SAS drives.
Third is the NIC's. Most servers have at least two NIC ports, newer ones have four or sometimes more. These are typically Intel NIC's or sometimes Broadcom. Both of which are supported by ESXI (and most other Linux/Unix builds) which is good.
It is worth mentioning that you CAN get some or all of these features on high end workstation/server boards from companies like Supermicro or Tyan. But your cost will go through the roof if doing that.
If at this point you've decided to build your own, here's the things you want to keep in mind. When running VM's, RAM is your friend unless you like waiting a lot. Personally I configure all my Windows VM's with at least 4Gb of RAM. Linux/Appliance VM's I may go less. CPU requirements is going to vary greatly depending on your usage but I would try to stick with at least 8 threads. For your NIC's stick with Intel. If your board doesn't have Intel NIC's onboard, plan on buying an add-on NIC. See end of this post for recommendations. You'll want at least 2 GbE ports. More if you are running network storage. Lastly is the disk requirements. If you're only running one or two VM's or don't care about performance, this isn't an issue. But if you're running half a dozen or more VM's or have expectations on disk performance keep in mind you'll either need several spindle drives or use SSD's.
If you've decided to buy used, here's some general recommendations.
1) Avoid going with anything older than a Xeon 5000 or Opteron 2300. They're slow, have a limited number of threads, and replacement parts are prohibitively expensive. Xeon 5500/5600 series or Opteron 6100 series would be greatly preferred for multiple reasons which I'll explain further below.
2) Avoid buying IBM. Their parts (as a general rule) are far more difficult to come by used and in my experience they are far less forgiving about being run in home environments. I could only run my memory banks half populated or the memory would overheat in my IBM x3650's.
3) Avoid being nickel and dime'd to death. When buying used servers pay very close attention to what you are and are not getting. Ask the seller if it's not clear. The used market is a bit odd and parts that were the big ticket items when new are now practically throw away. But parts that were almost an after thought can run you as much as you paid for the server. I can get a 6 core Hyper-threaded Xeon for a Poweredge R710 for $60 but a replacement/second power supply is $150. As previously mentioned Dell doesn't include a dedicated BMC NIC (DRAC Advanced) by default, that's another $30. Rail kits (if buying rackmount) are often almost $100 if bought separately. Drive blanks for empty bays (if you want them) can be $10/ea or more. All that adds up in a hurry. Memory will probably be your biggest component expense.
4) Avoid low density RAM. It's worth paying a small premium to get a server with larger DIMM's even if you're getting the same total capacity, especially if you're buying a server that uses DDR2. DDR2 FB-DIMMS run VERY hot and you may run into cooling issues if trying to run it with more than half the banks populated.
Now for specifics. I'm very partial to HP when it comes to used servers. I feel they give you the most bang for the buck. Newer Dell's are also decent however. I'm also partial to 2U for the additional expansion space and they are typically a bit quieter. Older servers are in a situation where their sum is greater than their parts.
Servers
Sub $300: HP DL360 G5 (1U) or DL380 G5 (2U). Don't get one with less than 16Gb of RAM, 32Gb would be preferred. Preferably using at least 4Gb DIMM's (less heat and electricity). Don't get one with only a single processor installed as you'll have to buy a VRM and Heatsink should you want to add a second. Expect the RAID controller battery to be due for replacement (although not strictly necessary) and possibly the CMOS battery. You can typically get one with a pair of Quad Cores and 16Gb of RAM for about $200 but it usually won't have rail kits and few or no drives. Be aware these servers use 2.5" drives (IE laptop sized drives) and have no place to mount 3.5" drives. Also be aware that the RAID controller included in these servers is a SAS 3.0Gb/s controller but SATA drives run at 1.5Gb/s speed so disk performance may not be what you expect. Lastly be aware that the noise, heat, and power consumption of this generation server is substantially higher than the next generation server we are looking at next. If you are going to be running this 24x7 and/or running it someplace where noise is a concern, I would STRONGLY recommend spending the extra cash for one generation newer.
Sub $600: HP DL380 G6 or Dell R710. Again, don't get one with less than 16Gb of RAM or only a single processor. If the price is within $80, get one that has Hex Cores rather than Quad Cores. Otherwise, spring for one with 32Gb RAM or more. Your usage plans will determine which of those two items is a higher priority. With a bit of shopping and maybe some patience you can get one with 32Gb of ram and a pair of hex cores for under $600. Your local Craigslist is going to be your best bet to find a REALLY good bargain. Most of the ones on eBay are recyclers. People that just want old stuff gone try Craigslist first. I just picked up a DL380G6 with 72Gb of RAM for $260. Some models of the Dell's were available with 3.5" drives but the HP's are 2.5" only. You can find low/no memory ones pretty consistently for $200-$250 these days. That combined with the substancialy lower heat/noise generation and lower power consumption makes it really hard to recommend the older G5's at any price.
Addons/Upgrades
CPU: It pays to shop around a bit here. Going one notch down on the processor will often reduce the price in half. For example, you can get a 6-Core Opteron for $25. You can get a lower power 6 Core Xeon (L5640) for $55. I strongly recommend going with the lower wattage processor. For a lab it's extremely unlikely you'll need the extra CPU horsepower, but the power and heat reduction will save you money over the long run.
Memory: Unfortunately hasn't really come down in price. This is why I recommend buying servers with sufficient memory to begin with. Expect to spend ~$200 for 32Gb of RAM for the above mentioned servers. More if you want higher density. For the servers running DDR2 you are limited to 8Gb modules. For the servers running DDR3 you can get 16Gb modules. If you buy memory, spend a bit extra for higher density modules unless you are matching modules already in the server.
NIC: Do not spend more than $100 on a NIC. You can get quad port Intel PRO/1000's for $100 or less on eBay. However, be aware if you bought a 1U server you may be limited to half height and/or half length cards (I'm looking at you Dell). Stick with Intel or Intel rebranded NIC's. Don't get Broadcom or off brand NIC's.
RAID/HBA: If you bought a complete server you probably already have a RAID card. But if you need a second one or are building your own you are hard pressed to beat the IBM ServeRAID M1015 for $95 which is a rebranded LSI 9220-8i. Be aware however that you will need a SAS breakout cable if you're connecting it to standard SATA drives instead of a SAS backplane. That will typically allow you to run up to 8 drives without an expander. If you are building a storage server, most people that run ZFS systems reflash the above mentioned card to IT (aka HBA mode). You will require a motherboard that does NOT have an onboard controller to compete this flash.
DRIVES: I strongly recommend booting off a USB thumbdrive whenever possible. It's cheap, fast, and doesn't generate extra noise/heat. If you want to stick with spindle drives, you can get 146Gb 10k SAS drives used for about $15/ea on ebay. With the decreasing cost of SSD's however, it's worth considering going that route. You can get a 128Gb Crucial MX100 SSD for $65 shipped on Amazon. It's going to be faster, quieter, cooler, and more efficient than a 10k or 15k SAS drive. If your server has 3.5" drive trays, be aware you may need a special tray to use a 2.5" drive. Most off the shelf adapters won't work as the drive wouldn't line up with the back plane. Also be aware that some SAS controllers don't run SATA drives at full speed.
SWITCHES: You don't have to break the bank here either. You can get a Cisco C2970G-24TS-E for under $200. That's a 24 port managed Gigabit switch. Be wary of running cheap switches if you're going to be building a larger lab. Many do not handle running at heavy load well. Be aware however that enterprise switches usually have a higher power consumption than home switches.
STORAGE: If you need more storage than the above servers will hold internally (or just want separate storage), you've got two options. Adding a storage enclosure or using networked storage. Unfortunately the only one of the "bargain" servers I've mentioned that includes an external SAS port is the IBM x-series and none of them have USB 3.0 or eSATA. However, if you can live with a 3Gb/s controller, you can get one with external ports for $50 and you can get an HP shelf (MSA) for $50-$60. Add another $30 for cables. That will give you room for 12x 3.5" or 10-25x 2.5" depending on model. Alternatively you can go network storage. For iSCSI you'll just need an OS that supports it and network connectivity. I would recommend a dedicated NIC for iSCSI however. Alternatively you can go with Fiber Channel. You can do a complete fiber channel setup including switch for under $200 that will give you 8Gb/s of connectivity between storage and host.
RACK: If you happen to be looking for a rack and live in a major city, hit up Craigslist. You'll often find people that just need the thing gone. I paid $100 for a complete HP 36U rack with doors, panels, wheels, and PDU's. $40 to rent a pickup from Uhaul. Don't cheap out and get a no-name 4 post rack or god forbid a 2 post rack.
Last edited:
HP makes some good servers, and I'd gladly take one to play with at home. I get to repair them from time to time at work as well.
Facebook
Twitter