Refurbished servers for business use

[MrDudeMan]

I'm setting up a virtualized environment for a small office (< 20 computers) to run the domain controller, a few application servers, and a few webservers. Note: I'm not interested in a SAN (pyramid of doom and all that), so I'll be using vSAN through ESXi.

I've been looking at new Dell R4XXs and they seem good, but the price tags quickly get out of scope once enough memory, hard drive space, etc. is added. I don't need anything crazy and I'll only be using vSphere Essentials Plus (max of 3 hosts and up to 2 sockets per host). I probably only need one socket per host and two hosts to carry the load.

With that said, I could easily get much more horsepower for the dollar from Server Monkey/xByte/etc.. Yes, the servers could be refurbished, but I could get three instead of two and I am going to be using fault tolerant mode anyway. The additional storage nodes will also provide more resiliency. I don't have any specific hesitations about buying refurbished machines especially with 3-5 year warranties, but I thought I'd check to see if anyone else could change my plan.

 

[XavierMace]

You're spending $9k on licensing but looking at buying refurbished servers? What exactly is the hardware budget? Is noise/power consumption a factor?

 

[MrDudeMan]

The license is 5,500 and my budget is as close to 10,000 as possible. I looked at HyperV and it seems like an awful solution compared to vSphere, but I can revisit that if you think it's worthwhile. I have experience with vSphere and I know it does what we want, which is why I chose it. I am the entire IT department, so I don't have time to mess with partial or substandard solutions, which is what HyperV looks like to me.

The main point of my post is that the refurbished servers have good warranties and vSphere features mitigate failure scenarios pretty well. I can get three servers with five year warranties and the license for 9k if I do it that way. Otherwise, I can get the license and two new servers with 3 year warranties right at 10k. The extra 1k can be used on a 10 gig switch in the first option and in the second option I'd have to wait to get that. The other benefit of the first option is that I could do it with two servers instead of three to allocate even more money to network upgrades.

I looked at getting vSphere standard licenses to try to save some money, but it ends up being close to the same price with a lot more hassle and limitations, so Essentials Plus makes more sense unless I'm missing something.

Noise/power are irrelevant. The R410 I specced on ServerMonkey is really similar to a new one from Dell for half the price including power usage, so it seems to be a moot point anyway.

 

[dave_the_nerd]

HyperV is not a substandard solution - although I don't like it as much as ESX/vCenter. But if you learn the Microsoft Way Of Doing Things™ it's just as good.

In your shoes, I'd be looking at implementing a KVM based (free!!!) solution so you can spend more money* on hardware. You can set up shared storage, failover, etc., and whatnot, it's just a bit more complicated. But if you're doing these sorts of budget implementations for people, that's going to be a good thing to have in your pocket.

*one of the hypervisors I use at work costs more than your entire implementation budget, and I have 15 of them. Hardware is important. I haven't used vSAN, although I've heard good things about it. But you can buy your way around the "pyramid of failure" too, with dual controllers, redundant SAS chains, etc.

 

[MrDudeMan]

I shouldn't have written substandard. I meant the documentation and presentation seems lackluster to me, but I don't know much about the actual hypervisor. Mainly it comes down to management. I'm having a hard time getting a feel for the HyperV management process because it isn't explained or documented online very well. Also, all of the comparisons I've come across are very one-sided or don't explain the implementation differences. I created a thread specifically for the hypervisor discussion here, though, as this post was intended to mainly be about refurbished hardware.

 

[dave_the_nerd]

Fair enough.

As far as hardware goes, a slightly used server with new HDDs is usually a pretty good deal, except you're missing out on support contracts, unless you pay even more, and it can be harder (eBay) to find older parts. The R410s are ~2009-2010 vintage, based on the CPU type, which is kind of... old. PSUs are likely to go soon, if a battery dies on the RAID controller you're SOL for that. Etc.

Without knowing exactly the load you intend to have (how many GB of RAM in use, how many CPUs, etc.) I can't really offer a good hardware recommendation. But redundant DCs, and a "few" VMs at 4GB of RAM each leave me assuming 64GB RAM setups minimum.

It looks like you can CTO an R630 with 64GB of RAM, 2x hex-core 1.9GHz CPUs, dual PSUs, an SD card reader (to boot the hypervisor) and 8x 1TB HDDs for ~$3800 each. A pair of those would probably do the trick for hardware.

Linux + KVM for a hypervisor (free), use the HDDs in RAID-10 for your datastore (4TB of storage) and set up block device duplication between the two systems (redundant storage and failover) and you're done.

 

[MrDudeMan]

Thanks for the info (both times).

The workload is pretty light as you inferred. Storage space is trivially important as most of the data on the VMs will be ephemeral and the rest will be on NAS network shares and Dropbox. I doubt I even need dual socket systems because the load will be so light, but it's hard to find single socket systems and it probably doesn't make much sense to go single socket anyway. I need about 20 GB per VM (actual) and I'll probably allocate 40-50 GB except for the DC, which will get 200 GB (allocated). The app servers actually run perfectly fine with 2 GB of RAM, but I'll allocate 4 GB just because.

The key is to make sure all of the VMs will be available with as close to 100% as possible. With as little as we need to run, it almost would make more sense to run them all on a single host with another host in the configuration as a hot spare. I don't think it works like that, but I could be wrong. That's just what the situation probably needs, which is why I was looking at refurbished hardware.

I'll take a look at the Dell CTO options again.

 

[sdifox]

why not just a bunch of servers? you can buy plenty of hardware instead of vmware licence. or go hyperv. 2012 Standard edition ought to be good enough for you. or go 2016

 

[MrDudeMan]

why not just a bunch of servers? you can buy plenty of hardware instead of vmware licence.

My two primary thoughts behind this: I'm remote (1000 miles from the office), which makes maintenance difficult at times, and up-time is important so I figured a hypervisor solution with high availability (e.g. ESXi) would help get things moving very quickly.

Part of why I'm waffling is because I keep having the same thought you probably are - why not buy a few servers and handle this a different way? I don't know and, unfortunately, I don't have enough experience to make this decision. I'm getting to the point where I want to find a consulting firm (or person) who I can pay to tell me what to do. I know enough to make various implementations viable, but I don't know enough to make a well-understood decision.

 

[sdifox]

My two primary thoughts behind this: I'm remote (1000 miles from the office), which makes maintenance difficult at times, and up-time is important so I figured a hypervisor solution with high availability (e.g. ESXi) would help get things moving very quickly.

Part of why I'm waffling is because I keep having the same thought you probably are - why not buy a few servers and handle this a different way? I don't know and, unfortunately, I don't have enough experience to make this decision. I'm getting to the point where I want to find a consulting firm (or person) who I can pay to tell me what to do. I know enough to make various implementations viable, but I don't know enough to make a well-understood decision.

Then just go AWS. Can't do HA on a budget. And why are your R410s so expensive? the drives?

My R710 was maybe US700 back in Dec 2014 with 6x4GB DDR3 and no hdd

 

[MrDudeMan]

Then just go AWS.

Are you saying put the DC on AWS as well? I've looked into cloud DCs and I'm not convinced either way yet. It would certainly make this a hell of a lot easier because I could remove all on-prem machines except the file server (NAS).

 

[sdifox]

Are you saying put the DC on AWS as well? I've looked into cloud DCs and I'm not convinced either way yet. It would certainly make this a hell of a lot easier because I could remove all on-prem machines except the file server (NAS).

you can host everything on AWS, including storage. Same thing on Google Cloud and Azure

 

[MrDudeMan]

Then just go AWS. Can't do HA on a budget. And why are your R410s so expensive? the drives?

Sorry - I responded before your edit, so I'm responding again. I can get an HA setup for 10k, but it has to be done with refurbished machines. I spent a few hours talking to VMware on the phone to figure out which options I would need to achieve my goals and I believe it's possible based on their advice. With that said, I'd love to have zero servers on-prem, but I didn't think that was a possibility because of the DC. I'd be happy to be proven wrong, though.

Maybe we don't even need a DC. The only thing we really use it for is AD and DNS. DHCP can be moved to the network hardware (which is probably where it should be anyway). I don't know what would happen with WSUS and GPOs (are GPOs part of AD? I'm not a Windows Server expert obviously). I would love to call someone and even pay them to discuss this, but who do I call? Does Microsoft even support this type of questioning? All of the local IT people I've tried are totally clueless. You guys are literally the only people I know will give me good information, but I need a more formal business relationship to get to the end of this and have it go well based on how things are going.

 

[sdifox]

Sorry - I responded before your edit, so I'm responding again. I can get an HA setup for 10k, but it has to be done with refurbished machines. I spent a few hours talking to VMware on the phone to figure out which options I would need to achieve my goals and I believe it's possible based on their advice. With that said, I'd love to have zero servers on-prem, but I didn't think that was a possibility because of the DC. I'd be happy to be proven wrong, though.

Maybe we don't even need a DC. The only thing we really use it for is AD and DNS. DHCP can be moved to the network hardware (which is probably where it should be anyway). I don't know what would happen with WSUS and GPOs (are GPOs part of AD? I'm not a Windows Server expert obviously). I would love to call someone and even pay them to discuss this, but who do I call? Does Microsoft even support this type of questioning? All of the local IT people I've tried are totally clueless. You guys are literally the only people I know will give me good information, but I need a more formal business relationship to get to the end of this and have it go well based on how things are going.

just example, you can call amazon and explain your situation and ask them what amazon can do for you

http://docs.aws.amazon.com/quickstart/latest/active-directory-ds/scenario-1.html

 

[MrDudeMan]

just example, you can call amazon and explain your situation and ask them what amazon can do for you

http://docs.aws.amazon.com/quickstart/latest/active-directory-ds/scenario-1.html

That's a good idea. I'll call them.

 

[MrDudeMan]

Here's what I'm able to configure on ServerMonkey as a reference:

Dell PowerEdge R620 4-Port Chassis $700
2 x 2.5 GHz Hex-Core Intel Xeon Processor with 15MB Cache--E5-2640 $35
64GB Memory Upgrade Kit (8x8GB) PC3-12800R $190
PERC H310 Lvl 0-50 RAID $45
4 x 1TB 7.2K RPM SATA 2.5" Dell Hard Drive $60
iDRAC7 Express License (600+ Series) - Included $0
Dell Broadcom 5720 Quad Port 1GbE Network Daughter Card $0
2 x Dell 750 Watt PSU $200
3 Year ServerMonkey Warranty $250
Total: $1,710​

and another option:

Dell PowerEdge R430 4-Port Chassis $1,000
2 x 1.9 GHz Hex-Core Intel Xeon Processor with 15MB Cache--E5-2609 v3 $280
64GB Memory Upgrade Kit (8x8GB) PC4-17000R $640
PERC H330 12Gb/s Lvl 0-50 RAID $100
4 x 1TB 7.2K RPM SATA 3.5" Dell Hard Drive $38
On-Board Quad Port 1GbE LOM $0
iDRAC8 Express Remote Access Card $0
2 x Dell 550 Watt PSU $350
3 Year ServerMonkey Warranty $250
Total: $3,067​

 

[dave_the_nerd]

The 430 config looks ok. The 620s are old enough that I'd be less inclined.

How much can you tell us about the web servers and applications? Is this going to be used for staging/development work or just production stuff that doesn't get moved around or reconfigured a lot?

The answer to whether or not you should have local vs. hosted services is dependent on what those services are. Your NAS, for hopefully obvious reasons, should be local, for instance. I would generally recommend that AD/Domain stuff and print servers be local as well. But a a web-based task tracking tool like BugZilla or JIRA could easily be on a hosted system.

The caveat being that you will most likely end up spending more on Amazon in the long run than you would on your own mini-datacenter, but there are fewer costs now and you don't have to mess with the hardware.

 

[sdifox]

The 430 config looks ok. The 620s are old enough that I'd be less inclined.

How much can you tell us about the web servers and applications? Is this going to be used for staging/development work or just production stuff that doesn't get moved around or reconfigured a lot?

The answer to whether or not you should have local vs. hosted services is dependent on what those services are. Your NAS, for hopefully obvious reasons, should be local, for instance. I would generally recommend that AD/Domain stuff and print servers be local as well. But a a web-based task tracking tool like BugZilla or JIRA could easily be on a hosted system.

The caveat being that you will most likely end up spending more on Amazon in the long run than you would on your own mini-datacenter, but there are fewer costs now and you don't have to mess with the hardware.

op wants ha and he is a thousand mile away.

 

[MrDudeMan]

The 430 config looks ok. The 620s are old enough that I'd be less inclined.

I kinda figured you'd say that based on your last post and I'm starting to see what you mean. I'm coming around slowly but surely.

How much can you tell us about the web servers and applications? Is this going to be used for staging/development work or just production stuff that doesn't get moved around or reconfigured a lot?

There will be 4-6 VMs total, one of which will be Server 2016 as a DC. I'm making an assumption that I don't need a secondary DC because I should be able to have the primary failover to the other host in the event of a host failure. If that's not true, then add a VM and assume the DCs will be statically bound to physical hosts.

The production apps are very lightweight webservers running mostly internal apps and potentially a few externally visible web apps for clients. I'm currently running the client-facing apps on AWS for obvious reasons, which is where I'll probably keep them unless there's a compelling need to pull them into the cluster. Two or three VMs will be used specifically for CAD (AutoCAD, Surfer, etc.) and one more will most likely be spun-up for Office applications. Like I said, this is very lightweight stuff for the most part and the number of users who will concurrently be using VMs is likely to never be greater than 7-10.

The guest OSes will be Server 2016, W10 Pro, and Ubuntu 16.04.

The answer to whether or not you should have local vs. hosted services is dependent on what those services are. Your NAS, for hopefully obvious reasons, should be local, for instance. I would generally recommend that AD/Domain stuff and print servers be local as well. But a a web-based task tracking tool like BugZilla or JIRA could easily be on a hosted system.

The NAS is and always will be a local file server. It majorly helped productivity because the DC used to be the FS as well and it was really struggling. Separating them was great, but the DC hardware is old and I'm constantly worried about what will happen when it blows up. [Enter virtualized DC]. As of right now, the internal web apps and CAD machines are hosted on craptastic Virtualbox VMs, which are running on the NAS. The NAS is beefy enough to handle it, but this is obviously bad for all of the reasons you already know. I had to make it work with what was available and I'm trying to make it right as fast as possible.

The caveat being that you will most likely end up spending more on Amazon in the long run than you would on your own mini-datacenter, but there are fewer costs now and you don't have to mess with the hardware.

I'm perfectly fine with long term costs that exceed on-prem ownership because of my lack of expertise and the likelihood that DR will go better with 'outsourced' IT (using AWS). Up front cost is an issue in as much as it could potentially prohibit good decisions from being made. That won't be a hard sell nor should it be, but I also need to have a plan regardless of which way we go.

 

[MrDudeMan]

op wants ha and he is a thousand mile away.

I'm in Colorado and the office is in Texas. I do have a relationship with a local (to Texas) IT company, but I get paid to handle this stuff, so I'm trying to make my life easier by having an HA cluster do some of the heavy lifting. That's partly why I'm trying to do an N+1 physical host setup along with all of the network infrastructure, etc. That's also why I was looking at refurbished servers; any money I can save will allow me to upgrade other things like switches that are sorely lacking in features and options.

 

[sdifox]

I'm in Colorado and the office is in Texas. I do have a relationship with a local (to Texas) IT company, but I get paid to handle this stuff, so I'm trying to make my life easier by having an HA cluster do some of the heavy lifting. That's partly why I'm trying to do an N+1 physical host setup along with all of the network infrastructure, etc. That's also why I was looking at refurbished servers; any money I can save will allow me to upgrade other things like switches that are sorely lacking in features and options.

well, refurbs dont have support contract. the warranty is usually exchange, so when something fails, someome has to determine what failed, contact seller, wait for exchange.may as well have a spare used part on site.

is it a standalone site, just remote to you? so not part of an ad tree with multiple sites?

 

[MrDudeMan]

well, refurbs dont have support contract. the warranty is usually exchange, so when something fails, someome has to determine what failed, contact seller, wait for exchange.may as well have a spare used part on site.

is it a standalone site, just remote to you? so not part of an ad tree with multiple sites?

I see what you mean. I suppose I'm okay with debugging the failure as long as there's another host to run the VMs in the interim. It's not ideal, but it could work. The budget is forcing a feature cut somewhere and support may be part of that.

It's a single site with a domain controller and a file server. The rest of the network is comprised of ~15 user desktops.

 

[sdifox]

I see what you mean. I suppose I'm okay with debugging the failure as long as there's another host to run the VMs in the interim. It's not ideal, but it could work. The budget is forcing a feature cut somewhere and support may be part of that.

It's a single site with a domain controller and a file server. The rest of the network is comprised of ~15 user desktops.

I think you should stick with windows server standard, which gives you two vm instance of the same os. so two server, two licence, two DC in A-A failover mode. Then your vms are in hyperv as a service. So you are not running a pure vm host.

Main concern is the data store for the webservers. unless you are willing to tackle db replication, you are looking at single point of failure any way you cut it.

I don't see vmware bring any value to this setup.


for what it's worth, I am running 2012R2 data centre at home. legit of course and I can spin up as many copy of it as I need
.

I setup a base 2012 and it sits on ssd. I then use difference disk to spin up vm instances. storage saving add up quickly.

 

[XavierMace]

The license is 5,500 and my budget is as close to 10,000 as possible.

Essentials Plus doesn't include vSAN. That licensing will eat up the rest of your budget alone. If somebody told me they wanted to build an HA cluster on $10k, I'd laugh at them.

 

[MrDudeMan]

Essentials Plus doesn't include vSAN. That licensing will eat up the rest of your budget alone. If somebody told me they wanted to build an HA cluster on $10k, I'd laugh at them.

Why would you laugh at someone for asking a question? That's incredibly stupid and says a lot more about you than anyone else. If the answer isn't what they want to hear or what they were expecting, explain it instead of acting like a superior douche. Besides, you're wrong because I spent a few hours on the phone yesterday with VMWare and Microsoft sales to get an array of opinions and potential solutions, which both of them were able to provide in my price range. I ordered two new servers from Dell because I'll need those in either case and I'll pick HyperV or ESXi after trying both. Regardless, I'll be around 10k and it will be an HA cluster.