Need Help! Price estimate on server build

[ViviTheMage]

Yeah ... 96GB of memory is overkill (unless you are processing/calculating on the machine). Get something with more space then memory. If you're cool, get some netapp stuff. If you're cheap, build a 24 slot supermicro server .. the most expensive part will be the disks themselves.

I've built something similiar to this in the past :

4U case with 24 hot swap trays , $1,000~
atom mobo - 2GB memory, $200~
2TBx24 - $300 a disk for WD RE4's ... unless you NEED something faster, like 10k/15k drives. Those start to cost a lot per GB.
raid card w/BBU- $800~


The biggest drives I can find @ 15k RPM, are 600GB .. which run almost $1,000 each. Maybe get some 500GB-1TB enterprise SSD's if you're funds are "unlimited" and performance is a HUGE factor?

 

[sdifox]

Buy a ton of NAS and gigabit switches? I am assuming you are not accessing hundreds of sequences at the same time.

 

[IndyColtsFan]

The biggest drives I can find @ 15k RPM, are 600GB .. which run almost $1,000 each. Maybe get some 500GB-1TB enterprise SSD's?

If you can find a 500 GB - 1 TB enterprise level SSDs for $1,000 each, I'm all ears. The last time I looked at enterprise-level SSDs ( a year to 18 months ago), you could buy a small car for the price of a moderately-sized one. I'm sure prices have dropped, but surely not that much.

 

[ViviTheMage]

haha, you can't find 500gb - 1TB SSD's for 1k each, I was talking about 15k RPM drives in the 600GB area.

 

[zinfamous]

ah, thanks for the tips. As for the memory, I know that our collegue was pushing 128gb RAM in some Apple servers that he purchased a year ago (or at least, that was his plan), so my thought was to look at something comparable. Indeed, 96 is probably overkill.

A couple of things that I mentioned to Platypus over the phone: We currently have 6 users who'd be accessing our database and doing alignment, running analysis, etc. There will be some new postdocs and grad students within the year, plus we have a handful of collaborators nearby, and possibly remote, that may want access at any given time. So, our access demand would indeed be quite low, no?

I'm not sure if the switches on campus are 1gb or 10gb--being a huge research institute, I'm guessing 10gb, but it could be 1gb. So, a lot of the components installed in this quote are priced for budgeting purposes at this stage, not absolute need. I do know that most of our users are currently denied access if one is running, say, quality scores, across newly-downloaded sequences. Not terribly complicated, but it can be intense for our current system--nearly 2 hours for these 15-20gb files. The current thought is that we are way low on RAM, but as others have said, the real issue could be bandwidth and disc speed?

Also, part of those quotes include 5 year service options, annual on-site maintenance, installation, remote service, all that business.

This is still a few months off, so we certainly do have a good bit of time to consider our real needs. The proposal was due last night at midnight, and I was looking at a 30-40k budget.

I got a thumbsup, so....

Now I'm going to get out the IT dictionary and try to translate your questions, so as to be more prepared when we really do approach this.

 

[Lifted]

1) With RAID6 you're getting 8TB (7TB after formatting) of storage. That's a far cry from 20 - 40TB you were talking earlier.

2) What apps are you using that need 96GB of RAM? You're running this task on a desktop now, which I assume has 4 - 8GB. Are you sure you're not wasting $10k RAM that will never be used.

3) Where are you backing up all of this data? Is the data changing every day? Will a loss of 2 - 3TB of data result in having to repeat many days/weeks of work? You should have a meeting with the IT department and the stakeholders in this project to determine what the RPO is (recovery point objective) as a failure of the storage array could result in a complete failure of the project if it will take months to generate this data again. Backing up 2TB of data per day is not an impossible task, but you will need to work with the IT department to design a backup schema that meets your needs and implement it. If commercial software will be required, don't forget to factor that into your budget as well. You may be able to get a low end 5 - 10TB backup solution $10k - $15k once all of the hardware and software has been accounted for.

 

[IndyColtsFan]

Here's a question that I haven't seen asked or mentioned (though I didn't read all the thread): does your institution already have a large datacenter with a SAN in place? Could you perhaps just expand their SAN and connect to it? That might be a cheaper option for you.

 

[Gooberlx2]

I'm a fan of the Dell/Equallogic iSCSI arrays, combined with a 1U rack server (we use a Dell R610)...works great.

We've also built storage servers through pogolinux. You can quote out a build on their website.

 

[Lifted]

I see a lot of people mentioning SAN's, but it appears that only the server is accessing the data directly. If that is the case, a SAN will be much more expensive than DAS, and will be slower to boot. Why would anyone use a SAN with only 1 node attached to the LUN's? Do any of you folks work for the US government? :sneaky:

 

[IndyColtsFan]

I see a lot of people mentioning SAN's, but it appears that only the server is accessing the data directly. If that is the case, a SAN will be much more expensive than DAS, and will be slower to boot. Why would anyone use a SAN with only 1 node attached to the LUN's? Do any of you folks work for the US government? :sneaky:

If his institution has a datacenter with a SAN, he should leverage that SAN and maybe expand it rather than buying a new one. Also I don't know that it was ever really established that he is only talking about one server until later in the thread.

 

[sourceninja]

Honestly, I'd stay away from Sun hardware unless you're intending to run Solaris.

No, trust me, just stay away from Sun no matter what. Oracle has fucked it up way to much to even bother.

 

[BoberFett]

MD3000 disk storage array

Qty 1 Configured with two single-port controllers
PowerVault MD3000 --Primary Hard Drive Ten 1TB 7.2K RPM Universal SATA 3Gbps --Server connectivity SAS 5/E HBA, PCI-Express, 2x4 connectors
--5x 500GB 7.2K RPM Universal SATA 3Gbps 3.5-in HotPlug Hard Drive
(12.5 TB is actually 3x our current read usage)
--300GB 15K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug Hard Drive, Cust. Kit
TOTAL: $13,139.39

Keep in mind that you're going to RAID those drives so you will lose some storage capacity. So you're at more like 10TB, 25% of your initial requirements. Put threemore of those on the bill. This is why people were scoffing at the $20K-30K you mentioned.

 

[Lifted]

If his institution has a datacenter with a SAN, he should leverage that SAN and maybe expand it rather than buying a new one.

That may be an option, but your still talking big $ when it comes to just disks. 2TB SATA will run you $2k list if it's from EMC or the like. On top of that, he'll still likely be limited to 1GbE.

He can buy 2TB enterprise SATA himself for < $300 and throw them into a DAS array. Same for backup.

Disks are cheap, storage is expensive.

OP:

Promise has DAS arrays such as these that can be used for cheap local storage.

This one has the controller
http://www.provantage.com/promise-technology-vr1840s~7PROM1C7.htm

This is just expansion - can be attached to the above unit.
http://www.provantage.com/promise-technology-vj1840~7PROM1CE.htm

32 x 2TB enterprise SATA drives and the two above items can be had for ~ $12k. That would give you 32TB RAID1/0.

A couple of these (one for backup) and your 96GB beast of a server would come in at about $44k.

 

[zinfamous]

Keep in mind that you're going to RAID those drives so you will lose some storage capacity. So you're at more like 10TB, 25% of your initial requirements. Put threemore of those on the bill. This is why people were scoffing at the $20K-30K you mentioned.

so 20-30TB of storage may be overkill. We're currently only using 4 TB of read data on our desktop server, and only recently did the boss mention that we might be slowing down the sequencing projects...so we probably won't be producing at the same rate.

Then again, all of that could change on a whim. such is life...

10TB may just suit us for some time.

 

[zinfamous]

That may be an option, but your still talking big $ when it comes to just disks. 2TB SATA will run you $2k list if it's from EMC or the like. On top of that, he'll still likely be limited to 1GbE.

He can buy 2TB enterprise SATA himself for < $300 and throw them into a DAS array. Same for backup.

Disks are cheap, storage is expensive.

OP:

Promise has DAS arrays such as these that can be used for cheap local storage.

This one has the controller
http://www.provantage.com/promise-technology-vr1840s~7PROM1C7.htm

This is just expansion - can be attached to the above unit.
http://www.provantage.com/promise-technology-vj1840~7PROM1CE.htm

32 x 2TB enterprise SATA drives and the two above items can be had for ~ $12k. That would give you 32TB RAID1/0.

A couple of these (one for backup) and your 96GB beast of a server would come in at about $44k.

that is awesome, will keep this in mind.

:thumbsup:

first link is broken, btw.

 

[Elbryn]

that is awesome, will keep this in mind.

:thumbsup:

first link is broken, btw.

get the first one, not the second. the second doesnt offer any raid potential, only jbod. while you'll be able to use different sized disks, you will not be able to raid them together to get data protection from disk failure. the only thing i'd be worried about is service on the device. depends on how you feel about maintaining/supporting it if there is a problem. paying a bigger vendor extra money for a known support quantity can be worth it. EMC charges an assload for support/maint. but when it's time to get a problem solved, we can and do make them bend over backwards to get it fixed.

if the server is the only one running the data, it's probably the only one running the sequencing jobs as well? (based on the past guy always building overpowering machines) does your data maintain a database that requires the full dataset be readily available? ie: do you have to keep all of your data active or do you run jobs only on active data, the rest being archived and can possibly be taken off the server and reloaded when needed?

if that's the case, you can go even lower tech. you can go the road of a loaded standalone desktop/server. 2 quad cpus with a bunch of ram and say 4 1tb disks raid 5 will get you 3tb online storage. then you get a couple of external hard drives with 2 tb drives in them off the shelf and once a year archive old data off to the portable hard drives. you could spend 15k on a loaded machine, take the remaining 15-25k and buy a 72" plasma screen to use for the primary monitor.

will your software work in a virtualized environment such as hyperV or vmware? this may be getting another layer deeper but if you potential have the need to run multiple sequence jobs at the same time on different sets of data, you may be able to utilize vitualization tech to accomplish that.

 

[jackace]

Contact, Dell, HP , or IBM. Contact their enterprise sales and they will take care of you. It ain't gonna be cheap, but it will be correct.

This is the answer to the question. Plus you should get some kind of support in case of failures.

 

[Lifted]

get the first one, not the second. the second doesnt offer any raid potential, only jbod. while you'll be able to use different sized disks, you will not be able to raid them together to get data protection from disk failure.

The second unit can be cascaded off the first unit and will utilize the controller in the first. No different than EMC.

the only thing i'd be worried about is service on the device. depends on how you feel about maintaining/supporting it if there is a problem. paying a bigger vendor extra money for a known support quantity can be worth it. EMC charges an assload for support/maint. but when it's time to get a problem solved, we can and do make them bend over backwards to get it fixed.

This is the main issue, so he really needs to size his storage requirements as best as possible. Large vendor with 40TB is not going to happen in his budget, but for 5 - 10TB it's possible.

then you get a couple of external hard drives with 2 tb drives in them off the shelf and once a year archive old data off to the portable hard drives.

That is not a backup solution. What happens if there is a failure between the once per year backup? Backups need to happen at least daily, and with databases, i prefer every 10 - 15 minutes.

 

[Goosemaster]

SAN,DAS,iWARP,ZFS,CLUSTERING,RAID..... all completely pointless to this discussion.

OP, while everyone is trying to help, you are going about this ALL wrong. Everyone will try to help based on your snippets of info and integrating the final solution will be a nightmare.

THIS is what you do to get the Job done right:

1. Accept that you don't know what you are doing
2. Find the senior-most IT person at the institution that is willing to talk with you. IF this is not an option, find out who their general vendors are and schedule a conversion (CDW, etc).
3. To vendor's or IT, describe the following:
3a. Provide details on your existing hardware and software
3b. Describe your computational requirements, how your existing infrastructure has met your past needs, and how it will fail to do so in the future.
3ba. Make sure to provide details on your workflows/task licefcycles so they get a clear picture of what your running and vault requirements are
4a. If IT is available: find out if they can contribute any existing resources to your department
4b. If IT is not available try to work with the vendors to see what the institutional fabric is and how you can connect to it in the mest efficient and effective way possible for your department
5. Notify them of your budget (grant) so that they know that you cannot go above that amount
7. LISTEN TO WHAT THEY SAY
8. Consider speaking directly with hardware suppliers for additional ideas.


For now I would ignore most, if not all of the recommendations made on here as they will only serve to confuse you and the vendors by putting words in your mouth that shouldn't have been there in the first place.

For all we know you might end up with DAS islands for different parts of the data lifecycle with a SAN for the long term... or campus IT might have a nice slice in their SAN just waiting for you.

Accept that you need to delegate the research to someone else - seek out those who know what to do and you will be rewarded for your delegation

P.S. We'll be glad to critique their proposals once you have them

 

[Homerboy]

As has alrady been posted here... what you do in this situation is you go to a VENDOR and have their SALES TEAM and DESIGN TEAM come up with a vaible, workable solution that fits your immediate and future needs. This is what they get paid to do. GO TO A VENDOR

 

[Homerboy]

As has alrady been posted here... what you do in this situation is you go to a VENDOR and have their SALES TEAM and DESIGN TEAM come up with a viable, workable solution that fits your immediate and future needs. This is what they get paid to do. GO TO A VENDOR

I'm going to quote myself and reiterate that you are STUPID if you are not working with a vendor to come up with a product for your needs.

 

[mvbighead]

I'm going to quote myself and reiterate that you are STUPID if you are not working with a vendor to come up with a product for your needs.

Not stupid if he has a connection to get a better deal with an academic program agreement. We have the same SAN he quoted with much less storage and our deal wasn't anywhere close to that.

 

[manlymatt83]

Why not use Amazon S3 or Rackspace Cloud Files? You'll also have good redundancy. Your type of work is exactly what virtualization/elastic storage was made for.

Otherwise, I'd say a SAN. Call EMC.

 

[Goosemaster]

Not stupid if he has a connection to get a better deal with an academic program agreement. We have the same SAN he quoted with much less storage and our deal wasn't anywhere close to that.

[sarcasm]

Exactly! 30% off of a SAN connected via 4Gbps to the Director that he didn't think to buy will be awesome!

[/sarcasm]


OP:
1) speak to solution provers now
2)haggle or switch vendors later

 

[BoomerD]

Yeah, I know it's probably the wrong forum, but my boss is looking for a quick estimate to build "a large server for data storage" within the next hour or two.

I've only built regular desktop PCs, and we do have a regular desktop acting as a data server (currently 6 TB+, and growing rapidly), but I think she's looking for something more serious...like a rack with an array, or something.
I'm not sure where to start with these--Really expensive is quite fine (this is a grant proposal), and high performance is a must. We are pushing~500gb data/week. (we do a lot of whole genome/transcriptome sequencing)/

Any ideas on where to start looking?

Thanks in advance!

Pics of boss and her rack??