• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

remote data access solutions

3

3sls

Junior Member
I do scientific computing on large data (e.g. 100 gigabytes at a time)

Currently I have a local machine stuffed with 4 big hard drives and I run everything locally.

I am leaning towards getting a dedicated NAS that will allow me to hot swap hard drives in case of failures. I anticipate my data needs will increase over time (20TB in a year or two) and would also like recommendations on what I should be doing (a big tower with lots of drives or something more efficient)

I would also like to learn if solutions exist for me to access the data when I work remotely- currently I just remote desktop into the local machine but if I could link directly to the NAS drive, that would allow me to work more efficiently.
 
I would recommend looking at Supermicro cases. They have some with 24 bays or even more. Something like this:

http://www.ncix.com/detail/supermicro-846be16-r920b-4u-rm-24bay-86-92733.htm

Norco is a cheaper option, but after reading enough reviews of DOAs and stuff with the backplane and other parts, I personally don't want to deal with that so it's worthwhile to pay more.

That's just the case though. You need a motherboard, cpu, and some SAS/SATA controllers, enough to handle all those drives.

I recently built one and it came up to about 3 grand, but with all that capacity it will last a long time. They already have 6TB drives on the horizon so you can always make arrays with 3TB drives then grow them later by adding bigger drives. I use Linux and use mdadm raid with NFS for sharing out. There are lot of options when going custom. iSCSI is an option too.
 
That seems quite expensive for something that's not even rackmount or redundant power. Though Synology does have good reviews in general.

The nice thing about DIY though is that you are not tied to proprietary stuff. You can just keep expanding with any hardware you like.
 
Depends on exact needs. Based on personal experience and skills, I'd probably just build a small file server, even with 20TB files. Probably running win8.1 or server 2013 (just personal inclination and skills). Depending on up time needed and data redundancy required I'd probably look at 6x4TB drives in RAID5 so one drive could die and still have it running if I need full up-time. That plus a UPS to run it and my networking gear (or seperate UPSs).

I'd also have a backup server or at least USB3 enclosure with 5x4TB drives as JBOD.

Or wait a little bit and go in on 4x5TB drives once those become consumer mainstream.

Or if I had the money and time, I'd go with a duplicate file server of the first and use the backup as a warm/hot standby in case of primary server failure and also use the warm/hot standby as the primary's backup target.

Of course you also need to think about required performance. If a regular gigabit link isn't going to work for you, you are going to need to start going a little more exotic. A couple of 10GbE adapters and a direct network connection between your work station and the server doesn't cost that much. Or you can go win8/8.1 or server 2013 and use multiple aggregated gigabit network cards/ports between the machines (SMB3 on windows adds SMB Multichannel which will aggregate multiple NICs for higher throughput. I have a pair of NICs on my server and desktop and regularly get 235MB/sec performance between machines and my RAID arrays).

For remote access, other than RDP, you could setup a VPN connection back to your network and connect over NFS/SMB like a regular network share. VPN tends to have its own overhead on top of your network connection though and depending on what you want and need to do, manipulating the data over the local network by remoting in to a machine on your LAN would likely be much more efficient than working with multi gigabyte files over the internet where I assume you may need to be transfering large amounts of data to do that.

NFS and CIFS/SMB do not work all that well over the internet. They can and do, but not well. The extra induced lag causes them to behave badly and you are also talking soda straws instead of sewer pipes trying to move data around. I personally like VPN over the internet for making an NFS/SMB share virtually local only for things like backing up data and moving files around (albeit slowly) I don't like it for directly opening/manipulating files.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top