• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

Anyone use RAM drives these days?

Jeff7181

Jeff7181

Lifer
I believe I found a use for a RAM drive at work. Basically a coworker generates a lot of logs - on the order of ten 300 MB log files every hour (sometimes up to 500 MB each). He needs to analyze those logs and record the results in a database for reference. The problem with this is that they all need to be analyzed in parallel - the process reads all 10 files and starts recording results based on date stamps on each record in the log file. This is an extremely I/O intense process as the application that does this doesn't appear to read the file from disk into memory to do it's analysis, it reads directly from disk.

We've debated moving his process from SATA disk to 15k SAS disk and even SSD on our SAN. We know that will speed it up, but to allocate a few hundred GB of SSD on our SAN to analyzing logs seems wasteful.

So I came up with the idea that maybe if he created a RAM drive of say, 10 GB, he could copy those 10 log files to the RAM drive and run his analysis from there, which would be many orders of magnitude faster than doing that from disk, even on our $2.5 million SAN.

So I did a quick Google search and came across this: http://memory.dataram.com/products-and-services/software/ramdisk

I actually downloaded the trial version and installed it on a test PC I have in my office that's actually running Windows 8 RTM and it works perfectly. The contents of the drive are wiped out on reboot, but the drive persists so it doesn't need to be recreated if the server is rebooted. The fact that the contents will be lost is of no concern as the logs he analyzes are archived on our SAN.

Anyone have any experience with something like this? Seems like it would be incredibly useful for just this sort of thing.
 
Generally I'm not a fan of them, but this seems like a legitimate use as long as the time to copy the files from the SAN doesn't offset the gains from using the RAM disk.

But I can't speak to that particular product as I've never use it, I've just used the tmpfs/ramfs filesystems that are built into Linux whenever I've played with them.
 
I just give sql server a sitton of memory and all indexes and database (compressed or not) reside in ram, then only writes to log have to happen in real time (database writes are lazy).

8gb Rdimm ECC $65 retail qty 1, 18 slots in G7 (24 in gen8) hp's that's alot of ram for peanuts. Plus disk writes (other than log) go WAY down for some reason the more ram sql server has - it seems to get lazier. I'm sure mysql works similar but not sure how comparable the soft-numa is
 
That's a great example of when RAM drives come in handy. I do a lot of temporary stuff in RAM: compiling code, zipping/unzipping archives, repeatedly searching for strings in a large number of source files. Not only is it faster, it conserves HDD life by not repeatly reading/writing stuff to the HDD that you don't actually care to store permanently.

The only problem you need to consider is how much total RAM you have on your computer. If your computer has 16GB of RAM, and you allocate 10GB for your RAM drive, that leaves you with 6GB of RAM left for your other apps. If your computer is a server, you need to find out if 6GB is enough for your server's background apps.
 
Eeqmcsq said:
That's a great example of when RAM drives come in handy. I do a lot of temporary stuff in RAM: compiling code, zipping/unzipping archives, repeatedly searching for strings in a large number of source files. Not only is it faster, it conserves HDD life by not repeatly reading/writing stuff to the HDD that you don't actually care to store permanently.

The only problem you need to consider is how much total RAM you have on your computer. If your computer has 16GB of RAM, and you allocate 10GB for your RAM drive, that leaves you with 6GB of RAM left for your other apps. If your computer is a server, you need to find out if 6GB is enough for your server's background apps.
Click to expand...

This particular server has more than enough RAM, so that's not a concern. I was just curious if there were any "gotcha's" that I might be missing.

Oh... and someone mentioned SQL... our largest databases are over 4 TB (we're a SaaS company)... not even HP's DL980's can be configured with that much memory, so... yeah. RAM drives won't be useful for us in that respect.
 
Jeff7181 said:
This particular server has more than enough RAM, so that's not a concern. I was just curious if there were any "gotcha's" that I might be missing.
Click to expand...

Hmm, let's brainstorm about possible gotchas...

- Losing the contents in the RAM drive - You already covered this - the original logs are stored elsewhere.

Of course, I don't know how "tech literate" your coworker is. He needs to understand that anything he dumps in the RAM drive can be LOST on a power failure. So although the original logs are stored elsewhere, he might starting using the RAM drive to create other files for himself, and he may not understand the risks of doing this on a RAM drive.

- As mentioned above, the time it takes to copy 10 x 500MB logs from your SAN to your RAM drive outweighs the benefits of the RAM drive - some performance measurements would be interesting to see here between directly analyzing the logs vs copying to RAM first.

- Not enough RAM leftover for other apps - You answered this by saying there will be enough.

- Running out of space on the 10 GB RAM drive - That will be up to your coworker to delete the previous logs in the RAM drive before copying the new log files into the RAM drive. Alternative, he can write a script to automate this.

- Drive letters - It's been a while since I've worked in Windows. If I remember correctly, a RAM drive is assigned a drive letter, and I think some RAM drive software blindly picks the next available letter. I don't know if you or your software need the RAM drive to be assigned at a certain drive letter. If yes, you will need to reboot the server and see if the RAM drive maintains the same drive letter.

That's all I can think of.
 
Eeqmcsq said:
Hmm, let's brainstorm about possible gotchas...

- Losing the contents in the RAM drive - You already covered this - the original logs are stored elsewhere.

Of course, I don't know how "tech literate" your coworker is. He needs to understand that anything he dumps in the RAM drive can be LOST on a power failure. So although the original logs are stored elsewhere, he might starting using the RAM drive to create other files for himself, and he may not understand the risks of doing this on a RAM drive.

- As mentioned above, the time it takes to copy 10 x 500MB logs from your SAN to your RAM drive outweighs the benefits of the RAM drive - some performance measurements would be interesting to see here between directly analyzing the logs vs copying to RAM first.

- Not enough RAM leftover for other apps - You answered this by saying there will be enough.

- Running out of space on the 10 GB RAM drive - That will be up to your coworker to delete the previous logs in the RAM drive before copying the new log files into the RAM drive. Alternative, he can write a script to automate this.

- Drive letters - It's been a while since I've worked in Windows. If I remember correctly, a RAM drive is assigned a drive letter, and I think some RAM drive software blindly picks the next available letter. I don't know if you or your software need the RAM drive to be assigned at a certain drive letter. If yes, you will need to reboot the server and see if the RAM drive maintains the same drive letter.

That's all I can think of.
Click to expand...

In reference to my coworker - he definitely understands the risks. Besides, all servers reside in our data center and connected to redundant UPS's and a diesel generator.

We use 4 Gb fibre channel, so I'd be shocked if it took more than 20 seconds to copy 5GB from the SAN to the server.

I believe the server has 48 GB of RAM... this task is the most intense operation it performs. It normally has over 40 GB of unused RAM.

It'll be a while before the logs consume more than 10 GB as they're web server logs and we don't expect a twofold increase in traffic anytime over the next year.

Drive letters O through Z are available.

So all those areas seem to be covered. Thanks for the feedback. 🙂
 
Even if your RAM drive is order(s) of magnitude faster than an SSD, is there a chance your analysis could become CPU bound? Depending on the complexity of the analysis, your CPU may outpace a hard drive, but not necessarily an SSD. I've seen extracts that peg the CPU at 100% processing maybe 20-30MB/sec (which would be borderline for a spinning drive depending on load).
 
There's a chance... but it's not really a CPU intensive task. It's basically tallying numbers in log files from 10 different web servers to get aggregate numbers. Plus, I think the two Xeon X5680's in that server are robust enough to handle even heavy encryption or compression. Although these files are neither encrypted nor compressed.
 
I use the Dataram as well and have been pleased with it, I use it with an older version of Photoshop for its scratch drive.
 
Jeff7181 said:
Why would you use a RAM drive to benchmark storage?
Click to expand...

ramdrives are nice the only problem is you do not have always so much ram to make a big one
in 1998 era i had a pc pii 233 with 384 ram
i used 256 ram as a ramdrive and i was booting from that
damn windows was so fast i even had the office installed and had 20-30 mbyte free space!!!!
i was booting from a cd i made the ramdrive copy the files to the ramdrive and then continue booting to windows
ramdrive was takign the c letter so all was fine 🙂
 
I have an OLDE dell server (I believe c640, but don't remember model for sure), it came with 128mb of ram and a celeron.

Anyhow, now its a p4 with 1408MB of ram. I set up a 512mb ramdisk for a small minecraft server, just for storing the map. It runs much faster vs using an old slow IDE hard drive for the map. Should help to keep the hard drive alive a bit longer as well since it's writing a backup of like 160-170mb every hour to disk rather than writing to the disk nonstop while the game runs....

I'm one of the few people who still run slackware ... lol
 
The last time I played around with a RAM drive was about 5-6 years ago I think, and again about 5 years before that. I used a program called RAMdiskNT, which has a recoverable RAM drive feature (ie. an image of the drive's contents is saved to disk on shutdown and loaded on startup), but I found it quicker to script the unzipping of a zip file into a blank RAM drive on startup and only update the zip file when necessary.

This was on Win2k/XP, when Firefox might take 10 seconds or so to start when loading over IDE/SATA1. Thunderbird wasn't so bad, but OpenOffice took 10-15 seconds to cold start, so I installed them all on to the RAM disk, backed up the install folders into a zip file and updated it whenever a new version came out. The RAM disk allowed cold start times of about 1 - 4 seconds (4 being for OpenOffice IIRC).

At the time I was limited by available/maximum RAM for my system. It all worked pretty well though.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top