Multi-Threaded Hashing

[BChico]

Looking for a multi-threaded command line executable that can run CRC or MD5 hashing on large volumes of data. (5TB+)

Anyone have experience working with an enterprise class type of tool? Most of the stuff that comes up on Google and shareware junk it looks like.

Thanks

 

[jrjiri]

Honestly when I need to work on large data sets I'll break it up by the # of cores on the server and just run all the jobs in parallel.. I've got some scripts for breaking up the data that we repeat runs on.. Everything else comes down to "is it worth the time to split it up" decisions.

I'd love to see what replies you get regardless as it's a great question. 😉

 

[kamikazekyle]

If you're running on Windows and want a quick and dirty solution, you can use BATPROC to launch n number of jobs for logparser. Logparser has a has function (HASHMD5_FILE). You can use any hashing tool in place of logparser.

You can also utilize Powershell, create an MD5 function (or use the Powershell Community Cmdlets), and then use it's background Jobs function to multithread the script. I a "multithreaded" Powershell script to collect audits from all our workstations.

As for an all-in-one-not-a-script solution, I haven't run across any that I've personally tried. I rarely have to hash a bunch of data, so trying to find a tool that's multithreaded wasn't a priority 😛

Though like jrjiri, I'm curious as to what you or others come up with.