• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

Mass storage of files

A

AFB

Lifer
I have this issue where I need to store large amounts of files (compressed sparse matrices) in a highly structured, efficient way. My problem is these files are produced from a large number of heterogeneous scripts. This would be simple, except they use different languages. I have some using C/Fortran libraries, and others using matlab.

I've researched this topic and come up with a number of ideas, but there's not a whole lot of research in this area other than image storage.

Idea 1: Write a C lib that stores the output file in DB itself. This could be used in matlab, c, but I don't think fortran.

Idea 2: Write a standalone program that should be able to be called from any of the above that takes a file location as input and does the rest.

Idea 3: Have each script write to the specific temp folder (did I mention this has to scale?) and then another program collect the output and store it.


Suggestions?
 
If you want it to scale, do not store large amounts of binary data in your database via a blob field or similar. It would certainly be smart to use a database to help organize the data though 🙂

 
e.g., with gzip:

gzcat myfile.gz | myscript_or_myprogram | gzip > myoutputfile.gz

I presume you'd like to replace gzip with your own representation... should be pretty trivial.

One thing C, Fortran, & most scripting languages can do: Read/Write to/from stdin/stdout.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top