Read Binary File Into 2D Array

[Gamingphreek]

Hey Guys,

I am trying to read a binary file (I believe it is a flat binary structure). The file is a bunch of velocity and position vectors that I can't get into; however, the file is ~515MB.

The array needs to be partially dynamically allocated if that makes any sense.
The array for my particular data set is [17, SIZE].

Since I am working with position vectors and what not, the values that need to be read in are doubles; however, it seems that ifstream.read() only supports reading in char*.

Can anyone help with this?

I know that in the C99 standard you do not need to call new to allocate that memory so in my header I have:

double* memBlock;

and in the file I have:

memBlock[17, chunkSize];

After that anything I try to use to read in the data crashes at run time? Any suggestions?

Thanks,
-Kevin

 

[EagleKeeper]

Set a pointer to the array structure.

Use a file read of x bytes.

If the data was written out in a binary block in the same structure format that you are using, there should be no problem.

Below is a code snipit that I am using for processing avionics data

CFile *pSegFile
...
AWSs2 *hdrSegs = new AWSs2[pHdr->segcount];

pSegFile->Read(hdrSegs, pHdr->segcount * sizeof(AWSs2));

 

[Gamingphreek]

Originally posted by: Common Courtesy
Set a pointer to the array structure.

Use a file read of x bytes.

If the data was written out in a binary block in the same structure format that you are using, there should be no problem.

Below is a code snipit that I am using for processing avionics data

CFile *pSegFile
...
AWSs2 *hdrSegs = new AWSs2[pHdr->segcount];

pSegFile->Read(hdrSegs, pHdr->segcount * sizeof(AWSs2));

The file, I believe, is unformatted and just continuous streams of doubles 1 after another.

outFile->read( reinterpret_cast<char*>(&memBlock), sizeof(double) ) SHOULD work for me, but for some reason it doesn't.

-Kevin

 

[slugg]

What I would do, if this were me programming... I'd create a struct whose members are those 17 components you're talking about, then make a STL Vector out of the struct type I defined, then you're ready to read in as many lines as you want. The STL will take care of everything else for you.

 

[Ken g6]

What do you mean by, "in the C99 standard you do not need to call new to allocate that memory"?

I believe that means you can have the following in a function:

double memBlock[17][chunkSize];

But in C++ you can't just create a pointer, start accessing a location, and expect it to be created for you. I've also never seen that comma syntax you used in C++.

 

[Gamingphreek]

Originally posted by: Ken g6
What do you mean by, "in the C99 standard you do not need to call new to allocate that memory"?

I believe that means you can have the following in a function:

double memBlock[17][chunkSize];

But in C++ you can't just create a pointer, start accessing a location, and expect it to be created for you. I've also never seen that comma syntax you used in C++.

Well - lets just say that I was tired - very tired this morning.

So basically, I have no idea why I had a comma instead of a second bracket set.

At any rate, my reinterpret_cast<char*> was a major problem. Instead I just referenced the char array with a double *, thereby indirectly casting the value.

With this I made a vector<double> theArray[17] to store the 17 rows by 'n' columns.

Everything works great now. I'm not entirely sure why the reinterpret cast was having problems; however, in talking to my boss, that is a risky function to call given that it is, essentially, a forced cast.

-Kevin

 

[Net]

did I hear dynamic array.

look at C++ vectors

 

[degibson]

Originally posted by: Gamingphreek
I am trying to read a binary file (I believe it is a flat binary structure) ...

mmap() to the rescue! You don't even need to malloc() 🙂

 

[sao123]

get and read are unformatted input operators... IE they read in chars.

to read into a double, you still need to use the extraction operators >>


inFile >> double;

 

[Venix]

Originally posted by: sao123
get and read are unformatted input operators... IE they read in chars.

to read into a double, you still need to use the extraction operators >>


inFile >> double;

>> is for formatted text I/O. As the OP stated numerous times, his file is binary.

 

[sao123]

Originally posted by: Venix

Originally posted by: sao123
get and read are unformatted input operators... IE they read in chars.

to read into a double, you still need to use the extraction operators >>


inFile >> double;

>> is for formatted text I/O. As the OP stated numerous times, his file is binary.

sorry, missed that in the OP.



I found an example online and I tested it, which seemed to work, and im sure all your coding is correct... but I also found an interesting tidbit with it, which could explain your problem:

It's also good to mention that the file you produce is only guaranteed to be readable by a program written using the same version of the same compiler that the output program was written with. A lot of people don't like that portability issue, and will read and write their binary values manually instead of using read() and write().

Not exactly helpful since your file is in binary, but a potential issue you might be facing.


Binary Example:

#include <iostream>
#include <fstream>

int main()
{
using namespace std;

ofstream ofs( "atest.txt", ios::binary );

if ( ofs ) {
double pi = 3.14;
double i_sqd = -1;

ofs.write( reinterpret_cast<char*>( &pi ), sizeof pi );
ofs.write( reinterpret_cast<char*>( &i_sqd ), sizeof i_sqd );
// Close the file to unlock it
ofs.close();

// Use a new object so we don't have to worry
// about error states in the old object
ifstream ifs( "atest.txt", ios::binary );
double read_in;

if ( ifs ) {
ifs.read( reinterpret_cast<char*>( &read_in ), sizeof read_in );
cout << read_in << '\n';
ifs.read( reinterpret_cast<char*>( &read_in ), sizeof read_in );
cout << read_in << '\n';

}
}

return 0;
}