How to write a spec to store floats in a data file?

[IHateMyJob2004]

How would you spec the storage of floats in a data file (binary data file). It's not as easy as you might think as there is no standard for storing a float in memory or anywhere pretty much (lie a network which is big endien). So, floats are not easily cross platform.

BDC is one option but that would blow up the file size. I need to keep this file compact so I need to use only 4 bytes.

The Interface document needs to specify exactly how to store a float.

NOTES:
http://en.wikipedia.org/wiki/IEEE_754-1985

Some people might not understand the problem, but trust me ... it is a real world issue. I need to do this because I am dealing with two vendors and 2 much different architectures.

From my testing, Wintel machines place things in the order shown here:
http://en.wikipedia.org/wiki/F...ating_Point_Format.svg
Bits 0..7 are byte 1
Bits 8..15 are byte 2
Bits 16..23 are byte 3
Bits 24..31 are byte 4

 

[Ken g6]

You act like the binary representation in memory for both platforms is the same. If that's the case, can you just cast your float to an unsigned int (or unsigned byte, and access it as a length-4 array)?

Or are you not using C?

 

[Crusty]

Originally posted by: Ken g6
You act like the binary representation in memory for both platforms is the same. If that's the case, can you just cast your float to an unsigned int (or unsigned byte, and access it as a length-4 array)?

Or are you not using C?

I think the point is that he doesn't care what the in memory format is, he needs to specify the serialized format so that interface implementers know how to parse the float from the datafile into memory.

 

[EagleKeeper]

This is what we use for some FAA documentation

s:8:23 Integer representation of float (s = sign:8bit = exponent: 23 bit = significand)

Notes on Number Types
Based on the example of an ?Integer representation of float?, an application must do the following to use this value.
a) Extract the value into a 32bit signed data type (Int32).
b) Divide the value by 2^23. (2^23 == dec: 8388608, hex: 0x800000)
c) Store into a Double.

 

[DaveSimmons]

"IEEE 754-1985, 64-bit, little-endian" ?

Parsing won't be needed for x86, and for almost everything else they'll at most need to read then reverse the byte order before casting to a double.

 

[degibson]

Store the whole thing in Network byte order.
When you write out the floats, use htonl() on each one.
When your read in the floats, use ntohl() on each one.

 

[IHateMyJob2004]

Originally posted by: Crusty

Originally posted by: Ken g6
You act like the binary representation in memory for both platforms is the same. If that's the case, can you just cast your float to an unsigned int (or unsigned byte, and access it as a length-4 array)?

Or are you not using C?

I think the point is that he doesn't care what the in memory format is, he needs to specify the serialized format so that interface implementers know how to parse the float from the datafile into memory.

In actuality, it is probably a non-issue but I want to have a well defined interface. There are going to be more hardware platforms down the road.


degibson,
Network byte order is big endien. There is no notion of endieness in floats. The problem lies in the fact that there is no cross platform standard for storing floats. Because of this, if you did what you said with htonl() and ntohl(), you could have two pieces of hardware talking to each other and the floats will come out garbled on the other end.

What really needs to be done in an interface document is to explicitly state what each bit in the 4 bytes are.

Common Courtesy kinda hit hte nail on the head.



In my document, I decided that I will state what the IEEE spec is. State what the bits mean from 0 ... 31. Then state specifically how those bits will be packed into the file. When in doubt, be as detailed as possible in documentation. The last thing i need is a20 meg file that doesn't work when integration testing comes.

 

[degibson]

My bad for not parsing your OP (pun intended).

What is the problem with SPEC: "/cite IEEE 794"? Too verbose for your spec? There is certainly something of a gamut between Common Coutesy's post an 794 -- how much detail do you want?

 

[imported_Dhaval00]

Originally posted by: degibson
My bad for not parsing your OP (pun intended).

What is the problem with SPEC: "/cite IEEE 794"? Too verbose for your spec? There is certainly something of a gamut between Common Coutesy's post an 794 -- how much detail do you want?

Let the man take credit for it, hater! If not pay raises, then at least we can earn some [more] goodwill this year .

 

[IHateMyJob2004]

Seems some are still confused.

The IEEE spec only states that floats are 1 sign bit, 8 bits for the exponent and 23 bits for the decimal. That leaves hardware vendors free to store these bits in memory in an bit order they would like. So, when you "stream" the 4 bytes to disk on hardware A, the result can be much different than hardware B. Therefore, when you write a specification for a data file or network message for example, you need to explicitly state what the bits are that are being streamed.

If drive space were not an issue on my project, I'd probably just use BCD. Thee is little confusion as to what BSD does granted you need to write that down in documentation to.

 

[degibson]

Originally posted by: IHateMyJob2004
That leaves hardware vendors free to store these bits in memory in an bit order they would like...

I had no idea 794 left actual hardware bit positions up for grabs. I feel like I spent a lot of time in college memorizing exponent and mantissa orders for no good reason now

However, now that we're on the same page... why don't you write your spec in the form of a C bit field, unioned with a 32-bit int? If that bitfield happens to correspond to x86's 794 flavor, so much the better. This is essentially what CommonCourtesy said, except you'll get 32-bit representation of 32-bit floats.

 

[EagleKeeper]

For those of you the choose to fly vs drive, your lives are dependant on the documentation that was previously posted.

 

[DaveSimmons]

> The IEEE spec only states that floats are 1 sign bit, 8 bits for the exponent and 23 bits for the decimal.

Then how about:

"IEEE 754, 64-bit, x86 bit order" ?