Reading a binary file without knowing type of data stored inside

[hyperfine]

Hi all, hope this title was descriptive enough. I'm a bit new to binary files so if I'm not explaining things properly please let me know.

Basically, I have a type of data file that is stored in binary format. The data file consists of a large header file, followed by a time series of voltage and current values. The software that writes the data from the equipment to the binary file also has a method for converting the binary file to a plain text file. Unfortunately, the plain text files are large and unwieldy. Therefore, I'd like to be able to open the binary file myself and process the data.

The problem I'm having is that it seems like if you don't know the type of data a sequence of bits is supposed to represent, it's difficult to convert the bits to a usable form.

For example, if I try to read in a certain number of bits into type char and then output the chars, the result I get is this:

A
B
F
2  //ABF2 is the type of this binary file--this part makes sense

//unintelligible ASCII symbols follow 

As I understand it, this is because only the first 4 bytes have a sensible char representation; the rest of the bit sequence may consist of any number of data types.

So, to tackle the problem I've defined a template function that will read in a particular data type T's worth of bits, and output the result:

template<typename T>
void readElement(std::ifstream& inputstream)
{
    T t = 0;
    if(inputstream.is_open())
    {
        inputstream.read((char*)&t, sizeof(T));
        std::cerr << "t = " << t << "\n";
    }

    else
    {
        std::cerr << "Couldn't open file!\n";
    }

    return t;
}

So I'm trying to treat this like a guessing game, basically read in a sequence of bits assuming they are supposed to represent a particular data type, and if something sensible comes out, that means I've figured out a certain part of the header.

Finally, my questions: Is this a worthwhile pursuit? Is it possible to figure out the format of the file by proceeding in this way? Is there a better way of figuring out how I should read the file?

Thanks!

[Peter87]

I think it's very hard to figure out how things are stored the way you are doing it. I mean, how do you know that a value is sensible? It's just numbers.

Isn't the file format documented somewhere? If you have access to the source code of the program that saves the file you can look at it to understand the format.

[hyperfine]

I think it's very hard to figure out how things are stored the way you are doing it. I mean, how do you know that a value is sensible? It's just numbers.

Well, the header should contain some information that I know beforehand. For instance, part of it should tell me what the sampling rate frequency is which I know beforehand. Also, once I get to the actual data it should be very easy to get the rest of the data since it should be formatted something like time->voltage->current, each with a constant fixed number of bits.

Isn't the file format documented somewhere? If you have access to the source code of the program that saves the file you can look at it to understand the format.

It's proprietary software, and as far as I can tell the company hasn't released much info on how to read the binary file. I've found some links that mention that such documentation is available, but all lead to dead links.

I know that some programs to open this file format have been written, but I'd like to learn myself how it could be done if possible.

[jlb]

If you have both a binary and a text file that was generated from the exact same data you can possibly, through trial and error and looking at hex dumps, be able to figure out the binary file format. However it would probably easier to just read and parse the text file instead.

[andywestken]

Is your ABF2 file an AXON BINARY FILE?

If so, it looks like Molecular Devices want you to use the DLL they provide to access AXON BINARY FILE data. But I see it's only available for Windows (and DOS!)

From the ABF User Guide
http://mdc.custhelp.com/euf/assets/software/FSP_ABFHelp_2.03.pdf

The AXON BINARY FILE has a proprietary format, however the files can be read (and created) by third-party developers by using the ABFFIO library.

which can be downloaded here:

Axon pCLAMP ABF File Support Pack Download Page
http://mdc.custhelp.com/app/answers/detail/a_id/18881/~/axon%E2%84%A2-pclamp%C2%AE-abf-file-support-pack-download-page

Is there a reason you don't want to use their library?

Andy

[dhayden]

The software that writes the data from the equipment to the binary file also has a method for converting the binary file to a plain text file. Unfortunately, the plain text files are large and unwieldy.

Well disk space is pretty cheap these days. I say spend the extra 1/10 of a penny and go with the text files. If necessary, compress them on disk and then when you process the data, decompress them and pipe the results right into your program. I fear that trying to reverse engineer the file format could take weeks and even then, how will you be certain that you've done it right?