Fastest way to input/output dynamic array of arrays into a file.

[Fadey]

First example: input/output a array of std:string.

Second example: input/output a class which nests other classes inside.
As long as there aren't any dynamically allocated variables this works:
fwrite(&classAndClasses, 1, sizeof(classAndClasses), file);
HOWEVER, I don't know if that is safe.

Sizeof seem to return a class size which depends on what's in there.
One int32 and size is 4byte,
2 int32 and the size is now 8bytes and so on...

If we nest dynamically allocated variables in, a function must be written
what instructs the fwrite/fread properly so instead of writing just a pointer, it will write it's content and size. That special read/write function in that class will call read/write on it's child classes and child classes on their childs and so on...

Problem comes in that when stream flow is not continues, process slows down drastically. Assuming it is because other processes get a chance to use device and force it to switch to a different position. It's the position switching what slows down HDD, I heard.

One solution would be to use stream buffering.
Have one big array what we prepare with data and then write that to minimize fread/fwrite calls.

I suspect that at least for fwrite, there already is some buffering.
It's buffer size doesn't seem to be big tho.
I'm afraid that if I make my own buffering, there will be double data buffering.

Thoughts?

[salem c]

How about you post an example of what works.

Before beginning much speculation and heresay on what might be "faster".

I mean you start with a statement about std::string, then launch into talk of fwrite, which is NOT compatible with std::string without a lot of messing about.

[jonnin]

the fastest way is a binary file and you just directly read and write the bytes, as you are showing.

it depends on the hard drive. SSD don't care if you jump around as much as a rotating disk, which is like an old time record player, if you want to play a specific song you lift the head, move it, put it back down, takes time to align it ... SSD just go there, no head to move. Raids and such can generate significant boost as well. Lots of things can slow down disk access. Fragmentation of the files, type of disk, sector size, device contention, and more. One point to make here is that if you REQUIRE top notch hard disk throughput, you need a dedicated system that you control; an embedded solution not a typical PC user solution. Otherwise, you MUST make your code smarter so that it works well when the throughput is terrible (and it is, on many untuned home systems that have fragments, viruses, background downloads/streams, web and OS writing to the disk nonstop, and such all going on).

sizeof is fine until you change platforms, then you need to watch it more carefully if you want your files to be readable cross platform due to byte alignment of structs/classes (which you can govern in compiler options too). Do you understand the container/pointer problem and fwrite? If you fwrite a class with a pointer, it write the pointer (the address) not the DATA. (this is the string problem salem mentioned as well, containers that have pointer data also need TLC).

buffers work. Ive been known to read big swaths of a big file into memory as just a byte array and hop around in that, swap it out if I need a different chunk of the file (be smart here). But I am kind of a code barbarian ... I just take the direct approach to stuff when I get annoyed with the tools.

[dhayden]

If you do your own buffering, then use open/close/read/write instead of fopen/fclose/fread/fwrite. But I wouldn't do my own buffering. Instead, see if your system supports setbuffer() to set the buffer used by a FILE. If you make it 1MB or so, you should be fine.

Open the file in binary mode so the library doesn't have to do any end-of-line translation.

These two things should allow you to read/write as fast as the disk will let you (or nearly so).

If we nest dynamically allocated variables in, a function must be written
what instructs the fwrite/fread properly

Yup. You're the one who has to write the functions :). I recommend creating read() and write() methods for each of the classes. This makes the code straightforward:

class MyClass {
public:
    size_t write(FILE *fp);
    int i;
    double d;
    OtherClass member;
};

// write MyClass to a file. Return the number of bytes written.  Sets errno to zero on
// success and non-zero on failure.
size_t MyClass::write(FILE *fp)
{
    size_t total=0;
    errno = 0;
    total += fwrite(&i, 1, sizeof(i), fp);
    total += fwrite(&d, 1, sizeof(d), fp);
    if (errno) return total;  // check for error since member.write() below might clear errno
    total += member.write(fp);
    return total;
}