Fastest way to write a file

[rachitagrawal]

I have a file to be written:

So to minimize the number of system calls to write(), I am dumping my lines into stringstream ss and then after n lines I am dumping them into the file. This does give me some performance gain wrt to writing line by line.

But, I am not sure if it correct or not, but I had read it somewhere c++ functions are slower than c function calls. So, will it be a better method to use fprintf() or any other method?

In short, I want to know the best method to achieve file writing wheere I have million lines to write to a file?

Thank you.

[kbw]

The stream's buffering will improve performance by reducing the number of OS write calls. You could increase the stream's buffer size to improve performance.

[KrishnaV]

if you have access to a memory mapped file library, thats another way to improve performance significantly.

[helios]

The more bytes you write in a single I/O call, the less time you'll spend waiting for the device to catch up. If you can store the entire data set in memory and write it just once, then do so. Otherwise, just write to a buffer (e.g. an std::string) and once it reaches a certain limit, flush it to the file. Basically, exactly what you're doing, but the limit should be in bytes or characters, not lines, and be as large as possible. 1 MiB would probably be fine, but the larger the better.

EDIT: Oh, and it's also important to call reserve() to minimize reallocations.

[rachitagrawal]

Actually what I noticed is that after certain amount of buffering there is no change in the performance. So, with experimentation, I chose 1000 lines which counts around 16K bytes.

But, my main concern was, does it help in using c function rather than using c++ function. Because I had read it somewhere fprintf is faster than cout. So, if that is true which is fastest of all possible ways?

[choisum]

suggest you could try it. The test program is straightforward. An actually, the result may be compiler/library dependent, so goot to verify for yourself on the actual platform.

[helios]

Maybe if you're making a lot of small calls. After a certain point, the real bottleneck is the device you're writing to.

[rachitagrawal]

@helios Actually, as I was increasing the number of line I buffer, from 1 to 1000, performance got better but after that no change. So, I guess it can't be small calls.

Here is something I am trying to write a binary file instead of ascii for performance and space, but I am not getting the binary file written. I can open and read the file as text file. So, it seems somehow it is still writing in text(ascii) mode. Can someone point out the mistake?

int main()
{
    int i;
    stringstream str(stringstream::out|stringstream::binary);

    for(i = 0; i < 1000; i++)
    {  
        str<<i<<" "<<(i%2)<<" "<<i+100<<endl;
    }
    ofstream datFile;
    datFile.open("test.dat", ofstream::binary);
    datFile.write(str.str().c_str(), str.str().length());
    datFile.close();

    return 0;
}

[anotheruser]

try:

datFile.open("test.dat", ios::binary);

datFile.write((char *)&text, text.length());

where text is a string variable with your string content. Anyway you'll need something more to read the file, like saving the length somewhere i think, won't you?.

[Dudester]

@anotheruser
Why not just use string::c_str()?

[firedraco]

Um, (char*)&textwill not do ANYTHING similar to what you think it will.

c_str() is the ONLY correct way to get a char* out of a string (like Dudester stated)

[choisum]

So, it seems somehow it is still writing in text(ascii) mode. Can someone point out the mistake?

When you stream an integer into a stringstream it's converted into text.

[rachitagrawal]

@choisum But since the stringstream is also opened in binary mode should it convert it into binary rather than text?

If that is the mistake, what should be the correct way to write a binary file using stringstream??

Thanks

[helios]

Oh! You were expecting i%2 to be written as it binary representation? No wonder I couldn't find the problem.
No, the integer will be converted to a decimal representation as soon as you push it into the std::stringstream. It doesn't matter how a stream is opened, only how data is sent to it. operator<<() always sends human-readable data. std::ios::binary merely affects how newlines are handled. Specifically, whether LF will be converted to the system default (no change on UNIX, convert to CRLF on Windows, and to CR on MacOS). std::ios::binary performs no conversion, while the default mode does.

This should work:

long temp=/*...*/;
str.write((char *)&temp,sizeof(temp));

Note that the file will only be readable by a machine with the same endianness. If this is not acceptable, you'll have to do the integer-to-byte-stream conversion yourself by shifting bits.