Hello, I have been looking everywhere for some information on how to improve the output speed of large text files. Right now it takes ~3.25 minutes to write a formatted text file (~62MB). Commenting out all the outfile writes the calculations finish within seconds. My hard drive light hardly glows during this entire process, unlike when copying a file of similar size.
It's not something I have tried in C++, but my first thoughts would be to try building a whole line and then using a single outfile << theLine type statement to send it to the disk.
Agreed. The << operator actually has quite a bit of overhead.
The other thing to do is not flush (don't use endl) until you are finished with all output.
Thank you both for your insight. Minimizing the frequency of the << operator has had significant effects on how fast the text file is outputted. The convenience of the << operator << is useful in providing formatted text output, however, this apparently comes with significant cost in the output time efficiency in large files.
By formatting the text output myself into a string (acting as a buffer), and then moving the contents of the string to the file using the << operator, the output time dropped significantly from ~3.25min to ~1min for a 62MB file. Changing the size of the buffer (in lines) did not significantly increase the output time of the file. In other words, by changing the amount of lines that the string holds before dumping its content into the file did not affect the rate.
To determine the processing bottleneck that was taking place, I decided to forego the line formatting and see how fast it would take to output the same amount of lines consisting of a constant string. The answer: 7 seconds. So the next bottleneck has become my own text formatting code. At least it is significantly more efficient than formatting text using the << operator!
So my next endeavor will be to figure out a way to optimize my own text-formatting code. A good lesson here: Use << for small files for ease of output and coding; format the text yourself for large files and control over efficiency.
Thanks again both of you. If I get my time down any further I will try to finish up the thread with my code. As always comments are welcome. Thx :)