Access whole file at once with ifstream?

[mahkoe]

So I want to make a data compression program to test out my idea, but I'm met with a problem. The algorithm I will eventually develop requires that I have access to ALL of the file to be compressed at once, which means RAM usage would be through the roof. Am I going to have to redesign the entire algorithm or is there some tricky way to get around this?

Extra notes:
If you need to know/are interested in the idea I plan on using for this project, you can find it here:
https://sourceforge.net/p/c2a/home/Data%20compression%20idea/

Okay, that's it, bye bye!

[Mathhead200]

Why does the file need to be in memory all at once? Can't you read it piece by piece, and then re-read it over if you need to go back. The file's not going anywhere as long as your program has an open stream attached to it...

Can you describe your problem a bit more?

[mahkoe]

Okay, the idea takes the numeric value of a file and translates it into a series of mathematical operations. To do this I need to know the numeric value of the file and therefore need to have access to it in its entirety at all times

[ModShop]

If I read your article correctly, your algorithm will take the integer value of each byte in the file, and combine to one huge integer, correct?

file:
!!!
resulting int (from int value of each byte)
323232 or '32' '32' '32'

Then, you find the quickest formula that generates this number, and store it into the compressed file, right?

Perhaps a bigint library such as GMP could help you, if I have the right idea.

[mahkoe]

thanks for the reply

No, I take the integer value, such as 10000000002 and "reduce" it to a series of mathematical operations, such as 10^10+2. Also, if you could elaborate on this biginit or send a link, that would be nice.

Thanks again

[ModShop]

http://gmplib.org/

That is what I was trying to say:)

For loading, you may just have to load the whole file at once. Perhaps you could load up to a certain size (say... 500MB), then compress that chunk, then load another. The compression ratio for HUGE files would be hurt, but not too bad I wouldn't think.

[mahkoe]

Okay, I think I have all I need. Thanks for the help, I am now marking this thread as solved

[Duthomhas]

There is also the concept of memory-mapped files. Boost has something to help you with them cross-platform.

[Galik]

I would think you can just use std::istream::seekg() to locate any part of the file you need:

std::ifstream ifs("to-compress.txt");

char c; // char to read
ifs.seekg(30); // move read cursor to char #30 in the file 
ifs.get(c); // read char #30 

http://cplusplus.com/reference/iostream/istream/seekg/

[ModShop]

I made a dummy program to test this concept, with *awesome results. I managed to get a 894 byte text file into just over 34 bytes. However, my formula generating algo took almost 40s to execute, and I'm pretty sure its not creating the smallest (or close to) formula. I already have some optimizations in mind that will reduce size and execution time, but have yet to implement them. I will post some code when everything is done.


*with some not so awesome results as well

EDIT: I'm an idiot. The measured time was from a test run containing debug output, significantly slowing down the program. Without the output, the program took just over 10s to compress a 5.14kb file.