How can I read 2G bin file to memory?

[Neverbibi]

It seems that "new" cannot ask for the memory larger than some kind of number. Is there something I didn't know? How should I do that?

[kbw]

It's not a fixed number. It's the amount of free virtual memory on the machine. WIN32 has a 4G physical limit, 2G is available to user programs.

[Neverbibi]

Oh, I'm programming in win32 environment, so is it impossible to read such big binary file to memory? Can I reset the virtual memory of my machine?

[kbw]

No. You cannot read a file that large into memory all at once. There's simply no room for it. Do you really need it all in memory at once?

[Neverbibi]

Yes, I do really need loading them at once for some calculation. Can I find some way to use the virtual memory? And a question I encountered is that My machine's memory has 4G capacity, why I still cannot ask for sufficient space?

[z05DSL3A]

http://www.brianmadden.com/blogs/brianmadden/archive/2004/02/19/the-4gb-windows-memory-limit-what-does-it-really-mean.aspx

[Incubbus]

http://msdn.microsoft.com/en-us/library/aa366556%28VS.85%29.aspx

[chrisname]

There's no way you'll be able to; unless you have a 64-bit machine and more than 2GB RAM free; you'll never get that much in RAM all at once. I tried merging some half-GB files (opening them in reverse order and appending them to the end of the first file, then closing it to go on to the next one) and even that wouldn't work; and I had 3-4 GB free at the time...

[jRaskell]

There's no way the computer itself can physically process 2G of data all at the same time. I'm sure there is some way of streaming the data in and out of some sort of processing pipeline. How that works is entirely dependant on the type of processing you want to do. Whether we're talking about audio data, image data, audio/video data, some form of sensor data, encrypted and/or compressed data, etc. it's possible to process it without loading it entirely into memory. The code to do so will likely be a bit more complicated though.

[chrisname]

I guess you could load parts of it at a time into memory; if you don't have to process the whole file at once you could do it line by line, or 100MB by 100MB or something.

[Incubbus]

while using windows u could use the file mapping techniques showed in the link above

[kbw]

File mapping doesn't magically get arround address limits.