Allocating a big quantity of memory

[BrunoAmaral]

Hi,

I'm programming a project in c++ which is being develop by another person
using the QtCreator IDE. I'm using Code::Blocks.

I need to allocate a matrix of 51000 x 51000 floats. As its a simetric matrix,
it's half the value. But the problem is that, the other person using QtCreator
released the software and it doesn't happen a memory overflow error, it makes a
memory swap to disk (paging). In my case i can't do that, and my release is always
terminated with an runtime error due this lack of memory.

How can I force this swaping?

ps: the performance reduction is acceptable, so we just want it to go on, and
do the necessary allocation of that matrix.

Thank you all.

Bruno

[helios]

In my case i can't do that

Why not?

[andywestken]

How are you trying to allocate your array?

[BrunoAmaral]

Thank for the replies.

When i said i can't do that, it's because my application terminate with a Runtime Error, and don't
try to swap the data between heap and disk.

I'm allocation the matrix as follows:

this->distanceMatrix = new float*[dataset->getNumElements()];
for(int i = 0; i < this->dataset->getNumElements(); i++) {
      this->distanceMatrix[i] = new float[i+1];
}

Where dataset->getNumElements(), in this case returns ~51000.

ps: I'm allocating half of the matrix.

Thank you again.

Bruno

[helios]

Well, unless you have 4.84 GiB ((51000^2+51000)/2*4 bytes) of memory and a 64-bit system, you'll never be able to allocate that much memory. You'll need to store part of the matrix in a file and manually copy sections of it between memory and storage as go work with it. If you think you won't need to completely fill the matrix, a sparse matrix is a possible solution.

[BrunoAmaral]

Hi helios,

I agree with you, I don't have that useful memory, so it crashes.
The problem is that another release of the same application running in the same system doesn't
crash. Instead of this, my OS makes swap between heap and disk, without explicit management
of files.

I'd want to know if it's possible to configure it, and where I could do it.

Thanks.

Bruno

[helios]

another release of the same application

What happened between that release and the version you're working on that would cause it to just start crashing?

my OS makes swap between heap and disk

If you're running a 64-bit system, then the program shouldn't crash no matter how much memory you try to allocate (except in the absurdly high cases, of course). If you're not, no matter what you do, you'll never be able to get the system to cooperate with you. It's a physical limitation of the addressing system and there's no way around it.

[BrunoAmaral]

What happened between that release and the version you're working on that would cause it to just start crashing?

The versions is the same. But the release is build using QtCreator, and in my case it's been built in Code::Blocks, using minGW. However i tried QtCreator or gcc in linux, but it's the same.

I have 6GB of Ram, so It's not an absurdly high case. That's why the other release runs with no crash, but a little slower than expected.

What makes me think its not my system is because it cooperates with the other release,
but not with mine.

[helios]

I have 6GB of Ram

But when I said that allocating that much data required at least 4.84 GiB, you said

I don't have that useful memory

Does this run without crashing? Warning: this will allocate 5 GiB and try to keep as much of it in physical memory as possible.

#include <cstring>

int main(){
	const size_t n=5*1024;
	const size_t size=1024*1024;
	char **array=new array[n];
	for (size_t a=0;a<n;a++){
		array[a]=new char[size];
		memset(array[a],0,size);
	}
	for (size_t a=0;a<n;a++)
		delete[] array[a];
	delete[] array;
	return 0;
}

[andywestken]

As you mention MinGW, you must be running on Windows.

So is the "paged file" you refer to a memory mapped file?

[BrunoAmaral]

helios:

my notebook has 6GB, but the avaiable memory is less than this value, of course.

The code didn't run without crashing. It appeared the same Runtime Error as in my application.

andywestken:

Yes, i'm running Windows, but i didnt mention "paged file".
The question is that, with no changes in any configuration by myself, the code that I
try to program crashes.

I've heard in many places that the swap between memory RAM and Disk (secondary memory),
isn't efficient, but it's an option. In my case, all that I want is this option, cause the slower process
is acceptable in our application.

Thanks for the help, again.

Bruno

[Athar]

You need a 64-bit CPU, a 64-bit compatible version of Windows and you need to compile your program for 64-bit for this to work.
On Windows, 32-bit applications are limited to 1.5 GB of available address space by default. 4 GB is the theoretical maximum addressable with 32-bit, so 5 GB is off-limits either way.

[helios]

my notebook has 6GB, but the avaiable memory is less than this value, of course.

If you have n bytes of memory, all your other processes are using m bytes, and you need to run another process that requires o bytes, and o>(n-m), the system will swap out the memory of background processes to make room for this new process, effectively giving approximately n bytes of memory to it. So it doesn't matter what other processes are running on the system.

The code didn't run without crashing.

You did compile it for x86-64 (also known as x64) and not x86, right? Remember that I said that only 64-bit systems can allocate that much memory? The entire system -- CPU, OS, and program -- has to be 64-bit.
(Sorry. I really should have mentioned this earlier. I don't know why I didn't.)

[zb0S216C]

Are you sure it's crashing due to a memory allocation failure and not a heap corruption? Either situation can cause a program crash. What's your debugger got to say about it?

_Wazzak