Size limit on three-dimensional vector?

[jorgen]

I have a program where I want to insert random numbers into a three-dimensional vector. It works fine up to a certain size and then I get Aborted (core dumped) when running the program (it compiles fine with a warning about converting from double to int, which is intentional).

Is there actually a size limit on vectors? If so, is there a workaround?

Below is the program. It works fine with these values and cs up to about 8000; any larger than that and it aborts.

int main() {

	int rs=100, s=200, cs=8000, i, j, k, seed = time(0);
	vector<vector<vector<int> > > r(rs, vector<vector<int> >(s, vector<int>(0)));
	CRandomMersenne RanGen(seed);
	
	for(i=0;i<rs;i++) {
		for(k=0;k<s;k++) {
			for(j=0;j<cs;j++) {
				r[i][k].push_back((int)cs*RanGen.Random());
			}
		}
	}	
						
	return 0;
}

[hamsterman]

The memory used by your vector is = rs*s*cs*sizeof(int) = 100*200*8000*4 = 640000000 B = 610 MB (plus some more).
I can see why that would not work..
How much memory do you have at all?

[jorgen]

Right... Hadn't thought of that. I actually have 3 GB of memory (this is RAM right?) but still this must be the problem. I gotta find another way to do it then. Thanks!

[Computergeek01]

(this is RAM right?)

I for one am glad you asked this because 70 out of a 100 people will tell you "Yes, your system memory is your RAM". But that's not correct, it's just what that guy they know who is really good with computers told them.

Your system memory is actually you physical RAM PLUS your systems Page File. The Page File is basically space on your Hard Drive that your computer uses to store data it doesn't need right away but that is still allocated to a running process.

EDIT: This is relavent to you because it means that even if you are allocating twice as much space as that you should still have plenty more memory left.

Where is the "Aborted" message coming from? Is it the IDE's debugger by chance?

[kev82]

The `plus some more' is only 600K if I've worked it out correctly.

610mb is not that much to allocate. It runs quite happily on Mac OS X. I've had memory issues on Windows XP before, but I think it was more than 610mb.

My guess is the vector reallocation policy is doubling as you fill it. This leads to lots of new/delete and a very fragmented memory space.

Add a new line between line 8 and 9 which says

r[i][k].reserve(cs)

See if that makes a difference.

[Computergeek01]

@ OP: Again I personally don't think it's a memory limit issue because, on Windows at least, you would get a very different error. But to put that theory down for good bring up Process Explorer (Or it's *nix equivalent) and tell us how much memory is allocated to the process just before it crashes.

[jorgen]

Hmm I guess there must be another problem then.. Good, still a chance it might work. Thanks for replies!

Re Computergeek01: Ok I see.. The message comes from the cygwin bash shell I use to run the program (with the command ./<progname> ). It is compiled using g++. It goes up to 650000KB before it crashes (found using task manager).

Re kev82: I tried inserting that command (I don't know vectors very well) but still get the same error.

[kev82]

First thing is to rule out memory, write another program that just allocates a single dimensional array.

int *x = new int[610*1024*1024/4];

and see if it runs or gives the same error.

[jorgen]

Re kev82: Yeah I do get the same problem doing it like that. However I get a hint because now the compiler warns about integer overflow.. So maybe that's it. However even with long long int the same thing happens, so I dont know.

Anyhow I found another way to do it in the actual program I'm writing.

[quirkyusername]

may I ask why you need such an enormous vector of ints? Do you need every single one of those elements? If it's the case that you want to use a vector because it represents the kind of structure you want very intuitively, but you don't really need every single element, make it a vector of pointers to ints and then just allocate on the fly if you need to use an element.

[jorgen]

I need it in connection with a bootstrap computation. The cs's number different data sets, the s's number different data in each set and rs is the number of resamples in the bootstrap.

Now this data is to be used to compute some different quantities, thus being inside yet another loop. At first I thought I need the resample to be the same for each of these quantities, thus I stored what data set should be used for each resample of each diferent data type; the integers then number data sets.

But now I'm thinking that since the standard deviation of each quantity is computed independently I don't need the resamples to be equal - and therefore I can pick random numbers every time and forget the whole vector.

In answer to your question, yes I think I would need every element if I did it with the first method.

Did that make sense? Hope so.

[kev82]

Just out of curiosity, why are you using C++ instead of R?

[jorgen]

I never heard of R before now actually. So a language good for statistical computing?

The main part of my program does a monte carlo metropolis computation. I don't know if that changes the usefulness of R or not, but anyway I've been working on the program for too long to change language now:) I will make sure to check it out later though.

[kev82]

You can have R call C functions for particular calculations if implementing them in R is too slow/difficult.

Oh well. No matter. Worth looking into in the future though, because I know it has modules developed for bootstrap sampling, amongst a million other things.