Program crash on pointer to array of pointers

[jwoolsto]

I am working on an engineering program that analyzes potentially billions of different designs to determine the best configuration and am having some trouble with an array. The largest contiguous array I have been able to successfully run the program with is 836,501,504 entries of type double (~6.23GB if I'm figuring right). At the moment this is sufficient but future expansion will require at least an additional order of magnitude. I wanted to keep all of the data in program memory rather than a database or binary file of some sort if possible (seems like I should be able to, maybe Im wrong.) Either way, as I said, I can create the on contiguous array.

My solution to getting a larger array was to create an array with the pointers to the first entry of a bunch of smaller arrays, that way I wouldn't run into not being able to find a super large contiguous block of memory. The attached snippet compiles fine, but reports the following error when I run it:

"EXC_BAD_ACCESS (SIGBUS) KERN_PROTECTION_FAILURE at 0x000000000000000a". The system is OSX 10.5. The error occurs on the second line (the new function). My suspicion is just that I've run out of room on the heap, but I would like someone's input on the matter, both regarding the nature of the problem, as well as any possible solutions.

Much appreciated,
Jared

//Creating an array of pointers to arrays. This is how the large memory is handled. Each entry is a smaller array, allowed to be strewn through memory in multiple blocks rather than one solid block. 
	double **grossWeightArray;
	*grossWeightArray = new double[17];
	
	grossWeightArray[0] = grossWeightArray1;
	grossWeightArray[1] = grossWeightArray2;
	grossWeightArray[2] = grossWeightArray3;
	grossWeightArray[3] = grossWeightArray4;
	grossWeightArray[4] = grossWeightArray5;
	grossWeightArray[5] = grossWeightArray6;
	grossWeightArray[6] = grossWeightArray7;
	grossWeightArray[7] = grossWeightArray8;
	grossWeightArray[8] = grossWeightArray9;
	grossWeightArray[9] = grossWeightArray10;
	grossWeightArray[10] = grossWeightArray11;
	grossWeightArray[11] = grossWeightArray12;
	grossWeightArray[12] = grossWeightArray13;
	grossWeightArray[13] = grossWeightArray14;
	grossWeightArray[14] = grossWeightArray15;
	grossWeightArray[15] = grossWeightArray16;
	grossWeightArray[16] = grossWeightArray17;

[helios]

On line 3, you're doing the equivalent of initializing grossWeightArray[0] (*grossWeightArray is the same as *(grossWeightArray+0), which is the same as grossWeightArray[0]), but you're not actually initializing grossWeightArray, which you should do like this:
double **grossWeightArray=new double*[17];

I also want to point out the futility of loading 6.23 GiB onto memory, when each element will only be used once. It's much more efficient to just load blocks from a file (assuming that's where you're getting your data from), or, if you're generating the data on the fly, to just not store it.
I mean, if you've determined that's not the best configuration so far, why keep it in memory?

I also think there's probably be a better approach to this problem than brute force. Specially considering the size of the space (i.e. the amount of possible configurations).

[jwoolsto]

The program is highly iterative, and multi-threaded. While it is certainly true I don't need to have ALL entries in memory at once, my programming skills are admittedly not sufficient to attempt to synchronize everything for a compile time indeterminate number of configurations to files. Due to the complexity of the actual analysis, I would like to get that portion working first, with the intention of modifying the management code later for more efficiency....hopefully by then having the knowledge sufficient to keep everything under control, but minus one variable to worry about screwing it up. Working with smaller data sets now isn't an option either as they would not demonstrate that all portions of the solver were working correctly. For what its worth, I suspected what you've confirmed helios, so many thanks for it.

Thanks,

Jared

[Zaita]

I'd like to point out, that the method you've described doesn't seem to be easily multi-threaded. Especially if your going to be doing pointer manipulation without having critical sections or mutexes. This will surely lead to undefined behaviour and undefined results.

[jwoolsto]

Well along those lines Zaita, Ive made two changes since. I took helios' suggestion and started dumping the data to a binary file. Also, while messing around with getting XCode to find the boost libraries, I determined that what I want is really more along the lines of multiprocessing, which leads me to another question which Ive posted in a new thread. Thanks for the input.

[Zaita]

I think your confusing multi-threading and multi-processing.

Multi-threading will work on a system with multiple cores, or multiple CPUs. The OS will handle moving the threads to new CPUs/Cores without your help.

Multi-processing is done by running different instances of the applications on separate machines and using inter-process communication (IPC) to communicate between them.

I don't think your problem is multi-processing. And if it was, it'd be ideal to multi-thread the application first to make use of 4 and 8 core systems before looking at multi-processing (as it's a lot slower).