Not enough memory for c++ code???

[studentffm]

Hi,

I have the following problem: I have a program which reads from a tex file and makes some calculations. The text file consists of several thousand data. When I wrote the program I used a few observations for testing purposes (in order to spend time) Now I run the program with the whole data set and after some times i get the error: The program is not responding. I assume this error is due to lack of computer memory.

In my view it is not reasonable to post code here because I have several hunderd lines...

I would be happy if someone can give me a clue what should I look for in the code in order to fix it (a general idea what could be the reason for the problem)

May be some more background info:
The code reads the text file in parts. When it makes the calculations with the data fom the first part it takes the data from the second part makes calculations ans so on... When I use very few data (e.g. 100 observations within 5 parts) everything works, but when I use the whole data (50000 thousand observations within 5 parts) the program stops working after making the calculations in the first part
Maybe the following code could be important:

	const int maxcount1 = 6000;
	const int maxcount2 = 6500; 
	static double data_parts_options[maxcount1][maxcount2];// Array for the data in each part
	int parts = 0;

10x

[chrisname]

You allocating 6000 * 6500 * sizeof(double) = (probably) 312,000,000 bytes on the stack might have something to do with it.

I'm fairly sure your stack, which is probably about 16 kiB can't store 300 MiB... Allocate that on the heap (it probably still won't work for people will less than 512 MiB memory though).

Use the new operator: http://www.cplusplus.com/reference/std/new/operator%20new%5B%5D/

[helios]

http://www.cplusplus.com/doc/tutorial/dynamic/

[DrakeMagi]

Found this on the web. Wonder if this is the right!

#include <pthread.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
   size_t stacksize;
   pthread_attr_t attr;
   pthread_attr_init(&attr);
   pthread_attr_getstacksize (&attr, &stacksize);
   printf("Default stack size = %d\n", stacksize);
   
}

// gcc -o stacksize stacksize.c -lpthread 

[tummychow]

I'm not sure what pthread is, but dynamic memory is part of the language where as that is probably nonstandard. DMA should work fine.

[chrisname]

pthread.h is the header file for the POSIX extension for threading. OP is likely using windows, and therefore won't be able to use pthread. At any rate, threading has nothing to do with memory allocation, so it's of no use to us at all...

that is probably nonstandard

It's not technically 'standard', but then, neither is win*.h. It's part of the operating system API.

@OP,
just use the new and delete operators as per the articles me and helios linked you to to allocate that memory on the heap and not on the stack.

[studentffm]

How can I do it by using realloc or more specifically how does realloc work with 2D array?

This an extraction from my code (hopefully this is enough in order to understand what I want to do...)

FILE *f_data;
double input_data
//some other code
while((fscanf_s(f_data, "%lf", &input_data) != EOF))
       {
		    count++;
			data_parts_options = (double**)realloc(data_parts_options, count*sizeof(double*)) //how does realloc work with 2D array???
//some calculations
}

and data_parts_options[parts][count] is a 2D array

And this is the problem...

I have the following problem: I have a program which reads from a tex file and makes some calculations. The text file consists of several thousand data. When I wrote the program I used a few observations for testing purposes (in order to spend time) Now I run the program with the whole data set and after some times i get the error: The program is not responding. I assume this error is due to lack of computer memory.

[chrisname]

You'd have to realloc() for each second dimension, e.g.

for (int i = 0; i < dimensions_in_array; i++)
    for (int j = 0; j < elements_per_dimension; j++)
        // Do the realloc here 

Like I said, it's not due to not enough memory, it's because you don't have enough stack space. I can't remember if the stack is usually stored in data cache or memory (my bet is memory, because it's larger; but cache would be alot faster). That's why we're using DMA, right?

[studentffm]

But realloc() is a kind of DMA isn't it??

should realloc consider the 2D or not how is the right synthax in a 2d case:

data_parts_options = (double**)realloc(data_parts_options, count*sizeof(double*))//now only count is considered

or

data_parts_options = (double**)realloc(data_parts_options, count*parts*sizeof(double*))//now both count and parts are considered

10x for answering my questions

[chrisname]

Yes, it is. I thought you were pushing these things on the stack. My mistake.

[studentffm]

I tried with new and the first loop works again fine but when the second loops should starts i get the following error:

First-chance exception at 0x0133b3c3 in program.exe: 0xC0000005: Access violation writing location 0x01190000.
Unhandled exception at 0x0133b3c3 in program.exe: 0xC0000005: Access violation writing location 0x01190000.

my code looks like this:

double **data_parts_options = NULL;
// declaration of some other variables
data_parts_options = new double*[maxcount1];
for( int d = 0 ; d < maxcount1 ; d++ )
{
data_parts_options[d] = new double[maxcount2];
}
	count = 0;
while((fscanf_s(f_data, "%lf", &input_data) != EOF))
{
count++;
if (input_data == 9.999)
{
//do something
}
else
{
data_parts_options[parts][count-1]=input_data; //At this point I get the error
}
}//close loop
for( int i = 0; i < maxcount1; i++)
{
delete [] data_parts_options[i];
}
delete [] data_parts_options ;

The values for maxcount1 and maxcount2 are 15000 for each variable... It should be enough...

[chrisname]

You're writing to memory you don't own. I don't understand why that matters with segmentation and paging, what with the virtual address space: you should be able to pretend you own everything from 0x0 to 0xFFFFFFFF; but then, if that memory doesn't exist in the physical address space... you can't map it.

Anyway, that's the problem. You're trying to access memory you don't own. I've never worked with multi-dimensional arrays; so wait for someone who has.