Increasing cache memory

[tomasito]

Hello everybody!

I saw something strange in a program I coded, however I am not sure if it is really a problem.

So, what I want to do in the program; I have 2 arryas in which I store results of calculations, they are getting that huge, that I am running out of RAM. So at a certain point I save the arrays to a file, put them back to 0 and continue my calculations.I open the files before my calculation loops and close them after all the calculation, so they stay opened for the whole time.

To use as much RAM as possible, I was observing the memory consumption on a system tool (called KDE system guard), there I saw, that the program reserves physical memory which stays equl during the whole time, however this program shows me also the cache memory and this one increases at each time I have written the data to the file. It seems that the function 'write' from the <fstream> library, takes a bit more cache memory each time(I figured out, that it has to be this command by quitting the others command, without this command it doesn't occur).

Do I have to worry about the cache memory? Why is it increasing with this function?

I am using Lunix (Red Hat) with a gcc compiler.

Here the code:

   double *array1;
    array1 = new double [500*9216*40];
    fill(array1, array1 + 500*9216*40, 0.0);
    double *array2;
    array2 = new double [500*9216*40];
    fill(array2, array2 + 500*9216*40, 0.0);
    
   ofstream array1_file;
   array1_file.open("array1.dat", ios:: trunc | ios::binary | ios::out);
       
   ofstream array2_file;
   array2_file.open("array2.dat", ios:: trunc | ios::binary | ios::out);
       
 
   int counter=0;
   int ID2;
   for (int ID1 = 0; ID1 < 9216; ID1++)
	{
          
        for (ID2 = ID1+1; ID2 < 9216; ID2++)
		{
			//calculations
			//...

                        array1[counter]=result1;
			array2[counter]=result2
                        counter++;
		}
                if ((ID1%500==0 | ID1==9215)){
                     array1_file.write (reinterpret_cast<char*>(array1),counter-1*sizeof(double));	
                     array2_file.write (reinterpret_cast<char*>(array2),counter-1*sizeof(double));	
                     fill(array1, array1 + 500*9216*40, 0.0);
                     fill(array2, array2 + 500*9216*40, 0.0);
                     counter=0;
                }
			
	}
       
        array1_file.close();
        array2_file.close();

        delete[] indices_row;
        delete[] values_row;

Do anyone understand what happens there?
Thanks!

[jsmith]

I believe that you are referring to the page cache. The ext2 and ext3 filesystems (as do all others) use a cache such that:

1) whenever you try to read a file, if the data isn't already in the cache,
it is read into the cache from disk and then given to you.
2) whenever you write to a file, Linux may cache the data in the page
cache rather than write it to disk the moment you write() it.

Both of the above are to try to minimize the amount of physical disk access required since disk access can kill system performance.

Linux will grow the page cache as long as it feels there is memory to support it.

If you want your data flushed to disk immediately, then you'll need to call flush():

array1_file.flush();
array2_file.flush();

after the writes.

Generally speaking, your application should not need to worry about the page cache unless you are implementing a database.

[tomasito]

Thanks for the answer, this clarifies the issue!