Strange issue with memory consumption on UNIX

[adzajac]

I'll need to store huge amount of heterogeneous data in the table like structure, where each object reflect to single cell. To do this I've created small example :

typedef boost::variant<int, double, std::string> variant;
typedef std::vector<variant *> RecordPtr;
static int rows = 20000;
static int columns = 50;
int main(){
	std::vector<RecordPtr *> recordset;
	recordset.reserve(rows);
	for (int i = 0; i<rows; ++i){
		RecordPtr *record = new RecordPtr();
		record->reserve(columns);
		for (int j=0; j<columns; ++j){
			variant *v = new variant( (double)1 );			
			record->push_back(v);
		}
		recordset.push_back( record );
	}
}

This structure is using about 40MB of RSS measured on UltraSpark Solaris, x86 solaris, and Linux 64 while the example below is using only 16MB could somebody can help to explain this strange behaviour.

typedef boost::variant<int, double, std::string> variant;
typedef std::vector<variant> RecordPtr;
static int rows = 20000;
static int columns = 50;
int main(){
	std::vector<RecordPtr *> recordset;
	recordset.reserve(rows);
	for (int i = 0; i<rows; ++i){
		RecordPtr *record = new RecordPtr();
		record->reserve(columns);
		for (int j=0; j<columns; ++j){
			variant v;
			v = (double)1;
			record->push_back(v);
		}
		recordset.push_back( record );		
	}
}

Thx in advance ... :)

[kfmfe04]

well, first of all, in C++, when you call a constructor with no arguments, you don't use parentheses:

RecordPtr *record = new RecordPtr; // not new RecordPtr()

[adzajac]

Doesn't matter. Sample was taken from the wider context and some obstacles left :). But thx for the advice.

[kbw]

You have an extra 4M of pointers, the heap may add a little in block management, but you have 20M unaccounted for. Interesting.

[jsmith]

Remember that the minimum allocation unit is 16 bytes, even if you request only 1.
What is sizeof( variant )?

[adzajac]

I agree sizeof( variant ) is 16 bytes ( on most architectures ) except x86solaris where it takes 12 bytes. But the main question is how many memory is taken by the system to manage the heap. If I'll create 1milion pointers to doubles it will not take strict 8MB for doubles and 4MB for pointers but system will use a little bit more for heap managing and the question is how many .... :)

[kbw]

main question is how many memory is taken by the system to manage the heap

That would be implementation\OS dependent.

I ran your code under Visual Studio 2008 in Debug mode (so I could look at the heap).
The first version with variant* does 40,429 calls into malloc and uses 44,204,032 bytes.
The second version with variant does 1,040,429 calls into malloc and uses 112,406,528 bytes.