I'll need to store huge amount of heterogeneous data in the table like structure, where each object reflect to single cell. To do this I've created small example :
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
typedef boost::variant<int, double, std::string> variant;
typedef std::vector<variant *> RecordPtr;
staticint rows = 20000;
staticint columns = 50;
int main(){
std::vector<RecordPtr *> recordset;
recordset.reserve(rows);
for (int i = 0; i<rows; ++i){
RecordPtr *record = new RecordPtr();
record->reserve(columns);
for (int j=0; j<columns; ++j){
variant *v = new variant( (double)1 );
record->push_back(v);
}
recordset.push_back( record );
}
}
This structure is using about 40MB of RSS measured on UltraSpark Solaris, x86 solaris, and Linux 64 while the example below is using only 16MB could somebody can help to explain this strange behaviour.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
typedef boost::variant<int, double, std::string> variant;
typedef std::vector<variant> RecordPtr;
staticint rows = 20000;
staticint columns = 50;
int main(){
std::vector<RecordPtr *> recordset;
recordset.reserve(rows);
for (int i = 0; i<rows; ++i){
RecordPtr *record = new RecordPtr();
record->reserve(columns);
for (int j=0; j<columns; ++j){
variant v;
v = (double)1;
record->push_back(v);
}
recordset.push_back( record );
}
}
I agree sizeof( variant ) is 16 bytes ( on most architectures ) except x86solaris where it takes 12 bytes. But the main question is how many memory is taken by the system to manage the heap. If I'll create 1milion pointers to doubles it will not take strict 8MB for doubles and 4MB for pointers but system will use a little bit more for heap managing and the question is how many .... :)
main question is how many memory is taken by the system to manage the heap
That would be implementation\OS dependent.
I ran your code under Visual Studio 2008 in Debug mode (so I could look at the heap).
The first version with variant* does 40,429 calls into malloc and uses 44,204,032 bytes.
The second version with variant does 1,040,429 calls into malloc and uses 112,406,528 bytes.