Best implementation for complex and huge shared arrays of struct ?

[lalebarde]

Hi all,
I need to build a shared memory with the following specifications :
1) It contains several arrays of millions of elements of complex struct types.
2) These arrays shall be read-only accessible from several applications in the same time.
3) It shall be very efficient in access time, so I don't want a database server.
4) It shall be able to be launched as a service at boot time.
5) It shall be Linux compliant.
6) No portability constraints.

I need your recommandations please on the best solutions.

[o1vk4iN6]

Several arrays which each contain millions of complex structures? Are you sure you even have enough memory to accommodate for that? If you have 20 arrays and each one has 1 million elements of 16 bytes (just how complex, if each one has a std::string you have to take into account the memory the string allocates), that's already 320 mb.

[mik2718]

I have used SysV shared memory before. Apparently mmap is now the way to go.

You say complex structs.
The main issue is are they POD data types?
If so then you are OK, if not you might have a problem since you would have to also control where pointed to data ends up.

[rapidcoder]

I need your recommandations please on the best solutions.

Buy Terracotta license... They have such solution. Costs probably lots of $$$$.

[lalebarde]

xerzi : I can afford some Gb of data in RAM.

mik2718 : thanks for the info. I am going to check mmap.

BTW, I organized my data to be POD compliant. A class with objects with attributes of different length can be re-written in a class with quantity of data as an attribute, plus an index to a POD array. So that everything becomes POD compliant.

Example :

struct T { P* mpX;}
T t[2000000];

can be rewritten :

struct T {size_t mIndexBeginningInX; size_t mQuantityOfX;}
P x[10000000];
T t[2000000];

rapidcoder : thanks for the tip. It looks great, but I will check first mmap.

[mik2718]

Looks like youshould be able to get it sorted.

Not sure about mmap but in SysV the base address of the shared memory window is NOT the same for the different processes.

So pointers are a problem - use offsets from the window base address, or something.

[lalebarde]

For information, here is the test I have managed to setup - at least, everything works fine as expected :

#include <iostream>
#include <string>
#include <cstdlib>

// Shared memory
#include <sys/types.h>
#include <sys/mman.h>
#include <fcntl.h>

// Gestion des erreurs
#include <errno.h>

using namespace std;

int main(int argc, const char *argv[]) {
	switch (atoi(argv[1])) {
		case 25 : { // Test shared memory with shm_open & mmap - http://kahdev.wordpress.com/2010/05/15/using-shared-memory-in-linux-programming/
			int *result = NULL;
			int integerSize = sizeof(int);
			// Open the shared memory - "manager"
			int descriptor = shm_open("/test25", O_CREAT | O_EXCL | O_RDWR, S_IRUSR | S_IWUSR);
			if (errno == EEXIST) {
				cout << "Instance : " << argv[2] << " - The exchange file test25 already exists" << endl;
			}
			if (descriptor == -1) {
				cout << "Instance : " << argv[2] << " has failed." << endl;
				exit errno;
			}
			/* Size up the shared memory. */
			if (ftruncate(descriptor, integerSize) != 0){
				cout << "Instance : " << argv[2] << " - ftruncate has failed with error code : " << errno << endl;
				exit errno;
			}
			result = (int*) mmap(NULL, integerSize, PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED | MAP_POPULATE, descriptor, 0 );
			if (result == MAP_FAILED) {
				cout << "Instance : " << argv[2] << " - mmap failed with error code : " << errno << endl;
				exit errno;
			}
			// Init the shared memory
			*result = 161803;
			string endCondition = "";
			while (endCondition != "quit") cin >> endCondition;
			// clean
			if (munmap(result, sizeof(int)) != 0) {
				cout << "Instance : " << argv[2] << " - munmap has failed with error code : " << errno << endl;
				exit errno;
			}
			if (shm_unlink("/test25") != 0) {
				cout << "Instance : " << argv[2] << " - shm_unlink has failed with error code : " << errno << endl;
				exit errno;
			}
			break;
		}
		case 26 : { // Test shared memory with shm_open & mmap
			int *result = NULL;
			int integerSize = sizeof(int);
			// Open the shared memory - "user"
			int descriptor = shm_open("/test25", O_RDONLY, S_IRUSR);
			if (errno == ENOENT) {
				cout << "Instance : " << argv[2] << " - The exchange file test25 does not exist" << endl;
			}
			if (descriptor == -1) {
				cout << "Instance : " << argv[2] << " has failed." << endl;
				exit errno;
			}
			result = (int*) mmap(NULL, integerSize, PROT_READ, MAP_SHARED, descriptor, 0 );
			if (result == MAP_FAILED) {
				cout << "Instance : " << argv[2] << " - mmap failed with error code : " << errno << endl;
				exit errno;
			}
			cout << "Value = " << *result << endl;
			if (munmap(result, sizeof(int)) != 0) { // unnecessary because the program exits - cf http://www.advancedlinuxprogramming.com/alp-folder/alp-ch05-ipc.pdf
			cout << "Instance : " << argv[2] << " - munmap has failed with error code : " << errno << endl;
			exit errno;
			}
			// shm_unlink("/test25"); // As I undersqtand it, that should be performed only by the "manager"
			break;
		}
	}

	return 0;
}

In shell #1 :

$ ./Release/test 25 111

In a root shell :

# ls -l /dev/shm/test25 
-rw------- 1 alain alain 4  5 mars  14:37 /dev/shm/test25

In shell #2 :

$ ./Release/test 25 222
Instance : 222 - The exchange file test25 already exists
Instance : 222 has failed.
$ ./Release/test 26 333
Value = 161803

In shell #1:

quit
$

$ ./Release/test 26 333
Instance : 333 - The exchange file test25 does not exist
Instance : 333 has failed.