mapped file for read-only shared data

[dinama]

hello!
has a large array of c-structures writed binary file (big data table).
which method is best performance for share this data between processes (read-only , binary search etc)
- simple map (open|mmap) whole file into memory, and cast to array
- create a shared memory (shmat ... ) and copy data
?

[shalky]

Here is a suggestion:
Create an hash table in memory for your data. If you refer to the data according to the index, then the hash table is accessed by the index. If you refer to the data according to some parts of the data, then use these data to generate hash table and its content point to the position of the data.

[Cubbi]

dinama wrote:

- simple map (open|mmap) whole file into memory, and cast to array
- create a shared memory (shmat ... ) and copy data

copy to the shmat memory from where? From mmap'd file? From opened file? Either way, direclty sharing a read-only mmap means one or two less copies, respectively. As for how that affects performance.. measure.

[kbw]

It sounds more like an architectural problem.

How much of this data will each processor require? If it's a small subset, it may be better to hide it behind a server, and let each processor request the parts they need. That way you can scale horizontally by adding more servers.

Only if the sheer cost of getting all that data to each processor was a factor, would I consider allowing them direct access to it in any form.

[dinama]

>copy to the shmat memory from where? From mmap'd file? From opened file? Either way, direclty > a read-only mmap means one or two less copies, respectively. As for how that affects performance.. measure.

file contain preordered constant c-structures for read-only random access.
simple way - mmap this file, cast void* to array pointer and use them. its workin ok.