Disk based container?

[steve yates]

We are in the process of re-writing an old utility that was written in house. It's win32 console program so it has a file size limit smaller than we currently need. And we don't have all the supporting libraries to link it anymore, so we can't make changes to the original utility.

It uses RogueWave's Tools.h++ RWBTreeOnDisk (a btree stored on disk). So we can store key/value pairs on disk while it's running. Two reasons it was originally designed this way. Back when it was written RAM was an issue and multiple indexes of a five+ million records couldn't be stored in memory - not as much as an issue these days. But we also need to retain the key/value pairs after the program closes - which is why when we re-write it we need a disk based container.

We have MySQL servers and thought about going that route. But we would prefer a stand alone program. And RogueWave's Tools.h++ doesn't exist anymore. I think it might be part of RogueWave's SourcePro C++, but that's not in the budget.

So anybody know of an open source disk based container? Cross platform would be nice (Linux and Windows), but we could live with it if it was just a Windows solution.

Doing some searching it sounds like Berkeley DB might be an option. But I need to do some more reading on it to see if it will do what we need.

[choisum]

try sqlite -- the data can be on disk or in memory.

btw, if it really is a straightforward key/value storage, map would suffice.

[Duthomhas]

If you want to make it work on disk, instanciate a std::map (or some other container) with a custom allocator.

You can google around for more information. Here's something I just found that looks like a promising read:
http://www.codeguru.com/Cpp/Cpp/cpp_mfc/stl/article.php/c4079/

Good luck!

[steve yates]

From what I've been reading sqlite is similar to Berkley DB, the data can be on disk or in memory, at least as far as we'll be using it. Not sure about sqlite, but what I've been reading about Berkley DB it can be accessed using a key/value pair.

As for using an STL container with a custom allocator, I'll look into it. But I'm not to confident about my abilities to optimize something like that as compared to using an established tool like Berkley DB or sqlite.

And if I did use an STL container it would be a multimap.

thanks for the replies!

[choisum]

sqlite gives you an sql interface, so it might be too heavy. Berkley DB sounds better for straight key/value pairs.

[Duthomhas]

Agreed.