Help me choose the best container

[Konstantin2]

I have about 10 billion records (x86-64). Size of every record is 16 bytes.
Every hour I have to add about 1 million new records and remove about 1 million old records. For calculations I need fast linear addressing of records, i.e. buf[i] must be fast. Subj.

1. vector<my_record>.
The best, I think. It has fast linear addressing. New elements are added fast when I use reserve(). But cropping container from the begin means move about 160 Gb in memory. This take much time cause I use swap. I'm not happy owner of computer with 1 Tb RAM. This is the only lack.

2. deque<my_record>.
Is there way to reserve() additional memory and mass pop_fronts() without many memory freeing and reallocations?

3. list<my_record>, map<my_record>.
It has slow linear addressing (find buf[i]).

4. Other suggestions?

[MiiNiPaa]

2. deque<my_record>.
Is there way to reserve() additional memory and mass pop_fronts() without many memory freeing and reallocations?

deque does not do reallocations (aside from rare control structure reallocations).

Every hour I have to add about 1 million new records and remove about 1 million old records.

Do you add and remove it in chunks (whole million is added at the same time, when the time comes exactly that million will be removed), you might want to roll-out your own deque-like structure based on list of vectors. That vay you can have fast insertion and removal of elements without sacrificing much or random access speed.

[Ericool]

other suggestion : Did you calculate the memory it will requires ? I guess not.

Do you need to keep allocate all data on runtime ? I doubt .

Save every records , then if we request a record read it and send it.

Have you considered using filters ?

Have you considered compressing data ?

Have you considered using multithreading for resources ?

Did you look at Big O notation for each of these data structures ?

Swapping ? what are the 2 elements location ?

Have you considered sorting ? (obviously using filters)

Have you considered using pools architecture ?

So many questions. Without knowing your hardware , the type of connections , etc. We can just presume.