When we push_back() to the vector, the vector may need to allocate fresh storage and move the contents to that storage. Pointers/references to elements and iterators would be invalidated if this happens.
Either reserve() enough space in the vector so that reallocation is not required.
Or use a different container; for instance a std::list<> (or std::deque<> if there are no insertions or erasures in the middle of the sequence).
Another thought: it looks like group_vector owns the strings and keys_group_map refers to strings within it, so maybe change keys_group_map to map char to an index in the group_vector array:
map<char, size_t> keys_group_map;