In order to "remove" the vector elements the algorithm needs to be able to copy them to a different location. But once a reference is set you can't change the object it refers to. Since your class has at least one reference member the copy assignment operator is implicitly deleted.
You could add a copy assignment operator that copies the value of one referenced object to another if that's feasible. But, if you want to stick with a vector, you may as well just use pointers (or unique_ptr or shared_ptr).
Alternatively, you could use a list (or forward_list, which is more efficient if you don't need it to be doubly-linked) instead of a vector since list elements don't need to be moved around when removed.