Or do I have to take care to only put either dynamically or staticallay allocated objects into my vector?
Well, you need some way of identifying which pointers point to objects which are dynamically allocated, and deleting only those objects.
If you're mixing pointers to dynamically and statically allocated classes in the same vector, that suggests to me that there's something very confused about the way you've designed your code.
I think if you use smart pointers then you won't need to delete even the dynamically allocated ones.
But if you make a smart pointer to statically allocated memory, then it will cause the same problem - it will try and delete the object (when the reference count drops to zero), and will most likely cause a crash.
So a vector of smart pointers to a mix of dynamically and statically allocated memory will still have the same problem.