Well, you've just created a whole set of problems. The question is, "How does delete [] p know how many elements to delete?"
That is implementation dependent, but, a common scheme is to store the number in the buffer itself. When you call delete [] p on an array that wasn't allocated with new [], the memory layout (type) won't match and you may delete less than you expect or call delete on elements beyond the range you're thinking about.
means if pointer is modified then delete won't work properly
Yes, that's exactly what it means. You can only use delete with an address to the start of the memory that was originally allocated by new. Anything else results in undefined behaviour.