It's a two step process. You have to remove the list and you have to delete the object.
If you know that you entries in two lists that point to an object that you need removed, you remove the pointer from the two lists and release the pointer once. The order you do this in doesn't really matter.