<> tells the compiler to look in predefined header directories only (generally where the STL is kept).
"" tells to compiler to look in the local directory, and usually the predefined header directories as well
As for string vs string.h...string is the C++ version, which you probably want. string.h is the C version, which if you want to include in a C++ program you should instead include cstring. In C++, all the standard headers do not have an extension.
The standard (C++98) says that these both map "in an implementation-defined manner to headers or to external source file names". Which means it's up to whoever wrote your preprocessor. Your compiler does not ever see these #include directive, only the output from the preprocessor.
1 2 3
Lastly, I've seen these two examples (both work):
#include <string>
#include <string.h>
Those are two completely different header files. One of them is the C++ string header, giving you std::string. The other is the Standard C library header giving you various C Standard definitions and so forth. They both work because both files exist, but those particular ones are very, very different. There is also <cstring>, which provides you the same as string.h but makes some C++ changes to improve things (you'll note there are many other C Standard library headers available as part of the C++ standard, with a "c" on the front and no".h", providing the same functionality but in a better, C++ way).