Of cource. It can have anything you want to put in it: classes, standalone functions, global variables.
can users directly access those classes directly?
As you decide. If your library is precompiled one, you can give header with only entities you want user to have access to. If it is source code library, you can close usage of some entities by not providing access through a header/ use of unnamed namespaces.
Is reading the source code of a standard library a good way to learn how to write libraries?
No. It is usually uses badly named helpers, does not care about maintainability and sacrifices readability for any speed increase (as standard library is expected to be fast)
I personally found it pretty hard to find example C++ Libraries, and mostly figured out good practice by messing around. They are often not to hard to make, but I found that recompiling already made libraries were a nice way of figuring out what they are about.