What is the good of limiting them until you use them?
All header files that you include have to be compiled for each of your translation units.
Obviously it won't improve compile times if you include tons of unnecessary headers.
It takes memory, space, and time to include directories. If you included a bunch of a directories for a simple program that, say, output a table of sine values, it would take forever to compile and take up way too much space to perform such a simple operation. However, bigger programs do tend to include a lot of directories.