If you want to close the program you can use std::exit: http://en.cppreference.com/w/cpp/utility/program/exit
If you want to exit just function, return to main, use return; // yes, it will work in void function
Just break would work, but that shows what you meant.
And don't use system(). It's slow, unsafe. Use std::cin.get() instead.
Your use of switch is corect. Program is probably not running properly 'cuz you don't ask user for input. If you do it in code you didn't post, then another reason might be how you calculate areas.
Are you sure that area of square is sqrt(side of square)? If side = 4 then sqrt(4) = 2. It should be pow(side, 2) or just
side * side
, shouldn't it? Also area of triangle is
ah/2
, a is triangle's side and h is it's height.
EDIT
Please don't delete posts 'cuz maybe there is someone who has the same problem as you. Now your making your post unhelpful.