use a STACK to check the braces/paranthesis,
it will be easier if you stick with Reverse Polish Notation,
if you dont know what stack and RPN are, take a look at these links
http://en.wikipedia.org/wiki/Reverse_Polish_notation
http://en.wikipedia.org/wiki/Stack_%28data_structure%29
there are more ways to do this... just think creatively.....
cheers...
p.s.
if you know about tree data structures this will be another hint,
the funny thing is that if you write your mathamatical expression to a tree, you can get both RPN and the normal notation by just changing the way you traverse on the tree...
http://en.wikipedia.org/wiki/Binary_tree
http://math.hws.edu/eck/cs225/s03/binary_trees/
EDIT---
ok, now i see that you are heading a harder way...
trying to figerout the numbers inside a string and then use them will take so much time and space(memory)... what you have to do is,
just read the string as a stream of characters... ( use a while loop to read until the char is NULL )
and put every number you meet inside of some where( stack is best )....