I'm having trouble with tokens. I'm reading through chapter 7 of programming principles and practice using c++, and I've read through chapters 6 and 7 several times, but I just can't grasp the concept of tokens very well. Tried searching for a while too, but there aren't very many tutorials on tokens and token streams.
Here's the code I'm trying to understand, from stroustrup's principles and practice using c++. It was too long to paste it in directly, so here's the link to it on pastebin
http://pastebin.com/RJzYFUVP
Lets give an example of 5+3/2
Main() starts off by calling calculate(). Calculate() accepts inputs from cin, token t would thus be 8+8/8, with the 8 being the kind for numbers. No prints or quits, so it continues on to statement(). Statement then calls expression(), which calls term(), which calls primary().
Here, token t = ts.get(). Would that mean that all the tokens from the token stream are in token t?
If that's the case, then primary() would see that the first token is a number, so it would return it's value to term(), which is 5. Term() would set left=8 and get the next token, which is +. At this point, I'm positive I'm interpreting something incorrectly. This program can "look ahead" and find and do the division before the addition, but how does it do this? I'm completely stumped on this right now. Can somebody help me out here, or point me towards a tutorial that explains tokens better? :(