Book suggestion to understand Tokens

[BigRedMachine]

I'm reading "Principles and Practice Using C++" chapter 6, it's about using Tokens and as if the concept of Tokens were not already confusing the whole chapter is just a mess and gives me the impression that Bjarne was just not in the mood when writing this chapter....
Can you guys suggest any book that also covers Tokens(to complete beginners)?
I can't find anything good on the internet....

[Mathhead200]

I'm not sure if I'd classify understanding tokens as a beginner topic. Tokens are simply the smallest sub-strings of a program that need to be read together to have meaning. For example: int x = 501; In that code the five tokens (in order) are:
int
x
=
501
;
The characters i and 5 are not a tokens however, because it they are part of the tokens int and 501 (respectively).

http://en.wikipedia.org/wiki/Lexical_analysis

[BigRedMachine]

This is the first problem I'm encountering(example from the book):

#include <std_lib_facilities.h>

class Token {
public:
        char kind;
        double value;
        Token(char ch = char())
            :kind(ch), value(0){}
        Token(char ch, double val)
            :kind(ch), value(val){}
};

Token get_token(); //read a token from cin
vector<Token> tok; //we'll put the tokens here

int main()
{
    while(cin){
        Token t = get_token();
        tok.push_back(t);
    }
}

It has not defined get_token(); and I have no idea what should be in the function's body to define it myself....

[tcs]

get_token() in your example has to return a value of type Token, which consists of the two attributes kind and name. Depending on the input stream a simple tokenizer could be written as:

Token get_token()
{
    Token token;

    cin >> token.kind;
    cin >> token. value;

   return token;
}

A corresponding input stream may be f.e.:

a 17.5
b 32.0
a 0.0
c 1234.5678

[mutexe]

If it helps, i've never heard of the concept of 'tokens' before.

[tcs]

You may also find tokens in communication systems like a real time bus, a so called token-bus. There are several potentiell communication participants, but only one at a time is allowed to send messages. This one is called the token holder. If ready or its maximum sending time is reached, it will forward the token, say the right to send, to another participant.

[BigRedMachine]

Thanks everyone....
found this on Bjarne website:

Token get_token()    // read a token from cin
{
    char ch;
    cin >> ch;    // note that >> skips whitespace (space, newline, tab, etc.)

    switch (ch) {
 //not yet   case ';':    // for "print"
 //not yet   case 'q':    // for "quit"
    case '(': case ')': case '+': case '-': case '*': case '/': 
        return Token(ch);        // let each character represent itself
    case '.':
    case '0': case '1': case '2': case '3': case '4':
    case '5': case '6': case '7': case '8': case '9':
        {    
            cin.putback(ch);         // put digit back into the input stream
            double val;
            cin >> val;              // read a floating-point number
            return Token('8',val);   // let '8' represent "a number"
        }
    default:
        error("Bad token");
    }
}

It's a very complicated messy chapter, I may either skip it or change the book....