Confuse abt token!!

[kunal101]

****stackarr.cpp****
#include "stackar.h"
#include "fatal.h"
#include <stdlib.h>

#define EmptyTOS ( -1 )
#define MinStackSize ( 5 )

struct StackRecord
{
int Capacity;
int TopOfStack;
ElementType *Array;
};

/* START: fig3_48.txt */
int
IsEmpty( Stack S )
{
return S->TopOfStack == EmptyTOS;
}
/* END */

int
IsFull( Stack S )
{
return S->TopOfStack == S->Capacity - 1;
}

/* START: fig3_46.txt */
Stack
CreateStack( int MaxElements )
{
Stack S;

/* 1*/ if( MaxElements < MinStackSize )
/* 2*/ Error( "Stack size is too small" );

/* 3*/ S = malloc( sizeof( struct StackRecord ) );
/* 4*/ if( S == NULL )
/* 5*/ FatalError( "Out of space!!!" );

/* 6*/ S->Array = malloc( sizeof( ElementType ) * MaxElements );
/* 7*/ if( S->Array == NULL )
/* 8*/ FatalError( "Out of space!!!" );
/* 9*/ S->Capacity = MaxElements;
/*10*/ MakeEmpty( S );

/*11*/ return S;
}
/* END */

/* START: fig3_49.txt */
void
MakeEmpty( Stack S )
{
S->TopOfStack = EmptyTOS;
}
/* END */

/* START: fig3_47.txt */
void
DisposeStack( Stack S )
{
if( S != NULL )
{
free( S->Array );
free( S );
}
}
/* END */

/* START: fig3_50.txt */
void
Push( ElementType X, Stack S )
{
if( IsFull( S ) )
Error( "Full stack" );
else
S->Array[ ++S->TopOfStack ] = X;
}
/* END */


/* START: fig3_51.txt */
ElementType
Top( Stack S )
{
if( !IsEmpty( S ) )
return S->Array[ S->TopOfStack ];
Error( "Empty stack" );
return 0; /* Return value used to avoid warning */
}
/* END */

/* START: fig3_52.txt */
void
Pop( Stack S )
{
if( IsEmpty( S ) )
Error( "Empty stack" );
else
S->TopOfStack--;
}
/* END */

/* START: fig3_53.txt */
ElementType
TopAndPop( Stack S )
{
if( !IsEmpty( S ) )
return S->Array[ S->TopOfStack-- ];
Error( "Empty stack" );
return 0; /* Return value used to avoid warning */
}
/* END */

***************
I have to count the number of tokens in this file stackarr.cpp via file handling
i am confused in a lot of things.
how many tokens does this hav:
1.#include
2.->
3."This is a string ";
nd many more...

[Duthomhas]

Ok, please use code tags. (I had to copy and paste and format so my poor eyes could read it through the "codeblindness" [as in "snowblindness"...])
[code]
code goes here
[/code]

The stuff you listed is just a stack. It has nothing to do with actually parsing tokens.

The example you list has 7 tokens.
1. #
2. include
3. ->
4. "
5. This is a string
6. "
7. ;
Notes:
- A string literal is a single token.
- The # and everything following it on a line is not technically a C or C++ tokenizable sequence, since the preprocessor handles them and the compiler itself never sees it. However, for your purposes you can tokenize them normally. (The preprocessor does.)

Get a character at a time from the file and select the longest-recognizable token you can get out of it. Each time you've finished recognizing a token, push it onto your stack.

By "longest recognizable" I mean don't stop for the small stuff.
In C++, template parameters often need a space between them:

std::list<std::pair<int,char> >

Notice how the last two '>'s had to be separated by a space. If they were together:

std::list<std::pair<int,char>>

then the last two '>'s would have been recognized as the >> operator. You need to do the same thing, don't stop for a particular token until you are sure you've got the whole thing.


If this is a school assignment then you should get clarification on what constitutes tokens. If it is for work you should get clarification on what constitutes tokens.

Hope this helps.