a problem :(

[Ola]

hii all ,
i have an assignment to implement scanner phase of some compiler based on some dfa.it should read from a file and output to a file..

i have a problem i made the function as
Token getToken(string str)

and then converts str to array of chars and it works well with individual words such as hi , for , if ...etc but when i enter

str ="int x ;"

it just considered the int and ignores the rest ..i don' know to figure the problem ..thx in advance:):)

[helios]

Lexers are typically written so that they take characters from a stream (e.g. a FILE* or an std::istream), and each successive call returns a new token.

[Ola]

yes i know ..but in my code it takes the string before the first white space and ignores the rest :-s ..i don' know what 's the probelmm???

[helios]

Is the string being set inside the function?

[Ola]

i don' understand what u mean sorry???!! but if u mean that i did that

getToken(string str)...so yes it gets the string and then i copy it to an array of chars to process it char by char..

[Duthomhas]

It should ignore the rest.

Your argument should be either a string& or a istream&.

The point of tokenizing is to take a piece off of the input stream and return what it is.

So, given an input of int x;

getToken --> IntTypeToken, remaining input --> x;

getToken --> VariableToken "x", remaining input --> ;

getToken --> StatementTerminatorToken, remaining input -->

Hope this helps.

[Ola]

it supposes to,but realy the function i wrote doesn' do that ..it terminates after the first space .............it's massive function so i can't figure where that break comes from !!!