C++

Forum

 
Can't get past the second token in a String Tokenizer function

motez23 (9)
So I have to write a function to get tokens (separated by spaces " ") from a string line. I get the first token just fine, and can get the second one as well, but after that I get an infinite loop where it keeps on returning the second token and I don't understand what I'm doing wrong.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
 
  string Tokenizer( string s, string d, bool first)
{
	string token = "";
	static int i;
	int k = 0;
	if (first == true)
	{
		i = s.find(d);
		if ( i == string::npos )
		{
			return "";
		}
		token = s.substr( 0, i );
	}	 
	else 
	{
		k = (s.substr( (i + 1), s.length()-i - 1 )).find( d );
		if ( k == string::npos )
		{
			return "";
		}
		token = s.substr( (i + 1), k);
	}
	return token;
}


this is the context I'm using the function in 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
	string token;
	string delims = " ";
	bool first = true;
	token = Tokenizer(line, delims, first);
	if (token != "")
	{
		wordcount++;
		uniquewordcount(token, Uwordcount, position, unique);
		first = false;
		while (token != "")
		{
			token = Tokenizer(line, delims, first);
			if (token == "")
			{
				break;
			}
			else
			{
				wordcount++;
				uniquewordcount(token, Uwordcount, position, unique);
			}
		}
	}


sidenote:
my delimeters should be a space " " and a period ".", would I just test to see if the first one fails and use string indexing to test the second delimeter or?...
motez23 (9)
I tweaked it a bit and now this tokenizes the first two words without the infinite loop. mild progress.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
 
string Tokenizer( string s, string d, bool first)
{
	string token = "";
	static int i=0, j=0;
	int k = 0;
	if (first == true)
	{
		i = s.find(d);
		if ( i == string::npos )
		{
			return "";
		}
		token = s.substr( 0, i );
		j=i;
	}	 
	else 
	{
		k = (s.substr( (j + 1), s.length()-j - 1 )).find( d );
		if ( k == string::npos )
		{
			return "";
		}
		token = s.substr( (j + 1), k);
		j+=k;
	}
	cout << token;
	return token;
}
Topic archived. No new replies allowed.