problem with infinite loop

[coolbran]

Hi everyone,
I am basically trying to write my own version of strtok() but with c++ string objects. The problem I am having is that i keep getting an infinite loop when i try to call my function strtok(). I cannot figure out what the problem is.

#include <fstream>
#include<iostream>
using namespace std;
#include <cstring>
#include <iomanip>
#include <string>

string tokenizer (string line, bool first,  size_t &i )
{	 
	size_t j;
	string token;
	//Test to see if I am looking for the first word in a line,
	//store the number of characters up to the first space into i
	if (first == true)
	{
		i = line.find(' '); //store the number of characters up to the first space into i
		if ( i == string:: npos)
		{
			cout<<"\no more tokens"<<endl;
			//return an empty string if thre are no more tokens			
			token =""; //Going to look for an empty string to stop
			return token;  //my loop in main()
		}
		token = line.substr(0, i); //retrieve token
		return token;
	}
	//**if NOT looking for the first token of a line *//
	// always go (count) from i -----> j when tracking down number of characters
	
	// from the place of the last retrived token (i+1),
	// go find next spce, save that number of spaces in between, save that number into j
	// basically *****|i----->j|****
	j = line.substr(i+1, line.length()-i-1 ).find(' '); 
	if ( j == string:: npos)
	{
		cout<<"\no more tokens"<<endl;
		token ="";
		return token;
	}
	token = line.substr(i+1, j);//extract token
	i = j;//Position of j is the starting place for the next token
	return token;

}

int main( )
{
	string line ;
	bool first = true;
	ifstream read;
	read.open("test.txt");
	if (read.fail() )
	{
		cout<<"error in input, exiting"<<endl;
		return -1;
	}
	ofstream out;
	out.open ("output.txt");
	if (out.fail() )
	{
		cout<<"output file error, exititng..";
		return -1;
	}
	size_t i, j;
	string pi;
	int k = 0;
	while (!read.eof() )
	{
		getline(read, line );
		cout << line << endl;
		out  << line <<endl; //Print lines to output file as i get them
		i = 0; //since we are tokenizing a new line, restart position of i at 0, 
		first = true;
		string token;
		token = tokenizer (line, first, i);
		cout << "\nfirst call of tokenenizer, it is: "<<token<<endl;
		while ( token.length() !=0) //Looking for an empty string to stop the loop
		{
			first = false;
			token = tokenizer ( line, first, i);
			cout<<endl<<"ins "<<token<<endl;
		}
	}

	return 0;
}

Help would be greatly appreciated

[helios]

Do you mean that you get an infinite loop by renaming your function to strtok()?

[coolbran]

I do not understand your question.

[Lynnabelle]

What helios means is that there is no function named strtok() here; perhaps you are referring to the string tokenizer function.

I'm confused by your while loop here:

while ( token.length() !=0) //Looking for an empty string to stop the loop
		{
			first = false;
			token = tokenizer ( line, first, i);
			cout<<endl<<"ins "<<token<<endl;
		}

Shouldn't first be made true, not false, while token is not empty? Maybe it should be:

if (token.length() == 0)
{
first = true;
// etc.
}

Just consider checking this part of the code... I didn't actually go through the logic carefully.

[champost]

Check out istringstream::str (http://www.cplusplus.com/reference/iostream/istringstream/str/), i think its just what you need and keeps your code short and simple. By the way the example at that link also works for data types other than integers.