C++ String Tokenizer equivalent?

[packetpirate]

I recently fell in love with the String Tokenizer in Java while working on an assignment for my Java course. I was just wondering if there are any equivalents of the String Tokenizer in C++? If so, where can I find the documentation?

[hanst99]

There is no inbuilt tokenizing functionality (that I am aware of), but it's not hard to implement it yourself - what functionality do you need (I am not very fluent with java, but the actual tokenizing of strings can be done with a few lines of code)?

[jsmith]

You can look at the boost.tokenizer library:

http://www.boost.org/doc/libs/1_46_1/libs/tokenizer/index.html

[filipe]

I don't know the Java tokenizer, but the C standard library (which is part of C++) has strtok(): http://www.cplusplus.com/reference/clibrary/cstring/strtok/

If you want a tokenizer which also categorizes tokens, you'll have to write one. But it's pretty easy to do.

[mof]

Yeah, it's very easy, almost trivial. e.g.

#define UNKNOWNCHAR _T('_')
#define ISTOKEN(x) (iswalpha(x) || (x == UNKNOWNCHAR))

/-----------------------------------------------------------------------------

int ReturnToken(LPTSTR str, int i, LPTSTR token)
{
	int j=0;

	token[0] = 0;
	while(ISTOKEN(str[i]))
	{
		token[j++] = str[i++];
		token[j] = 0;
	}
	if(str[i] == 0) return -1;
	return j;
}

Just an example from something I wrote. But you can tailor ISTOKEN to include any tokens you like. Please excuse the Visual C++ syntax.

[packetpirate]

filipe, thanks. I'll make sure to read that.