string tokenization

[hooshdar3]

Hello.
I know there is strtok() to do the tokenization;but all strtok() examples I have seen only output one token at a time, moving the pointer ahead. Can't the tokens be saved somewhere?

[Yay295]

Yes, if you tell it to save them somewhere.

[hooshdar3]

Yay! could you be more descriptive please?
I want each token stored in a variable or something.
Could you please provide mee with an example?

[Yay295]

#include <iostream>
#include <cstring>

int main ( )
{
	char str[] = "This is a string.";

	std::cout << str << '\n';

	int i = 0;

	char * word[10];

	word[i] = strtok ( str, " " );

	while ( word[i] )
	{
		std::cout << word[i] << '\n';
		word[++i] = strtok ( nullptr, " " );
	}
}

The problem is that you have to know the number of words beforehand. I know there must be a way around that, but I'm not good enough with cstrings to know what it is.

[cire]

Why use strtok in C++?

// http://ideone.com/N0X8DO
#include <iostream>
#include <string>
#include <vector>

bool check(std::size_t pos)
{
    return pos != std::string::npos;
}

std::vector<std::string> tokenize(const std::string& text, const std::string& delimiters = " ")
{
    std::vector<std::string> tokens;

    std::size_t start_pos;
    std::size_t end_pos = 0;

    while (check(start_pos = text.find_first_not_of(delimiters, end_pos)) )
    {
        end_pos = text.find_first_of(delimiters, start_pos);
        tokens.emplace_back(text.substr(start_pos, end_pos - start_pos));
    }

    return tokens;
}

int main()
{
    {
        auto t = tokenize("8 (4 ounce) fillets salmon,1/2 cup peanut oil,4 tablespoons soy sauce,"
            "4 tablespoons balsamic vinegar,4 tablespoons green onions (chopped),3 teaspoons "
            "brown sugar,2 cloves garlic (minced),1 1/2 teaspoons ground ginger,2 teaspoons "
            "crushed red pepper flakes,1 teaspoon sesame oil,1/2 teaspoon salt", ",");

        for (auto& token : t)
            std::cout << token << '\n';

        std::cout << '\n';
    }

    {
        auto t = tokenize("Test 1,string 1,Test 2,string 2:Test 3:string 3", ",:");

        for (auto& token : t)
            std::cout << token << '\n';

        std::cout << '\n';
    } 

    {
        auto t = tokenize("Mary had a little lamb");

        for (auto& token : t)
            std::cout << token << '\n';
    }
}

[Duthomhas]

Strtok is evil.
http://www.cplusplus.com/faq/sequences/strings/split/

[NoXzema]

Or just use a tokenizer like Flex, Quex, or hand made.