How to Tokenize a String in C++ by delimiter

C-Plus-Plus-String

In this article, we are going to learn, how to Tokenize a String in C++ by delimiter spaces, commas (,). We will learn this by using the below techniques.

  • Using find() and substr Functions to Tokenize a String in C++
  • Using std::stringstream and getline Functions to Tokenize a String in C++

1. Using find and substr Functions to Tokenize String in C++

In this example, we are tokenizing the string. We are creating tokens of each word that exists in the string. We are using the find function to find the spaces and on each space, we are saving the token to a vector. Once we reach the end we are done with the string. All tokens are pushed to a vector so we can print the vector elements with all tokens in it. All tokens are pushed to a vector so we can print the vector elements with all tokens in it.

C++ Program to Tokenize a String by space delimiter

#include <iostream>
#include <string>
#include <vector>


using namespace std;

int main()
{
	string str = "One Two Three Four";
    string delim = " ";
	
    vector<string> Tokens{};

    size_t index = 0;
	
    while ((index = str.find(delim)) != string::npos) 
	{
        Tokens.push_back(str.substr(0, index));
        str.erase(0, index + delim.length());
    }
	
    if (!str.empty())
        Tokens.push_back(str.substr(0, index));
	
	
	cout << "String Broken in to Tokens:" << endl;
    for (const auto &tok : Tokens) 
	{
        cout << tok << endl;
    }

    return 0;
}

Output

String Broken in to Tokens:
One
Two
Three
Four

In another example, we are using the ‘,’ as a delimiter to tokenize the string. we are looking for the ‘,’ to separate the words as each token.

C++ Program to Tokenize a String in C++ by comma delimiter 

#include <iostream>
#include <string>
#include <vector>

using namespace std;

int main()
{
	string str = "One,Two,Three,Four";
    string delim = ",";
	
    vector<string> Tokens{};

    size_t index = 0;
	
    while ((index = str.find(delim)) != string::npos) 
	{
        Tokens.push_back(str.substr(0, index));
        str.erase(0, index + delim.length());
    }
	
    if (!str.empty())
        Tokens.push_back(str.substr(0, index));
	
	cout << "String Broken in to Tokens:" << endl;
    for (const auto &tok : Tokens) 
	{
        cout << tok << endl;
    }

    return 0;
}

Output

String Broken in to Tokens:
One
Two
Three
Four

2. Using std::stringstream and getline Functions to Tokenize a String in C++

In this example, we are taking the string as argument input in the form of a std::stringstream . Using the getline() function we can easily break a string into tokens. On each token, we are pushing it to a vector to collect all tokens. Once done then we can print the tokens using a simple for loop.

C++ Program to Tokenize a String in C++ by a delimiter

#include <iostream>
#include <string>
#include <sstream>
#include <vector>


using namespace std;

int main(){
    string str = "One Two Three Four";
    char del = ' ';
    vector<string> Tokens{};

    stringstream sstream(str);
	
    string substr;
    while (std::getline(sstream, substr, del))
       Tokens.push_back(substr);

	cout << "String Broken in to Tokens:" << endl;
    for (const auto &tok : Tokens) {
       cout << tok << endl;
    }

    return 0;
}

Output

String Broken in to Tokens:
One
Two
Three
Four

Summary

In this post, We have learned how to Tokenize a String in C++ by delimiter.

Related Posts