StringTokenizer in Java

In Java, working with strings often involves splitting or parsing them into smaller parts, known as tokens. One of the tools provided by Java for this purpose is the StringTokenizer class. Although newer methods like split() and String.split() are more commonly used today, StringTokenizer remains a useful utility for certain scenarios. In this article, we’ll explore what StringTokenizer is, how it works, and when to use it.

What is StringTokenizer?

StringTokenizer is a class in the java.util package that allows you to break a string into tokens. A token is a substring separated by a delimiter (e.g., space, comma, or any other character). By default, the delimiter is a space, but you can specify custom delimiters as needed.

The StringTokenizer class is particularly useful when you need to process strings in a simple and efficient way without using regular expressions.

Key Features of StringTokenizer

1. Tokenization: Splits a string into smaller parts (tokens) based on a delimiter.
2. Custom Delimiters: Allows you to specify one or more delimiters.
3. Iterative Access: Provides methods to iterate through tokens one by one.
4. Legacy Class: It’s part of Java’s older utility classes but is still functional and supported.

Constructors of StringTokenizer

The StringTokenizer class provides three constructors:

1. StringTokenizer(String str)
   Creates a tokenizer for the given string using the default delimiter (space).

2. StringTokenizer(String str, String delimiter)
   Creates a tokenizer for the given string using the specified delimiter.

3. StringTokenizer(String str, String delimiter, boolean returnDelims)
   Creates a tokenizer for the given string using the specified delimiter. If returnDelims is true, the delimiters are also treated as tokens. 

Common Methods of StringTokenizer

1. hasMoreTokens()
   Returns true if there are more tokens available.

2. nextToken()
   Returns the next token from the string.

3. countTokens()
   Returns the total number of tokens remaining in the string. 

Example Basic:

​​import java.util.StringTokenizer;
public class StringTokenizerExample {
public static void main(String[] args) {
String str="Java is a powerful programming language";
// Create a StringTokenizer with default delimiter (space)
StringTokenizer tokenizer =new StringTokenizer(str);
// Iterate through tokens
while (tokenizer.hasMoreTokens()) {
System.out.println(tokenizer.nextToken());
        }
    }
}

Output:

Java  
is  
a  
powerful  
programming  
language 

Example: Using Custom Delimiters

​import java.util.StringTokenizer;
public class CustomDelimiterExample {
public static void main(String[] args) {
String str="Java,Python,C++,JavaScript";
// Create a StringTokenizer with a comma as the delimiter
StringTokenizer tokenizer = new StringTokenizer(str, ",");
// Iterate through tokens
while (tokenizer.hasMoreTokens()) {
System.out.println(tokenizer.nextToken());
        }
    }
}

Output:

Java  
Python  
C++  
JavaScript

Example: Returning Delimiters as Tokens

import java.util.StringTokenizer;
public class ReturnDelimitersExample {
public static void main(String[] args) {
String str= "Java|Python|C++|JavaScript";
// Create a StringTokenizer with '|' as the delimiter and returnDelims set to true
StringTokenizer tokenizer = new StringTokenizer(str, "|", true);
// Iterate through tokens
while (tokenizer.hasMoreTokens()) {
System.out.println(tokenizer.nextToken());
        }
    }
}

Output:

Java  
|  
Python  
|  
C++  
|  
JavaScript 

Advantages of StringTokenizer

1. Simple to Use: Easy to tokenize strings without complex regular expressions.
2. Lightweight: Minimal overhead compared to other string manipulation methods.
3. Backward Compatibility: Useful in legacy codebases where StringTokenizer is already in use. 

Limitations of StringTokenizer

1. Deprecated in Modern Java: While still functional, StringTokenizer is considered a legacy class. The split() method from the String class or java.util.regex is preferred in modern Java.
2. Limited Functionality: It doesn’t support advanced features like regular expressions.
3. Immutable Strings: Since Java strings are immutable, StringTokenizer doesn’t modify the original string but creates new tokens, which can be less efficient for large strings. 

Alternative: Using String.split()

The split() method from the String class is a modern alternative to StringTokenizer. It supports regular expressions and is more versatile.

When to Use StringTokenizer

While StringTokenizer is considered outdated, it can still be useful in the following scenarios:

1. Legacy Code: When working with older Java applications that already use StringTokenizer.
2. Simple Tokenization: For quick and straightforward tokenization without the need for regular expressions.
3. Backward Compatibility: When maintaining compatibility with older Java versions.