StringTokenizer Class in Java

StringTokenizer, found in java.util, is one of Java's oldest utility classes for breaking a string into smaller pieces called tokens, based on a set of delimiter characters. It predates Java's regular expression support and is considered a legacy class today — it is worth knowing for reading older codebases, but new code should generally prefer String.split().

Why it matters

StringTokenizer is a legacy class that you will still meet in older codebases, textbooks and exam questions. You need to be able to read it and to know how it differs from split() — in particular that it silently skips empty tokens — so that you can replace it safely when maintaining old code.

Basic Usage and Methods

You construct a StringTokenizer with the string to split and (optionally) a set of delimiter characters — if no delimiter is given, it defaults to whitespace (space, tab, newline). The class exposes a small, simple API for walking through the tokens one at a time.

• hasMoreTokens() — returns true if at least one token remains to be read.
• nextToken() — returns the next token as a String and advances the internal position.
• countTokens() — returns the number of tokens remaining (without consuming them).

Why String.split() Is Preferred Today

String.split(regex), added later, is far more powerful because it accepts a full regular expression rather than a fixed set of delimiter characters, so it can split on patterns like "one or more spaces" or "a comma optionally followed by a space." It also returns a String array, which integrates naturally with modern Java code (streams, for-each loops, and so on), whereas StringTokenizer implements the older Enumeration-style iteration. For any new code you write, split() (or, for more complex parsing, the java.util.regex package) is the right tool; StringTokenizer is mainly useful today for recognizing it when reading legacy code.

Example

import java.util.StringTokenizer;

public class StringTokenizerDemo {
    public static void main(String[] args) {
        StringTokenizer tokenizer = new StringTokenizer("Java,Python,C++,Kotlin", ",");

        System.out.println("Token count: " + tokenizer.countTokens());

        while (tokenizer.hasMoreTokens()) {
            System.out.println(tokenizer.nextToken());
        }
    }
}

Token count: 4
Java
Python
C++
Kotlin

Exercise

Use a StringTokenizer to print each word of a sentence on its own line, followed by the number of words.

Expected output: learn, java, with, webnest, then Words: 4

import java.util.StringTokenizer;

public class WordList {
    public static void main(String[] args) {
        String sentence = "learn java with webnest";
        StringTokenizer tokens = new StringTokenizer(sentence);

        // TODO: remember how many tokens there are, then print each one

    }
}

import java.util.StringTokenizer;

public class WordList {
    public static void main(String[] args) {
        String sentence = "learn java with webnest";
        StringTokenizer tokens = new StringTokenizer(sentence);

        int total = tokens.countTokens();
        while (tokens.hasMoreTokens()) {
            System.out.println(tokens.nextToken());
        }
        System.out.println("Words: " + total);
    }
}

Quick quiz

Interview questions