Amit Choubey
← Back to Posts

Byte Pair Encoding (BPE) Explained with the Banana Bandana Example

August 12, 2025

Byte Pair Encoding (BPE) — The Banana Bandana Story

Before we get into the fun part, let's start with a quick fact:
Computers work in bytes. A byte has 8 bits, so it can represent 2^8 = 256 different values — numbered from 0 to 255.
When tokenizing text, each unique token is assigned an ID in this range until the vocab grows larger, at which point new tokens get IDs like 256th, 257, and so on.

One of the smartest ways to decide which tokens to create is Byte Pair Encoding (BPE) — a method used by GPT, RoBERTa, and many others. It merges the most frequent pairs of symbols into bigger tokens again and again, keeping the vocab small but expressive.

And instead of boring definitions, let's make it fun with a banana wearing a bandana.

1. The Idea Behind BPE

  • Start with characters (including spaces).
  • Find the most frequent pair.
  • Merge it into a single token.
  • Repeat until you've built your vocab or hit a limit.

Why this works:

  • Captures common subwords like "ing", "tion", "ana".
  • Handles rare words & typos better than splitting only on spaces.
  • Keeps the vocabulary size manageable.

2. Step-by-Step Example: "banana bandana"

Instead of capital letters, let's use numbers for clarity:

1 = b
2 = a
3 = n
4 = d
5 = (space)

Our string "banana bandana" becomes:

1 2 3 2 3 2 5 1 2 3 4 2 3 2

Step 0 — Start

[1] [2] [3] [2] [3] [2] [5] [1] [2] [3] [4] [2] [3] [2]

Step 1 — Merge most frequent pair (2 3 → 23)

[1] [23] [2] [3] [2] [5] [1] [23] [4] [2] [3] [2]

Step 2 — Merge most frequent pair (1 23 → 123)

[123] [2] [3] [2] [5] [123] [4] [2] [3] [2]

Step 3 — Merge (23 2 → 232) and (123 4 → 1234)

[123] [232] [5] [1234] [232]

3. Use Cases of BPE

  • Efficient Tokenization — fewer tokens, same meaning.
  • Typo Resilience — bananna still splits into familiar parts.
  • Cross-Language Support — works for languages without spaces.
  • Subword Coverage — learns reusable building blocks for words.

4. Try It Yourself (Minimal)

Here's a tiny snippet to try BPE on "low low lower" using <256th> token IDs:

tokens = ["l", "o", "w", " ", "l", "o", "w", " ", "l", "o", "w", "e", "r"]
for step in range(3):  # limit merges
    pairs = [(tokens[i], tokens[i+1]) for i in range(len(tokens)-1)]
    most_common = max(set(pairs), key=pairs.count)
    tokens = [tokens[i] + tokens[i+1] if i < len(tokens)-1 and (tokens[i], tokens[i+1]) == most_common else tokens[i]
              for i in range(len(tokens)) if not (i>0 and (tokens[i-1], tokens[i]) == most_common)]
    print(f"Step {step+1}:", tokens)

Run this and watch your tokens merge — the first new token after the original 0–255 set would get ID <256th>.

Conclusion

BPE isn't just "token splitting" — it's the magic that lets large language models keep a tiny vocab while speaking fluently in countless contexts. By understanding how BPE works, you gain insight into one of the fundamental techniques that powers modern NLP systems.

The beauty of BPE lies in its simplicity: it learns the most common patterns in your text and builds a vocabulary that captures the essence of language without overwhelming the model with rare tokens. This makes it an essential tool for anyone working with language models or interested in how computers understand human text.

But BPE is just one approach to tokenization. Are there other techniques that might be even better? What about SentencePiece, WordPiece, or character-level tokenization? We'll explore these alternatives and their trade-offs in future blog posts. Stay tuned to discover the full spectrum of tokenization strategies that power today's language models.