Cleaning Up Messy Text: Remove Duplicates, Sort Lines, Strip Whitespace and Diff the Result

Someone sends you a list of 1,400 email addresses from three different exports. Some are duplicated, some have trailing spaces, some are in capitals, a few have a tab where a space should be, and the file mixes Windows and Unix line endings. You need one clean, deduplicated, sorted list by lunchtime.

This comes up constantly: mailing lists, SKU lists, log extracts, keyword lists, URLs from a crawl, names from a form export. Spreadsheets handle some of it badly and the command line handles it well but not everyone has one open. Here is the order of operations that works, and a browser tool for each step.

#Key takeaways

• Clean in a fixed order: whitespace first, then case, then duplicates, then sort. Doing it in another order leaves duplicates behind.
• "Duplicate" needs a definition. Alice@Example.com and alice@example.com are the same address and different strings.
• Sort naturally when the data has numbers in it, or item10 lands before item2.
• Diff the before and after. It takes ten seconds and catches the time you removed something you needed.

#Why order matters

Suppose the list contains these three lines:

alice@example.com
Alice@example.com
alice@example.com   ← trailing spaces

Remove duplicates first and nothing is removed, because all three strings differ. Trim whitespace and lowercase first, and all three become identical, so deduplication leaves one. The cleaning steps have to run before the comparison step.

The full order:

1. Normalise whitespace and line endings
2. Normalise case (if the data is case-insensitive)
3. Remove duplicates and blank lines
4. Sort
5. Diff against the original

#Step 1: Whitespace and line endings

Paste the text into the Whitespace Cleaner. It trims leading and trailing spaces on each line, collapses runs of spaces and tabs, removes blank lines and normalises line endings.

Two things it will catch that are invisible in most editors:

• Non-breaking spaces (U+00A0), which come from web pages and Word. They look like spaces, and "alice" ≠ "alice " as far as any program is concerned.
• Mixed line endings (\r\n and \n in the same file), which make some tools see every other line as blank.

If you only want to trim and keep internal spacing (for example, addresses where double spaces are meaningful), use the trim-only option rather than collapsing.

#Step 2: Case

Decide whether case matters. For email addresses, usernames and most identifiers it does not, and normalising to lowercase before deduplicating is correct. For names and product titles it does, and you should skip this step and use a case-insensitive duplicate check instead.

The Text Case Converter handles lower, UPPER, Title Case, Sentence case, camelCase, snake_case and kebab-case on the whole block at once. Title Case is the one to be careful with: it capitalises "Of" and "The" in the middle of names, which may not be what you want.

#Step 3: Duplicates and blanks

The Duplicate Line Remover keeps the first occurrence of each line and drops the rest. Options worth knowing:

• Case-insensitive, which treats Alice and alice as the same. Use this when you decided in step 2 to keep the original case.
• Trim before comparing, so trailing spaces do not create false uniques. Harmless to enable even if you ran the whitespace cleaner.
• Remove blank lines at the same time.

The tool reports how many lines were removed. If that number surprises you (you expected a handful and it removed 600), stop and look at what was removed before continuing. A column of identical placeholder values, or a list that was accidentally pasted twice, both produce big numbers for legitimate reasons, but a wrong setting does too.

#Step 4: Sort

The Text Line Sorter sorts alphabetically, numerically, by length, naturally, and in reverse.

Natural sort is the one most people want and do not know to ask for. Plain alphabetical sort puts file10 before file2, because 1 sorts before 2. Natural sort compares the numbers as numbers: file2, file10, file100. Use it for anything with version numbers, invoice numbers or SKUs.

Numeric sort is for lines that are entirely numbers. Sort by length is useful for keyword lists (short head terms first) and for spotting outliers: the longest "email address" in a list is usually two addresses stuck together.

Sort after deduplicating, not before. Sorting first does not help the duplicate check and makes the diff in the next step harder to read.

#Step 5: Diff the result

Open the original and the cleaned text side by side in the Text Diff Checker. Every removed line is highlighted. Skim it. You are looking for lines that were removed as "duplicates" but were not, which usually means the case or trim setting was more aggressive than the data could take.

This step feels unnecessary right up until the time it finds the problem. It is ten seconds.

#Worked example

The email list from the introduction, 1,400 lines from three exports:

Three minutes, and the diff showed that 480 of the removed lines were genuine duplicates and two were malformed entries (name@@example.com) that needed a human decision anyway.

#When you need more than line operations

Some clean-up is inside the lines, not between them:

• Extracting the domain from each address, or the number from each line: the Regex Builder and Regex Tester.
• Counting what you have (how many lines, words, characters): the Word Counter.
• Escaping the result for use in JSON, a URL or HTML: String Escape.
• Turning a list of URLs or titles into slugs: the Slug Generator.

#Frequently asked questions

#Do these tools upload my text?

No. The whitespace cleaner, case converter, duplicate remover, sorter and diff checker all run in your browser. The text stays on your device, which matters when the list is customer data.

#Why does my duplicate check miss obvious duplicates?

Almost always whitespace or case: a trailing space, a non-breaking space, or a capital letter. Run the whitespace cleaner first, and enable case-insensitive comparison if case is not meaningful for your data.

#How do I keep the last occurrence instead of the first?

Reverse the order with the sorter, remove duplicates (which now keeps what was originally last), then reverse again.

#Is there a size limit?

These are browser tools, so the limit is your browser's memory. Lists of a few hundred thousand lines are fine. For multi-gigabyte files, use sort -u on the command line.

#Tools used in this guide

• Whitespace Remover & Text Cleaner
• Text Case Converter
• Duplicate Line Remover
• Text Line Sorter
• Text Diff Checker