Remove Duplicate Lines

How duplicates are found

The tool compares whole lines. Two lines are duplicates when they contain exactly the same characters, with two adjustments you can switch on or off:

• Ignore case treats Apple, apple and APPLE as the same line. It uses Unicode case rules, so the German straße also matches STRASSE.
• Ignore spaces at the start and end treats " apple" and "apple " as the same line. This is on by default because stray spaces are invisible and usually accidental. The line that is kept is not changed, so its spaces stay as they were.

Accented letters are compared after Unicode normalisation, so an é typed as one character matches an é typed as e plus an accent mark. Lines that differ in any other way, such as colour and color, both stay.

Which copy is kept

With First occurrence, every line stays where it first appears and later copies are removed. With Last occurrence, the last copy stays instead, which is useful when newer entries are added at the bottom of a list. The remaining lines keep their original order unless you choose A → Z or Z → A. Sorting uses natural order, so item 2 comes before item 10, and empty lines that are kept go to the end.

Empty lines

By default an empty line is treated like any other line, so only the first one stays. Keep all leaves every empty line where it is, which keeps paragraph breaks in ordinary text. Remove all deletes them. Lines that contain only spaces count as empty.

Worked example

Take this list of six lines:

apple
Banana
apple
 banana
Cherry
apple

With the default options the result has four lines: apple, Banana, " banana" and Cherry. Two extra copies of apple were removed. The line " banana" stays because its small b differs from Banana; the space in front does not matter. Turn on Ignore case and it goes too, leaving three lines. With Last occurrence instead, the result is Banana, " banana", Cherry, apple, because the last apple is on line 6.

Switch on List the repeated lines to see each line that appeared more than once and how many times, for example to check which e-mail addresses were entered twice.

Files and line endings

You can open a plain-text file or drop it on the box. If an unchanged file uses Windows line endings (CRLF), the download keeps them; otherwise the download uses Unix line endings (LF), which Windows Notepad, Excel and other current programs read without problems. If the file starts with a byte order mark (a hidden marker that says the file is Unicode text), the download starts with one too.

Text boxes become slow with very many lines, so when you paste or open a list of more than 20,000 lines, the box shows its first 20,000 lines and cannot be edited, and the result box also shows at most 20,000 lines. The tool still uses every line, and Copy and Download include the whole result. The list is processed in your browser and is not uploaded.