Remove Duplicate Lines
Delete repeated lines from lists, emails, keywords or logs while keeping the first copy in its original order. Exact, case-sensitive matching. Free and private.
What is the Remove Duplicate Lines?
Merged two email lists, exported keywords from several tools, or copied a log full of repeats? Remove Duplicate Lines deletes every line that has already appeared and keeps the first copy exactly where it was. The order of your list stays intact, so the output is ready to paste back into a spreadsheet, CRM, code file or text editor.
Matching is exact and case-sensitive. Apple and apple are treated as different entries, and a line with a trailing space is not the same as one without. That strictness means nothing is removed by accident. Windows, Mac and Unix line endings are handled alike, blank lines collapse to a single empty line, and you can open a local text file instead of pasting and download the cleaned list as a text file.
How to use the Remove Duplicate Lines
- Paste your list into the One entry per line box, with each item on its own line.
- Or press Open a text file to load a local .txt, .csv, .json, .md, .html, .xml or .log file of up to 5 MB into the box; a live character count shows how much text is loaded.
- Press Process to remove repeated lines.
- Review the cleaned list in the result area; the first occurrence of each line remains in its original position.
- Press Copy result to take the list with you, Download to save it as unique-lines.txt, or Reset to clear everything.
Features
- Removes every repeat of a line and keeps only its first occurrence.
- Preserves the original order of the remaining lines; nothing is sorted.
- Exact, case-sensitive comparison where spaces count.
- Normalises Windows (CRLF) and old Mac (CR) line endings before comparing.
- Processes lists up to 1,000,000 characters, pasted or loaded from a text file, and downloads the result as unique-lines.txt.
Why use this remove duplicate lines
- Cleans mailing lists and contact exports before an import, avoiding double sends.
- De-duplicates keyword, URL and tag lists collected from different sources.
- Saves time compared with spreadsheet formulas or manual scanning.
- Keeps customer data private because the list never leaves your browser.
How it works
Normalize CRLF/CR to LF; preserve the first occurrence of each exact line using a set.
Examples
Repeated words with different case
Input: apple pear apple Apple
Result: apple pear Apple Exact, case-sensitive comparison; the first occurrence and one empty line are retained.
Blank lines and a trailing space
Input: red blue red red
Result: red blue red Exact, case-sensitive comparison; the first occurrence and one empty line are retained. The last line keeps its trailing space, so it is not a duplicate of red.
Results are estimates for general information, not financial, medical, legal, tax, or property advice. Confirm important figures with the relevant institution or a qualified professional.
Remove Duplicate Lines: frequently asked questions
Does it keep the original order of my lines?
Yes. The tool walks through your text from top to bottom and keeps each line the first time it appears. Later copies are dropped, but nothing is moved or sorted. If your list was in priority or date order, the cleaned version is still in that order, just without the repeats. Sort the list separately if you need it alphabetical.
Is the comparison case-sensitive?
Yes. Apple, apple and APPLE are three different lines and all of them are kept. This avoids merging entries that genuinely differ, such as codes or passwords where case matters. If you want case-insensitive removal, first convert the whole list to lowercase with a case converter, then run it through this tool.
Why were two lines that look identical both kept?
They probably differ in invisible characters. A trailing space, a tab or a non-breaking space makes a line unique, so red and red followed by a space are both kept. Lines are not trimmed before comparing. Run your list through the whitespace cleaner first if spacing differences should be ignored, then remove duplicates.
What happens to empty lines?
Empty lines are treated like any other line, so the first blank line is kept and every later blank line is removed as a duplicate. In practice that means a list separated by several blank lines ends up with only one empty line, located where the first blank appeared. Delete that remaining blank line by hand if you do not want it.
Can I remove duplicates from a CSV file?
Yes, if each row is on one line. Load the file with Open a text file and the tool removes rows that are exactly the same from start to finish, including every column. It does not compare a single column, so two rows with the same email but different names are both kept. The header row is kept as long as it is not repeated.