Beyond Exact-Match Deduplication
A plain "remove duplicate lines" tool only catches entries that are byte-for-byte identical - it would completely miss that "Spider-Man: Homecoming" and "Spiderman Homecoming" are the same movie with different punctuation and spacing. This tool measures how similar two lines are using Levenshtein edit distance (the minimum number of single-character insertions, deletions, or substitutions needed to turn one string into the other), normalized into a 0-100% similarity score, then groups every line that's at or above your chosen threshold.
It's genuinely useful for cleaning up messy real-world data: scraped movie/product lists with inconsistent formatting, CSV exports with typos, or user-submitted entries that should have been deduplicated but weren't.
Tuning the Match
Frequently Asked Questions
No - every comparison runs entirely in your browser using plain JavaScript. Your list of lines is never uploaded or sent to a server.
It's 1 minus (Levenshtein edit distance divided by the length of the longer of the two lines), expressed as a percentage. Two identical lines score 100%; two completely different lines of similar length score closer to 0%.
Edit-distance similarity is purely about how many characters differ, not meaning - "The Dark Knight" and "The Dark Knight Rises" share most of their characters and can score fairly high even though they're different movies. Raise the similarity threshold if you're seeing false-positive groupings like this.
Comparison is done pairwise (every line against every other un-grouped line), which gets slower as your list grows - the tool processes up to 1,500 non-empty lines and tells you if your input was longer than that.