Smart Duplicate & Fuzzy Matcher

Smart Duplicate & Fuzzy Matcher

80%

Beyond Exact-Match Deduplication

A plain "remove duplicate lines" tool only catches entries that are byte-for-byte identical - it would completely miss that "Spider-Man: Homecoming" and "Spiderman Homecoming" are the same movie with different punctuation and spacing. This tool measures how similar two lines are using Levenshtein edit distance (the minimum number of single-character insertions, deletions, or substitutions needed to turn one string into the other), normalized into a 0-100% similarity score, then groups every line that's at or above your chosen threshold.

It's genuinely useful for cleaning up messy real-world data: scraped movie/product lists with inconsistent formatting, CSV exports with typos, or user-submitted entries that should have been deduplicated but weren't.

Tuning the Match

Adjustable ThresholdRaise it for only near-identical matches, lower it to catch looser variations - the right value depends on how messy your data is.
Case-Insensitive OptionTreats "Inception" and "INCEPTION" as identical when comparing, on by default.
Ignore Punctuation OptionStrips punctuation before comparing, so "Spider-Man:" and "Spiderman" compare on their letters alone.
100% LocalAll comparisons run in your browser - your list is never uploaded anywhere.

Frequently Asked Questions

Does this tool upload my list anywhere?

No - every comparison runs entirely in your browser using plain JavaScript. Your list of lines is never uploaded or sent to a server.

How exactly is the similarity percentage calculated?

It's 1 minus (Levenshtein edit distance divided by the length of the longer of the two lines), expressed as a percentage. Two identical lines score 100%; two completely different lines of similar length score closer to 0%.

Why are two clearly-different lines sometimes grouped together?

Edit-distance similarity is purely about how many characters differ, not meaning - "The Dark Knight" and "The Dark Knight Rises" share most of their characters and can score fairly high even though they're different movies. Raise the similarity threshold if you're seeing false-positive groupings like this.

Is there a limit to how much text I can paste?

Comparison is done pairwise (every line against every other un-grouped line), which gets slower as your list grows - the tool processes up to 1,500 non-empty lines and tells you if your input was longer than that.