TL;DR
- Duplicates return after a cleanup when the forms, tools and imports behind them keep running.
- Reviewing 1,000 duplicates by hand, one by one, can take a full day.
- Each source of duplicates leaves its own pattern, so each source needs its own search.
- Merging new duplicates automatically on a schedule works best once each search has been tested.
- Dedupely bulk merges the backlog and can merge each source's new duplicates automatically.
Duplicates keep coming back because the forms, integrations, imports and people that created them are still adding records. A cleanup clears the backlog, and those sources start a new one the next morning.
If you're the admin, you know the cycle. You spend days merging, the CRM looks clean for a week, and then the same kinds of duplicates show up again. Working faster doesn't change the outcome, because the work never ends.
In this post, you'll see why manual cleanup takes so long and why it can't keep up. You'll also see how to find each source of new duplicates, build a search for it and start merging automatically on a schedule.
Why cleanup takes hours or days
Merging by hand is slow because every set needs a decision. You open the records, compare the fields, pick what stays and merge. Then you do it again for the next set.
Teams that do this at scale describe the cost in plain terms. One team estimated a full day of work for every 1,000 duplicates reviewed one by one. Others merge around 300 a day by hand for weeks and still have thousands left.
At that scale, merging one by one can take hundreds of hours.
The time also grows with uncertainty. Exact email matches take seconds to confirm. Near matches, shared phone numbers and records with blank fields take longer, and those are the sets that get left for "later".
Why duplicates come back after a cleanup
A cleanup fixes records. It doesn't change what creates them, so the inflow continues at the same rate as before.
The sources are still running
Web forms, booking tools, phone and call-tracking apps, enrichment tools, list imports and syncs with other systems all add records. Many of them create a new record when they don't find an exact match on file. Each source is covered in more detail in CRM deduplication across tools.
People create records too
Reps add contacts from their inbox, import their own lists and create a company they couldn't find in search. Each habit adds a few records a week, and they add up across a team.
Clearing one layer shows the next
Some teams find that once they clear one batch, another set shows up.
The first pass usually catches exact matches. Then near matches become visible: a typo in an email, a phone number in another format, a company with two domains. To the admin, it looks like the problem got bigger.
Manual merging runs at a fixed pace
One person can merge a set number of records a day. New duplicates arrive every day, including weekends and the admin's days off. At 25 new duplicates a day, a week away means 175 sets waiting on your return.
How to break the cycle
The fix is to treat the backlog and the inflow as two different jobs. The backlog is cleared once. The inflow is handled by searches that run on their own, one per source.
Find where new duplicates come from
Look at the duplicates created in the last few weeks, not the whole database. For each set, check the created date, the source or original source field and the user or integration that created it. Two or three sources usually stand out quickly.
Write them down in a short list. That list decides which searches you need and which fixes belong upstream, like a form that should update records instead of creating them.
Build one search per source
Each source tends to create duplicates with the same shape. A single search for everything either misses those patterns or pairs records that aren't duplicates. A search built for one source can be strict where that source is reliable.
| Source | What its duplicates often look like | Fields that often match them |
|---|---|---|
| Web forms and booking tools | Same person, same email, new record on each submission | Email, plus first and last name |
| Phone and call-tracking apps | Same number in a different format, often with no name or email | Phone, compared without formatting |
| Enrichment and prospecting tools | Company name written differently, same domain | Core company domain, plus company name |
| List and event imports | Same name and company, different or missing email | First name, last name and company |
| Reps creating records | Personal and work email for one person | Name and phone, or name and company |
What counts as a duplicate is case by case, so adjust these to your data. Two people who share a household phone, for example, aren't one person.
Test each search before you trust it
Merge a few sets from the search by hand and open the results in the CRM. Then merge a page in bulk and check a sample. Reject every false match you find so it stays out of later runs.
A search is ready when it stops surprising you. If one source keeps producing false matches, tighten its search rather than reviewing every set by hand.
Get alerts first, then merge automatically
Start with alerts. A search can check for new duplicates on a schedule and email you when they reach a number you set. That shows how fast each source produces them, without merging anything.
Once a search has proven itself, merge its new duplicates automatically on a schedule, and leave unclear sets for review. The trade-offs between catching duplicates at entry and on a schedule are covered in catching CRM duplicates at entry vs. on a schedule.
Keep a short review routine
Automation handles the patterns you know. A short regular check catches new ones, like a new integration or a form that changed. How often to run that check depends on how much new data you get, as covered in how often you should deduplicate your CRM.
How Dedupely keeps duplicates from coming back
In Dedupely, you build a duplicate search for each source and bulk merge the backlog, with a preview of each merge. Once a search is tested, you can get an email when new matches reach a set number, then merge them automatically on a schedule. Large groups and placeholder data are held back by default, so they wait for a person.
From cleanup projects to a routine that keeps up
Duplicates come back because their sources never stopped. Merging faster by hand only shortens the gap between one cleanup and the next.
Finding each source, giving it a tested search and merging what you're certain of on a schedule turns cleanup into a routine. The hours go into the cases that need a person, and the rest runs without you.
Frequently asked questions
Why do duplicates keep coming back in my CRM?
Because the forms, integrations, imports and people that created them are still adding records. A cleanup removes existing duplicates but doesn't change the sources. Find which sources create most new duplicates, build a search for each one and merge their new duplicates regularly.
How long does it take to merge duplicates by hand?
One team estimated a full day for every 1,000 duplicates reviewed one by one, and it depends on how certain the matches are. Exact email matches are quick to confirm. Near matches, shared phone numbers and records with blank fields take longer, because each set needs a closer look before you merge.
How do I find out what is creating duplicates?
Look at duplicates created in the last few weeks. For each set, check the created date, the source field and the user or integration that created the record. Two or three sources usually account for most of them, and those are where to start.
Is it safe to merge duplicates automatically?
It can be, for searches you've tested. Merge a few sets by hand, then a page in bulk, and reject any false matches. Once results are stable, merge new duplicates you're certain of, like exact email matches, automatically on a schedule, and keep unclear sets for review.
Can I stop duplicates from being created at all?
You can reduce them. Forms can update existing records by email, imports can check for matches first and reps can search before creating a record. Some duplicates still get through, like typos and second email addresses, so a regular search for new duplicates is still needed.
Or sign up with Google
Pipedrive Marketplace 
