Ongoing Deduplication
Ongoing deduplication is the practice of finding and merging duplicate records in a CRM continuously or on a set schedule, after the existing duplicates have been cleaned up once. Also called automated deduplication or scheduled deduplication, it treats duplicates as a steady flow to manage rather than a backlog to clear every year or two.
A one-time cleanup removes the duplicates that exist today. Ongoing deduplication handles the ones that arrive next, from web forms, list imports, integrations and records added by hand. It often combines three parts: automated detection that finds new duplicate sets, automatic merging for the rules a team already trusts, and a regular review of everything else. The main decisions shift from how to clear a backlog to how often each check runs, which rules merge on their own and who reviews the rest.
See how Dedupely keeps deduplication running in your CRM
What Is Ongoing Deduplication?
Ongoing deduplication, also called continuous or automatic deduplication, is the routine that keeps a CRM close to its cleaned state. It covers each object a team deduplicates: contacts and companies in HubSpot, accounts and leads in Salesforce, people and organizations in Pipedrive, leads in Close, and contacts in Nimble and Google Contacts.
A simple example: after a cleanup, a team runs three checks on contacts. An exact email check merges new pairs, such as two records for jsmith@acme.com created by a webinar form and a list import. A looser name-and-company check sends an alert when it finds new sets, so someone can confirm that J. Smith and John Smith at Acme are the same person before merging. Once a quarter, a broader sweep looks for patterns the regular checks don't cover.
Teams set this up with different levels of automation, and each has its strengths. Alert-based setups find new duplicates automatically and leave every merge to a person, which keeps a human decision in front of each set. Automatic merging handles trusted rules without anyone starting a run, which keeps new duplicates from staying in the CRM for long. Periodic sweeps, run by hand on a calendar, can use looser criteria and look across the whole database, which catches what a narrow automated rule misses. Many teams combine all three, matched to how much each rule can be trusted.
How Ongoing Deduplication Works
Ongoing deduplication repeats a cycle that's set up once and adjusted over time:
- Start from a clean baseline. The existing backlog is cleared first, by hand or in bulk. Otherwise the automated checks spend their runs on old duplicates, and new ones are hard to spot among them.
- Sort the rules by trust. Rules that have returned real duplicates over many rounds, often built on strong fields like email addresses or customer IDs, can merge on their own. Rules on looser criteria send alerts instead, and the broadest patterns are left for periodic sweeps.
- Set a cadence for each rule. How often a check runs depends on how fast records arrive and how soon a duplicate causes trouble. A busy web form may need frequent checks, while a slow-moving object may need only a weekly one.
- Keep the data current. A check only finds duplicates in the records it can see. Tools that work alongside the CRM refresh their data at intervals, so the gap between a record's creation and its first check depends on that refresh.
- Route the exceptions. Uncertain matches go to a review queue, and risky ones, such as very large sets or placeholder values, are held back from automatic merging.
- Review the results and adjust. Merge counts, held-back matches and review queues show whether a rule is working. A rule that keeps producing false matches gets stricter, and one that proves reliable can move up a tier.
Why Ongoing Deduplication Matters for CRM Teams
A CRM starts drifting from its cleaned state the day after a cleanup. Each new duplicate counts one customer twice in reports, splits activity across two timelines and can send the same lead to two reps. In CRMs that bill by contact or record count, it's also a record paid for twice. Between one-off cleanups, those effects grow with every form fill and import.
Small, regular merges are easier to manage than large, rare ones. A check that finds a handful of new sets is quick to act on, and the context behind each one, such as the import or campaign that created it, is still easy to trace. A cleanup that waits a year faces a far larger set of duplicates, harder decisions about which values to keep, and a sudden shift in reported numbers when hundreds of records merge at once.
Ongoing deduplication also makes the work predictable. Instead of a project that competes with other priorities, it becomes a short, recurring review with a known scope: the sets a rule couldn't merge on its own. And because the same rules run each time, the results are consistent from one month to the next, which keeps reports comparable over time.
Getting Started With Ongoing Deduplication
Begin after the first cleanup, not before. Clear the existing duplicates first, then look at where new records come from, such as web forms, imports, integrations and manual entry, and how many arrive each week. That pace sets how often each check needs to run.
Decide which rules to automate based on what you've already seen. A rule whose results you've merged by hand over several rounds, with no sets rejected or changed, is ready to merge on its own. A rule that finds useful matches but still needs a person is better set to alert you. Patterns that need looser criteria can wait for a periodic sweep on a calendar, such as once a quarter. Set what each merge keeps, and hold back risky matches, before any rule runs on its own.
Then make the review a habit. Set a regular time to look at what merged, what was held back and what's waiting for a decision. Compare a few merged records with their originals, tighten rules that produce false matches, and move a rule to automatic merging once its results hold up.
In Dedupely, Auto Match and Auto Merge carry out this approach on each Duplicate Search: Auto Match notifies you when new matches reach the threshold you set, and Auto Merge merges a trusted search's new duplicates on your schedule, sending low-confidence matches to Manual Review. The Duplicate Report email, sent weekly or monthly, shows the merges, Auto Merge activity and the matches that need your decision in each connected CRM.
Key Takeaways: Ongoing Deduplication
Ongoing deduplication keeps finding and merging duplicates after the first cleanup, continuously or on a schedule, so new duplicates from forms, imports and integrations are handled while they're few. It works best from a clean baseline, with rules sorted by trust: proven rules merge automatically, looser ones send alerts, and the broadest patterns wait for periodic sweeps. Each check runs at a pace that matches how fast records arrive, and a regular review keeps the rules accurate. In Dedupely, Auto Match and Auto Merge are set per Duplicate Search: Auto Match notifies you of new matches, and Auto Merge merges a trusted search's new duplicates on your schedule. Safety Settings hold risky matches back, Manual Review holds the uncertain ones, and the Duplicate Report shows what happened.
Frequently Asked Questions About Ongoing Deduplication
How often should CRM deduplication run?
Run each check as often as new duplicates arrive and as often as someone can act on what it finds. A CRM that takes in many form fills or a steady flow of leads may need checks several times a day, while one that changes slowly may need only a weekly check. The answer can differ by rule: a strict email rule that merges on its own can run often, a looser rule that sends alerts can run less often so each review holds enough sets to be worth the time, and a broad sweep can run monthly or quarterly. In Dedupely, Auto Match and Auto Merge are set per Duplicate Search, so each rule keeps its own pace: a strict search can run Auto Merge as soon as possible, in a continuous loop of scan, merge and scan again, while a looser search stays on Auto Match and notifies you when new matches reach the threshold you set. Auto Merge can also run every set number of minutes, hours, days, weeks or months, or between chosen hours.
Can automated deduplication run on new records as they're created?
Yes, automated deduplication can check new records soon after they're created and merge the clear matches. How soon depends on the setup. Some tools check a record at entry and warn about a likely duplicate, others compare new records in frequent batches, and others run in a continuous loop that scans and merges as records arrive. Checks at entry catch some duplicates before anyone works on them, and frequent batch checks catch the ones that arrive through imports and syncs, which an entry check may not see. The setup that fits depends on where a CRM's records come from, whether the CRM is HubSpot, Salesforce, Pipedrive or another. In Dedupely, new records are detected within 1 to 5 minutes, and synced data refreshes every 10 to 30 minutes. Auto Merge can run as soon as possible, so a trusted Duplicate Search merges new duplicates shortly after they reach Dedupely, while low-confidence matches go to Manual Review instead.
What's the difference between a one-time cleanup and ongoing deduplication?
A one-time cleanup clears the duplicates that exist today, and ongoing deduplication handles the ones that arrive afterward. A cleanup is a project: it looks across the whole database, often with several rules and looser criteria, and it needs time to review large sets and decide which values to keep. Ongoing deduplication is a routine: a few trusted rules run on a schedule, new duplicates are handled in small batches, and a person reviews only what the rules can't settle. The two work in sequence. The cleanup sets the baseline and shows which rules can be trusted, and those rules then become the automated checks. Without the routine, a cleaned CRM drifts back as forms and imports add records. In Dedupely, a cleanup often uses Bulk Merge to merge many sets at once, and Auto Match and Auto Merge then keep the trusted Duplicate Searches running. Merges can't be undone in either case, so Merge Rules and Safety Settings should be in place first.
Which duplicates should be merged automatically in ongoing deduplication?
Merge automatically only the duplicates your rules already pair correctly, and send the rest to a person. Good candidates are matches on strong identifiers, such as an exact email address or a customer ID combined with a second field, from sources that create duplicates often, like repeat form fills. Keep matches on looser criteria, such as similar names, on alerts, along with records where a wrong merge costs more, such as accounts tied to contracts or billing. Large sets and placeholder values, like "test@test.com," should be held back from automatic merging, because they point to bad data rather than one record. The split can change: a rule that proves reliable can move to automatic merging later. In Dedupely, Safety Settings exclude matches with more than 20 records (the default) or built on poor or example data from Auto Merge, and Auto Merge sends low-confidence matches to Manual Review instead of merging them. Dedupely Intelligence can't be automated, so its matches wait for you to review and merge them.
How do you monitor ongoing deduplication?
Track what merged, what was held back and what's waiting for a decision, and check them at a regular time. Merge counts over time show whether duplicates are arriving faster or slower. Held-back matches show whether safety rules are catching bad data or blocking real duplicates. A growing review queue means the uncertain sets need more attention, or a rule needs stricter criteria. A sample of merged records compared with their originals shows whether the merge settings keep the right values. In Dedupely, the Duplicate Report email, sent weekly by default or monthly, shows merges with their trend, Auto Merge activity, time saved by Auto Merge and matches found, plus excluded matches to review and, for each connected CRM, the matches that need your decision. Each person sets their own reports. Merge History records each merge with the original records and their values, and the Audit Log, which only admins can see, records merges, scans and scheduled Auto Matches.
