How to automate CRM deduplication without losing data

Once a CRM is clean, duplicates start coming back. Forms, imports, integrations and reps typing in records add new ones every week, and nobody wants to merge them by hand forever.

Automation is the answer, but it's also where cleanups go wrong, since an automatic merge runs without anyone looking. A merge changes your live CRM, and a loose search on a schedule can do a lot of damage overnight.

In this post, you'll see how to automate CRM deduplication in stages. That means alerts first, then tested searches, rules for what each merge keeps, safety limits and a way to stop everything quickly.

Two things you can automate

Deduplication has two parts, and you can automate each one on its own.

• Finding duplicates: a search runs on a schedule and tells you when new matches appear. Nothing in your CRM changes.
• Merging duplicates: a search runs on a schedule and merges what it finds. Your CRM changes every time it runs.

Finding can be automated early and for almost every search. Merging should be automated late, only for searches you trust completely.

Our post on catching duplicates at entry vs. on a schedule covers where scheduled work fits next to your CRM's own deduplication.

Get alerts before you merge anything

Start every search in a notify-only mode. In Dedupely, that's Auto Match. It checks for new duplicates on a schedule and emails you when they reach a number you choose, without merging.

That gives you two things. You see how many new duplicates each source produces, and you see what they look like before any rule acts on them. For some searches, alerts are all you'll ever need.

Test each search by hand

A search earns automatic merging by proving itself first. Before you switch one on, work through it manually.

• Merge one or two sets by hand and open the results in your CRM.
• Merge one or two pages of sets in bulk and check a sample.
• Reject every false match you find, so it stays out of later runs.
• Watch the results for a while. A search is ready when it stops surprising you.

Test the search on the edge cases you know too. Family members sharing an email, branches sharing a domain and placeholder values like test@test.com should not show up as matches.

Choose which duplicates to automate

Automate the duplicates you'd merge without looking. Exact email matches for contacts are the usual first choice. Three or four strong fields that agree are a good standard for anything else.

What counts as a duplicate is case by case, so the right searches depend on your data. As a rule of thumb:

Keep the looser searches on alerts and review their results by hand.

Decide what each merge keeps

An automatic merge follows your rules with nobody checking the result. Set those rules before it runs. In Dedupely, they're your Merge Rules.

• Which record survives: the oldest, the newest, the most recently updated or the most complete.
• Which values win: for owner, lifecycle stage, email and phone, decide field by field.
• Blank fields: an empty value should never overwrite a filled one.

Preview a few merges with those rules before switching anything on. Our post on merging CRM records without losing data covers each choice in depth.

Merge automatically on a schedule

Once a search is tested and its rules are set, let it merge new duplicates automatically on a schedule. In Dedupely, that's Auto Merge. The schedule can be as often as possible, every few hours, daily, on chosen weekdays or within a time window.

Even the shortest schedule isn't instant. New records take a few minutes to reach a deduplication tool, so a rep may see a duplicate briefly before it's merged.

Pick the schedule by how fast duplicates cause harm. A form that creates duplicates all day may need frequent runs, while a weekly import may only need one run after it lands. Our post on how often to deduplicate your CRM covers cadence in more detail.

Keep safety limits on

Even a tested search can meet data it hasn't seen. Safety limits catch the riskiest matches before they merge.

• Large groups: a set of more than about 20 records is rarely one customer. It's usually a placeholder or a shared value.
• Placeholder data: values like test@test.com, John Doe or 0000000000 shouldn't drive a merge.
• CRM limits: some CRMs cap how many associations a record can hold or how often it can be merged. Decide in advance whether those merges should wait for you.

Risky matches should wait in a holding area where you can merge, reject or review them by hand. In Dedupely, the Safety Settings move them to the Excluded tab. Sets you've sent to review or rejected should never merge automatically.

Watch it, and know how to stop it

Automatic merging still needs an owner. Check the merges it made, especially in the first weeks, and keep a record of every merge with the original values. Dedupely's Merge History does that, and you can download it as a CSV.

If something looks wrong, stop first and investigate second. Cancel any merges still running, then pause or turn off the schedule so the same mistake doesn't repeat while you look.

From there, find the affected merges, restore the original values from your merge record and reject the false match. Then tighten the search and put it back on alerts until it earns your trust again. Our playbook on what to do if you merged the wrong records walks through each step.

Automate in stages

Automating deduplication without losing data comes down to order. Alerts come first, then searches tested by hand, then rules for what each merge keeps, then automatic merging on a schedule.

Safety limits and a quick way to stop make the rest of the risk manageable. Automate only what you'd merge without looking, and the uncertain sets stay with the people who can decide them.

Frequently asked questions