How to build a deduplication system for RevOps

Many RevOps teams have run a big CRM cleanup. It takes weeks, the duplicate count drops, and months later the same problems are back.

The cleanup worked. What was missing was a system around it. New records keep arriving from forms, syncs, enrichment tools and imports, and a one-time project can't keep up with sources that never stop.

In this post, you'll see the parts of a deduplication system and how they connect. That covers searches, rules for which value wins, alerts, scheduled merging, review by owner, reporting and a record of every merge.

What makes deduplication a system

A project has an end date. A system has inputs, rules and outputs, and it keeps running while people work on other things.

For RevOps, that means each part of deduplication has a clear job. Searches find duplicates, rules decide the outcome, automation handles what's certain, people handle what isn't, and reports show whether it's working.

Who owns the system is a separate question. Our post on CRM deduplication for RevOps teams covers ownership and how to learn what each team needs. This post is about how the system is built.

The parts of a deduplication system

Start with a map of your sources

Every duplicate comes from somewhere. In a RevOps stack, that's usually a mix of forms, enrichment tools, call apps, helpdesk syncs, imports and other CRMs.

List each source with three facts: what it creates, which field it uses to decide a record already exists, and who owns it. That list becomes the plan for your searches.

Sources also change. Add a line to the map every time a new tool connects to the CRM, before its first sync.

Build one search per source or scenario

A single search that tries to catch every duplicate is hard to test. When it pairs the wrong records, you can't tell which part of the criteria caused it.

Smaller searches are easier to trust. Give each source or scenario its own search, with criteria that fit the data it produces.

• By source: an enrichment tool that creates contacts with a name and company domain gets a search on those fields.
• By scenario: contacts with the same email, companies with the same domain, or leads that match an existing contact's phone.
• By certainty: a strict search for exact matches, and a looser one for near matches that need review.

Filters keep each search focused, for example on records created after a date or on one lead source. What counts as a duplicate is still case by case, so test each search on sets you already know.

Write the rules for which value wins once

Searches can be many, but the outcome of a merge should follow one set of rules. Otherwise, the same customer ends up with a different owner depending on which search found them.

Decide which record survives, then go field by field. Owner, lifecycle stage, lead source and email preferences need the most care, since different teams report on them.

Keep the rules in one place and change them on purpose. A rule change affects future merges only, so note the date when you change one.

Alert first, then merge on a schedule

New searches shouldn't merge anything on their own. Start each one on alerts only: you get notified when matches reach a number you choose, and you merge by hand.

That period tells you how the search behaves on new data. Merge one set, then a page of sets, and reject any match that turns out to be two different records.

Once the results stay stable, you can merge new duplicates you're certain of, like exact email matches, automatically on a schedule. Looser searches stay on alerts, with a person deciding.

The trade-offs between catching duplicates at entry and on a schedule are covered in catching CRM duplicates at entry vs. on a schedule.

Split the review by record owner

Automation covers the duplicates you're certain of. The rest needs a person, and the system has to say which person.

Two patterns come up often. In the first, one person owns cleanup and sales reps stay out of it, so the CRM changes in one predictable way. In the second, sets are split by owner.

• Same owner on every record: the merge doesn't change who owns the customer, so these sets are good candidates for bulk or scheduled merging.
• Different owners: the merge decides who keeps the account, so these sets go to review.

When owners differ, the owner of the surviving record or their manager should confirm the result. Write down who decides, so a disputed account doesn't wait for weeks.

Report on the system, not only the cleanup

A system needs a signal that it's working. A short report on a regular cadence, weekly or monthly, is enough for most teams.

Look at new duplicates per search and per source, sets merged, sets waiting for review and matches rejected. A search that suddenly finds many more matches often points to a source that changed. For the numbers worth tracking, see duplicate rate and other metrics for a clean CRM.

Keep a record of every merge

A record of every merge, with the original values and the values that won, is how you answer questions later.

It tells a rep why their account changed and shows a manager which search merged what. If a merge was wrong, it holds the values you need to restore. More on this in why you need a record of every merge.

A system outlasts the cleanup

A big cleanup fixes the past. The sources that created those duplicates keep running, so the fix only lasts if something keeps running too.

Map the sources, give each one a search, decide outcomes once and let automation handle only what's certain. People review the rest, and a report and a merge log show the whole team that the CRM is being looked after.

Frequently asked questions