Ultimate Salesforce Duplicate Management Guide (Best practices)

Duplicates in Salesforce cost time and revenue. This guide covers where they come from, what Salesforce does about them natively, and how to clear the ones already in your org.

It is a plan in three steps: clear what is there, stop new ones entering, then automate.

Table of Contents:

1. Salesforce Duplicates 101
   1. How duplicates enter Salesforce
   2. Why are Salesforce duplicates costly and who should worry?
   3. Should I worry about duplicates today?
   4. What Salesforce does to reduce duplicates
      1. Preventing sales reps from creating duplicates
      2. Showing potential duplicate records (de-dupe on the fly)
      3. Running duplicate jobs
2. Step One: De-dupe existing duplicated Salesforce records
   1. Finding duplicates in Salesforce
   2. Start merging one-by-one
   3. Using merge rules to save time
   4. Set your primary records
   5. When should I bulk merge or turn on auto merge?
   6. Audit your merges
   7. “Help! I’m still seeing duplicates”
   8. Best practices to prevent merge disasters
   9. Initial de-dupe start to finish
3. Step Two: Preventing Duplicate record entry
   1. Review the common duplicate sources
   2. Educating your team on data etiquette
4. Step Three: Automate
   1. Use auto merge to pick up daily duplicates
5. Budgeting your Salesforce De-dupe
   1. How de-dupes are priced
   2. Realistic time frames for initial de-dupe
   3. Who should be involved in the de-dupe?
   4. How do I calculate the ROI of de-duping Salesforce?
6. In-house vs. AppExchange Solutions
   1. API solution built by an in-house developer
   2. Salesforce Native Duplicate Management
   3. “Heck, it’s the sales team’s job!”
   4. Why AppExchange solutions are a no-brainer
   5. What to look for in a de-dupe provider

Salesforce Duplicates 101

How duplicates enter Salesforce

Duplicates enter Salesforce four ways:

• Imported data.
• Third-party integrations that sync or generate new records.
• Web forms with insufficient checks.
• In-house API usage that needs tweaking to make sure it’s not creating dupes.

Why are Salesforce duplicates costly and who should worry?

Everyone who touches the customer or the brand.

These roles feel it first:

• Sales reps.
• Sales managers.
• Sales and Marketing VPs and CMOs.

Why? Data duplication issues costs businesses in various ways:

• Duplicate leads means salespeople end up contacting the same person.
• Call records and notes go missing.
• Employees lose up to 50% of their time to routine data cleanup (MITSloan).
• Prevention costs $1, correction costs $10 and leaving duplicates cost up to $100 (SiriusDecisions).
• Bad data affects lead generation, marketing, finance and customer relationships.

A study on lead data health shows that 15% of company sales leads are duplicated. Another study shows that companies can recover up to 70% of their revenue just on clean data alone.

Duplicate records reach customers before anyone else notices them. That makes this everyone's problem, not the admin's.

Should I worry about duplicates today?

The best answer is: no, yesterday! The second best answer is, right now.

You might say, “our next newsletter isn’t until next month, we can start fixing duplicates then…”

Waiting to the last minute to take care of your duplicates is the perfect recipe for disaster. I’ve seen customers come to us time and time again needing 300,000+ records de-duped right now. This can be very dangerous. You’re risking your data and rushing a process that takes time and tweaking to get right.

Don’t wait until your next deadline to get serious about fixing your duplicate issues. Start a few weeks ahead of time, at least. This will give you a decent head start.

What Salesforce does to reduce duplicates

Salesforce gives you three native tools for duplicates.

Preventing sales reps from creating duplicates

Duplicate Rules alert a rep, or block them, when they are about to save a record that already exists.

You can learn more about how to create duplicate rules at the Stop Users from Creating Duplicate Records page.

Showing potential duplicate records (de-dupe on the fly)

Salesforce shows Potential Duplicates on the record itself, so your team sees them while they work.

To learn how to set this up go to the Show Duplicate Records in Lightening Experience page.

Running duplicate jobs

Duplicate Jobs scan the records already in your org and collect what they find into duplicate record sets.

That gives you the list. Merging it is still manual.

Duplicate jobs can be learned about in greater detail here.

Native duplicate management in Salesforce is quite extensive and all areas are covered in this document.

Step One: De-dupe existing duplicated Salesforce records

The first cleanup is where you learn which duplicates you actually have. Everything you work out here is what you automate later.

Sign up for Dedupe.ly

Go to the Salesforce integration page and sign up for Dedupely. Once there, you’ll be asked to sign in using your Salesforce account. After that, you’ll be asked which objects you want to sync. Select the objects to sync and click “Download” and you’ll be on your way.

Finding duplicates

Start with the fields you want to match on. First and last name is enough to begin; add fields later to narrow what the search returns.

When the search finishes, read down the list of matches and look for anything that is not a duplicate.

A wrong match tells you how far to trust the Match Options behind it. Tighten them before you merge anything in bulk.

Start merging one-by-one

Merging one at a time at the start shows you which fields actually differ between duplicates, and which ones matter.

Manual Review shows every match before it merges, and lets you set which record is the Primary and which value wins each field.

Using merge rules to save time

Merge Rules make that choice for you. Set one per field: keep the newest value, take a value over a blank, or pin the field to the Primary.

That is what makes bulk merging and Auto Merge safe to run.

Set your primary records

The Primary is the record that survives a merge. The Secondaries are absorbed into it, and Merge Rules decide field by field which of their values come across.

When should I bulk merge or turn on auto merge?

This depends entirely on how confident you are in your matches. One wrong match means there are others you have not seen. Tighten the Match Options before you merge in bulk.

Build Match Options that leave almost no room for error. First name, last name and email together is safe: two people rarely share all three.

Once the matches hold up, merge in bulk and turn on Auto Merge.

Audit your merges

Merge History lists every merge and what it kept. Refresh the record in Salesforce to see the result there.

“Help! I’m still seeing duplicates”

Normal. No first pass catches every duplicate. Start by widening the Match Options.

Dedupely sets text fields to similar matching by default. Each matcher finds a different shape of duplicate.

Change it and new matches appear:

• Exact match is the strictest match type. It ignores uppercase and lowercase but must match the text exactly.
• Similar match is also fairly strict but ignores punctuation marks among other commonalities that deem removable.
• Match first similar word/match last similar word both work to match the first or last words. This can work well for company names or first names that have the middle name added.
• Fuzzy match is the most aggressive match type and works roughly on sound-alike matches. However, this is the most inaccurate match type and should never be relied on to produce correct matches.

Then look at the prefixes and suffixes in your data. Matching improves once you start ignoring common terms. By adding ignored terms you reduce the extra noise preventing proper matching.

Best practices to prevent merge disasters

Disasters happen, and they are all preventable. These are the habits that stop one:

1. Always have backups of your data handy. Create a special backup before large bulk merges.
2. Always test and audit your matching setup. It’s easy to make a mistake by rushing. Dedupely does everything possible to prevent users from making simple mistakes. However, always closely review before bulk merges and audit the changes after the merge.
3. Be aware of how and when your data evolves. Adapt your match setup and merge rules accordingly to avoid collisions with changes in your Salesforce data.
4. Be in-tune with errors and nuances in your data. Having defaults like “000-0000” in a phone number which would match all records with blank phone numbers. To a computer non-blank defaults don’t look blank. Dedupely does try to make sure each input passes validation of what it’s suppose to look like.
5. Never automate merges that you haven’t tested and reviewed. Never run bulk merges until you’ve looked over a good portion of the duplicates and are confident of the results.
6. Take attribution into account. Decide how lead and contact owners are preserved through merge rules. Sam’s lead might become Jack’s lead because Jack has a duplicate of Sam’s lead. Who wins the ownership in this case?

Initial de-dupe start to finish

The initial de-dupe can take anywhere from a day to two weeks. It really depends on the shape of your data and how much time is dedicated to the task to completion.

The initial cleanup will give you a close look at your duplicate data and how fields should be preserved.

You can learn more about how to use Dedupely here in our support center.

Step Two: Preventing duplicate record entry

Prevention costs the least of the three steps. Salesforce gives you several ways to do it.

Dedupely does not prevent duplicate entry in Salesforce. Salesforce's own Duplicate Rules are the tool for that, and the AppExchange lists others.

Review the common duplicate sources

Knowing where duplicates come from is what stops the next batch.

If your developers write against the Salesforce APIs, ask them whether their code creates records without checking first.

Check every third-party app that creates records: web-to-lead forms, and any integration that syncs across platforms without a duplicate check.

Educating your team on data etiquette

Anyone who enters or edits records in Salesforce should know:

• How a field should be formatted
• How to check for a record before creating one
• Which tool to use to find or merge a duplicate
• How merging affects sales attribution

A team that understands this stops most duplicates before they exist.

Step Three: Automate

Step one is done and step two is in place. Now automate.

Prevention is the best way to solving most your duplicate problems (and maybe all problems). However dupes are inevitable and automation will pay dividends over the days, weeks, months and years your team is interacting with Salesforce. They will thank you as well!

Use auto merge to pick up daily duplicates

Auto Merge clears the obvious duplicates so nobody has to look at them.

You still review week to week. Dedupely flags what the non-automated searches catch, and you decide on those.

Between your team and a few preventative measures, you stay ahead of it.

Budgeting your Salesforce De-dupe

How de-dupes are priced

Record count is the largest factor in what any vendor charges.

Ask each one how they count records, whether the price is annual or one-off, and what happens when your database grows.

Dedupely bills on records synced and charges no per-user fee.

Costs are also influenced by:

• the level of customization of your Salesforce instance
• consulting bases instead of self-serve
• the amount of time required to complete the de-dupe

Vendors price differently enough that the only comparison worth making is a quote against your own record count.

Realistic time frames for initial de-dupe

Time scales with record count the same way pricing does.

A small org takes an hour or so. A large one takes days, because there is more data, more care needed, and more shapes of duplicate to catch.

Start weeks before a deadline, not hours. What decides how long it takes:

• How fast whoever you contract works.
• The testing and care it takes to avoid losing data.
• Errors already in the data that have to be fixed first.
• Time to sync and move records around.

The expensive ones are the surprises that land on the day of a launch. Be safe, do it ahead of time.

Who should be involved in the de-dupe?

Whoever owns the data, the sales managers, and anyone whose reporting changes when records merge.

How do I calculate the ROI of de-duping Salesforce?

Plenty of studies show what duplicate data costs. None of them justify a line in your budget.

The simplest measure is the hours your team loses finding and merging duplicates by hand. You can measure the amount of time it takes to find one group of duplicates and merge them by hand. Then ask your reps how many times a day/week they have to fix duplicates. Multiply those together and you have an hour figure over a certain period of time. How much are those hours costing your company instead of benefiting it?

The calculation could look like this:

((Number of duplicates merged by reps daily) * (minutes to merge a duplicate / 60) = (lost hours)) * (hourly base pay)

Time one person merging one group of duplicates. Multiply by the groups you have, then by the number of people doing it.

(20 * 4 minutes = 80 minutes) * 30.00/hr = $39 per day

Put your own hourly cost against that total and you have the monthly price of leaving the duplicates where they are .

Add lost opportunities if you want to. If the number still looks small, you do not have a duplicate problem. If you do it should be pretty easy to convince your team and peers to get on board with you in your de-dupe efforts.

This is an estimate, not an audit. It is enough to decide whether the spend is worth making.

In-house vs. AppExchange Solutions

Everyone has been down the DIY rabbit hole and ended up paying someone else anyway.

Then there are scenarios where in-house just makes so much sense.

What are the options we have for effective de-duping?

API solution built by an in-house developer

You have developers. Why not put them on a solution that fits every requirement exactly? Not so fast!

Some developers love writing their own apps. Believe me, at one point I would have designed our entire software stack–and nearly did–before I was reminded of the true costs of in-house.

Your developers are there to build what you sell. The first version takes 1) time of initial development 2) time of bug fixes and specs back and forth 3) maintenance and updates over the years 4) costs of hosting and running.

Then it needs maintaining every time the CRM's API changes, and it is months before it runs unattended.

Salesforce Native Duplicate Management

Salesforce has more native duplicate management than most CRMs.

Where it stops short:

• Matching Rules compare on exact or fuzzy, and nothing else.
• Active rules are capped per object.
• Duplicate Jobs need a higher Salesforce edition.
• Nothing merges automatically, and nothing merges in bulk.
• A merge takes three records at a time.
• It adds up to a lot of clicking.

The Salesforce native duplicate management, while a cut above other CRMs, still misses the mark in terms of time saving duplicate management. While it does the job it still doesn’t completely solve the problem in a significant way.

“Heck, it’s the sales team’s job!”

When was the last time “data cleansing and cleaning up duplicates” was in the job description of a sales rep? Do any of your sales reps think their job is cleaning up bad data?

Training personnel to clean up after themselves is pretty basic. Everyone should actively clean up the data as they enter it into Salesforce. It’s just proper etiquette not to mention respect for fellow team members.

However if you’re not aiding sales in gathering, handling and cleaning data you’re making their lives harder, which in turn make sales harder and mean less sales for your company. By “aiding” I mean providing them with the proper tools to do their job.

Why AppExchange solutions are a no-brainer

There are dozens of de-dupe apps on the Salesforce AppExchange. Some of them are not so great and others are pretty awesome. These apps are built by companies that dedicate large amounts of resources on building solutions to save you insane amounts of time and money. They are more affordable and give a higher return than the above alternatives.

You skip the DIY detour, the internal project that pulls your team off customers, and native features that stop short of merging.

What to look for in a de-dupe provider

Most listings on the AppExchange handle the basics. What separates them is what happens during the merge itself.

With a consultancy, make sure they understand your fields. Take nothing for granted: they will not know what X and Y are for, or that dropping one costs you a report.

Self-serve puts your team in control of the whole process. Expect a learning curve and some documentation.

Whatever you choose should do all of this:

• Match on any field, exact or similar, with common terms ignored.
• Allow you to merge in bulk, customize merges and automate merges once you’ve done the homework and are confident in your settings.
• Merge Rules that cover Primary selection, attribution and your custom fields.
• Tell you about new duplicates as they appear in Salesforce.
• Answer support quickly.

If self-serve does not fit, our team sets it up with you at no extra cost. Take a closer look at the Salesforce Dedupely connector here.