TL;DR
- A large CRM database takes longer to sync and can't be reviewed set by set.
- CRM API limits can slow the first sync of a database with millions of records.
- Very large duplicate sets usually come from placeholder values or shared inboxes, so hold them back.
- Merge a large database in stages, starting with exact matches on email or domain.
- Dedupely syncs databases of millions of records and bulk merges thousands of matches in minutes.
A CRM with millions of records has the same kinds of duplicates as a small one. What changes is everything around them. Loading the data takes longer, searches return more sets and more teams feel each merge.
At that size, nobody can look at every match. The cleanup has to rest on criteria you trust, safeguards that catch what the criteria miss and a record of every change.
In this post, you'll see what changes when a database gets large and how to plan the first sync. You'll also see how to handle oversized matches, merge in stages and keep a trail that holds up to questions later.
What changes when a CRM database gets large
Size on its own doesn't make a cleanup hard. Our post on complex CRM deduplication at scale explains why clear criteria matter more than record counts. Size does change how you plan the work.
| What changes | Why it matters |
|---|---|
| Loading the data takes longer | The first sync of a large database can take hours, so plan it before the cleanup starts |
| Searches return thousands of sets | You decide on criteria and samples instead of each set |
| Bad values repeat at scale | One placeholder phone number can tie hundreds of records together |
| More systems feed the CRM | New duplicates keep arriving from forms, imports and integrations while you clean |
| More people rely on each record | A wrong rule affects many teams at once, so changes need an owner and a record |
Start with the objects you're cleaning now
A large CRM rarely needs every object cleaned at once. Pick the objects where duplicates cause the most trouble today, often contacts and companies, and start there.
Count the records in each one before you begin. Many tools that connect to a CRM sync whole objects, and some price by the number of records you sync. Turning on only the objects you're working on keeps the first sync shorter and the count lower.
Work through one object at a time, and finish its stages before you turn on the next one.
Plan around the first sync
Before any tool can find duplicates, it has to read your records. For a database with millions of them, that first sync can take hours.
Your CRM's API limits can make the first sync take longer, so give it room in your plan.
- Start the first sync when you don't need results the same day.
- Avoid running a large import or migration at the same time.
- Tell your CRM admin, so a slower integration doesn't look like an outage.
Hold back oversized matches
In a large database, a single bad value can create a set with hundreds of records. These sets are rarely one customer. They usually share a placeholder, a test value or a shared inbox.
- Placeholder phone numbers like 0000000000 or 1234567890.
- Example data like test@test.com or "John Doe".
- Shared inboxes like info@ or sales@ that many people used on forms.
- Company domains that belong to a large group with many separate offices.
Hold back any set above a size you set, such as 20 records, and look at it separately. Our post on placeholder values that make records look like duplicates covers how to find them.
Merge in stages
A large cleanup done in one pass leaves no room to learn. That's why almost every team with a large database merges in stages, checking each round before the next one runs.
| Stage | What merges | What to check |
|---|---|---|
| 1. Test | One set, then one page of sets from your strictest search | Every merged record, opened in the CRM |
| 2. Exact matches | Exact email or domain matches, in bulk | A random sample of merged records |
| 3. Combined fields | Looser matches confirmed by a second field, like name and company | A larger sample, including edge cases |
| 4. Review | Sets that need a person, including the oversized ones | Each set, by someone who knows the records |
Leave time between stages for the people who use the CRM to notice problems. Our post on what to check after a large merge lists what to look at after each round.
Keep a trail of every change
With thousands of merges, someone will ask about one of them. A sales rep can't find an account, or finance sees a customer count drop.
Two kinds of records answer those questions. A merge history shows what each merge changed, with the original values. An activity log shows who ran each merge, changed a search or turned on an automation, and when.
Decide where both live and who can see them before the first bulk merge. Our post on why you need a record of every merge covers what a useful record shows.
Keep new duplicates from growing back
A large CRM keeps taking in records during the cleanup. If the sources that created the duplicates stay the same, the count grows back.
Once your rules are tested, you can merge new duplicates you're certain of, like exact email matches, automatically on a schedule. Send the rest to review regularly, so the backlog doesn't build up again.
How Dedupely handles large CRM databases
A Dedupely customer has synced over 18 million records, and bulk merging a few thousand matches takes minutes. The first sync takes 3–10 hours for 600,000 to 1 million records, and longer for the largest databases.
By default, Dedupely excludes risky matches, like sets of more than 20 records or placeholder data. They wait in a separate tab for you to merge, reject or review.
Every merge is recorded with its original values and can be exported as a CSV, and an audit log shows who did what. You choose which objects to sync, so you can start with the ones you're cleaning.
Plan for size, decide on criteria
A large CRM database doesn't need a different idea of what a duplicate is. It needs a plan that accounts for its size: a first sync scheduled ahead, oversized sets held back and merges run in stages.
Above all, it needs criteria you trust and a record of every change. With those in place, millions of records become a series of stages you can check, one at a time.
Frequently asked questions
How do you deduplicate a CRM with millions of records?
Start with one object, like contacts, and plan the first sync ahead, since it can take hours. Hold back oversized matches built on placeholder values. Then merge in stages, from exact email or domain matches to looser ones, checking a sample of merged records after each stage.
Why does syncing a large CRM take so long?
Every record has to be read through the CRM's API, and your CRM's API limits can make the first sync take longer. For a database with millions of records, a first sync of several hours is normal. Schedule it ahead, and avoid running a large import or migration at the same time.
Can you bulk merge thousands of duplicates at once?
Yes, many deduplication tools can merge thousands of duplicate sets in one action, so the risk sits in the criteria you merge on. Test the search on one set and one page, and check the results in your CRM. Then bulk merge the rest of that search with the same criteria.
Why does one duplicate set contain hundreds of records?
Usually because the records share a value that isn't real, like a placeholder phone number, test@test.com or a shared inbox like info@. These sets are rarely one customer. Hold them back from bulk merges and review them separately, or leave that value out of the match.
How do you keep track of thousands of merges?
Keep a merge history that shows the original records and the values that won, ideally exportable. Pair it with an activity log of who ran each merge and changed each search. Decide who can see both before the first bulk merge, since questions usually come weeks later.
Or sign up with Google
HubSpot Marketplace 
