# Deduplication at Scale – Definition, FAQs & How Dedupely Helps > Deduplication at scale explained: how to deduplicate a large database, in what order to merge, and how to keep oversized duplicate sets out of a merge. - URL: https://dedupe.ly/glossary/deduplication-at-scale - Section: glossary - Site index: https://dedupe.ly/llms.txt - Any page: append `.md` to its URL for this plain-text form --- [Glossary](https://dedupe.ly/glossary) Deduplication at Scale ## Deduplication at Scale Deduplication at scale is the process of finding and merging duplicate records in a large CRM database, such as hundreds of thousands or millions of contacts, companies or accounts. At that size, the work is planned as a series of passes, from the clearest duplicates to the least certain, with checks that hold back the riskiest matches before anything merges. Size changes more than the number of duplicates. Comparing each record with each other record takes far more work as a database grows, a shared value such as a company phone line can tie hundreds of records into one set, and loading the data into a tool can take hours instead of minutes. Merge tools built for a few records at a time stop being practical. Large-scale deduplication answers this with a fixed order of work, many duplicate sets per action and safeguards that keep one weak rule from repeating across thousands of merges. See how Dedupely deduplicates large databases in your CRM [Try for free](https://v2.app.dedupe.ly/signup) [Talk to sales](https://meetings.dedupe.ly/meetings/gaby-salinas/talk-to-sales) Overview Deduplication at scale is finding and merging duplicates in a CRM database too large to clean one record at a time. It relies on a set order of work, matching that stays fast on large data, checks for oversized duplicate sets and merging many sets in one action. On this page - What Is Deduplication at Scale? - How Deduplication at Scale Works - Why Deduplication at Scale Matters for CRM Teams - Getting Started With Deduplication at Scale - Key Takeaways: Deduplication at Scale - Frequently Asked Questions About Deduplication at Scale - Related Terms and Concepts See how Dedupely deduplicates large databases in your CRM [Try for free](https://v2.app.dedupe.ly/signup) [Talk to sales](https://meetings.dedupe.ly/meetings/gaby-salinas/talk-to-sales) ### What Is Deduplication at Scale? Deduplication at scale, also called large-scale or bulk deduplication, is CRM deduplication applied to a database where size shapes every decision. The steps are the same as in a small CRM: match the records, decide what each merged record keeps and merge. What changes is how many records each step touches, how long it takes and how far a single mistake can spread. A short example shows both sides. A CRM holds 400,000 contacts. A search on exact email finds 12,000 duplicate sets, most of them two records, such as "J. Smith" and "Jon Smith," both at jon@acme.com. One set holds 900 records that share info@acme.com. Merging those sets one by one would mean 12,000 separate merges, and merging the 900-record set would turn 900 people into one contact. The first problem calls for merging many sets per action, the second for a check that holds the set back. It applies to every large object: contacts and companies in HubSpot, accounts and leads in Salesforce, people and organizations in Pipedrive, leads in Close, and contacts in Nimble and Google Contacts. Teams take a few approaches. A single cleanup project clears the backlog in one planned push, which gives a clean starting point quickly. Phased passes work object by object and search by search, which keeps each batch small enough to check. Continuous runs keep searches going on a schedule, which stops a new backlog from forming. Many teams combine them: phased passes first, then scheduled runs. Resources: - [How to Approach CRM Deduplication](https://dedupe.ly/knowledge-base/how-to-approach-crm-deduplication) (KB) - [What is a duplicate?](https://dedupe.ly/blog/what-is-a-duplicate) ### How Deduplication at Scale Works Deduplication at scale follows the same matching and merging stages as any cleanup, with a few steps added to handle the size: - **Load the data.** A tool has to read the records before it can compare them. For a large database, the first sync or export can take hours, and later updates often bring in only the records that changed. - **Keep the comparisons manageable.** The number of possible pairs grows much faster than the number of records: doubling the records roughly quadruples the pairs. Tools often narrow the work, for example by comparing only records that share a key such as an email domain. - **Order the passes.** The strictest searches run first, on strong fields such as email or a customer ID. Looser matching comes later, on the smaller set of records that's left. - **Hold back oversized sets.** A set with dozens or hundreds of records often points to a shared inbox, a company phone line or a placeholder value. Those sets are set aside for a person to check. - **Merge in large batches.** Many sets are merged per action with the same rules for the surviving record and each field. A large batch often works through its sets one after another, so it takes time to finish. - **Check and keep going.** A sample of merged records is checked in the CRM, and the searches that proved reliable keep running as new records arrive. Resources: - [Finding Duplicates](https://dedupe.ly/knowledge-base/find-duplicates) (KB) - [Syncing Records](https://dedupe.ly/knowledge-base/syncing-records-with-dedupely) (KB) ### Why Deduplication at Scale Matters for CRM Teams In a large database, duplicates distort numbers that many people depend on. When thousands of customers exist as two or three records, contact counts, segment sizes and pipeline totals are all overstated, and attribution splits across records. Reports built on that data feed forecasts, territory plans and budgets, so the error travels well beyond the CRM. Day to day, the same customer can get one email twice or be worked by two reps at once. The cost of fixing duplicates by hand grows with the database. Clearing a few dozen sets is an afternoon. Clearing tens of thousands, one merge at a time, isn't something a team can finish, so the backlog stays and keeps growing as forms, imports and integrations add more records. In tools billed by contact count, each of those extra records is also paid for. Scale raises the stakes of each rule. A merge rule that keeps the wrong email, or a search that groups different people, repeats its error in every set it touches, and a large batch can produce thousands of wrong merges in one run. Many merges are hard or impossible to reverse. That's why order, held-back sets and a small first batch matter as much as speed when the database is large. Resources: - [Why your CRM database is turning into a mess (and how to fix it)](https://dedupe.ly/blog/why-your-crm-database-is-turning-into-a-mess) ### Getting Started With Deduplication at Scale Start by sizing the job. Note how many records each object holds, which objects matter most for the work at hand, and how long the first load into a tool will take. Plan to clean one object at a time, since companies and contacts need different matching. Then set the order. Run the strictest search on each object first, set the rules for which record survives and which value each field keeps, and merge a small batch. Check those records in the CRM before merging the rest of that search. Move to looser searches only once the clear duplicates are gone, and keep the uncertain sets for a person to review. Look at the largest sets before every big batch. A set with far more records than a real duplicate often comes from a shared address or a placeholder, and it belongs outside the merge. Once the backlog is cleared, put the searches you trust on a schedule, so new duplicates are merged before a second backlog forms. In Dedupely, Bulk Merge carries out these passes at scale: "Select all X matches" merges a Duplicate Search's matches in one action, using the Primary Record and Merge Rules you've set. Safety Settings hold back matches with more than 20 records (the default) or built on poor or example data, so oversized sets stay out of the selection and wait in the Excluded tab. Resources: - [Merging duplicates](https://dedupe.ly/knowledge-base/merge-duplicates) (KB) - [Merge Safety](https://dedupe.ly/knowledge-base/merge-safety) (KB) ### Key Takeaways: Deduplication at Scale Deduplication at scale is finding and merging duplicates in a database too large to clean one record at a time. It works through passes, from the strictest searches to the loosest, merges many sets per action and holds back oversized sets that point to shared or placeholder values. Because one bad rule repeats across thousands of merges, a small first batch and a check in the CRM come before the rest. In Dedupely, the first sync takes from a few minutes to 10–12 hours beyond 1 million records. Bulk Merge's "Select all X matches" merges a search's matches in one action, Safety Settings hold back matches over 20 records by default, Dedupely Intelligence finds complex duplicate patterns on Contacts, Companies and Leads, and Auto Match and Auto Merge keep trusted searches running. Resources: - [Dedupely product overview](https://dedupe.ly/product) ### Frequently Asked Questions About Deduplication at Scale #### How long does it take to deduplicate a large CRM database? It depends on the size of the database, how many passes the cleanup needs and how much of it a person reviews. The time goes into three parts: loading the records into a tool, running the searches, and merging and checking the results. Loading is often the longest wait for a very large database, while review time depends on how many uncertain sets there are. A cleanup that starts with strict searches moves fastest, because those sets need little review and can be merged in large batches. Looser passes take longer, since more of their sets are worth opening. In Dedupely, the first sync takes 2–15 minutes for up to 50,000 records, 15–90 minutes up to 300,000, 1–3 hours up to 600,000, 3–10 hours up to 1 million and 10–12 hours beyond that. After the first sync, Dedupely refreshes every 10–30 minutes and detects new records within 1–5 minutes. #### In what order should you merge duplicates when deduplicating at scale? Merge the clearest duplicates first and the least certain last. Start with searches on strong fields, such as an exact email address or a customer ID, and merge those sets in large batches once a small first batch checks out. Then move to combined fields, such as name + company, and leave matches on a single weak field, such as a name alone, for last, with more review. Work one object at a time, because each object needs its own matching, and check the largest sets before each batch. Once the backlog is cleared, schedule the searches that proved reliable so new duplicates don't form a second one. Dedupely's guide How to Approach CRM Deduplication groups searches into Obvious, Conditional and Complex duplicates and suggests three phases: Clean Legacy Data, Clean Current Data and Keep Future Data Clean. Auto Match and Auto Merge handle that last phase. #### What is the safest way to deduplicate contacts and companies at scale? Run strict searches first, set the merge rules before selecting anything, hold back oversized and placeholder sets, and merge a small batch before the rest. At scale, one weak rule repeats across thousands of sets, so the protection has to come before the merge. Check the largest sets in each search, since a shared inbox or company phone line can tie many unrelated records together, and send the uncertain sets to a person. In Dedupely, merges can't be undone. Safety Settings keep matches with more than 20 records (the default) and matches built on poor or example data out of bulk selection and Auto Merge, and a rule for very short values can be turned on. Auto Merge sends low-confidence matches to Manual Review instead of merging them. Merge History records every original record and its values for each merge and downloads as a CSV, so a mistake can be traced and the values restored by hand. #### Can you deduplicate millions of records in a CRM such as HubSpot or Salesforce? Yes, with a tool that compares records efficiently and merges many duplicate sets per action. Native merge tools often handle a few records at a time, which suits the occasional duplicate a rep notices but not a database of millions. At that size, the work runs in passes, one object at a time, with oversized sets held back and a small batch checked before each large one. The first load also takes longer, so it helps to plan around it. Dedupely syncs CRMs with more than a million records, with a first sync of 10–12 hours beyond 1 million, and Bulk Merge's "Select all X matches" merges a search's matches in one action. In Salesforce, it can merge thousands of records that way. Matching happens within each object, so a Lead isn't merged with a Contact, and a CSV file uploaded to Dedupely can hold up to 1,000,000 rows. #### What is the difference between bulk merge and deduplication at scale? A bulk merge is one action that merges many duplicate sets at once, while deduplication at scale is the whole process of cleaning a large database. That process covers loading the records, choosing and ordering the searches, setting merge rules, holding back risky sets, merging in batches, checking the results and keeping new duplicates in check afterward. Bulk merging is the step that makes the work possible at that size, but on its own it doesn't decide which sets are safe or in what order to merge them. In Dedupely, Bulk Merge merges several sets, a page or "Select all X matches" in a Duplicate Search, and leaves out sets in Manual Review and matches excluded by Safety Settings. Dedupely Intelligence, Dedupely's AI duplicate matching, runs on Contacts, Companies and Leads, and for safety it has no database-wide "Select All." Auto Match and Auto Merge then keep the trusted searches running on a schedule. ### Related Terms and Concepts #### [Bulk merge](https://dedupe.ly/glossary/bulk-merge) Bulk merge is the single action that makes deduplication at scale possible, merging many sets with the same rules. Deduplication at scale decides which sets go into each bulk merge and in what order. #### [Merge safety](https://dedupe.ly/glossary/merge-safety) Merge safety holds risky sets out of bulk and scheduled merging. In a large database, it's what keeps an oversized set built on a shared value from merging hundreds of people into one. #### [Record limit](https://dedupe.ly/glossary/record-limit) A record limit sets how many records a tool or account can work with. In a large database, duplicates take up part of it, and merging them brings the count back toward real customers. #### [Ongoing deduplication](https://dedupe.ly/glossary/ongoing-deduplication) Ongoing deduplication keeps searches running after the first cleanup. It's the phase that keeps a large database from building a new backlog once the old one is cleared. #### [CRM sync](https://dedupe.ly/glossary/crm-sync) CRM sync brings records into a deduplication tool. In a large database, the first sync is one of the longest steps, so it shapes the timeline of the whole cleanup. #### [Native deduplication](https://dedupe.ly/glossary/native-deduplication) Native deduplication is the duplicate management built into a CRM. Its tools are often built for a few records at a time, which is where deduplication at scale needs a different approach. Skip to content