Deduplicating a Large Database

A large database, 480,000 records and up, needs the same steps as any deduplication project, done in a set order: sync only the objects you need, plan around the first sync, test before merging, merge in stages and automate last. Dedupely handles databases of millions of records, with plans up to 20 million and bulk merges of thousands of matches at once. This Playbook shows how to run it so every stage is checked before the next one starts.

Step 1: Sync only the objects you need

A whole object syncs, so every record in an object you turn on counts toward your record limit. Before you connect, decide which objects you'll dedupe first.

• Turn on only those objects in CRM Settings. You can turn others on later.
• Start with the object that causes the most problems, usually Contacts or Companies.
• Sandbox records count too, like any other synced record.

Up to 10% over your record limit, everything keeps working for 10 days while you upgrade or turn objects off. More than 10% over, or still over after 10 days, your account becomes read-only: syncing and scans keep running, but merging stops. Dedupely never upgrades your plan or charges you for going over. See Billing & Usage.

Step 2: Plan around the first sync

The first sync of a large database takes hours, not minutes:

After the first sync, Dedupely refreshes every 10–30 minutes and finds new records within 1–5 minutes. Syncing never merges, edits or deletes anything in your CRM, so your team can keep working while it runs. See Syncing Records.

Step 3: Test before you merge

Treat merges as permanent: at this size a wrong rule repeats thousands of times.

• Salesforce: connect a Sandbox and run your searches and Merge Rules there first, to see how merges affect your Flows, Apex triggers and integrations. See the Salesforce Deduplication Guide.
• Every CRM: run 1–2 Single Merges, check them in your CRM, then bulk merge 1–2 pages and check again.
• Check the Merge Preview in View Match Details before each new search, so you know which values survive.

Step 4: Merge in stages

Don't merge the whole database at once. Merge one search at a time, from the safest to the riskiest:

1. Exact duplicates first, on at least three fields (for example First Name, Last Name and Email on Exact).
2. Companies and accounts, with Company Name on Similar Words (Any Order) and Website on Domain Root.
3. Looser searches last, like name-based searches or Dedupely Intelligence, reviewed match by match.

Bulk Merge can merge a whole page or Select all X matches at once, with no maximum. Cancel all stops a running bulk merge; merges already in progress finish. Matches with more than 20 records and placeholder data go to the Excluded tab, so they're never merged in bulk. See Merge Safety.

How to merge thousands of HubSpot companies

Duplicate companies are often the biggest job in a large HubSpot portal. To merge them at scale:

• Match on name and domain: Company Name on Similar Words (Any Order), with Ignored terms Inc., LLC, Ltd., plus Website on Domain Root. Add Country and City to Visible Fields to spot branches.
• Choose your Merge choices first. HubSpot limits how many associations one record can have and how many times a record can be merged. Set "Too many associations to move" and "Too many past record merges in same record" before you bulk merge. Auto Merge can't ask, so it skips matches set to Always ask.
• Expect HubSpot's own behavior: HubSpot usually gives the merged company a new Record ID (old IDs redirect), and both records leave active workflows.

See How to Dedupe Companies and Accounts.

Step 5: Automate last

Once the backlog is merged, keep new duplicates from piling up again:

1. Turn on Auto Match on each search you've tested, so someone reviews new matches on a schedule. See Auto Match.
2. Move a search to Auto Merge once it has run without false matches, with the As soon as possible schedule. See Auto Merge.

Example setup

A HubSpot portal with 520,000 contacts, 210,000 companies and 40,000 deals, about 770,000 records, on the Enterprise plan (1 million records).

1. Day 1: Contacts and Companies turned on. Deals stay off until contacts and companies are done. First sync: about 3–10 hours.
2. Day 2: Merge Rules set: Primary Record set to Last Updated Record; Lose Rules on Contact owner for former reps. Merge choices set to "Merge and keep the associations that fit" and "Merge by copying the data over".
3. Day 2: 2 Single Merges and 1 bulk-merged page on the exact contact search (First Name, Last Name, Email on Exact), checked in HubSpot.
4. Week 1: exact contact search bulk merged with Select all X matches. Then companies on Company Name (Similar Words, Ignored terms) and Website (Domain Root), page by page for the first 10 pages.
5. Week 2: name-based contact searches and Dedupely Intelligence reviewed; unclear matches sent to Manual Review.
6. Week 3: Deals turned on and deduped the same way. Auto Match on every search; Auto Merge (As soon as possible) on the exact searches.

Common mistakes

• Syncing every object at once. It fills your record limit and makes the first sync longer. Start with one or two objects.
• Merging during the first sync. Wait until the sync finishes, so your searches see every record.
• Testing on one record and merging everything. Test 1–2 Single Merges, then 1–2 pages, for every new search.
• Skipping Merge Rules. At this size, the default winner repeats on thousands of records. Set the Primary Record and the fields that matter first.
• Turning on Auto Merge before the backlog is done. Automate searches that have already run without false matches.

FAQs about large databases

• What counts as a large database? 480,000 records or more. The steps work for any size, but the order matters more as the database grows.
• Is there a limit on how many matches I can merge at once? No. Bulk Merge can merge every match in a search with Select all X matches.
• Can I test in a sandbox? Yes, in Salesforce. Sandbox records count toward your record limit.
• Does a large first sync change anything in my CRM? No. Syncing never merges, edits or deletes records.
• What happens if I go over my record limit? Up to 10% over, you have 10 days to upgrade or turn objects off. After that, or if you're more than 10% over, your account is read-only until you do.