Deduplication Scripts
Deduplication scripts are custom code written to find duplicate records in a CRM and merge them. A script reads records through the CRM's API or from an export, compares them with matching logic written by the team, and then merges the records it pairs, or produces a list for a person to merge.
The term covers a range of DIY deduplication: a one-off script run against an export before an import, a scheduled job that checks new contacts each night, and a workflow automation that looks for an existing record before a new one is created. What they share is that the team owns every rule, from which fields count as a match to which value survives the merge, and also owns every result. That makes scripts flexible, and it makes their safety depend on how well the code is written, checked and kept up to date.
See how Dedupely finds and merges duplicates without code in your CRM
What Are Deduplication Scripts?
Deduplication scripts are code that does the two jobs of deduplication, matching and merging, with logic the team writes itself. A short example: a script compares contacts on email address and finds "J. Smith, jsmith@acme.com" and "John Smith, JSmith@acme.com." It lowercases both addresses, pairs the records, keeps the older record and fills its blank fields from the newer one. Each of those choices is a line of code that someone decided on.
Scripts take a few common forms. A standalone script runs against a CRM export or through the CRM's API, which gives full control over the logic and suits a one-time cleanup. A scheduled job runs the same logic on a timer, which keeps new duplicates in check without anyone starting it. A workflow automation runs a check when a record is created or updated, which catches some duplicates close to the moment they appear. All three can be built for exactly the data a team has.
Scripts can work on the objects a CRM's API makes available: contacts and companies in HubSpot, leads and accounts in Salesforce, people and organizations in Pipedrive, and leads in Close. What a script can do on each object depends on what the CRM's API allows, and that varies from one CRM to the next.
How Deduplication Scripts Work
Most deduplication scripts follow the same stages, whatever they're written in:
- Pull the records. The script reads records from an export or page by page through the CRM's API. CRM APIs often limit how many requests a script can make in a set time, so large databases take longer and need the script to pause and retry.
- Clean the values. Before comparing, the script evens out differences that don't change the meaning, such as capitalization, spaces, phone number formatting or legal suffixes on company names.
- Compare and group. The script compares records on the fields it's written to check. Exact comparisons are simple to code; looser ones, such as catching typos or names that sound alike, take more work to get right without pairing different people.
- Choose what survives. For each group, the code picks the record to keep and the value to keep in each field. Unless it's written to skip them, a blank value can overwrite a real one.
- Merge. The script calls the CRM's merge, if it has one, or rebuilds the merge itself by updating one record, moving related activity and deleting the others. Merge support varies: Close's API, for example, has no contact merge.
- Record the result. A well-built script writes a log of each merge, with the original records and their values, since many CRM merges are hard or impossible to reverse.
Each stage is a place where the logic can be adjusted, and also a place where a bug can change data across many records in one run.
Why Deduplication Scripts Matter for CRM Teams
Scripts give a team full control over how duplicates are found and merged. They can follow rules no built-in tool offers, handle an unusual object or field, and fit into processes the team already runs. For a one-time cleanup with a clear rule, such as removing exact email duplicates from an export before an import, a short script can be a quick fix.
The costs show up over time. A script depends on the CRM's API, fields and limits, so a renamed field, a new required property or an API change can break it or, worse, change what it does without an error. It also depends on the person who wrote it, and when that person moves on, the logic can become hard to read or change.
The bigger risk is to the data itself. A script often merges without a preview, so a wrong rule shows up only after records are merged. A loose comparison or a placeholder value such as test@test.com can group unrelated contacts, and a merge step that doesn't carry over related activity can leave notes, deals or tasks behind. Because many CRM merges can't be reversed, recovering from a bad run often means rebuilding records by hand from logs or backups, if they exist. Reports, contact counts and billing based on those records are off until the repair is done.
Getting Started With Deduplication Scripts
Write down the rules before writing any code: which fields decide a match on each object, which record survives, and which value wins in each field, including what happens to blank and placeholder values. Rules on paper are easier to review with the people who use the data, and they make the code easier to check later.
Run every change as a dry run first. Have the script list the groups it would merge and the result it would produce, without merging anything, and check a sample by hand. Look hardest at the largest groups and at matches built on one weak field. When the results hold up, merge a small batch, check those records in the CRM, and only then run the full set.
Log each merge with the original records, their values and the record that survived, and keep the log somewhere the team can find it. Plan for upkeep too: someone needs to own the script, update it when the CRM changes and watch each run.
In Dedupely, Duplicate Searches and Merge Rules set the matching and merge logic without code, and a preview shows the result of each merge before anything changes in the CRM. Every merge is then recorded in Merge History, with each original record and its values, ready to download as a CSV.
Key Takeaways: Deduplication Scripts
Deduplication scripts are custom code that matches and merges CRM records with logic a team writes and maintains itself. They're flexible and can suit a one-time cleanup, but they depend on the CRM's API and limits, often merge without a preview, and leave the team to log and repair every merge. Dry runs, small batches and a log of each merge make them safer. Dedupely carries out the same steps without code: Duplicate Searches set the matching, Merge Rules decide which values win, with a preview before each merge, and Merge History keeps each original record and its values. Safety Settings hold risky matches out of Bulk Merge and Auto Merge, and Auto Match and Auto Merge keep a trusted search running on a schedule.
Frequently Asked Questions About Deduplication Scripts
How do you deduplicate CRM records with a script?
You pull the records, clean the values, compare them on chosen fields, decide what survives and then merge through the CRM. The script reads records from an export or through the CRM's API, evens out capitalization and formatting, groups records that match, and picks the surviving record and the winning value in each field. The merge step depends on the CRM: some offer a merge call, while others, such as Close for contacts, don't, so the script has to rebuild the merge itself. Writing the matching takes the most care, because looser comparisons find more duplicates and also more false matches. In Dedupely, the same steps run without code. A Duplicate Search sets the fields and gives each one a Match Option, such as Exact, Similar or Fuzzy, and Merge Rules decide which record and values win. Changing a search never changes CRM data, because merges happen only when you say so.
Why are DIY merge scripts risky for CRM data?
Because a mistake in the code can merge the wrong records across a whole database before anyone sees the result. A script often merges without a preview, so a loose comparison, a placeholder value like test@test.com or a shared inbox can group unrelated people into one record. Unless the code handles it, a blank value can overwrite a real one, and related notes, deals or tasks can be left behind. Many CRM merges can't be reversed, so a bad run means rebuilding records by hand. Scripts also break when the CRM changes a field or an API limit, and they depend on whoever wrote them. In Dedupely, merges can't be undone either, so the checks come first: a preview shows each merge's result, blank values never win, and all activities and associations are preserved. Safety Settings hold matches with more than 20 records (the default) or built on poor or example data out of Bulk Merge and Auto Merge.
When do deduplication scripts make sense instead of a dedicated tool?
Scripts make sense when the job is small, one-time and simple, and someone on the team can write, check and own the code. Removing exact email duplicates from an export before an import is a good example, as is a rule so specific to one business that no tool offers it. A dedicated tool fits better when deduplication is ongoing, when the matching needs to catch typos, nicknames or formatting differences, when people who don't code need to see and change the rules, or when merges need a preview and a record of what changed. Upkeep often decides it: a script needs someone to update it whenever the CRM changes. In Dedupely, matching and merging are set up without code in Duplicate Searches and Merge Rules, and Auto Match and Auto Merge keep a search running on a schedule. Auto Match notifies you when new matches reach a count you set, and Auto Merge can run as soon as possible.
How do you test a merge script safely before it runs on live CRM data?
Run it as a dry run first, then merge a small batch and check the records before running the full set. In a dry run, the script lists the groups it would merge and the result for each one, without changing anything. Check a sample by hand, starting with the largest groups and the ones matched on a single field. Some CRMs offer a sandbox, which lets a script merge real-looking records without touching live ones. Back up the records before the first live run, and stop the run if the results look wrong. In Dedupely, you can clone a Duplicate Search to try new criteria without changing the active one, then click "Scan" to see the matches. Two previews show the result before a merge: winning values in bold in the results, and the Preview Merge panel. A Custom merge, where you pick each winning value by hand, is a good way to check Merge Rules on edge cases.
Which details should you log when a script merges records?
Log the original records with every field value, the record that survived, the value that won in each field, the date and the rule that grouped them. That's the information needed to explain a merge later or rebuild records after a wrong one, since many CRM merges can't be reversed. Log record IDs from before and after the merge, because some CRMs, such as HubSpot, create a new record ID on merge. Note which run and which version of the script made each merge, and who started it, so changes in behavior can be traced. Keep the log somewhere the whole team can find it. In Dedupely, Merge History records every merge made in Dedupely, with the date, each original record and its values, and a link to the record in the CRM, and it downloads as a CSV. Who made each merge is in the Audit Log, which is admin only.
