deduplicate customer data

Table of Contents

Background

You can bulk deduplicate people, companies, deals automatically using rules.

Because there could be multiple reasons for the creation of duplicates, you'd probably use several templates to find and deduplicate records, and then group those templates together in a deduplication Recipe for automation.

In some cases though you may need more granular customization for picking duplicate records to include - or exclude - from the deduplication process. Typically you'd use a Filter in Step 1 to segment records, but if the relevant records do not have anything in common to filter by, and there are too many of them to include/exclude by ID, you can use the approach below to customize the deduplication process in bulk.

Similarly, for picking the master, you'd typically apply a set of rules in Step 4. But if you have a pre-defined list of records that you and the team have already decided to pick as master or that you'd like to pick master records individually, and those master selections cannot be applied using rules because they'd be different for each duplicate group, you can use that same approach below to customize the master selection in bulk.

Approach Overview

As part of the customization process you'd run deduplication in preview mode, analyze the CSV report of the duplicates, and "tag" records for exclusion or as masters.

Then you'd import the "tags" into the CRM as custom fields.

Lastly, you'd run deduplication and use those "tag" fields in the filter and master selection rules.

This approach would enable you to customize the deduplication process in bulk, and have complete granular control over deduplication exclusions and master selection.

Step 1: Deduplicate in Preview Mode

You can simulate the deduplication process for analysis purposes without modifying any data in the CRM by running deduplication in Preview Mode.

The preview run will produce a CSV report of the duplicates. The CSV report will have a row per record and include the following columns:

  • Record ID

  • Deeplink to open the record in the CRM.

  • All fields selected in Step 1

  • All fields selected in Step 4

Tip: since every field that is specified in Step 4 Master Selection is also included in the CSV report, you can include additional fields to get more context about the records in the CSV report. For example, add "Industry exists" as the last rule to have the "Industry" field included in the CSV report.

Open the CSV report in Google Sheets or Excel and add two new columns:

  • Deduplication Exclude

  • Deduplication Master

Now given the context from all the fields available in the report, update the CSV as needed:

  • Enter TRUE in the "Deduplication Exclude" column for any row that you'd like to exclude from deduplication.

  • Enter TRUE in the "Deduplication Master" for the row that you'd like to designated as master from within a deduplication group.

When you're done updating the CSV report with exclusion and master selection, remove all columns from the CSV except for the following:

  • Record ID

  • Deduplication Exclude

  • Deduplication Master

Then save the report to a CSV file.

Step 2: Create Custom Fields in the CRM

Create two custom fields in the CRM:

  • Deduplication Exclude

  • Deduplication Master

For the field type pick "Boolean" or "Checkbox".

After creating the new custom fields in the CRM, refresh the data in Insycle by logging out and then logging back in, or click "Sync" under Settings>Accounts.

Step 3: Use Magical Import to Tag Duplicates

Open the Magical Import module, and load the CSV file you saved in Step 1.

Map the ID column to the relevant record ID field. The other two fields will map automatically.

In Step 2 of Magical Import, pick "Only update existing" because those records already exist in the CRM, you're simply "tagging" those existing records for deduplication.

Then, click to import the CSV.

Step 4: Deduplicate in Update Mode with Customizations

Now that records are tagged with deduplication customizations you can use the custom fields to customize the process.

To exclude records from deduplication, in Step 1, click Filter and enter: "Deduplication Exclude" not any of "Yes" or "True".

To designate record as master, in Step 4, remove all rules and add just one rule: "Deduplication Master" is "Yes" or "True"

That it.

You can run deduplication in Preview mode again to confirm that it works as expected with all the customizations and then run it in Update mode to merge the duplicates in the CRM.

Additional Resources

Did this answer your question?