Duplicate data poses serious problems for companies of any size.
Duplicate records inhibit your marketing team from effectively segmenting and personalizing your communications. Sales teams step on each other's toes and lack vital context in conversations. Support teams miss important information, and analysis and reporting are skewed.
Insycle helps you to merge duplicate contacts, companies, and deals—flexibly and powerfully.
Table of Contents
How It Works
Insycle analyzes your database and identifies duplicates with flexible matching rules, using any field in your database, to help you identify and merge more duplicates in bulk.
Once Insycle identifies the duplicate records, you set rules for determining the master record that other duplicates will be merged into—such as the first record created, record with the most email opens, or any other attribute that would be relevant. You can also set merging logic on a field-by-field basis.
Insycle provides a complete report of what was identified as a duplicate, what was merged, and what the outcome was in your master record.
First, you use a filter to identify duplicates, such as contacts with matching names and companies. Then you can review those duplicates, and set advanced rules for determining what data is retained in the master record after the duplicates are merged.
With Insycle, you can run deduplication processes in bulk, then automate your deduplication templates to keep your database free from duplicates moving forward.
Step 1: Find Duplicates
Navigate to the Merge Duplicates module. From the top of the screen, choose whether you will be deduplicating contacts, companies, deals, or custom objects.
For a quick start, explore the default templates. There might be a template that already solves exactly what you need. You can also use an existing template as a starting point and adjust it to your requirements.
In order to find duplicates, you need to define how to match records.
For example, to find duplicate people you might match using "First Name", "Last Name", and "Email Domain" fields. In that example, people with the same first name AND same last name AND email domain will show as possible duplicates.
Insycle allows for deep control in defining how to match the values of those fields:
Exact Match vs. Similar Match
Exact Match looks for other records that match exactly, with no differences from one record to the next. This is great for identifying surface-level duplicates while remaining highly accurate.
Similar Match analyzes data and identifies customer records that are similar, but not exact matches. It works by analyzing the “closeness” of two different data points.
Closeness is determined by measuring the number of changes necessary to make the two data points match.
insertion: bar → barn
deletion: barn → bar
substitution: barn → bark
Ignored allows you to pick parts of the value to ignore. For example, ignore symbols and whitespace when comparing phone numbers, or ignore HTTP, www, subdomain, or top-level domain (.com vs co.uk) when comparing websites or email domains. This is great for catching more advanced duplicates.
Match Parts allows you to compare specific portions of a particular field. For instance, you can pick whether to compare the entire value, any two words, or just the first 5 letters, or X last characters, etc. This is great for specific use cases.
Adding Match Fields
You can add fields by clicking the "+" button, or remove fields by clicking the X button. When you click "Find", Insycle will identify duplicate records based on the matching filter.
Step 2: Review Duplicates
In this Step, Insycle allows you to preview the identified duplicates and double-check to make sure that your filter is producing accurate results. That way, you can avoid merging any records that are not actually duplicates.
Click the "Find" button in Step 1, Insycle will generate a list of the identified duplicates for you to review.
When two or more records represent the same entity (person, company, or other), we refer to them as a Duplicate Group. For example, if you had 4 records for the same person, it would count as 1 Duplicate Group with 4 records.
In this view, each row represents a Duplicate Group. When you click a row, it expands and shows the records that belong to that group.
Here, you can dig deep into your identified duplicate groups, down into the individual records, to check for accuracy and get a better idea of the best way to determine the Master Record that other records will be merged into, which we will do in Step 4.
You can add more columns to the view using the ⚙ button on the right, in order to get more context and analyze the records.
Step 3: Choose Operation
In Step 3, you can merge duplicates in bulk using rules, or manually.
When merging in bulk, Insycle limits the duplicate group size to 5 and skips groups that contain more records. You can adjust the default value of 5 as needed, up to 100 records per duplicate group. This is a safety precaution to ensure that if you use a duplicate matching filter that is too broad in Step #1, you do not accidentally merge many non-duplicate records together.
If you select Manual Mode, you'll have complete control over which records are merged together by selecting them from the Record Viewer in Step 2. Here, you get a full preview of the record and can choose which records will be merged and which will not.
In Manual Mode, you can also individually select the Master Record that other records will be merged into on a case-by-case basis.
In Bulk Mode, you set rules for determining the master record automatically in Step 4.
Step 4: Master Selection
Insycle offers extremely flexible controls from determining the master record when bulk merging.
The goal is to pick one record from each Duplicate Group as the master, and the remaining records in that group will get merged into it. For example, if you had 4 records that represent the same person, it would count as 1 Duplicate Group with 4 records, all of which will be merged into 1 master record.
You can make your master selection on the record level, such as merging all duplicates in a group into the earliest created record.
Or, you can customize your master selection down to the field level. For example, maybe you want the "Email" field to come from the last modified record, but you want First Name, Last Name, and Phone Number to come from the earliest created record.
Insycle gives you complete control over how data is merged into your master record using advanced rules.
Master Selection By Record
When merging in bulk, the master record from each Duplicate Group is determined using rules via an elimination process. For each Duplicate Group, Insycle eliminates records that do not meet the rule criteria.
Rules are read in order, from top to bottom. So if a Master Record meets the first criteria (Highest marketing emails clicked in the example below) the others on the list will not impact the selected Master Record.
For example, let's say that we have 4 records that represent the same person — Britany Calvert. That is 1 Duplicate Group that consists of 4 records.
Here is the data that we have for this duplicate group:
We haven't sent any emails to Britany yet, so when Insycle processes the first 3 rules — emails clicked, email opened, and email bounced — Insycle cannot eliminate any record because they all have the same value of zero.
In the next rule about contact owner, records 1000, 1001, and 1003 are eliminated since no contact owner exists for those records. Now there is only one record remaining 1002 and therefore that's the master record.
You can add or remove rules using standard and custom fields. And since the order of the rules is important, you can also reorder the rules as needed using the arrow buttons.
Priority Match vs. Absolute Match
On the right hand side of Step 4, you will find the option to select Priority and Absolute Match.
Priority Match. Go down the master selection rules in order, one-by-one. As soon as a record meets one of the criteria, it becomes the master record. A vast majority of duplicate templates should use Priority Match.
Absolute Match. The master record must meet all of the listed rules in the "Record" tab in Step 4. If a record does not match every rule listed, no master record will be identified. Absolute match is less commonly used, but can be useful in specific use cases.
PLEASE NOTE: Choosing Absolute Match can often result in no master record being identified, since the record has to match every rule listed.
The vast majority of the time, you should select Priority Match.
Master Selection By Field
Insycle also gives you the ability to pick which data to retain in the master record on a field-by-field basis.
Then, the master record will be decided based on the rules that you set but may pull values in fields from several different records from the duplicate group, based on the rules that you set in the Fields tab.
For instance, you might want the First Name and Last Name fields to be retained from the earliest created record in the duplicate group. But, you may want the email retained to come from the most recently updated record. With Insycle, you can. In this situation, the master record will retain it's data for all additional fields, but the First Name, Last Name, and Email will be pulled from other records in the group based on your settings.
Let's take a look at another example:
You have multiple options for determining criteria:
From master record. The value for this field will always be retained from the master record that you set in the "Record" tab in this step.
From record where value. For example, retain "Employee #" field from the record with the most employees.
From record based on other field. Retain a field based on criteria for a secondary field. In the example above, we retain the data in the Contact owner field from the record with the earliest create date.
Combine and append all values. Additionally, you can append data into master selection fields as well. This is useful for "Notes" fields or multi-select picklist fields.
In this example, we have set by-field master selection rules for 5 different fields; Annual Revenue, Employees, Contact owner, Lifecycle Stage, and My Custom Field.
In the case of Lifecycle Stage, there are two rules. Insycle will look at the first rule in the list first, and if it finds a record that matches the criteria, the second rule will be ignored. In the example above, if Insycle were to find a record in the duplicate group that had the "Lifecycle Stage" of "Customer," then the following rule looking for "SQL" would be ignored.
Using field-by-field master selection rules gives you unparalleled control over what data is retained in your master record during a bulk merge.
Step 5: Preview Or Update
Now, with our Filter and Master Record selection in place, we can run the Merge Duplicates module to update our selected records.
Click the Review button which will open up a pop up window where you can choose between Preview Mode and Update Mode.
Preview Mode simulates the deduplication process for analysis purposes without modifying any data in the CRM, and
Update mode runs the operation and updates the data in your CRM.
Initially, it is a good idea to run in Preview Mode and ensure that your deduplication template is working as you would expect. Once you have confirmed that it is, then run in Update Mode.
Click the "Next" button to go to the Notify screen, where you can select who will receive an email report. You can also add additional context here.
Finally, click the "Next" button to choose whether you will schedule this deduplication process to run automatically on a set schedule, or one-time.
Here, you can choose to schedule this operation to run hourly, daily, weekly, or monthly. When scheduled, you'll receive email the reports after each run. You can view all scheduled automations on the “Automations” page on your dashboard.
The CSV report generated provides a complete look at your entire merge operation, down to individual duplicate groups.
In the "Result" column, the "Master (After)" for each duplicate group will show the data the final master record will contain, including master selection settings both for by-record and by-field configurations. You can see the entire duplicate group and the data retained in the master record.
To correlate merged records, you can see the "Duplicate Group ID" field in any report generated in the Merge Duplicates module. This is also the Record ID that all other records were merged into. The Duplicate Group ID will be identical for the entire duplicate group that was successfully merged into the master record.
When a field says "(Default)," it means that the CRM will be using its default processes for dealing with the field—this is typically seen for blank fields, system IDs, and other specific situations.
Tip: Every field that is included in Step 4: Master Selection is also included in the CSV report, and so you may want to specify additional fields to get more context about the records in the CSV report. For example, add "Industry exists" as the last rule to have the "Industry" field included in the CSV report.
Once you are satisfied with how the operation runs, you can save your settings as a template for future use by clicking on the "+" and "Save" buttons on the Template Menu.
For situations where you need more granular customization for picking duplicate records to include - or exclude - from the deduplication process, or for picking the master record, and there are no common rules you can apply for all or some of the records, you can customize bulk deduplication using exclusions and pre-defined masters.
Separately, it is also possible to customize the merge logic behavior. For example, to instruct Insycle to copy values from field A to field B as part of the merge, or combine multiple fields into one. Customized merge logic requires an Enterprise plan. Please reach out via chat to discuss your specific requirements.
Preview Changes Before They Go Live
With Insycle's Merge Duplicates module, you can preview the changes that you are making to your data before those changes are pushed to your live database. That way, you can check to ensure your deduplication template is working as expected.
You can set up ongoing deduplication for HubSpot companies and Salesforce accounts automation with Insycle on the module level, using Recipes, or integrating with HubSpot Workflows. You can view all scheduled automations on the “Automations” page on your dashboard.
You can schedule your Merge Duplicates templates to run on an automated, set schedule.
You do this by clicking the Review button at the bottom of the module page. Then, you go through a three-step process to run the operation. In the third step, you can choose the "Automate" tab, and schedule your template to run on a set schedule.
Doing this, you can consistently ensure that duplicates are identified and merged in your database.
Audit Trail and History
The Activity Tracker lets you review all changes made through Insycle. At any time you can download a CSV report of the operation and records affected.
Frequently Asked Questions
I already have a list of duplicates, can Insycle bulk merge them?
Yes. You can use a customized list of duplicates, as detailed in this help article, and use Insycle to deduplicate the list in bulk. Include ID numbers from your connected CRM in your CSV.
Why can I only process 50 duplicate groups at a time?
Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. Insycle shows 50 records on the module screen as a preview. You do not necessarily need to select the duplicate groups individually, because you will be able to select all relevant records after clicking the "Review" button to begin the operation.
How do I ensure that I am not merging non-duplicate records together?
Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.
First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure your that your Merge Duplicates template is working as expecting and not merging non-duplicate records together.
Additionally, you can reduce risk when merging duplicates by narrowing your filter in step 1. Use Exact Match instead of Similar Match. Make sure that you are using true uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.
How many duplicates can I merge into one master record?
You can merge as many duplicates as you would like into one single master record. This setting is available in Step 3 of the Merge Duplicates module.
This setting is here to ensure that you if you set too broad of a filter, that you do not accidentally merge many non-duplicate records together on accident. This is precautionary, but you can change the setting as needed to match the size of your duplicate groups. If you are going to set this number at a high level, it is a good idea to run in Preview Mode first to make sure your deduplication template is operating as you intend.
Does the amount of identified and merged records have any limits in the paid subscription?
All plans include unlimited usage, unlimited users, and unlimited operations, see the pricing page. Only during the free trial is the number of records updated, cleansed, or merged is capped at 500 records.
Insycle is having trouble determining a master record. What could be causing this issue?
First, check Step 4 to ensure that you have "Priority Match" selected and not "Absolute Match."
With Priority Match, the rules are processed in order and your master record only has to match one rule. In Absolute Match, your master record would have to meet all of the criteria in the "Records" tab in Step 4.
The vast majority of the time it is best to be in Priority Match.
Can I select which data is retained in my master record in a field-by-field basis?
Yes, Insycle allows you to select which fields are retained in the master record based on pre-determined rules. To learn more please see this help article.
I need to exclude some records from deduplication. Can I do that?
Yes. You can exclude records from deduplication by including a "Deduplication Exclude" field in your CSV, as detailed in this help article.
My team needs to review and approve the master, can I accommodate that with Insycle?
Yes. You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. Consult with this help article to learn more.
Alternatively, you can run the Merge Duplicates module in preview mode, then deliver the preview CSV that Insycle generates. This CSV shows exactly how the merge would take place, but does not update your live data. Then they can approve the merge based on this document, before running Merge Duplicates in Update Mode.
Additionally, you can choose the master manually choosing the "Manual" tab in Step 4.
Do my matching fields have to match each other exactly?
How can I find duplicates when one field is empty?
Insycle requires the values to match in the "Step 1: Find Duplicates" step, and if the value for the matching fields listed in step 1 are blank, Insycle will not include those record in the deduplication.
The workaround to having an empty matching value, is to remove that condition/field from Step 1 and add it in Step 4. Then, the field will be included in the report generated for you to review and identify when duplicates weren't merged due to a missing field.
For example, let's say in Step 1 you have the matching fields: First Name, Last Name, Phone Number. Sometimes, the Phone Number field may be empty.
Remove the Phone Number Field, and add the "Phone Number Exists" as the last rule in Step 4. Now, Insycle will only use First Name and Last Name to identify the duplicates, but the Phone Number field will be included in the report for those contacts. Then you can use a customized approach to deduplication to merge records that are missing the phone number effectively.