Release Information

Automerge Failure Process Update

Occasionally, failures will occur when merging due to issues in the platform with both first and third party plugins. The Automerge processes has been re-evaluated to offer better insight should failures occur.

Duplicare Copilot

Build your Dedupe+ Rules with AI using the new Duplicare Copilot. Create a new Dedupe+ Rule and use natural language to describe what records you want to match, such as "active contacts with the same name and either phone number or email address" and let the AI convert it to a full Dedupe+ Rule ready for you to review.

Work Around "TransactionCurrencyId needs to be supplied to format the money field" error

A recent Microsoft change can trigger this error when updating the value of a money field on the master record during a merge. While hopefully Microsoft will resolve the underlying problem soon, this change works around the error within Duplicare as well.

Added Intersect to Dedupe+ Jobs

Previously intersect could only be used in real time but this feature has now been extended to batch jobs.

Improved Duplicates Detected +

Further fixes around Duplicates Detected + that were affecting using taking very specific steps during the data entry. These steps have been fixed to have the expected behaviour.

Handling Errored Metadata

There are some situations in fields provided by Microsoft out of the box that some fields can miss metadata that should be required. A fix was deployed to handle the few situations where it may not be configured quite as it should be.

API Request Improvements

A number of improvements in decreasing the API requests used by duplicare when performing cross entity matching were added.

Merge Window Composite Updates

Composite fields now break down to their unique field names and render those names accordingly i.e. firstname and lastname instead of fullname.

Improved Duplicates Detected +

When choosing "Ignore and Save" on Duplicates Detected +, a second duplication check was taking place causing the out of the box duplicates detected window to also display. This has now been fixed.

Customizable Behaviour When Merging Custom Tables

When a merge was taking place for a table that isn't normally mergeable out of the box (account, contact, lead, case) the behaviour of related records can now be amended on a per relationship basis - this is for situations where you don't want re-parenting to take place.

Improved Address Standardization for businesses

There was a bug preventing businesses being treated as businesses despite having the company name field specified in the address field. This has now been rectified.

Introduced async merging in QuickMerge Dedupe+ Jobs

Dedupe+ Jobs can now optionally be enabled to run async in QuickMerge Dedupe+ Jobs. Each merge that we find will be run as a system job asynchronously so the dedupe+ job itself will finish much quicker whilst the merges are still completing in the system job queue. 

Introduced optional "title" to fuzzy name match dedupe rule

The fuzzy name match rule can now have an optional "title" field added to it, allowing for even better matching as well as removing some false positives.

Introduced optional "Company Name" to standardized address & fuzzy address rule.

An optional company name field has been introduced when created an address field. This name won't change the standardized result if you have it enabled but does allow the "company name" field to be taken into account during a fuzzy address match to improve matching logic, particularly with a B2B focus.

Introduced an optional "Opt in" field to standardized address.

In address fields if you have standardize enabled then by default we will standardize every address, both historically and in the future. We have an introduced a new option for "opt in" which is a boolean field you can specify. If this boolean is set to true, we will standardize that address otherwise we will leave it as it was. There are occasions, or routes data takes that means sometimes standardizing of addresses requires to be skipped and this new fields allows this to be configured as such.

Fixed issue when combining static value and OR rules

An issue was found when multiple static value rules are combined with an OR in a rule. This has now been rectified in both real time and batch.

Efficiency changes when editing Dedupe+ Rules

The system was performing some unnecessary maintenance when editing dedupe+ rules, particularly those without global search enabled. Whilst this wasn't an issue, the bug was fixed to save time on the UX.

Fixed tracking of merging when merging tables that don't have OOTB merge

The "Duplicate Management Dashboard" and respective "Duplicate Report" tables were not tracking when a merge was taking place for a table that isn't normally mergeable out of the box (account, contact, lead, case).

Server Side Efficiency Changes

A number of changes were made at Data8 server side, where batch procedures are ran so jobs are picked up and ran quicker..

Merging Orders

When merging sales order records, the subordinate order records are now correctly marked as canceled. An "Order Close" activity will be created to indicate that it is lost because it has been merged into another order.

Merging records with missing currency

When merging records that have currency-type values, a warning will now be shown if the selected currency is blank

Duplicates Detected+ and special ribbon buttons

The Duplicates Detected+ window is no longer shown when a record is updated by one of the special ribbon buttons such as Activate, Deactivate, Qualify, Disqualify etc. It will still be shown as normal when the standard Save button is used instead.

Closed Quote/Order/Invoice records

When merging custom entities that have related quote/order/invoice type records, any such records that are in a closed state will automatically be moved to the master record without requiring system administrator privileges.

Merge error diagnostics

If an error occurs during a merge, additional information is now shown alongside the main error message. If the error is triggered by a plugin, the name of that plugin will be shown. The option to download a log file with the full error message and other details is also available.

Merging Quotes

When merging quote records, the subordinate quote records are now correctly marked as lost. A "Quote Close" activity will be created to indicate that it is lost because it has been merged into another quote.

Use Master Merge Rule

The "Use Master" merge rule now has a new option to ignore any blank values. When this option is enabled, additional rules can then be added to perform other merging logic when the master record doesn't contain a value, e.g. take the most popular value from the other records being merged.

Expression Validation

Additional validation is now applied to expressions being used for the master selection rule and sort/filter expressions in merge rules. This avoids issues that could lead to unexpected values being used during a merge. For example, if you use a sort expression in a Filter & Sort merge rule to prefer taking values from records based on a specific industry, the expected syntax would be something like:

CASE WHEN industrycode = 100001 THEN 1 ELSE 0 END

Common errors we see are when the numeric value is enclosed in quotes, or a text value is entered instead:

CASE WHEN industrycode = '100001' THEN 1 ELSE 0 END
or
CASE WHEN industrycode = 'Agriculture' THEN 1 ELSE 0 END

In both these cases, the equality expression would have always evaluated to false and therefore the sort order would not have had the expected result.

Both these two incorrect examples will now trigger an error in the merge rule editor and will need to be corrected to the first correct example before the rule can be saved.

Automerge Updates

When an automerge fails due to the Dataverse configuration, rather than stopping the entire dedupe+ job we will now continue. Any failed updates will be pushed in and the error be shown on screen to allow users to fix and retry. Dedupe+ Jobs will also now finish with a status of "Completed with Errors".

Assign Duplicate Groups

Duplicate groups can now be reassigned. They can either be reassigned using the standard "Assign" from a subgrid or they can be reassign from the multi record page.

Fixed an issue around lead -> contact merging

A behind the scenes change in the platform introduced a breaking change for some lead to contact merging. This has now been rectified.

NOT rule added

A new "NOT" rule has been added to dedupe+ rules.

Batch Scheduling Update

The batch schedule process has been rewritten to dramatically reduce API usage and become much more efficient.

New Version Alert

The new version alert is now instant and if you update, the alert will disappear straight away. Previously it took around 10 minutes to disappear.

Validate against metadata - Apply to memo field

The new validate against metadata options wasn't applying to memo fields, this is now rectified

Misc Bug Fixes

A number of smaller bugs were fixed around the global search and PCF controls included.

Validate against metadata

A new feature added to merge+ rules allows you to validate the potential merged data against the metadata in Dataverse. Examples such as:

• Invalid option sets will now be removed from calculations
• Sum rules will now limit to max values
• Concat rules will now truncate to max length.

This will default to one for new merge+ rules but will need to be enabled for existing merge+ to ensure backwards compatibility.

API Usage reduction bug fix

A very specific set of criteria existed that could cause the merge window to stop rendering, introduced in 6.2.0. This has now been fixed.

If you have 6.2.0 installed, please upgrade at your next available time to do so. An issue was identified that could cause the merge window to stop displaying.

API Usage dramatically reduced

A refactor of the core parts of the product has reduced the API calls required for general usage of the tool.

Most Popular Across Fields Merge Implementation

After the fix in 6.1.54 for the UI, an issue was uncovered in potential inconsistency in the result calculation depending on record sort order. This has now been rectified and a consistent sort order is now applied.

Domain Match Rule

In some scenarios, the domain match rule did not follow the specified exclusions when finding duplicates - this is now fixed.

Multi Record Merge through the API

An issue was found when calling the legacy multi record merge through the API i.e. via Power Automate. This method is different than the multi record merge action ran during the normal merging UX.

Merge Window Display Name

The merge window was showing the display name specified in Dataverse and was not using any form overridden values if specified - this is now rectified and display name is showing correctly.

Most Popular Across Fields Merge Rule

Fixes a bug when editing this merge rule type that could cause it to not save the value for the "Take the <n>th most popular value" setting.

Controls Update

A number of bug fixes were added across the controls throughout the solution.

Browser Storage changes

Occasionally Duplicare uses storage in a browser - this has been improved to minimize any browser issues.

On-Premise Updates

A number of updates have been made to the on-premise version to bring the performance in line with the online version of the solution.

Controls Update

A number of configuration controls were updated to move away from AutoComplete to dropdown boxes for a better user experience

New Assignment Strategy for Dedupe+ Jobs

Dedupe+ Jobs can now have a team specified as the assignment strategy, meaning you can evenly split merging records evenly across team members

Custom Duplicate ID storage

A specific field can now be specified for "Store Duplicate ID" in Dedupe+ Jobs making for easier analysis of job results. 

Merge+ Rule Editor Fix

Fixed a bug causing headers to not be displayed for some form layouts in the Merge+ Rule editor

Domain Matching Fix

Fixed error when saving records after removing a value from a field that has a domain matching rule associated with it.

Address Standardisation Error Handling

Handle any errors calling the address standardisation API to avoid errors saving records if no credits are available for address standardisation.

Duplicate Detected PCF Reworked

Designed to be more efficient with API calls and also now works across table/entity

Added Name Matching Option

A number of name matches were added to the fuzzy company name option for better international support

Miscellaneous Bug Fixes

A number of bug fixes are now included

Fixed Scoring Rule Bug

Fixed an issue in the UI which would occur when entering a letter into a number box

Fixed the confirm dialog on address/phone standardization

Fixed an issue that mean OK and Cancel didn't behave as intended when created or amending a rule.

Trial Bug Fixes

Fixed a number of issues that only occurred on Trial licenses when using non trial features.​

Miscellaneous Bug Fixes

A number of bug fixes are now included

Usability enhancements in Dedupe+ Rule editor:

• Ellipsis button "..." is highlighted when hovering over a rule
• Rule is highlighted when the ellipsis button is clicked
• Added option to move rules

Added country as an option in Fuzzy Company Name

Improved performance when merging large numbers of custom entity records

Worked around error causing a custom action to be inactive after the solution is imported into an online environment.

Added new Scoring rule type

Fixed display of merge screen when using forms that contain a "File" type column

Improved naming of automatically scheduled Dedupe+ Jobs

Fixed Merging Permission Issue

When a user has merging permissions via team inheritance, it was incorrectly showing no permission.

Global Search & Search All Records Behaviour

The behaviour across global search and search all records was improved to perform as expected. Additional notes were also added through to ensure proper usage.

Miscellaneous Bug Fixes

A number of bug fixes are now included.

Fixed Merge interface bugs

The merging interface will now handle vector icon images properly and will also wrap multi-line text fields.

Merge+ Rule Tweak

The "Merged Status Reason" will no longer be visible when configuring an out of the box merging enabled table.

Added new Dedupe+ Rule editor

The full dedupe+ rule editor got a face lift to look a little more modern.

Added new Merge+ Rule editor

The full merge+ rule editor got a face lift to look a little more modern.

Miscellaneous Bug Fixes

A number of bug fixes are now included.

Added columns to Global Search results

The new global search experience has been updated to enable configuration of additional fields to be shown on the results of global search.

NOTE: The legacy global search (web resource named: data8_GlobalSearch.html) is deprecated for online users. All online users should now use the web resource named data8_advancedlookup.index.html

Extended multi field rule

Additional options were exposed in multi field rule to allow a better match set up. You can now 

Allow Blank Match and Allow Non Conflicting Matches in multi field rule, much like you always could for Exact Match Rule.

Fixed a bug with permissions on Duplicate Detection Action

A bug meant specific system set ups could throw an error when searching for duplicates.

Fixed a bug on Duplicates Detected +

A bug meant the merge button would always show, even if users didn't have merge permissions. The user wouldn't have been able to persist the merge so it was a poor UX.

Miscellaneous Bug Fixes

A number of big fixes are now included.

Fixed UCI Issue with Dedupe+ Rules

Occasional transactional issues would occur when creating, updating or deleting dedupe+ rules due to a change in how the unified interface interacts with the data on screen. This has now been amended.

Improved International Fuzzy Company Name

Duplicates Detected+ Release

A brand new duplicates detected+ feature was added into the product. For more information, please refer to the installation guide for more information.

Added "IsGhostRecord" variable for Merge+ Rules

This allows for better rules when using lead to contact matching.

Miscellaneous Bug Fixes

A number of big fixes are now included.

Duplicates Detected+ Release

A brand new duplicates detected+ feature was added into the product. For more information, please refer to the installation guide for more information.

Improved International Fuzzy Company Name

A number of improvements were added to the fuzzy company name feature for better international matches.

Geo-Restriction on Address Standardization

The restriction for running address standardization in Azure regions has now been removed.

Handle status bug upon merge

Occasionally a record could be have been left as inactive incorrectly. This has now been fixed.

Better handle UTC issues on merge

Occasionally an issue occurred with dates when users spanning multiple time zones were merged.

Miscellaneous bug fixes and performance improvements

A wide range of minor bug fixes and performance improvements were added.

Lookup+

Lookup+ was added to the product. More information can be found here.

Duplicare SDK Client

A C# nuget package was introduced and can be found here. Help on how to use this can be found here.

Application Insights

Adding some basic application insights anonymous usage tracking. Also added the ability to enable advanced logging in the event of requiring assistance from Data8.

Minor UI Tweaks

A number of error messages have been improved to be more accurate and the user interface in the config area has been updated slightly.

Merge Permissions Fixed

The logic to calculate merging permissions has been updated

Improved Dedupe+ Job Statistics

The statistics generated for a Dedupe+ Job now correctly take into account any records that have been ignored due to previously-created exclusion rules.

Improved Merge+ Rule editing

When selecting an existing "Use most common value across a set of fields" rule, the previously selected fields that the rule will use are now highlighted.

Dedupe+ Rule Creation Fix

Resolves a "Business Process Error: Object reference not set to an instance of an object" error when creating a new Dedupe+ Rule.

Data Imports

Enhances duplicate detection during data imports. When data is imported that includes a lookup field that is important to duplicate detection, this may have led to the duplicates not being correctly identified in earlier versions. This version fixes this and will leave the duplicate record in a "Partially Imported" state within the import job. To trigger the solution to re-synchronise existing rules with the new functionality, you need to make a change to the rule definition (e.g. add a new "exact match" rule, save it, then remove and save it again)

Advanced Find

Displays the correct merge button on the Advanced Find window ribbon (requires deactivating and reactivating the Merge+ Rule and publishing customisations)

Merging Fixes

• Fixes issues with required field handling in the merge screen
• Fixes issues merging multi-select picklist records

Canadian Address Improvements

The calculations for using fuzzy matching on addresses can vary by country and a number of improvements have been made for canadian addresses based on official guidance.

Server side efficiency changes for batch jobs

Some efficiency changes where made on the server side, allowed for faster batch dedupe+ jobs

Updated latest version checker

The configuration page has a latest version checker and a bug was identified during the move to a 3 number based version number as of 4.0.0. This has now been fixed.

Updates/preparation for a not yet announced feature

More details released in due time.

Fixed bug with empty multi select picklists

Minor bug fix which prevented records with a blank multi select picklist being merged.

Azure data processing

Option to choose where your data is processed from a selection of Azure regions

Added new PCF component

New PCF custom control added for quicker feedback on duplicates, speeding up the duplicate detection process even quicker.

Extended duplicate detection to optional search across the whole system

Optionally use the system organisation to search for duplicates across the whole system, regardless of permissions. The duplicates returned will not share data it shouldn’t and will just give the count of duplicates.

Fuzzy Name Matching Improvements

Improvements were made to the fuzzy name, including an updated blacklist of names.

Bug Fixes & Performance Updates

A number of bugs fix and a number performance updates were added throughout, particularly in the web resources used to manage the duplicare specific rules.

Versioning

Versioning numbers where brought back to be inline, regardless of premise or online. Also moved to a 3 digit version number rather than a 4.

Persistent Form Selection

During Merge+ configuration you can now select a system form that all users will use as their merge+ layout.

Read Only in Merging

The multi record merge fields now adhere to the configuration of the form and will show as read only, as they are on the form

Required Fields in Merging

The multi record merge fields now adhere to the configuration of the fields and won't allow submission when a required field is blank.

Fix for many-to-many merging

This release fixes a bug that would cause Dedupe+ Jobs with Automerge targeting a custom entity with a many-to-many relationship to hang.

Change in implementation with Fuzzy Company Name

A change was made in the implementation of the fuzzy company name within Dedupe+ rules allowing you to optionally choose to allow if matching legal identifiers should match or not (i.e. Data8 Ltd matches Data8 PLC). Default behaviour would previously not allow this but default behaviour now will.

Please re-address your dedupe+ rules that use this rule and check the new option, if required.

Fixes for multi select picklist in cross entity

Fixed an issue in rendering the merge window when multi select picklists were used in a lead to contact set up

Increased maximum length of master select

Increased the upper limit of the master select rule in Merge+ rules

Added version checker

The main configuration page will now show you a warning if you are behind in your duplicare version.

Fixes for entities with special characters in names

Fixed an issue setting up Merge+ Rules and Global Search for entities that have a special character in their display names.

Versioning numbers now split out for online and on-premise to help us keep up support for the latest features.

Unified Interface Support

Introduce a PCF component to bring back AutoComplete functionality throughout within UI.

Add new icons to better improve the experience.

Fixed a number of bugs where the Unified Interface handles WebResources slightly differently.

Global Search

Fixed an issue where a global search only rule was being ignored.

Introduce Global Search

Global Search support added throughout. To enable Global Search, set “Global Search Enabled” to “Yes” on any Dedupe+ rules you want to apply it to.

Quick Merge Performance Improvements

A number of efficiency improvements have been added to the quickmerge process both in real time and in batch.

Address Standardization bug fix

Fixed an issue in address standardization when only a country is entered.

Auto Edit Forms support removed

The support in allowing duplicare to auto edit your forms by adding relevant fields and javascript has been removed.

Fixed issue with Associate/Disassociate when custom merging

Fixed Issue with Tasks when custom merging

Fixed Issue with Multi Select Picklist options when custom merging

Entities that cannot be merged through out of the box methods require advanced logic and a number of bugs were identified and rectified.

Dedupe+ Job without Merge+

Fixed a bug whereby you could run a dedupe+ job without a Merge+ rule for an entity that isn’t an out of the box mergable entity. A Merge+ rule is required due to the background fields being required to merge non out of the box.

Modified Currency Fields

Fixed handling of currency fields that have been modified when merging an unsaved record.

Inactive Address Standardization Handling

Fixed errors encountered when updating addresses that were covered by an inactive Address Standardization entry

Cross-Entity Batch Jobs

Fixed an error encountered when running a batch lead-to-account or lead-to-contact job that found duplicate leads without any corresponding accounts or contacts.

If you are upgrading from a version 2.0.0.0, 2.0.0.1 or 2.0.0.2 release, please deactivate any configuration entities e.g. Dedupe+ Rules, Merge+ Rules etc. first, then install this version, then reactivate those configuration entities again.

Merge+ Configuration

Addresses a number of bugs in the UI for creating a Merge+ configuration

Address Standardization Inconsistancies

Patched a bug around address standardization, in partcular UK postcodes.

If you are upgrading from a version 2.0.0.0, 2.0.0.1 or 2.0.0.2 release, please deactivate any configuration entities e.g. Dedupe+ Rules, Merge+ Rules etc. first, then install this version, then reactivate those configuration entities again.

Dedupe+ Rule Creation

Added a missing hidden field to the Dedupe+ Rule form to fix an error when creating a new Dedupe+ Rule.

Duplicate Detection with particular rule combinations

When making changes, we have required a single Unmanaged solution as a place to store those changes. Moving forward, each change will be stored in it's own unique managed solution - this will reduce any problems caused when moving solutions across environments.

Removed Managed Metadata Changes

Version 2.0.0.0 introduced a change where any field, SDK message processing step etc. that was created when configuring duplicare was done as a managed change. This was a breaking change that caused issues when enabling duplicare features on unmanaged entities, and has been rolled back in this release.

If you are upgrading from a previous version 2.0.0.x release, please deactivate any configuration entities e.g. Dedupe+ Rules, Merge+ Rules etc. first, then install this version, then reactivate those configuration entities again.

Business Unit handling during creation

Extending the recent changes surrounding business unit ownership to fully support "creation" situations - where the record hasn't been commited to the database yet

Improved PO Box matches in non UK countries

The logic around PO Boxes and fuzzy address matches has been extended to work better in non UK countries.

Merging of date & time values

Fixed the error Cannot convert the literal 'yyyy-MM-ddTHH:mm:ss' to the expected type 'Edm.Date' and Cannot convert the literal 'yyyy-MM-ddTHH:mm:ss' to the expected type 'Edm.DateTimeOffset' when merging records and changing the value of a date & time attribute.

Recalculating scheduled job times

When changing the hour of a scheduled job, the next job time is now updated correctly.

Back end updates in order to improve usability

A number of features have been reworked behind the scenes, allowing duplicare to make use of some of the bonus aspects of merge such as time constraints. 

User Interface Updates in Multi Record Merge

When commiting to something that has multiple stages i.e. merging 3+ records, a better progress indicator is displayed on screen so you can be sure it is progressing as you would expect.

Enable use of Owning Business Unit field for duplicate detection

When using the Owning Business Unit field in a Dedupe+ Rule, the business unit is correctly calculated when a record is being created/updated.

Microsoft Dynamics 365 Version 9.1 Compatibility

Leads in a locked marketing list can cause an issue on merge. A fix has been put in place that unlocks the marketing list, makes any changes and then locks the marketing list.

Microsoft Dynamics 365 Version 9.1 Compatibility

Resolves an error in v9.1 when starting or scheduling a batch job
“An resource with type ‘Microsoft.Dynamics.CRM.userquery’ was found, but it is not assignable to the expected type ‘Microsoft.Dynamics.CRM.savedquery’. The type specified in the resource must be equal to either the expected type or a derived type.”

Storing duplicate ID’s in a batch job

Each Dedupe+ Job now has an additional toggle field, allowing you to store the duplicate id within duplicate entity itself.

This will allow you to perform advanced finds and identify duplicates outside of the batch results – using the “data8_duplicatedetectedid” on the relevant entity (for example account or contact).

This will require us to modify any duplicate records found, meaning the modifiedon and modifiedby date would be updated using the account that you have authorised under your account within Data8.

Running batch deduplication with a subset against a bigger set

You can now take a group of records and dedupe it against a separate group of records.

This will allow you to perform much specialised and efficient dedupe jobs within your environment. Imagine this scenario: “I would like to dedupe all contacts created today against my entire list of active contacts”. Previously, you would only have been able to do a dedupe job on the “Active Contacts” saved view without any guarantee there was even any contacts created today – you may also get extra duplicates highlighted which wasn’t the main reason for you running the job.

Now, you are able to specify the following:

• “Filtered Master View” – this is the records you want to find duplicates for.
• “Filtered Candidate View” – this is the records you want to find the duplicate within.

In order to do the scenario mentioned above, you could make a saved view called “Contacts Created Today” and set that to be your “Filtered Master View”. You could then either leave “Filtered Candidate View” blank (meaning all records) or specify “Active Contacts”. The result would be that we would look for any duplicates only of the contacts created today.

This same functionality has been extended into schedules too so you can run this on a schedule also.

UCI

The whole duplicare solution is now fully developed tested within the Unified Client Interface  and now is at full feature parity with the original user interface.