Data Integrity: Data Quality Rules
Creating a Data Quality Rule
Data8 Data Integrity allows the creation of customised data quality scores for any table across Dataverse. Navigate to “Rules” table on the main sitemap and click “New” to create a new rule.
By default, new rules will be in a draft status, allowing them to be created and refined before publishing them. Published rules are read only to prevent issues with the score validity as rules can be amended over time. Each rule has a version number that is managed by the Data8 Data Integrity solution and this version number is stored for each recalculation.
When creating a new rule, you’ll need to input a name, select the table from the drop down menu before selecting the field in which to store the score – this field will need to be created by you and will need to of a “whole number” field.
Once you have selected the table name and the scoring field name, the “Rule Definition” box should load in the metadata and allow you to start building a rule.
Advanced Data Quality Rules Settings
You can to specify a “Next Calculation” field – this field is only used if you use a rule that is time based as the score will change over time. This should be a Date Time field.
Most Recent Snapshot Field Name:
You can to specify a lookup to the snapshot table – the snapshots are stored each time a score is changed and this offers the option for a lookup to be set on the source table. This may be because you want to use it in reporting, such as Power BI and you’d like a direct link to the latest snapshot.
By default, we store score changes for 1 year but this can be override here.
Store Tips in Snapshots:
When a snapshot is made, here you are given the option to persist the tips that display to the end user on the same snapshot. The tips are stored in the data8_tips column of the snapshot table and are stored as a JSON blob which can be transformed using popular reporting tools such as Power BI. Having these tips allows you to tell a story of how, and why, scores are changing over time.
Lookup Override Field & Lookup Override Field Logical Name:
There may be some situations where you cannot make a score on the table you want (think systemuser) – this option allows you to make a new table with a lookup and it will act like an interim table, so you may make a “System User Score” table which has a score and a look up to the systemuser. By setting these values, the rule definition context will be override from the logical name set earlier on, to this new override value.
For information on score types, Click Here.