Data Quality Rules That Prevent Bad Decisions

“Improve data quality” is too broad to implement. A useful quality programme defines which data must satisfy which rule, at what point, for which business decision, and who resolves a failure.

The goal is not perfect data everywhere. It is data that is fit for an explicit operational or analytical purpose.

Begin With Critical Data Elements

Identify the fields and records that directly influence money, customers, compliance, inventory, access, or executive decisions. Examples include customer identity, product unit, tax category, order status, invoice amount, payment reference, and stock location.

For each, name the owner, consumers, source, expected freshness, and consequence of error.

Use Six Rule Types

Completeness

Required data exists. Example: every shipped order has customer, address, item, quantity, and shipment reference.

Validity

Values conform to a domain or format. Example: currency belongs to the approved reference list and quantity uses the product's permitted unit.

Uniqueness

A real entity or event is not represented more than once. Example: provider transaction ID is unique within the merchant account.

Consistency

Related values do not contradict. Example: delivered date cannot precede dispatch, and an inactive product cannot be newly allocated without an approved exception.

Accuracy

Data reflects the real-world value. This often requires verification against an authoritative source, not only a format check.

Timeliness

Data is available within the period required by its consumer. Yesterday's inventory may be valid but unfit for checkout.

Put Rules at the Right Boundary

Prevent invalid data at entry where possible. Validate again at integration boundaries because source and contract can change. Apply analytical tests after transformation to catch joins, aggregations, and history errors.

Do not silently “fix” material business data in a pipeline. Route uncertain corrections to an owner and preserve the original value and rule result.

Define Severity and Response

Classify a failed rule:

• Reject: processing cannot safely continue.
• Quarantine: isolate the record for review.
• Warn: continue but create an owned exception.
• Observe: measure a pattern before enforcing.

Connect each severity to owner, service target, retry, and escalation. Thousands of alerts without a resolution process are not a quality control.

Monitor Quality as a Product

Track pass rate, affected records, business impact, time to resolution, recurrence, and source. Segment results by system, channel, supplier, team, or product to locate root causes.

Publish quality targets with the data product. Consumers should know freshness, completeness, and known limitations before using the data.

Test Transformations

For pipelines and warehouses, test:

• Source-to-target record counts.
• Required keys and referential integrity.
• Duplicate facts.
• Accepted value ranges.
• Balance and control totals.
• Slowly changing dimension behaviour.
• Incremental watermark and late arrivals.
• Deletion and retention.

Reconcile important financial and operational totals to authoritative systems.

Improve the Source Process

When the same rule fails repeatedly, investigate form design, default values, training, incentive, ownership, system constraints, supplier contracts, and integration mapping.

A dashboard showing poor addresses is less valuable than changing the checkout and verification process that creates them.

Avoid Misleading Quality Scores

One overall percentage can hide a critical defect. Report by dimension, data element, use case, and severity. A dataset can be 99% complete and still unusable if the missing one percent contains the highest-value transactions.

Establish a Quality Contract

For every critical data product, document owner, schema, definitions, rule set, thresholds, refresh target, incident contact, and change policy. Version rules alongside pipeline code and test them before release.

DualByte's IT consulting service can help translate business controls into enforceable data rules and operational exception workflows.

Sources

• Microsoft Purview and Profisee MDM Reference Architecture
• Great Expectations Documentation