Data validation: Processes and techniques explained.

Bad data does not announce itself. It quietly distorts reports, breaks downstream pipelines and erodes confidence in every decision that follows. Data validation is the systematic check that catches these problems before they compound.

In this article, you’ll learn:

• What is data validation?
• Why does data validation matter for business decisions?
• What are the main data validation techniques?
• How do data validation processes work step by step?
• How should organisations choose and implement a validation approach?
• Start building more reliable data pipelines

Data validation is the process of verifying that data meets predefined rules before it enters a system or drives a decision. Those rules cover format, completeness, range and consistency. When a date field accepts "13/32/2024" without complaint, it represents a validation failure and every record downstream of that field now carries a defect.

Every team that touches data encounters validation, whether they call it that or not. A marketing analyst importing a CRM export, an engineer loading an API response into a data warehouse or a data steward reviewing a third-party audience feed all depend on validation checks to catch errors at the source. The common thread is that each of these handoffs introduces risk: the sending system's definition of "valid" may not match the receiving system's requirements.

Validation becomes critical at three specific moments. First, when data enters a system for the first time. Second, when it moves between systems during data transformation, where format mismatches and encoding differences are common. Third, when it is aggregated for reporting, where missing values or inconsistent field definitions can silently skew totals and averages. Each handoff is a point where errors can propagate if no checkpoint exists.

It is worth distinguishing data validation from data cleansing. Validation identifies whether data breaks a rule. Cleansing corrects or removes the offending record. Validation answers, "Is this data acceptable?" while cleansing answers, "How do we fix it?" Both are steps in a broader data quality management workflow, but they serve different purposes and occur at different stages.

Key takeaway: Validation is the checkpoint. It does not fix data. It decides whether data is fit to move forward.

Downstream decisions are only as reliable as the data feeding them. Consider a customer segmentation model built on records in which 15 per cent of email addresses are malformed. The resulting segments will include unreachable contacts, wasting campaign spend on people who will never receive the message, much less convert. The model itself may look statistically sound, but its outputs are operationally useless.

• Validation reduces rework costs. Errors caught at ingestion are cheap to fix. An invalid post code flagged during a batch load takes seconds to quarantine. That same error discovered after a report has been distributed to leadership requires re-analysis, re-communication and recovery of trust. The time and credibility costs of late-stage error discovery dwarf the cost of an upfront check.
• Regulatory compliance depends on validated data. Privacy frameworks such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) require that personal data be accurate and current. Unvalidated records containing stale consent flags or incorrect identifiers create compliance exposure that audits will surface. Organisations applying data anonymisation techniques to protect personal information still need validated source records; anonymising an already incorrect record does not reduce regulatory risk.
• Validated data enables reliable data enrichment. You can only confidently append third-party attributes to a record when its core identifiers, such as email, phone or customer ID, have already passed validation rules. Enriching a record with a malformed email address means the appended attributes may attach to the wrong person or to no one at all.
• Unvalidated data distorts predictive models. A churn model trained on records with inconsistent date formats will misinterpret customer tenure, producing predictions that look precise but are built on noise.

Summary:

1. Bad data caught early costs less to fix.
2. Unreliable records waste media spend and distort models.
3. Compliance frameworks require provably accurate data.
4. Enrichment and standardisation only work on a validated foundation.

Validation techniques are not interchangeable. Each targets a specific class of error and applying the wrong technique wastes computing resources while missing the actual problem. The descriptions and table below map each technique to its error class, a concrete failure example and the business risk it prevents.

• Type validation confirms a field contains the expected data type. A revenue field that receives the string "N/A" instead of a numeric value fails type validation and must be flagged before it reaches a financial model. Without this check, aggregation functions will either generate an error or silently drop the record.
• Range and constraint validation checks that a value falls within an acceptable boundary. A customer age field accepting 847 as a valid entry indicates a missing range rule. Valid ages for a consumer audience typically fall between 18 and 120. Out-of-bounds values distort averages and corrupt segment definitions.
• Format validation enforces structural patterns using regular expressions or schema rules. An U.S. phone number field must match a 10-digit pattern. A record reading "555-CALL-NOW" fails format validation and cannot be reliably used for SMS outreach.
• Consistency validation (cross-field) verifies that related fields agree with each other. A record where "country" is "US" but "postal_code" contains a six-character alphanumeric string in Canadian format fails consistency validation. One of the two fields is wrong and without this check, the record may be routed to the wrong regional team.
• Uniqueness validation checks that records intended to be distinct are not duplicated. Two customer records sharing the same email address but carrying different loyalty tier labels will corrupt any downstream personalisation logic that keys on email.
• Completeness validation confirms that mandatory fields are populated. A B2B lead record missing "company_name" cannot be routed to the correct account team. Completeness validation catches this issue at ingestion rather than during handoff.
• Referential integrity validation ensures that a foreign key in one dataset points to a record that actually exists in the referenced dataset. An order record referencing a customer_id that does not exist in the customer master table will orphan the order in any join, making it invisible to revenue reporting.

A data validation process is a repeatable sequence applied consistently, not a one-time manual check. Organisations that treat validation as an ad hoc activity spend more time firefighting downstream errors than building reliable pipelines.

• Step 1: Define validation rules. Document what "valid" means for each field before writing any code. Rules must be agreed upon by both the data producer (the system or team sending data) and the data consumer (the system or team receiving it). Undocumented rules are the leading cause of rule drift, where the definition of "valid" changes over time without anyone noticing until a failure cascades.
• Step 2: Profile the incoming data. Run a statistical summary of the dataset before applying rules. Profiling reveals the actual distribution of values, the percentage of null values per field and unexpected value ranges. This step informs which rules are necessary and whether existing rules are calibrated correctly. For example, if profiling shows that 40 per cent of records in a "phone" field contain international formats, a rule expecting only U.S. 10-digit patterns will reject valid data.
• Step 3: Apply validation rules programmatically. Run the defined rules against every record. Automated rule engines catch errors at ingestion speed. Manual spot-checks catch only a fraction of errors and introduce human variability. A spot-check that samples 5 per cent of records will miss systematic errors that affect the other 95 per cent.
• Step 4: Classify and route failures. Not all validation failures require the same response. Records with missing noncritical fields may be quarantined for enrichment. Records with invalid identifiers may be rejected outright. Classification prevents both over-rejection (discarding recoverable data) and under-rejection (passing bad data into production systems).
• Step 5: Log, alert and report. Validation outcomes must be recorded. Failure rates tracked over time can reveal systemic upstream problems. A spike in format failures on a specific date point to a source system change that needs investigation. Without logs, the same errors can recur without anyone identifying the root cause.
• Step 6: Feed outcomes back to the source. Validation is most effective when failures are reported to the team or system that produced the data. Closed-loop feedback drives upstream improvement and reduces repeat errors. This is the step most organisations skip and skipping it means validation becomes a permanent cost centre rather than a driver of improving data quality.

Data standardisation and data cleansing typically follow this process. Once records are validated and classified, standardisation normalises formats (for example, converting all date fields to ISO 8601) and cleansing corrects or removes invalid records before they proceed to storage or activation.

Summary:

1. Define rules before touching data.
2. Profile first and calibrate rules to reality.
3. Automate rule application; manual checks do not scale.
4. Classify failures by severity before routing.
5. Log everything and feed results back upstream.

The right validation approach depends on three organisational conditions: data volume, the number of source systems and the technical maturity of the team managing the pipeline. Applying an enterprise-grade rule engine to a single-source, low-volume feed is overengineering. Applying manual spot-checks to a multisource, real-time pipeline is underengineering.

Decision framework:

• If your organisation ingests data from a single source at low volume (under 100,000 records per batch), manual or spreadsheet-based validation with documented rules is sufficient to start. The priority at this stage is rule documentation, not tooling.
• If your organisation ingests from multiple sources with different schemas, schema-level data validation tools with format and consistency rule libraries are necessary to prevent cross-source conflicts. Without schema enforcement, two sources can define "customer_id" differently and the conflict will not surface until a join fails.
• If your organisation operates real-time event streams (clickstream, transaction data), streaming validation that checks records at the point of collection is required. Batch validation alone allows bad data to enter downstream systems before the next batch window closes, which can mean hours of corrupted records.
• If your organisation manages regulated customer data (health, financial or identity data), validation must include completeness and referential integrity checks tied to compliance-defined field requirements, with immutable audit logs of every validation outcome.

Evaluation checklist for a validation solution:

• Does it support all seven validation techniques (type, range, format, consistency, uniqueness, completeness, referential integrity)?
• Does it validate data at the schema level before data lands in storage?
• Does it produce machine-readable failure logs?
• Does it integrate with the team's existing data modelling and data transformation workflows?
• Does it scale to the organisation's peak ingestion volume without introducing latency?

Adobe Experience Platform applies validation at the schema layer using the Experience Data Model standard, which enforces field type, format and completeness rules at ingestion, before data enters the unified profile. For enterprises managing customer data across multiple channels and source systems, this embedded validation reduces the manual rule-writing burden and provides a standardised ruleset aligned with common customer data structures. Data modelling decisions made upstream in the Experience Data Model schema directly determine which validation rules apply, making schema design and validation inseparable at enterprise scale.

For teams not yet at enterprise scale, the decision framework above still applies: start with documented rules, profile your data before deploying checks and build toward automation as volume and source complexity grow. Data hygiene best practices, including consistent rule documentation, closed-loop feedback to source teams and regular rule audits, remain the foundation regardless of tooling.

Key takeaway: Choose your validation approach based on data volume, source count and compliance requirements. Then select tooling that fits, not the reverse.