Great Expectations (GX): to check your data quality
7 min read

Data Quality Check, in its simplest terms, is the process of verifying a fundamental question: “Is our data reliable and fit for use?”
No matter how sophisticated your dashboards or how advanced your machine learning models are, the output becomes worthless if the underlying data is riddled with gaps or inconsistent formats. This is the classic “Garbage In, Garbage Out” problem. However, ensuring this integrity at the scale of Big Data is a significant challenge, especially when it comes to visual verification.
1. The 6 Core Dimensions of Data Quality
When we say data is “high quality,” we typically evaluate it based on these six foundational pillars:
- Accuracy
- Question: Does the data reflect real-world facts?
- Example: Is a customer’s age recorded as 200?
- Completeness
- Question: Is all the required data present without missing values?
- Example: Is the percentage of “Null” values within an acceptable threshold?
- Consistency
- Question: Is the information uniform across different systems or tables?
- Example: Does an “Order Completed” status in the sales table appear as “Payment Canceled” in the accounting table?
- Validity
- Question: Does the data adhere to defined formats or business rules?
- Example: Does an email address include an ”@” symbol, or is the date formatted as
YYYY-MM-DD?
- Timeliness
- Question: Is the data up-to-date and available when needed?
- Example: Does today’s dashboard accurately reflect the transactions that occurred yesterday?
- Uniqueness
- Question: Are there any redundant or duplicate records?
- Example: Was the same transaction ID generated twice for a single purchase?
2. Why Data Quality Matters?
Beyond simply “keeping data clean,” ensuring data quality is critical for several key business reasons:
- Increased Trust in Decision-Making: Inaccurate data leads to flawed strategic decisions by leadership. Data quality checks serve as a “seal of assurance” for the metrics and insights delivered by the data team, ensuring that every stakeholder can rely on the numbers.
- Cost Efficiency (Shift-Left Approach): Fixing data errors becomes exponentially more expensive and resource-intensive the further they travel downstream. By implementing proactive checks early in the pipeline, you can prevent costly “data disasters” before they impact end-users.
- Scalable Pipeline Governance: As data volume grows, manual verification becomes impossible. Leveraging tools like Great Expectations (GX) allows you to automate the validation process, enabling the system to autonomously detect and flag anomalies without human intervention.
3. Where to Implement Data Quality Checks
Data quality isn’t a one-time event; it should be verified at multiple touchpoints as data flows through the pipeline.
- Ingestion (Entry Stage): Validate the data the moment it is brought in from the source. At this stage, the primary focus is often schema validation — ensuring that data types are correct and the structure matches expectations.
- Transformation (Processing Stage): As you clean and aggregate data, you must verify the business logic. For example, you might check if the sum of individual transactions matches the total recorded value or if any calculated fields fall within a logical range.
- Serving (Delivery Stage): This is the final checkpoint before data reaches the end-user or a BI tool. Here, you verify the integrity of the final output to ensure that the processed data is consistent, complete, and ready for decision-making.
To automate these checkpoints and manage them as code (Validation-as-Code), Great Expectations (GX) has emerged as one of the most powerful and popular tools in the modern data stack.
Core Components of Great Expectations (GX)
To effectively manage data quality, GX is built upon several key components that work together seamlessly:

Data Context
The Data Context is the “central control tower” of a GX project.
It manages all project configurations, including settings files, datasource information, and Expectation Suites. It typically exists as a YAML file within the gx/ directory or as a Python object, defining where the data is located and where the results should be stored. You can configure the storage path to cloud services like Amazon S3, which allows validation results to be delivered directly as Data Docs reports.
Datasource
The Datasource defines which data GX will access
It handles connections to databases (PostgreSQL, BigQuery), cloud storage (S3, GCS), or local files (CSV, Parquet). Beyond simply specifying a path, it also determines the execution engine — such as Pandas, Spark, or SQLAlchemy — that will be used to process the data.
Expectations & Expectation Suite
These are the heart of GX and represent the actual validation rules
- Expectation: A granular “unit rule” defining a specific requirement for your data. You can find a comprehensive list in the GX Expectations Gallery.
- Expectation Suite: A collection of Expectations bundled into a single test scenario. For example, a “User Profile Validation Suite” might include rules ensuring that IDs are unique and email addresses follow a valid format.
For example, a “User Information Table Validation Suite” would include rules such as unique IDs and email format compliance.
Batch Request
A Batch Request defines exactly “which” data should be retrieved from a Datasource for validation.
It specifies the subset of data (a “Batch”) to be processed, such as a specific database table, a file from a directory, or a filtered slice of data (e.g., records from a specific date). Essentially, it acts as the bridge that selects the target data from the Datasource to be handed over to the validation engine.
Checkpoint
The unit of execution that applies defined rules (Expectation Suites) to actual data (Datasources) to perform validation
It defines: “Which data should be inspected with which rules, and what actions (such as sending alerts) should be taken once the results are generated?”
Example: Validating User Data
Let’s explore the lifecycle of a data validation process using Great Expectations.
# users.csv
user_id,user_name,email,age,signup_date,status
1,Seonggil,seonggil@example.com,28,2024-01-15,active
2,James,james@test.com,34,2024-02-10,active
3,Anna,anna_google.com,22,2024-03-05,pending
4,,secret@hidden.com,-5,2024-03-20,active
5,Chris,chris@example.com,150,2024-04-01,canceled
1,Duplicate,double@test.com,30,2024-04-10,active
6,Olivia,olivia@example.com,25,2025-12-31,unknown
- Identified Issues:
- Duplicate user IDs: (
user_id: 1) - Invalid email format: (
user_id: 3,email: anna_google.com)
- Duplicate user IDs: (
pip install "great-expectations>=0.18.22,<1.0.0"
Note: Syntax may differ in version 1.x.x or later.
Create Or Get Context
The Data Context manages execution results and test scenarios (checkpoints, expectations, validations) in JSON format. These can be reused based on their assigned names.
import great_expectations as gx
context = gx.get_context()
# context = gx.get_context(project_config=project_config)
Define Datasource
A Datasource contains information regarding “where the data is coming from.”
context.add_datasource(
name="sample_data",
class_name="Datasource",
execution_engine={
"class_name": "PandasExecutionEngine" # using Pandas
},
data_connectors={
"default_runtime_data_connector_name": {
"class_name": "RuntimeDataConnector",
"batch_identifiers": ["default_identifier_name"],
}
},
)
Create Expectations & Suite
You can apply specific Expectation Types to your data source. Explore the GX Expectations Gallery for more rules
GX Expectations Gallery — GX’s Expectation Gallery: a rich, collaboration-ready vocabulary for data quality. Start exploring!
from great_expectations.core import ExpectationConfiguration
suite_name = "users_df_suite"
# Validate User Id uniqueness
ex1 = ExpectationConfiguration(
expectation_type="expect_column_values_to_be_unique",
kwargs={"column": "user_id"},
)
# Validate Email format via Regex
email_regex = r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"
ex2 = ExpectationConfiguration(
expectation_type="expect_column_values_to_match_regex",
kwargs={
"column": "email",
"regex": email_regex
},
)
# Apply Suite to Context
context.add_or_update_expectation_suite(
suite_name,
expectations=[ex1, ex2]
)
# Verify the suite
context.list_expectation_suites()
# [ExpectationSuiteIdentifier::users_df_suite]
context.list_expectation_suites()
# [ExpectationSuiteIdentifier::users_df_suite]
print(suite)
# {
# "expectation_suite_name": "users_df_suite",
# "ge_cloud_id": null,
# "expectations": [
# {
# "expectation_type": "expect_column_values_to_be_unique",
# "kwargs": {
# "column": "id"
# },
# "meta": {}
# },
# {
# "expectation_type": "expect_column_values_to_match_regex",
# "kwargs": {
# "column": "email",
# "regex": "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$"
# },
# "meta": {}
# }
# ],
# "data_asset_type": null,
# "meta": {
# "great_expectations_version": "0.18.22"
# }
# }
Batch Request
A Batch Request defines the unit of data to be validated
import pandas as pd
from great_expectations.core.batch import RuntimeBatchRequest
target_df = pd.read_csv("./data/users.csv")
data_asset_name = "users_info_asset" # Name of the validation target
batch_request = RuntimeBatchRequest(
datasource_name="sample_data",
data_connector_name="default_runtime_data_connector_name",
data_asset_name=data_asset_name,
runtime_parameters={"batch_data": target_df}, # DF loaded at runtime
batch_identifiers={
"default_identifier_name": "default_id" # Using "default_id" as identifier
},
)
- data_asset_name: Defines “what” the data is (e.g., users)
- batch_identifiers: Defines the specific characteristics of the current batch (e.g., user data loaded on 2024–01–01)
Checkpoint
By defining a Checkpoint, you can group which
BatchRequestto validate against whichExpectationSuite, and determine subsequent Actions (e.g., updating docs, sending alerts).
checkpoint_name = "sample_checkpoint"
checkpoint = context.add_or_update_checkpoint(
name=checkpoint_name,
validations=[ # Multiple suites can be applied
{
"expectation_suite_name": suite_name,
}
],
action_list=[
{
"name": "store_validation_result",
"action": {"class_name": "StoreValidationResultAction"},
},
{
"name": "update_data_docs",
"action": {"class_name": "UpdateDataDocsAction"},
},
],
)
# Run Checkpoint
checkpoint.run(
run_name="sample_checkpoint_at_2026_04_18",
batch_request=batch_request,
)
Result (Data Docs)
Results are rendered in HTML, providing a user-friendly interface to review validation outcomes.



You can confirm that it accurately validates the invalid email format (anna_google.com) and the duplicate user ID (Non-Unique ID: 1) within the previously defined CSV.