seon99il

Data Engineering

Great Expectations (GX): to check your data quality

7 min read

Great Expectations
Great Expectations

Data Quality Check, in its simplest terms, is the process of verifying a fundamental question: “Is our data reliable and fit for use?”

No matter how sophisticated your dashboards or how advanced your machine learning models are, the output becomes worthless if the underlying data is riddled with gaps or inconsistent formats. This is the classic “Garbage In, Garbage Out” problem. However, ensuring this integrity at the scale of Big Data is a significant challenge, especially when it comes to visual verification.

1. The 6 Core Dimensions of Data Quality

When we say data is “high quality,” we typically evaluate it based on these six foundational pillars:

  1. Accuracy
    • Question: Does the data reflect real-world facts?
    • Example: Is a customer’s age recorded as 200?
  2. Completeness
    • Question: Is all the required data present without missing values?
    • Example: Is the percentage of “Null” values within an acceptable threshold?
  3. Consistency
    • Question: Is the information uniform across different systems or tables?
    • Example: Does an “Order Completed” status in the sales table appear as “Payment Canceled” in the accounting table?
  4. Validity
    • Question: Does the data adhere to defined formats or business rules?
    • Example: Does an email address include an ”@” symbol, or is the date formatted as YYYY-MM-DD?
  5. Timeliness
    • Question: Is the data up-to-date and available when needed?
    • Example: Does today’s dashboard accurately reflect the transactions that occurred yesterday?
  6. Uniqueness
    • Question: Are there any redundant or duplicate records?
    • Example: Was the same transaction ID generated twice for a single purchase?

2. Why Data Quality Matters?

Beyond simply “keeping data clean,” ensuring data quality is critical for several key business reasons:

  • Increased Trust in Decision-Making: Inaccurate data leads to flawed strategic decisions by leadership. Data quality checks serve as a “seal of assurance” for the metrics and insights delivered by the data team, ensuring that every stakeholder can rely on the numbers.
  • Cost Efficiency (Shift-Left Approach): Fixing data errors becomes exponentially more expensive and resource-intensive the further they travel downstream. By implementing proactive checks early in the pipeline, you can prevent costly “data disasters” before they impact end-users.
  • Scalable Pipeline Governance: As data volume grows, manual verification becomes impossible. Leveraging tools like Great Expectations (GX) allows you to automate the validation process, enabling the system to autonomously detect and flag anomalies without human intervention.

3. Where to Implement Data Quality Checks

Data quality isn’t a one-time event; it should be verified at multiple touchpoints as data flows through the pipeline.

  • Ingestion (Entry Stage): Validate the data the moment it is brought in from the source. At this stage, the primary focus is often schema validation — ensuring that data types are correct and the structure matches expectations.
  • Transformation (Processing Stage): As you clean and aggregate data, you must verify the business logic. For example, you might check if the sum of individual transactions matches the total recorded value or if any calculated fields fall within a logical range.
  • Serving (Delivery Stage): This is the final checkpoint before data reaches the end-user or a BI tool. Here, you verify the integrity of the final output to ensure that the processed data is consistent, complete, and ready for decision-making.

To automate these checkpoints and manage them as code (Validation-as-Code), Great Expectations (GX) has emerged as one of the most powerful and popular tools in the modern data stack.

Core Components of Great Expectations (GX)

To effectively manage data quality, GX is built upon several key components that work together seamlessly:

Components of Great Expectations
Components of Great Expectations

Data Context

The Data Context is the “central control tower” of a GX project.

It manages all project configurations, including settings files, datasource information, and Expectation Suites. It typically exists as a YAML file within the gx/ directory or as a Python object, defining where the data is located and where the results should be stored. You can configure the storage path to cloud services like Amazon S3, which allows validation results to be delivered directly as Data Docs reports.

Datasource

The Datasource defines which data GX will access

It handles connections to databases (PostgreSQL, BigQuery), cloud storage (S3, GCS), or local files (CSV, Parquet). Beyond simply specifying a path, it also determines the execution engine — such as Pandas, Spark, or SQLAlchemy — that will be used to process the data.

Expectations & Expectation Suite

These are the heart of GX and represent the actual validation rules

  • Expectation: A granular “unit rule” defining a specific requirement for your data. You can find a comprehensive list in the GX Expectations Gallery.
  • Expectation Suite: A collection of Expectations bundled into a single test scenario. For example, a “User Profile Validation Suite” might include rules ensuring that IDs are unique and email addresses follow a valid format.

For example, a “User Information Table Validation Suite” would include rules such as unique IDs and email format compliance.

Batch Request

A Batch Request defines exactly “which” data should be retrieved from a Datasource for validation.

It specifies the subset of data (a “Batch”) to be processed, such as a specific database table, a file from a directory, or a filtered slice of data (e.g., records from a specific date). Essentially, it acts as the bridge that selects the target data from the Datasource to be handed over to the validation engine.

Checkpoint

The unit of execution that applies defined rules (Expectation Suites) to actual data (Datasources) to perform validation

It defines: “Which data should be inspected with which rules, and what actions (such as sending alerts) should be taken once the results are generated?”

Example: Validating User Data

Let’s explore the lifecycle of a data validation process using Great Expectations.

# users.csv

user_id,user_name,email,age,signup_date,status
1,Seonggil,seonggil@example.com,28,2024-01-15,active
2,James,james@test.com,34,2024-02-10,active
3,Anna,anna_google.com,22,2024-03-05,pending
4,,secret@hidden.com,-5,2024-03-20,active
5,Chris,chris@example.com,150,2024-04-01,canceled
1,Duplicate,double@test.com,30,2024-04-10,active
6,Olivia,olivia@example.com,25,2025-12-31,unknown
  • Identified Issues:
    • Duplicate user IDs: (user_id: 1)
    • Invalid email format: (user_id: 3, email: anna_google.com)
pip install "great-expectations>=0.18.22,<1.0.0"

Note: Syntax may differ in version 1.x.x or later.

Create Or Get Context

The Data Context manages execution results and test scenarios (checkpoints, expectations, validations) in JSON format. These can be reused based on their assigned names.

import great_expectations as gx

context = gx.get_context()
# context = gx.get_context(project_config=project_config)

Define Datasource

A Datasource contains information regarding “where the data is coming from.”

context.add_datasource(
    name="sample_data",
    class_name="Datasource",
    execution_engine={
        "class_name": "PandasExecutionEngine" # using Pandas
    },
    data_connectors={
        "default_runtime_data_connector_name": {
            "class_name": "RuntimeDataConnector",
            "batch_identifiers": ["default_identifier_name"],
        }
    },
)

Create Expectations & Suite

You can apply specific Expectation Types to your data source. Explore the GX Expectations Gallery for more rules

GX Expectations Gallery — GX’s Expectation Gallery: a rich, collaboration-ready vocabulary for data quality. Start exploring!

from great_expectations.core import ExpectationConfiguration

suite_name = "users_df_suite"

# Validate User Id uniqueness
ex1 = ExpectationConfiguration(
        expectation_type="expect_column_values_to_be_unique",
        kwargs={"column": "user_id"},
      )

# Validate Email format via Regex
email_regex = r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"
ex2 = ExpectationConfiguration(
    expectation_type="expect_column_values_to_match_regex",
    kwargs={
        "column": "email",
        "regex": email_regex
    },
)

# Apply Suite to Context
context.add_or_update_expectation_suite(
    suite_name,
    expectations=[ex1, ex2]
)

# Verify the suite
context.list_expectation_suites()
# [ExpectationSuiteIdentifier::users_df_suite]
context.list_expectation_suites()
# [ExpectationSuiteIdentifier::users_df_suite]

print(suite)

# {
#     "expectation_suite_name": "users_df_suite",
#     "ge_cloud_id": null,
#     "expectations": [
#         {
#             "expectation_type": "expect_column_values_to_be_unique",
#             "kwargs": {
#                 "column": "id"
#             },
#             "meta": {}
#         },
#         {
#             "expectation_type": "expect_column_values_to_match_regex",
#             "kwargs": {
#                 "column": "email",
#                 "regex": "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$"
#             },
#             "meta": {}
#         }
#     ],
#     "data_asset_type": null,
#     "meta": {
#         "great_expectations_version": "0.18.22"
#     }
# }

Batch Request

A Batch Request defines the unit of data to be validated

import pandas as pd
from great_expectations.core.batch import RuntimeBatchRequest

target_df = pd.read_csv("./data/users.csv")
data_asset_name = "users_info_asset"  # Name of the validation target

batch_request = RuntimeBatchRequest(
    datasource_name="sample_data",
    data_connector_name="default_runtime_data_connector_name",
    data_asset_name=data_asset_name,
    runtime_parameters={"batch_data": target_df}, # DF loaded at runtime
    batch_identifiers={
        "default_identifier_name": "default_id"  # Using "default_id" as identifier
    },
)
  • data_asset_name: Defines “what” the data is (e.g., users)
  • batch_identifiers: Defines the specific characteristics of the current batch (e.g., user data loaded on 2024–01–01)

Checkpoint

By defining a Checkpoint, you can group which BatchRequest to validate against which ExpectationSuite, and determine subsequent Actions (e.g., updating docs, sending alerts).

checkpoint_name = "sample_checkpoint"

checkpoint = context.add_or_update_checkpoint(
        name=checkpoint_name,
        validations=[ # Multiple suites can be applied
            {
                "expectation_suite_name": suite_name,
            }
        ],
        action_list=[
            {
                "name": "store_validation_result",
                "action": {"class_name": "StoreValidationResultAction"},
            },
            {
                "name": "update_data_docs",
                "action": {"class_name": "UpdateDataDocsAction"},
            },
        ],
    )

# Run Checkpoint
checkpoint.run(
    run_name="sample_checkpoint_at_2026_04_18",
    batch_request=batch_request,
)

Result (Data Docs)

Results are rendered in HTML, providing a user-friendly interface to review validation outcomes.

users_df_suite
users_df_suite
Validation Results
Validation Results
Validation Result Detail
Validation Result Detail

You can confirm that it accurately validates the invalid email format (anna_google.com) and the duplicate user ID (Non-Unique ID: 1) within the previously defined CSV.

Previous posts

View all posts →

Comments