Data Validation with PyArrow

new in 0.33.0

PyArrow tables are a common exchange format between systems, and pyarrow.Table carries its own schema. Pandera supports validating them directly so you get dtype checking, value checks and class-based models on top of that schema.

Installation

pip install 'pandera[pyarrow]'

Validation is performed by pandera’s narwhals backend, so narwhals is installed alongside pyarrow.

DataFrameSchema

Import the pyarrow entry point and define a schema as you would for any other backend:

import pyarrow
import pandera.pyarrow as pa

schema = pa.DataFrameSchema(
    {
        "state": pa.Column(str),
        "city": pa.Column(str),
        "price": pa.Column(int, pa.Check.in_range(5, 20)),
    }
)

table = pyarrow.table(
    {
        "state": ["FL", "FL", "CA", "CA"],
        "city": ["Orlando", "Miami", "Los Angeles", "San Francisco"],
        "price": [8, 12, 10, 16],
    }
)

schema.validate(table)

pyarrow.Table
state: string
city: string
price: int64
----
state: [["FL","FL","CA","CA"]]
city: [["Orlando","Miami","Los Angeles","San Francisco"]]
price: [[8,12,10,16]]

validate returns a pyarrow.Table, so schemas drop into an existing pipeline without changing types.

DataFrameModel

class Schema(pa.DataFrameModel):
    state: str
    city: str
    price: int = pa.Field(in_range={"min_value": 5, "max_value": 20})


Schema.validate(table)

pyarrow.Table
state: string
city: string
price: int64
----
state: [["FL","FL","CA","CA"]]
city: [["Orlando","Miami","Los Angeles","San Francisco"]]
price: [[8,12,10,16]]

Annotate function signatures with pandera.typing.pyarrow.Table and use check_types() to validate inputs and outputs:

from pandera.typing.pyarrow import Table


@pa.check_types
def transform(df: Table[Schema]) -> Table[Schema]:
    return df


transform(table)

pyarrow.Table
state: string
city: string
price: int64
----
state: [["FL","FL","CA","CA"]]
city: [["Orlando","Miami","Los Angeles","San Francisco"]]
price: [[8,12,10,16]]

Supported data types

Columns accept native pyarrow types, python builtins, and their string aliases — all resolve to the same underlying pandera datatype:

pa.DataFrameSchema(
    {
        "a": pa.Column(pyarrow.int64()),
        "b": pa.Column(int),
        "c": pa.Column("int64"),
    }
)

<Schema DataFrameSchema(columns={'a': <Schema Column(name=a, type=DataType(Int64))>, 'b': <Schema Column(name=b, type=DataType(Int64))>, 'c': <Schema Column(name=c, type=DataType(Int64))>}, checks=[], parsers=[], index=None, dtype=None, coerce=False, strict=False, name=None, ordered=False, unique=None, report_duplicates=all, unique_column_names=False, add_missing_columns=False, title=None, description=None, metadata=None, drop_invalid_rows=False)>

Parametrized pyarrow types such as pyarrow.timestamp("us"), pyarrow.decimal128(10, 2) and pyarrow.list_(pyarrow.int32()) are supported.

Custom checks

Check functions receive a PyArrowData container holding the native table and the column key, mirroring PolarsData and IbisData on the other backends:

import pyarrow.compute as pc

schema = pa.DataFrameSchema(
    {"price": pa.Column(int, pa.Check(lambda data: pc.greater(data.table[data.key], 0)))}
)
schema.validate(table)

pyarrow.Table
state: string
city: string
price: int64
----
state: [["FL","FL","CA","CA"]]
city: [["Orlando","Miami","Los Angeles","San Francisco"]]
price: [[8,12,10,16]]

A check function taking two positional arguments receives the native table and the key directly, i.e. check_fn(table, key).

Validation depth

Unlike a polars.LazyFrame or an ibis.Table, a pyarrow.Table is always fully materialized in memory, so pandera runs both schema-level and data-level checks by default. Set PANDERA_VALIDATION_DEPTH=SCHEMA_ONLY (or use config_context()) to skip data-level checks.

Limitations

• coerce=True is not applied. Coercion is not yet implemented in the narwhals backend that serves pyarrow; a SchemaWarning is emitted and a dtype mismatch is reported as a WRONG_DATATYPE error instead.

• Data synthesis strategies are not supported, matching the polars and ibis backends.