Allow pandas 3.x in dependency constraints

[moomindani]

Summary

Relax the pandas version upper bound from <2.4.0 to <4.0.0 to allow pandas 3.x alongside databricks-sql-connector.

The pandas APIs used in this project (nullable extension dtypes like Int64Dtype, StringDtype, DataFrame.to_numpy, PyArrow-to-pandas conversion via types_mapper) are all compatible with pandas 3.0. The key behavioral change in pandas 3 — StringDtype() defaulting to PyArrow-backed storage — does not affect this code because results are immediately converted to numpy/Python objects via to_numpy(na_value=None, dtype="object").

Closes #732

Changes

• pyproject.toml: Raise pandas upper bound from <2.4.0 to <4.0.0 for both Python version groups
• tests/unit/test_pandas_compatibility.py: New test suite for _convert_arrow_table covering all mapped data types

Test Results

All existing and new tests pass on both pandas 2.x and pandas 3.x:

Environment	Existing unit tests	New pandas compat tests
pandas 3.0.2 + pyarrow 23.0.1 (Python 3.14.3)	575 passed, 4 skipped	10 passed
pandas 2.3.3 + pyarrow 23.0.1 (Python 3.14.3)	575 passed, 4 skipped	10 passed

New test cases (test_pandas_compatibility.py)

Tests exercise ResultSet._convert_arrow_table() with the following scenarios:

• test_integer_types — int8, int16, int32, int64 columns with nulls
• test_unsigned_integer_types — uint8, uint16, uint32, uint64 columns with nulls
• test_float_types — float32, float64 columns with nulls
• test_boolean_type — boolean column with nulls
• test_string_type — string column with nulls (validates StringDtype behavior change in pandas 3)
• test_mixed_types — table with int64, string, float64, bool columns simulating real query results
• test_duplicate_column_names — verifies the rename-before-to_pandas workaround
• test_empty_table — empty Arrow table
• test_all_nulls — all-null columns for int and string types
• test_disable_pandas_path — non-pandas code path (_disable_pandas=True)

[moomindani]

Friendly ping for review — this unblocks pandas 3.x users (see #732, which has multiple +1s including Ibis).

cc @jprakash-db @vikrantpuppala @tejassp-db — would any of you be able to take a look or assign a reviewer?

Summary of validation

All existing and new tests pass on both pandas 2.x and pandas 3.x:

Environment	Existing unit tests	New pandas compat tests
pandas 3.0.2 + pyarrow 23.0.1 (Python 3.14.3)	575 passed, 4 skipped	10 passed
pandas 2.3.3 + pyarrow 23.0.1 (Python 3.14.3)	575 passed, 4 skipped	10 passed

The pandas 3 behavioral change (StringDtype() defaulting to PyArrow-backed storage) does not affect this code because results are immediately converted to numpy/Python objects via to_numpy(na_value=None, dtype="object"). Test coverage explicitly validates this in test_string_type and test_all_nulls.

Happy to address any feedback. Thanks!

[dhirschfeld-ffma]

Could a maintainer please take a look.

[vikrantpuppala]

Thanks for the contribution! Mostly LGTM, with a few minor comments

[vikrantpuppala]

Bypassing __init__ via object.__new__ works because _convert_arrow_table only reads self.description and self.connection.disable_pandas but it might be fragile.

Can we construct via the normal constructor with mocked args as done here: https://github.com/databricks/databricks-sql-python/blob/main/tests/unit/test_client.py#L187-L196

[vikrantpuppala]

Can we perhaps add tests for these datatypes as well:

• pa.decimal128
• pa.date32 / pa.date64
• pa.timestamp
• pa.binary / pa.large_string.
• pa.list_ / pa.struct / pa.map_