Dependency version mismatch on pyarrow

[naoki-jarvisml]

version 3.22.0 uses ArrowDtype which is only available in pyarrow>=10.0.1, but dependency requirement specifies pyarrow >= 3.0.0.

Environment details

• OS type and version: Darwin mac-mini.lan 20.6.0 Darwin Kernel Version 20.6.0 (also on GCP AppEngine)
• Python version: Python 3.10.11
• pip version: pip 23.3.1
• google-cloud-bigquery version: 3.22.0

Steps to reproduce

note: google-cloud-aiplatform requires pyarrow 7.0.0 for python 3.10

1. install google-cloud-bigquery and google-cloud-aiplatform db-dtypes
2. call to_dataframe on RowIterator

Code example

# example
client = bigquery.Client()
df = bqclient.query(sql_statement).result().to_dataframe()

Stack trace

File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/cloud/bigquery/table.py", line 2283, in to_dataframe
    range_date_dtype = pandas.ArrowDtype(
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/pandas/core/dtypes/dtypes.py", line 2096, in __init__
    raise ImportError("pyarrow>=10.0.1 is required for ArrowDtype")
ImportError: pyarrow>=10.0.1 is required for ArrowDtype"

[tswast]

What version of pandas do you have?

I suspect older versions of pandas may be compatible with older pyarrow.

[naoki-jarvisml]

I have pandas 2.2.2. I've reported this issue since pip's dependency resolution didn't prevent this issue from happening. I'd expect google-cloud-bigquery 3.22.0 to require pyarrow>=10.0.1, or older version of pandas which maybe compatible with this change assuming you're right about it.

[tswast]

We test with minimum versions of all of our dependencies to make sure they are compatible and the latest versions of all of our dependencies.

For the in-betweens we have some version detection logic throughout the code, but it sounds like we need to extend it for this case.

[Linchin]

Thanks for raising the issue! Indeed I only accounted for when pandas is an older version. I will add a min version check for pyarrow too.