pull google-cloud-bigquery-storage and pyarrow into required dependencies

[tswast]

https://github.com/googleapis/python-bigquery-storage/pull/224/files#r667241381

We'll still want to keep bqstorage around in "extras" so that pip install google-cloud-bigquery[bqstorage] still succeeds, but the [bqstorage part will be a no-op.

[tswast]

This will likely result in a lot of code cleanup work to remove dead code paths that assume the BQ Storage client library is not available.

[plamut]

I'll be happy to clean that stuff, I never really liked it. :)

[plamut]

@tswast Do you know how many users use BigQuery without extras AND would not be able to install them if we make them required? In other words, would making some of the extras required require a major version bump?

[tswast]

I'm not too concerned about google-cloud-bigquery-storage, as we already directly depend on grpcio

python-bigquery/setup.py

Line 32 in 5deef6f

"grpcio >= 1.38.1, < 2.0dev", # https://github.com/googleapis/python-bigquery/issues/695

and proto-plus

python-bigquery/setup.py

Line 34 in 5deef6f

"proto-plus >= 1.10.0",

The rest is just pure Python, I believe.

pyarrow is the more concerning dependency. There were problems in the past where Spark had pinned an old version of PyArrow due to wire format changes (https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#compatibility-setting-for-pyarrow-0-15-0-and-spark-2-3-x-2-4-x) It appears they've been able to resolve this with an environment variable, but that'll break BigQuery as it uses the latest wire format version.

[tswast]

Looks like Spark >= 3.0 is using the latest wire format: dotnet/spark#370 (comment)

[plamut]

So the only major concern is users that use Spark < 3.0 alongside BigQuery without extras? And even they can either upgrade Spark, if possible, or cap the BigQuery version as a workaround?

[tswast]

That's the only concern that comes to mind. And yes, they should have workarounds as they can pin older versions that don't need arrow / google-cloud-bigquery-storage.

I suppose the other concern is if someone manually builds their own C++ Arrow library and tries to use it in the same environment as pyarrow that wasn't built with exactly the same compiler settings, but I think that's more a problem with pyarrow + pip than it is something we need to worry about.