feat: Add request body compression with optional brotli by mixalturek · Pull Request #927 · apify/apify-client-python

mixalturek · 2026-07-08T09:37:16Z

https://github.com/apify/apify-core/pull/28971 added support for brotli compression to BE.
Pros: higher compression, cons: more CPU intensive.
The brotli dependencies are defined as optional, one has to explicitly enable it and choose one. If none is available, the code falls back to gzip.
JS client doesn't compress requests that are too small, this Python client compresses just everything. No change done, I only noticed and stating it.

Closes: #942

Related PRs

codecov · 2026-07-08T09:40:08Z

Codecov Report

❌ Patch coverage is 97.11934% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.59%. Comparing base (7f61a10) to head (c810cfa).
⚠️ Report is 36 commits behind head on master.

Files with missing lines	Patch %	Lines
src/apify_client/_utils/http.py	86.36%	3 Missing ⚠️
src/apify_client/_utils/encoding.py	95.00%	2 Missing ⚠️
src/apify_client/types.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #927      +/-   ##
==========================================
+ Coverage   94.54%   94.59%   +0.04%     
==========================================
  Files          48       58      +10     
  Lines        5100     5239     +139     
==========================================
+ Hits         4822     4956     +134     
- Misses        278      283       +5

Flag	Coverage Δ
integration	`92.40% <80.65%> (-0.52%)`	⬇️
unit	`83.73% <97.11%> (+0.36%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- apify/apify-core#28971 added support for brotli compression to BE. - Pros: higher compression, cons: more CPU intensive. - The brotli dependencies are defined as optional, one has to explicitly enable it and choose one. If none is available, the code falls back to gzip. - JS client doesn't compress requests that are too small, this Python client compresses just everything. No change done, I only noticed and stating it.

vdusek · 2026-07-08T13:37:11Z

This has a significant impact. @Pijukatel, please check it as well.

- Remove `brotlicffi`, only CPython runtime is supported. - Add a test with bytearray as an input. This would break only in `brotlicffi`, which is no longer in the dependencies. `brotli` dependency handles it correctly and doesn't throw. - Add `brotli` to dev dependencies and use it directly in tests.

- apify/apify-client-python#927 (comment) - The default was 11 (max). Now it uses quality 6 - roughly 2–4× faster with only a modest increase in compressed size.

mixalturek · 2026-07-08T15:37:45Z

@vdusek Nearly all comments are fixed now, thanks again for your great review.

One more thing: I forgot to mention that the JS client skips compression if the payload size is below 1024 B. Do we want to apply it to the Python client as well?

- apify/apify-client-python#927 (comment) - The default was 11 (max). Now it uses quality 6 - roughly 2–4× faster with only a modest increase in compressed size.

vdusek · 2026-07-09T07:01:44Z

One more thing: I forgot to mention that the JS client skips compression if the payload size is below 1024 B. Do we want to apply it to the Python client as well?

Thanks for noticing. I opened an issue for that: #934. Let's resolve it in a separate PR so it is documented in the changelog.

Pijukatel

Not sure about the non-default compression quality , but it is not blocking for me.

vdusek

I removed brotli from the dev dependency group. Sorry, my previous comment wasn't entirely correct. We run uv sync --all-extras everywhere in CI, so there's no need to include it there. The initial mypy/ty override ignores confused me, but they were probably added based on your local setup, where you didn't have the brotli extra installed.

Other than that, one more thing: the documentation 🙂.

Please extend the installation section in the docs here: https://docs.apify.com/api/client/python/docs#installation. You can use a similar section to the one you added to the README.

Also, please create a new concept page for compression. Something like: explaining HTTP compression, how to switch between compression methods, and the pros and cons of each approach.

- apify/apify-core#28971 added support for brotli compression to BE. - Pros: higher compression, cons: more CPU intensive. - The code could be running on too old Node.js, so brotli compression is applied only if possible; the code otherwise falls back to gzip. Too small payloads and unsupported types are still not compressed - no change. ### Related PRs - apify/apify-core#28971 - apify/apify-docs#2750 - #962 - apify/apify-client-python#927 - apify/apify-sdk-python#1031

…dules

vdusek

I did a few additional improvements. @Pijukatel do you want to re-check it?

* feat: Add request body compression with optional brotli (apify#927) ## Description - https://github.com/apify/apify-core/pull/28971 added support for brotli compression to BE. - Pros: higher compression, cons: more CPU intensive. - The brotli dependencies are defined as optional, one has to explicitly enable it and choose one. - JS client doesn't compress requests that are too small, this Python client compresses just everything. No change done, I only noticed and stating it. ## Issues Closes: apify#942 ## Related PRs - https://github.com/apify/apify-core/pull/28971 - apify/apify-docs#2750 - apify/apify-client-js#962 - apify#927 - apify/apify-sdk-python#1031 --------- Co-authored-by: Vlada Dusek <v.dusek96@gmail.com> * chore(release): Update changelog and package version [skip ci] * chore: Automatic docs theme update [skip ci] * test: Deflake test_schedule_list and test_task_list by polling eventually consistent listings (apify#951) `test_schedule_list` and `test_task_list` failed in CI ([schedule run](https://github.com/apify/apify-client-python/actions/runs/29497290807/job/87617167353), [task run](https://github.com/apify/apify-client-python/actions/runs/29498108478/job/87619855853)) because they assert read-your-write on listing endpoints: they list resources immediately after creating them, and under load the listing can serve a view that hasn't yet caught up with the creates, so the fresh IDs are sometimes missing. The creates themselves succeeded in both cases, so these are eventual-consistency flakes, not client bugs. The fix wraps each list read in the existing `poll_until_condition` helper (30 s ceiling), waiting until the created IDs appear in the listing. The original assertions still run on the final page, so a real regression still fails. Follows the same deflaking pattern as apify#824, apify#831, apify#844, and apify#868. * docs: fix input guide example to bound the wait with wait_duration (apify#955) The "Pass input to an Actor" guide example claimed that `timeout=timedelta(seconds=60)` makes `call()` wait up to 60 seconds for the run to finish. It doesn't. On `call()`, `timeout` is the per-request HTTP timeout forwarded to `start()`, while the wait cap is `wait_duration` (default `None`, i.e. wait indefinitely). A user copying the example would block indefinitely on a long or never-finishing run. The sync and async examples now use `wait_duration=timedelta(seconds=60)` and the comment is reworded accordingly. The stale copies under `website/versioned_docs/` are left untouched on purpose. The versioning workflow deletes and re-snapshots `version-3.0` from `docs/` on every 3.x release, so the fix propagates there automatically. * chore: Automatic docs theme update [skip ci] * fix: offload async request body compression to a worker thread (apify#950) The async `ImpitHttpClientAsync.call()` serialized and compressed request bodies inline on the event loop. Compression is CPU-bound (and brotli is becoming the default via the Apify SDK), so it blocked the loop for the whole duration of the compression, stalling every other concurrent task (response reads, the platform events websocket, other in-flight requests). Request preparation is now offloaded to a worker thread via `asyncio.to_thread` whenever there is a body to compress. Bodyless requests (the common polling, listing, and get path) stay inline to avoid a needless thread-dispatch hop. The synchronous client is unchanged, as it has no event loop to block. * chore(release): Update changelog and package version [skip ci] * fix: Propagate last_run status/origin filters to chained storage clients (apify#954) The `status` and `origin` filters passed to `ActorClient.last_run(...)` and `TaskClient.last_run(...)` were silently dropped by the chained storage clients. For example, `actor.last_run(status='SUCCEEDED').dataset().list_items()` queried the most recent run's dataset regardless of status. This is a regression vs 1.x, where `_sub_resource_init_options` forwarded the parent's params to every child client; the typed-client refactor (apify#604) lost that. The fix passes the run client's default params to its four child-client factories (`dataset()`, `key_value_store()`, `request_queue()`, `log()`) in both `RunClient` and `RunClientAsync`, so the filters ride along on the `runs/last` storage endpoints again, matching 1.x and the JS client. Adds a parametrized regression test covering all four chained clients, sync and async. All 8 cases fail before the fix and pass after. * chore(release): Update changelog and package version [skip ci] * chore: Automatic docs theme update [skip ci] * fix: Propagate API token to custom HTTP clients (apify#956) `ApifyClient.with_custom_http_client(token=...)` (and the async twin) stored the token on the `ApifyClient` instance but never passed it to the injected HTTP client, so no request carried an `Authorization` header and every call failed with 401. The documented custom HTTP client example inherited the bug. - `with_custom_http_client` now sets `Authorization: Bearer <token>` on the injected client's default headers, unless the client already has an auth header configured (checked case-insensitively). - `HttpClientBase._prepare_request_call` now merges the client's default headers under the per-request headers, so any custom client using the helper (including a pre-built `ImpitHttpClient` passed as the custom client) actually sends them. For the default client the wire behavior is unchanged, since impit request-level headers replace the identical client-level ones. - The HTTPX guide examples now merge `self._headers` before delegating, and the `HttpClient` ABC docstring states that implementations must send the default headers with every request. Regression tests cover the token reaching the wire (custom sync/async clients and a pre-built tokenless `ImpitHttpClient`) and the no-clobber semantics for client-configured auth headers. * chore(release): Update changelog and package version [skip ci] * fix: Make batch_add_requests split batches by serialized payload size (apify#953) The 9 MB payload guard in `batch_add_requests` was inert: `constrained_batches` was called without `get_len`, so the default `len()` measured each request dict's key count (~4) instead of its serialized size. Batches were therefore split only by the 25-request count limit, and large requests shipped as one oversized POST that the API rejects with 413, failing the whole call. The guard now measures each request as its UTF-8 JSON byte length, using the same serialization flags as the HTTP client's request body path. It also passes `strict=False`, which preserves the previous contract for an individually oversized request: it's sent in its own batch and left for the API to judge, instead of raising a client-side `ValueError`. Both the size-based splitting and the oversized-singleton path are covered by new sync/async regression tests. *✍️ Drafted by Claude Code* * chore(release): Update changelog and package version [skip ci] * docs: Fix documentation mismatches with actual client behavior (apify#958) Fixes 12 documentation issues found by an engineering audit, where docs, examples, or docstrings contradict what the client actually does: - The custom HTTP client guide example (`HttpxClient`) now raises `ApifyApiError` for error responses instead of returning them raw, and the guide documents this part of the `call` contract (resource clients rely on it, e.g. to translate a 404 into a `None` return value). - The streaming concepts page no longer claims all three streaming methods yield a raw `impit.Response` — it now describes the actual yielded value per method (`stream_record` yields a `dict` with the response under `value`, `stream` and `stream_record` may yield `None`). - The pagination concepts page no longer lists `ListOfRequests` among page models exposing `total`/`offset`/`count`; it now explains its cursor-based pagination via `next_cursor`. - The logging formatter example no longer references `%(status_code)s` (absent on most records, causing logging errors) and no longer attaches a duplicate handler; the page notes which properties are present on every record. - The upgrading-to-v3 guide cross-links now include the site baseUrl (`/api/client/python/...`), fixing 6 links that 404ed on the published site. - The conda instruction for the brotli extra installs `brotli-python` (the Python bindings) instead of `brotli` (the C library). - `RunClient.resurrect` docstrings cite the real `SUCCEEDED` status instead of the nonexistent `FINISHED`. - `wait_for_finish` docstrings in `RunClient` and `BuildClient` spell the terminal status as `TIMED-OUT` (the real literal) instead of `TIMED_OUT`. - The quick-start page refers to the `Run` model's `default_dataset_id` attribute instead of the v2-era run dictionary with `defaultDatasetId`. - The README dataset example passes `fields` as `list[str]` per the signature instead of a comma-separated string. - The README quick-start examples handle the `Run | None` return of `call()` instead of accessing attributes on a possible `None`. - The timeouts concepts page states that `no_timeout` is capped at 24 hours by the default client instead of claiming it disables the timeout entirely. *✍️ Drafted by Claude Code* * chore(release): Update changelog and package version [skip ci] * docs: Version docs for v3.1.0 [skip ci] * chore: Automatic docs theme update [skip ci] * chore: Automatic docs theme update [skip ci] * fix: Add missing cannot-monetize-without-payout-billing-info error code (apify#960) - Updates the auto-generated Pydantic models and TypedDicts based on the proposed OpenAPI specification changes. - Based on apify-docs PR [#2785](apify/apify-docs#2785). * chore(release): Update changelog and package version [skip ci] * chore(deps): update pnpm to v11.15.1 (apify#967) * chore(deps): update dependency oxfmt to ^0.59.0 (apify#966) * chore(deps): lock file maintenance (apify#968) * fix: Normalize query params in dataset create_items_public_url (apify#963) `DatasetClient.create_items_public_url` (and its async twin) passed `_build_params` output straight to `urlencode`, bypassing the `_parse_params` normalization that the real HTTP request path applies. As a result, boolean and list query params ended up as Python reprs in the signed URL — e.g. `create_items_public_url(clean=True, fields=['title', 'url'])` produced `clean=True&fields=%5B%27title%27%2C+%27url%27%5D` instead of `clean=true&fields=title,url`. Consumers of the shared URL then got unclean/unfiltered items or an API error. The fix routes the params through `self._http_client._parse_params(...)` before `urlencode`, so the public URL matches exactly what an actual API request would send (bool→`true`/`false`, list→comma-joined, `None` dropped). Added regression tests for both the sync and async clients. *✍️ Drafted by Claude Code* * chore(release): Update changelog and package version [skip ci] * chore(deps): update actions/setup-node action to v7 (apify#969) * chore(deps): update actions/setup-python action to v7 (apify#970) * docs: Remove conda special instructions for brotli (apify#959) It is now included by default: https://github.com/conda-forge/apify-client-feedstock/pull/4/changes * docs: Update description of `request_url` in `Webhooks` (apify#974) - Updates the auto-generated Pydantic models and TypedDicts based on the proposed OpenAPI specification changes. - Based on apify-docs PR [#2794](apify/apify-docs#2794). --------- Co-authored-by: Michal Turek <michal.turek@apify.com> Co-authored-by: Vlada Dusek <v.dusek96@gmail.com> Co-authored-by: Apify Service Account <64261774+apify-service-account@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Josef Procházka <josef.prochazka@apify.com>

mixalturek force-pushed the brotli-compression branch 2 times, most recently from 49bf689 to aabd87c Compare July 8, 2026 10:05

mixalturek force-pushed the brotli-compression branch from aabd87c to f26780a Compare July 8, 2026 10:25

This was referenced Jul 8, 2026

feat: Compress requests using brotli algo apify/apify-client-js#962

Merged

feat: Document support for brotli compression apify/apify-docs#2750

Merged

mixalturek requested review from nmanerikar and vdusek July 8, 2026 11:04

mixalturek mentioned this pull request Jul 8, 2026

feat: Compress requests using brotli algo apify/apify-sdk-python#1031

Closed

vdusek requested changes Jul 8, 2026

View reviewed changes

vdusek requested a review from Pijukatel July 8, 2026 13:36

mixalturek added 3 commits July 8, 2026 17:10

feat: Refactoring of tests

0dd99a9

feat: Use brotli quality=6; lower but faster

3fffa62

vdusek mentioned this pull request Jul 9, 2026

Skip request-body compression for small payloads #934

Open

Pijukatel approved these changes Jul 9, 2026

View reviewed changes

Comment thread README.md Outdated

rm brotli from dev deps

37f5a47

vdusek reviewed Jul 9, 2026

View reviewed changes

vdusek changed the title ~~feat: Compress requests using brotli algo~~ feat: Add request body compression with optional brotli Jul 9, 2026

feat: Documentation for the compression

7617ed0

mixalturek requested a review from szaganek as a code owner July 9, 2026 11:31

feat: Formatting

28f34e2

feat: Polishing

25d7afa

mixalturek marked this pull request as ready for review July 14, 2026 13:15

mixalturek requested a review from vdusek July 14, 2026 13:15

mixalturek self-assigned this Jul 14, 2026

vdusek added 4 commits July 14, 2026 19:47

refactor: Split _utils module into a _utils subpackage

a0650b6

refactor: Delegate compressor quality validation to the underlying mo…

f7f6a34

…dules

docs: Polish the compression concept guide

9fc37df

test: Add compressor tests and parametrize charge tests by encoding

0cf66e3

github-actions Bot added this to the 145th sprint - Core Services team milestone Jul 14, 2026

github-actions Bot added the t-core-services Issues with this label are in the ownership of the core services team. label Jul 14, 2026

vdusek added 6 commits July 14, 2026 20:50

refactor: Unify compression naming and extract compressor resolution

0a76634

test: Align compressor tests with renamed param and terminology

29f5315

docs: Rename compression page to HTTP compression and trim install docs

1feee71

build: Format brotli optional-dependency on a single line

4674110

feat: Validate compressor quality range and fail fast

ff4f21e

test: Use pytest.param with explicit ids in compression tests

a90bb02

vdusek added t-tooling Issues with this label are in the ownership of the tooling team. adhoc Ad-hoc unplanned task added during the sprint. labels Jul 14, 2026

vdusek approved these changes Jul 14, 2026

View reviewed changes

szaganek approved these changes Jul 15, 2026

View reviewed changes

Comment thread docs/02_concepts/13_http_compression.mdx Outdated

Comment thread docs/02_concepts/13_http_compression.mdx Outdated

Comment thread docs/02_concepts/13_http_compression.mdx Outdated

vdusek added 5 commits July 15, 2026 09:50

docs: Apply review suggestions to HTTP compression page

72c4e74

refactor: Validate compression eagerly at client construction

dfbaca0

refactor: Align try_import with crawlee-python

6596cef

style: Keep future annotations import in try_import

273c09f

test: Add missing future annotations import to unit tests

c810cfa

vdusek merged commit a8a393b into master Jul 15, 2026
30 checks passed

vdusek deleted the brotli-compression branch July 15, 2026 08:21

vdusek mentioned this pull request Jul 15, 2026

Adopt apify-client v3.1 and enable brotli compression by default apify/apify-sdk-python#1045

Closed

Pijukatel mentioned this pull request Jul 18, 2026

Update to enable extras brotli compression conda-forge/apify-client-feedstock#3

Open

Uh oh!

Conversation

mixalturek commented Jul 8, 2026 • edited by vdusek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related PRs

Uh oh!

codecov Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vdusek commented Jul 8, 2026

Uh oh!

mixalturek commented Jul 8, 2026

Uh oh!

vdusek commented Jul 9, 2026

Uh oh!

Pijukatel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mixalturek commented Jul 8, 2026 •

edited by vdusek

Loading

codecov Bot commented Jul 8, 2026 •

edited

Loading