test: Fix flaky integration tests asserting membership in capped iterate()#824
Merged
Merged
Conversation
Drop the limit=10 cap so the iterator walks all of the user's queues; under the parallel integration suite, other tests creating queues concurrently could push the just-created ones past the 10-newest window and the inclusion assertion would fail.
vdusek
added
adhoc
github-actions
Bot
added
the
tested
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #824 +/- ##
=======================================
Coverage 94.56% 94.56%
=======================================
Files 48 48
Lines 4929 4929
=======================================
Hits 4661 4661
Misses 268 268
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
vdusek
changed the title
janbuchar
approved these changes
May 27, 2026
vdusek
added a commit
that referenced
this pull request
Jul 16, 2026
…ally consistent listings (#951) `test_schedule_list` and `test_task_list` failed in CI ([schedule run](https://github.com/apify/apify-client-python/actions/runs/29497290807/job/87617167353), [task run](https://github.com/apify/apify-client-python/actions/runs/29498108478/job/87619855853)) because they assert read-your-write on listing endpoints: they list resources immediately after creating them, and under load the listing can serve a view that hasn't yet caught up with the creates, so the fresh IDs are sometimes missing. The creates themselves succeeded in both cases, so these are eventual-consistency flakes, not client bugs. The fix wraps each list read in the existing `poll_until_condition` helper (30 s ceiling), waiting until the created IDs appear in the listing. The original assertions still run on the final page, so a real regression still fails. Follows the same deflaking pattern as #824, #831, #844, and #868.
pittmanbrandon20-design
added a commit
to pittmanbrandon20-design/apify-client-python
that referenced
this pull request
Jul 26, 2026
* feat: Add request body compression with optional brotli (apify#927) ## Description - https://github.com/apify/apify-core/pull/28971 added support for brotli compression to BE. - Pros: higher compression, cons: more CPU intensive. - The brotli dependencies are defined as optional, one has to explicitly enable it and choose one. - JS client doesn't compress requests that are too small, this Python client compresses just everything. No change done, I only noticed and stating it. ## Issues Closes: apify#942 ## Related PRs - https://github.com/apify/apify-core/pull/28971 - apify/apify-docs#2750 - apify/apify-client-js#962 - apify#927 - apify/apify-sdk-python#1031 --------- Co-authored-by: Vlada Dusek <v.dusek96@gmail.com> * chore(release): Update changelog and package version [skip ci] * chore: Automatic docs theme update [skip ci] * test: Deflake test_schedule_list and test_task_list by polling eventually consistent listings (apify#951) `test_schedule_list` and `test_task_list` failed in CI ([schedule run](https://github.com/apify/apify-client-python/actions/runs/29497290807/job/87617167353), [task run](https://github.com/apify/apify-client-python/actions/runs/29498108478/job/87619855853)) because they assert read-your-write on listing endpoints: they list resources immediately after creating them, and under load the listing can serve a view that hasn't yet caught up with the creates, so the fresh IDs are sometimes missing. The creates themselves succeeded in both cases, so these are eventual-consistency flakes, not client bugs. The fix wraps each list read in the existing `poll_until_condition` helper (30 s ceiling), waiting until the created IDs appear in the listing. The original assertions still run on the final page, so a real regression still fails. Follows the same deflaking pattern as apify#824, apify#831, apify#844, and apify#868. * docs: fix input guide example to bound the wait with wait_duration (apify#955) The "Pass input to an Actor" guide example claimed that `timeout=timedelta(seconds=60)` makes `call()` wait up to 60 seconds for the run to finish. It doesn't. On `call()`, `timeout` is the per-request HTTP timeout forwarded to `start()`, while the wait cap is `wait_duration` (default `None`, i.e. wait indefinitely). A user copying the example would block indefinitely on a long or never-finishing run. The sync and async examples now use `wait_duration=timedelta(seconds=60)` and the comment is reworded accordingly. The stale copies under `website/versioned_docs/` are left untouched on purpose. The versioning workflow deletes and re-snapshots `version-3.0` from `docs/` on every 3.x release, so the fix propagates there automatically. * chore: Automatic docs theme update [skip ci] * fix: offload async request body compression to a worker thread (apify#950) The async `ImpitHttpClientAsync.call()` serialized and compressed request bodies inline on the event loop. Compression is CPU-bound (and brotli is becoming the default via the Apify SDK), so it blocked the loop for the whole duration of the compression, stalling every other concurrent task (response reads, the platform events websocket, other in-flight requests). Request preparation is now offloaded to a worker thread via `asyncio.to_thread` whenever there is a body to compress. Bodyless requests (the common polling, listing, and get path) stay inline to avoid a needless thread-dispatch hop. The synchronous client is unchanged, as it has no event loop to block. * chore(release): Update changelog and package version [skip ci] * fix: Propagate last_run status/origin filters to chained storage clients (apify#954) The `status` and `origin` filters passed to `ActorClient.last_run(...)` and `TaskClient.last_run(...)` were silently dropped by the chained storage clients. For example, `actor.last_run(status='SUCCEEDED').dataset().list_items()` queried the most recent run's dataset regardless of status. This is a regression vs 1.x, where `_sub_resource_init_options` forwarded the parent's params to every child client; the typed-client refactor (apify#604) lost that. The fix passes the run client's default params to its four child-client factories (`dataset()`, `key_value_store()`, `request_queue()`, `log()`) in both `RunClient` and `RunClientAsync`, so the filters ride along on the `runs/last` storage endpoints again, matching 1.x and the JS client. Adds a parametrized regression test covering all four chained clients, sync and async. All 8 cases fail before the fix and pass after. * chore(release): Update changelog and package version [skip ci] * chore: Automatic docs theme update [skip ci] * fix: Propagate API token to custom HTTP clients (apify#956) `ApifyClient.with_custom_http_client(token=...)` (and the async twin) stored the token on the `ApifyClient` instance but never passed it to the injected HTTP client, so no request carried an `Authorization` header and every call failed with 401. The documented custom HTTP client example inherited the bug. - `with_custom_http_client` now sets `Authorization: Bearer <token>` on the injected client's default headers, unless the client already has an auth header configured (checked case-insensitively). - `HttpClientBase._prepare_request_call` now merges the client's default headers under the per-request headers, so any custom client using the helper (including a pre-built `ImpitHttpClient` passed as the custom client) actually sends them. For the default client the wire behavior is unchanged, since impit request-level headers replace the identical client-level ones. - The HTTPX guide examples now merge `self._headers` before delegating, and the `HttpClient` ABC docstring states that implementations must send the default headers with every request. Regression tests cover the token reaching the wire (custom sync/async clients and a pre-built tokenless `ImpitHttpClient`) and the no-clobber semantics for client-configured auth headers. * chore(release): Update changelog and package version [skip ci] * fix: Make batch_add_requests split batches by serialized payload size (apify#953) The 9 MB payload guard in `batch_add_requests` was inert: `constrained_batches` was called without `get_len`, so the default `len()` measured each request dict's key count (~4) instead of its serialized size. Batches were therefore split only by the 25-request count limit, and large requests shipped as one oversized POST that the API rejects with 413, failing the whole call. The guard now measures each request as its UTF-8 JSON byte length, using the same serialization flags as the HTTP client's request body path. It also passes `strict=False`, which preserves the previous contract for an individually oversized request: it's sent in its own batch and left for the API to judge, instead of raising a client-side `ValueError`. Both the size-based splitting and the oversized-singleton path are covered by new sync/async regression tests. *✍️ Drafted by Claude Code* * chore(release): Update changelog and package version [skip ci] * docs: Fix documentation mismatches with actual client behavior (apify#958) Fixes 12 documentation issues found by an engineering audit, where docs, examples, or docstrings contradict what the client actually does: - The custom HTTP client guide example (`HttpxClient`) now raises `ApifyApiError` for error responses instead of returning them raw, and the guide documents this part of the `call` contract (resource clients rely on it, e.g. to translate a 404 into a `None` return value). - The streaming concepts page no longer claims all three streaming methods yield a raw `impit.Response` — it now describes the actual yielded value per method (`stream_record` yields a `dict` with the response under `value`, `stream` and `stream_record` may yield `None`). - The pagination concepts page no longer lists `ListOfRequests` among page models exposing `total`/`offset`/`count`; it now explains its cursor-based pagination via `next_cursor`. - The logging formatter example no longer references `%(status_code)s` (absent on most records, causing logging errors) and no longer attaches a duplicate handler; the page notes which properties are present on every record. - The upgrading-to-v3 guide cross-links now include the site baseUrl (`/api/client/python/...`), fixing 6 links that 404ed on the published site. - The conda instruction for the brotli extra installs `brotli-python` (the Python bindings) instead of `brotli` (the C library). - `RunClient.resurrect` docstrings cite the real `SUCCEEDED` status instead of the nonexistent `FINISHED`. - `wait_for_finish` docstrings in `RunClient` and `BuildClient` spell the terminal status as `TIMED-OUT` (the real literal) instead of `TIMED_OUT`. - The quick-start page refers to the `Run` model's `default_dataset_id` attribute instead of the v2-era run dictionary with `defaultDatasetId`. - The README dataset example passes `fields` as `list[str]` per the signature instead of a comma-separated string. - The README quick-start examples handle the `Run | None` return of `call()` instead of accessing attributes on a possible `None`. - The timeouts concepts page states that `no_timeout` is capped at 24 hours by the default client instead of claiming it disables the timeout entirely. *✍️ Drafted by Claude Code* * chore(release): Update changelog and package version [skip ci] * docs: Version docs for v3.1.0 [skip ci] * chore: Automatic docs theme update [skip ci] * chore: Automatic docs theme update [skip ci] * fix: Add missing cannot-monetize-without-payout-billing-info error code (apify#960) - Updates the auto-generated Pydantic models and TypedDicts based on the proposed OpenAPI specification changes. - Based on apify-docs PR [#2785](apify/apify-docs#2785). * chore(release): Update changelog and package version [skip ci] * chore(deps): update pnpm to v11.15.1 (apify#967) * chore(deps): update dependency oxfmt to ^0.59.0 (apify#966) * chore(deps): lock file maintenance (apify#968) * fix: Normalize query params in dataset create_items_public_url (apify#963) `DatasetClient.create_items_public_url` (and its async twin) passed `_build_params` output straight to `urlencode`, bypassing the `_parse_params` normalization that the real HTTP request path applies. As a result, boolean and list query params ended up as Python reprs in the signed URL — e.g. `create_items_public_url(clean=True, fields=['title', 'url'])` produced `clean=True&fields=%5B%27title%27%2C+%27url%27%5D` instead of `clean=true&fields=title,url`. Consumers of the shared URL then got unclean/unfiltered items or an API error. The fix routes the params through `self._http_client._parse_params(...)` before `urlencode`, so the public URL matches exactly what an actual API request would send (bool→`true`/`false`, list→comma-joined, `None` dropped). Added regression tests for both the sync and async clients. *✍️ Drafted by Claude Code* * chore(release): Update changelog and package version [skip ci] * chore(deps): update actions/setup-node action to v7 (apify#969) * chore(deps): update actions/setup-python action to v7 (apify#970) * docs: Remove conda special instructions for brotli (apify#959) It is now included by default: https://github.com/conda-forge/apify-client-feedstock/pull/4/changes * docs: Update description of `request_url` in `Webhooks` (apify#974) - Updates the auto-generated Pydantic models and TypedDicts based on the proposed OpenAPI specification changes. - Based on apify-docs PR [#2794](apify/apify-docs#2794). --------- Co-authored-by: Michal Turek <michal.turek@apify.com> Co-authored-by: Vlada Dusek <v.dusek96@gmail.com> Co-authored-by: Apify Service Account <64261774+apify-service-account@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Josef Procházka <josef.prochazka@apify.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Several
*_collection_iterateintegration tests created a few resources, callediterate(limit=10, desc=True), and asserted each just-created ID showed up.limitis a total cap, not a per-page size — under the parallel suite, other workers creating the same resource type could push the fresh items past the 10-newest window before the assertion ran.Dropping
limit=10lets the iterator walk the full collection (server-side pagination still chunks the requests), so the inclusion check runs against the full set instead of a racy window.Failing CI runs