fix(crawlers): Keep Request mutations across handler runs and retries#2076
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Pull request overview
This PR fixes request state persistence across handler runs and retries by removing the per-handler deep-copied Request and instead executing handlers against the original Request instance. This aligns Python behavior with Crawlee for JS and also ensures AdaptivePlaywrightCrawler sub-crawler runs (browser/static probes) can contribute to the final request state.
Changes:
- Remove
Requestdeep-copy isolation fromRequestHandlerRunResult, so handler mutations (e.g.,user_data,headers) affect the real request and persist across retries. - Simplify
BasicCrawlerrequest execution by dropping theswapped_contextrestoration mechanism and not post-applying request changes. - Update unit tests to assert persistence of request/session mutations across retries and across adaptive sub-crawler runs.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/crawlers/_basic/test_basic_crawler.py | Expands coverage to verify Request.user_data and Session mutations persist across retries (handler + error handler). |
| tests/unit/crawlers/_adaptive_playwright/test_adaptive_playwright_crawler.py | Updates expectations to reflect shared-request behavior across adaptive probe runs and verifies final persisted request state. |
| src/crawlee/crawlers/_basic/_context_utils.py | Removes swapped_context helper that existed to restore original request after running handlers on an isolated copy. |
| src/crawlee/crawlers/_basic/_basic_crawler.py | Stops using an isolated request copy in handler execution and removes post-run “apply changes back” logic. |
| src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py | Ensures sub-crawlers share the same original request while still isolating storage-result tracking per sub-crawler run. |
| src/crawlee/_types.py | Removes request deep-copying and “apply request changes” behavior from RequestHandlerRunResult. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
vdusek
changed the title
Request mutations across handler and retriesRequest mutations across handler runs and retries
| Sub crawler pipeline call is dynamically created based on the `rendering_type`. | ||
| New copy-like context is created from passed `context` and `state` and is passed to sub crawler pipeline. | ||
| A new context is created from passed `context` and `state` and is passed to sub crawler pipeline. The original | ||
| `request` is shared, so its mutations persist. Only `result` and `use_state` are isolated per sub crawler. |
There was a problem hiding this comment.
Not an issue of this PR, but it is pretty hard to have a mental model of what can be mutated and what can not, and under which conditions...
There was a problem hiding this comment.
What do you suggest? Shall we open an issue and revisit it later? Or just document it somewhere?
Description
This PR ensures that mutations of
user_dataandheadersin aRequestpersist across request handler runs and retries. The behavior matches Crawlee for JS.In
AdaptivePlaywrightCrawler, both the static and the browser runs affect the final request state.Issues
context.request.user_datamutations are silently dropped on retry #2061Testing