python/docs/introduction/adding-more-urls

[giscus[bot]]

python/docs/introduction/adding-more-urls

Previously you've built a very simple crawler that downloads HTML of a single page, reads its title and prints it to the console. This is the original source code:

https://crawlee.dev/python/docs/introduction/adding-more-urls

[abrhaleitela]

nit: the package does not seem to exist in the correct path for latest crawlee version.
from crawlee.enqueue_strategy import EnqueueStrategy

works in:
from crawlee import EnqueueStrategy

[janbuchar]

This should be fixed now. Thanks for reporting!

[eyayaw]

This means that no new requests will be started after the 20th 10th request is finished.?

[vdusek]

If you're referring to the max_requests_per_crawl=10 parameter, then yes.

[eyayaw]

Yes, I was referring to the snippet above it.

crawler = BeautifulSoupCrawler(max_requests_per_crawl=10)

[ruaultadrien]

In the documentation you say that the transform_request function should either return a modified Request or None. But the code example below it does not seem to be consistent as we declare a function with the output signature: RequestOptions | RequestTransformAction

def transform_request(
    request_options: RequestOptions,
) -> RequestOptions | RequestTransformAction:

Is it an issue or am I understanding something wrong?

[Mantisus]

Hey, @ruaultadrien

Thank you for noticing that. Yes, there is an error in the documentation.

The correct behavior is specified in the code block.

Return either RequestOptions with changes from which the Request will be formed, or skip if this URL should be skipped, or unchanged if it applies without changes.

UPD: The documentation should already have been updated.

[ruaultadrien]

Thanks for the update :)