Skip to content

docs: Add "Run on Apify" button - #1127

Merged
vdusek merged 16 commits into
masterfrom
add-run-on-apify
Apr 3, 2025
Merged

docs: Add "Run on Apify" button#1127
vdusek merged 16 commits into
masterfrom
add-run-on-apify

Conversation

@vdusek

@vdusek vdusek commented Apr 1, 2025

Copy link
Copy Markdown
Collaborator

Description

  • Basically all the website code was copied from the Crawlee TS and adjusted where necessary for Python.
  • All code samples were manually adjusted to fit within the button width (while maintaining the same maximum width).
  • Removed asyncio.run calls to reduce noise (the example code runner on Apify executes the main coroutine).
  • Added "Run on Apify" buttons to relevant code examples (those containing crawler.run or some explicit interactions with storages). Initialization examples do not include these buttons.
  • I adjusted max_requests_per_crawl to 10 everywhere (I believe 10 is enough to demonstrate what the sample does).
  • I adjusted the contributing guide based on the discussion with @B4nan.

Example

image

Issues

Testing

  • The website was locally rendered and manually checked page by page.

Checklist

  • CI passed

@github-actions github-actions Bot added this to the 111th sprint - Tooling team milestone Apr 1, 2025
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Apr 1, 2025
@vdusek
vdusek requested review from B4nan, Pijukatel and Copilot April 2, 2025 16:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates several code examples to enhance the Apify integration by adding “Run on Apify” buttons and streamlining example code. Key changes include:

  • Reformatting import statements to use multi-line grouping.
  • Removing standalone asyncio.run invocation blocks from most examples.
  • Adjusting crawler initialization parameters in selected examples.
  • Updating documentation in CONTRIBUTING.md for clarity and consistency.

Reviewed Changes

Copilot reviewed 132 out of 135 changed files in this pull request and generated no comments.

Show a summary per file
File Description
docs/examples/code_examples/crawl_website_with_relative_links_all_links.py Reformatted imports and removed the asyncio.run(main()) block.
docs/examples/code_examples/crawl_specific_links_on_website_pw.py Removed the asyncio.run(main()) block.
docs/examples/code_examples/crawl_specific_links_on_website_bs.py Reformatted imports and removed the asyncio.run(main()) block.
docs/examples/code_examples/crawl_multiple_urls_pw.py Removed the asyncio.run(main()) block.
docs/examples/code_examples/crawl_multiple_urls_bs.py Reformatted imports and removed the asyncio.run(main()) block.
docs/examples/code_examples/crawl_all_links_on_website_pw.py Removed the asyncio.run(main()) block.
docs/examples/code_examples/crawl_all_links_on_website_bs.py Reformatted imports and removed the asyncio.run(main()) block.
docs/examples/code_examples/configure_json_logging.py Removed the asyncio.run(main()) block.
docs/examples/code_examples/capture_screenshot_using_playwright.py Removed the asyncio.run(main()) block.
docs/examples/code_examples/beautifulsoup_crawler_stop.py Reformatted imports and removed the asyncio.run(main()) block.
docs/examples/code_examples/beautifulsoup_crawler_keep_alive.py Removed the asyncio.run(main()) block.
docs/examples/code_examples/beautifulsoup_crawler.py Removed the asyncio.run(main()) block and reformatted imports.
docs/examples/code_examples/add_data_to_dataset_pw.py Removed the asyncio.run(main()) block.
docs/examples/code_examples/add_data_to_dataset_bs.py Reformatted imports and removed the asyncio.run(main()) block.
docs/examples/code_examples/adaptive_playwright_crawler.py Increased the max_requests_per_crawl value and removed the asyncio.run(main()) block.
docs/deployment/code_examples/apify/crawler_as_actor_example.py Reformatted imports and lowered the max_requests_per_crawl value.
CONTRIBUTING.md Updated guidelines and instructions for API documentation and local documentation setup.
Files not reviewed (3)
  • docs/examples/add_data_to_dataset.mdx: Language not supported
  • docs/examples/beautifulsoup_crawler.mdx: Language not supported
  • docs/examples/capture_screenshot_using_playwright.mdx: Language not supported
Comments suppressed due to low confidence (2)

docs/examples/code_examples/adaptive_playwright_crawler.py:16

  • The max_requests_per_crawl value was increased from 5 to 10; please confirm that this change is intentional as it may impact crawler concurrency and resource usage.
max_requests_per_crawl=10, playwright_crawler_specific_kwargs={'headless': False}

docs/deployment/code_examples/apify/crawler_as_actor_example.py:12

  • The max_requests_per_crawl value was decreased from 50 to 10; please verify that this adjustment aligns with the expected behavior for Actor-based execution.
crawler = BeautifulSoupCrawler(max_requests_per_crawl=10)

@vdusek
vdusek marked this pull request as ready for review April 2, 2025 16:19

@Pijukatel Pijukatel left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, just 2 tiny details.

Comment thread docs/examples/code_examples/playwright_crawler_with_camoufox.py Outdated
Comment thread docs/introduction/06_scraping.mdx Outdated
@vdusek
vdusek merged commit e7ebda7 into master Apr 3, 2025
@vdusek
vdusek deleted the add-run-on-apify branch April 3, 2025 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add "Run on Apify" button to doc code examples

4 participants