Skip to content

docs: add example resume a crawl - #1143

Merged
vdusek merged 8 commits into
apify:masterfrom
Mantisus:resume-crawl
Apr 10, 2025
Merged

docs: add example resume a crawl#1143
vdusek merged 8 commits into
apify:masterfrom
Mantisus:resume-crawl

Conversation

@Mantisus

@Mantisus Mantisus commented Apr 8, 2025

Copy link
Copy Markdown
Collaborator

Description

  • add example resume a crawl

Issues

@Mantisus
Mantisus requested a review from Copilot April 8, 2025 01:17
Copilot AI reviewed Apr 8, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

Files not reviewed (1)
  • docs/examples/resume_crawl.mdx: Language not supported

@Mantisus
Mantisus requested review from janbuchar and vdusek April 8, 2025 01:18
vdusek
vdusek requested changes Apr 8, 2025
View reviewed changes

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we can demonstrate this better:

  • Do not use enqueue_links, instead start with a predefined set of URLs.
  • Remove unnecessary calls and comments, like the export_data_json, to reduce the noise.
  • During the first crawl, stop the run manually using CTRL+C.
  • Execute a second run, let it finish, and compare the statistics (logs) of both runs.

Comment thread docs/examples/resume_crawl.mdx Outdated
Comment thread docs/examples/resume_crawl.mdx Outdated
Comment thread docs/examples/resume_crawl.mdx Outdated
Comment thread docs/examples/code_examples/resume_crawl.py Outdated
Mantisus and others added 2 commits April 8, 2025 16:01
janbuchar
janbuchar reviewed Apr 8, 2025
View reviewed changes
Comment thread docs/examples/code_examples/resume_crawl.py Outdated
@Mantisus Mantisus self-assigned this Apr 8, 2025
@Mantisus
Mantisus requested a review from vdusek April 9, 2025 01:38
janbuchar
janbuchar reviewed Apr 9, 2025
View reviewed changes
Comment thread best_crawler.py Outdated
janbuchar
janbuchar reviewed Apr 9, 2025
View reviewed changes
Comment thread website/static/img/resuming-paused-crawl/00.jpg Outdated
Comment thread docs/examples/code_examples/resuming_paused_crawl.py Outdated
Mantisus and others added 2 commits April 9, 2025 13:51
vdusek
vdusek approved these changes Apr 10, 2025
View reviewed changes

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek
vdusek merged commit 40edf70 into apify:master Apr 10, 2025
@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Apr 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@janbuchar janbuchar janbuchar approved these changes
@vdusek vdusek vdusek approved these changes

Assignees

@Mantisus Mantisus

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Explain how to resume a crawl

4 participants

@Mantisus @janbuchar @vdusek