Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/examples/code_examples/resuming_paused_crawl.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import asyncio

from crawlee import ConcurrencySettings, service_locator
from crawlee.crawlers import (
BeautifulSoupCrawler,
BeautifulSoupCrawlingContext,
)

# Disable clearing the `RequestQueue`, `KeyValueStore` and `Dataset` on each run.
# This makes the scraper continue from where it left off in the previous run.
# The recommended way to achieve this behavior is setting the environment variable
# `CRAWLEE_PURGE_ON_START=0`
configuration = service_locator.get_configuration()
configuration.purge_on_start = False


async def main() -> None:
crawler = BeautifulSoupCrawler(
# Let's slow down the crawler for a demonstration
concurrency_settings=ConcurrencySettings(max_tasks_per_minute=20)
)

@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')

# List of links for crawl
requests = [
'https://crawlee.dev',
'https://crawlee.dev/python/docs',
'https://crawlee.dev/python/docs/examples',
'https://crawlee.dev/python/docs/guides',
'https://crawlee.dev/python/docs/quick-start',
]

await crawler.run(requests)


if __name__ == '__main__':
asyncio.run(main())
35 changes: 35 additions & 0 deletions docs/examples/resuming_paused_crawl.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
id: resuming-paused-crawl
title: Resuming a paused crawl
---

import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';

import ResumeCrawl from '!!raw-loader!roa-loader!./code_examples/resuming_paused_crawl.py';

This example demonstrates how to resume crawling from its last state when running locally, if for some reason it was unexpectedly terminated.

If each run should continue crawling from the previous state, you can configure this using `purge_on_start` in <ApiLink to="class/Configuration">`Configuration`</ApiLink>.

Use the code below and perform 2 sequential runs. During the 1st run, stop the crawler by pressing `CTRL+C`, and the 2nd run will resume crawling from where it stopped.

<RunnableCodeBlock className="language-python" language="python">
{ResumeCrawl}
</RunnableCodeBlock>

Perform the 1st run, interrupting the crawler with `CTRL+C` after 2 links have been processed.

![Run with interruption](/img/resuming-paused-crawl/00.webp 'Run with interruption.')

Now resume crawling after the pause to process the remaining 3 links.

![Resuming crawling](/img/resuming-paused-crawl/01.webp 'Resuming crawling.')

Alternatively, use the environment variable `CRAWLEE_PURGE_ON_START=0` instead of using `configuration.purge_on_start = False`.

For example, when running code:

```bash
CRAWLEE_PURGE_ON_START=0 python -m best_crawler
```
4 changes: 2 additions & 2 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file added website/static/img/resuming-paused-crawl/00.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/resuming-paused-crawl/01.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.