diff --git a/docs/02_concepts/03_storages.mdx b/docs/02_concepts/03_storages.mdx index bb616813..f44a98c4 100644 --- a/docs/02_concepts/03_storages.mdx +++ b/docs/02_concepts/03_storages.mdx @@ -151,6 +151,8 @@ you can use the `KeyValueStore. In this section we will show you how to work with [request queues](https://docs.apify.com/platform/storage/request-queue). +By default, a request queue is optimized for a single consumer. To let multiple Actor runs process the same queue at once, see [single vs. shared request queue](./storage-clients#single-vs-shared-request-queue). + ### Adding requests to a queue To add a request into the queue, you can use the `RequestQueue.add_request` method. @@ -187,6 +189,6 @@ Behind the scenes, the SDK uses storage clients to communicate with the storage ## Conclusion -This page has covered the three storage types (datasets, key-value stores, and request queues): how they are emulated on the local filesystem, how to open named and unnamed storages, and how to read from and write to each through the `Actor` shortcuts and the storage classes. +This page has covered the three storage types (datasets, key-value stores, and request queues): how they are emulated on the local filesystem, how to open named and unnamed storages, and how to read from and write to each through the `Actor` shortcuts and the storage classes. When multiple Actor runs need to process the same request queue, choose between a [single and shared request queue](./storage-clients#single-vs-shared-request-queue). For comprehensive information about storage on the Apify platform, see the [storage documentation](https://docs.apify.com/platform/storage), including the pages on [datasets](https://docs.apify.com/platform/storage/dataset), [key-value stores](https://docs.apify.com/platform/storage/key-value-store), and [request queues](https://docs.apify.com/platform/storage/request-queue). diff --git a/docs/02_concepts/12_storage_clients.mdx b/docs/02_concepts/12_storage_clients.mdx index 221af2a0..7b953fd9 100644 --- a/docs/02_concepts/12_storage_clients.mdx +++ b/docs/02_concepts/12_storage_clients.mdx @@ -43,8 +43,8 @@ For details, see the [Crawlee storage clients guide](https://crawlee.dev/python/ `ApifyStorageClient` supports two ways of accessing the Apify request queue, selected via its `request_queue_access` argument: -- **`'single'`** (default) - optimized for a single consumer. It makes fewer API calls, so it is cheaper and faster, but it doesn't support multiple clients consuming the same queue concurrently. This is the right choice for the majority of Actors. -- **`'shared'`** - supports multiple consumers working on the same queue at the same time, at the cost of more API calls. +- **`'single'`** (default) - optimized for a single consumer. It makes fewer API calls, so it's cheaper and faster, but it doesn't support multiple clients consuming the same queue concurrently. This is the right choice for the majority of Actors. +- **`'shared'`** - supports multiple consumers working on the same queue at the same time, at the cost of more API calls. Use it when several Actor runs share one named request queue. For example, you can split a large crawl across parallel runs, or feed the queue from a producer run while worker runs consume it. To opt into the shared client, set it as the cloud client of the `SmartApifyStorageClient` in the [service locator](https://crawlee.dev/python/docs/guides/service-locator) before entering the Actor context: