Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/02_concepts/03_storages.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,8 @@ you can use the <ApiLink to="class/KeyValueStore#get_public_url">`KeyValueStore.

In this section we will show you how to work with [request queues](https://docs.apify.com/platform/storage/request-queue).

By default, a request queue is optimized for a single consumer. To let multiple Actor runs process the same queue at once, see [single vs. shared request queue](./storage-clients#single-vs-shared-request-queue).

### Adding requests to a queue

To add a request into the queue, you can use the <ApiLink to="class/RequestQueue#add_request">`RequestQueue.add_request`</ApiLink> method.
Expand Down Expand Up @@ -187,6 +189,6 @@ Behind the scenes, the SDK uses storage clients to communicate with the storage

## Conclusion

This page has covered the three storage types (datasets, key-value stores, and request queues): how they are emulated on the local filesystem, how to open named and unnamed storages, and how to read from and write to each through the `Actor` shortcuts and the storage classes.
This page has covered the three storage types (datasets, key-value stores, and request queues): how they are emulated on the local filesystem, how to open named and unnamed storages, and how to read from and write to each through the `Actor` shortcuts and the storage classes. When multiple Actor runs need to process the same request queue, choose between a [single and shared request queue](./storage-clients#single-vs-shared-request-queue).

For comprehensive information about storage on the Apify platform, see the [storage documentation](https://docs.apify.com/platform/storage), including the pages on [datasets](https://docs.apify.com/platform/storage/dataset), [key-value stores](https://docs.apify.com/platform/storage/key-value-store), and [request queues](https://docs.apify.com/platform/storage/request-queue).
4 changes: 2 additions & 2 deletions docs/02_concepts/12_storage_clients.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ For details, see the [Crawlee storage clients guide](https://crawlee.dev/python/

`ApifyStorageClient` supports two ways of accessing the Apify request queue, selected via its `request_queue_access` argument:

- **`'single'`** (default) - optimized for a single consumer. It makes fewer API calls, so it is cheaper and faster, but it doesn't support multiple clients consuming the same queue concurrently. This is the right choice for the majority of Actors.
- **`'shared'`** - supports multiple consumers working on the same queue at the same time, at the cost of more API calls.
- **`'single'`** (default) - optimized for a single consumer. It makes fewer API calls, so it's cheaper and faster, but it doesn't support multiple clients consuming the same queue concurrently. This is the right choice for the majority of Actors.
- **`'shared'`** - supports multiple consumers working on the same queue at the same time, at the cost of more API calls. Use it when several Actor runs share one named request queue. For example, you can split a large crawl across parallel runs, or feed the queue from a producer run while worker runs consume it.

To opt into the shared client, set it as the cloud client of the `SmartApifyStorageClient` in the [service locator](https://crawlee.dev/python/docs/guides/service-locator) before entering the Actor context:

Expand Down