diff --git a/README.md b/README.md index 3af17cc..9195620 100644 --- a/README.md +++ b/README.md @@ -43,34 +43,44 @@ Crawlera middleware won't be able to handle them. ### Settings -* `CRAWLERA_FETCH_ENABLED` (type `bool`, default `False`). Whether or not the middleware will be enabled, - i.e. requests should be downloaded using the Crawlera Fetch API. The `crawlera_fetch_enabled` spider - attribute takes precedence over this setting. +* `CRAWLERA_FETCH_ENABLED` (type `bool`, default `False`) -* `CRAWLERA_FETCH_APIKEY` (type `str`). API key to be used to authenticate against the Crawlera endpoint - (mandatory if enabled) + Whether or not the middleware will be enabled, i.e. requests should be downloaded using + the Crawlera Fetch API. The `crawlera_fetch_enabled` spider attribute takes precedence + over this setting. + +* `CRAWLERA_FETCH_APIKEY` (type `str`) + + API key to be used to authenticate against the Crawlera endpoint (mandatory if enabled) + +* `CRAWLERA_FETCH_URL` (Type `str`, default `"http://fetch.crawlera.com:8010/fetch/v2/"`) -* `CRAWLERA_FETCH_URL` (Type `str`, default `"http://fetch.crawlera.com:8010/fetch/v2/"`). The endpoint of a specific Crawlera instance -* `CRAWLERA_FETCH_RAISE_ON_ERROR` (type `bool`, default `True`). Whether or not the middleware will - raise an exception if an error occurs while downloading or decoding a request. If `False`, a - warning will be logged and the raw upstream response will be returned upon encountering an error. +* `CRAWLERA_FETCH_RAISE_ON_ERROR` (type `bool`, default `True`) + + Whether or not the middleware will raise an exception if an error occurs while downloading + or decoding a response. If `False`, a warning will be logged and the raw upstream response + will be returned upon encountering an error. * `CRAWLERA_FETCH_DOWNLOAD_SLOT_POLICY` (type `enum.Enum` - `crawlera_fetch.DownloadSlotPolicy`, - default `DownloadSlotPolicy.Domain`). + default `DownloadSlotPolicy.Domain`) + Possible values are `DownloadSlotPolicy.Domain`, `DownloadSlotPolicy.Single`, `DownloadSlotPolicydefault` (Scrapy default). If set to `DownloadSlotPolicy.Domain`, please consider setting `SCHEDULER_PRIORITY_QUEUE="scrapy.pqueues.DownloaderAwarePriorityQueue"` to - make better usage of concurrency options and avoid delays. + make better usage of concurrency options and avoiding delays. * `CRAWLERA_FETCH_DEFAULT_ARGS` (type `dict`, default `{}`) + Default values to be sent to the Crawlera Fetch API. For instance, set to `{"device": "mobile"}` to render all requests with a mobile profile. ### Spider attributes -* `crawlera_fetch_enabled` (type `bool`, default `False`). Whether or not the middleware will be enabled. +* `crawlera_fetch_enabled` (type `bool`, default `False`) + + Whether or not the middleware will be enabled. Takes precedence over the `CRAWLERA_FETCH_ENABLED` setting. ### Log formatter