Skip to content
This repository has been archived by the owner on Dec 28, 2023. It is now read-only.

Commit

Permalink
Readme format changes
Browse files Browse the repository at this point in the history
  • Loading branch information
elacuesta committed Apr 10, 2021
1 parent ee88226 commit 1492753
Showing 1 changed file with 22 additions and 12 deletions.
34 changes: 22 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,34 +43,44 @@ Crawlera middleware won't be able to handle them.

### Settings

* `CRAWLERA_FETCH_ENABLED` (type `bool`, default `False`). Whether or not the middleware will be enabled,
i.e. requests should be downloaded using the Crawlera Fetch API. The `crawlera_fetch_enabled` spider
attribute takes precedence over this setting.
* `CRAWLERA_FETCH_ENABLED` (type `bool`, default `False`)

* `CRAWLERA_FETCH_APIKEY` (type `str`). API key to be used to authenticate against the Crawlera endpoint
(mandatory if enabled)
Whether or not the middleware will be enabled, i.e. requests should be downloaded using
the Crawlera Fetch API. The `crawlera_fetch_enabled` spider attribute takes precedence
over this setting.

* `CRAWLERA_FETCH_APIKEY` (type `str`)

API key to be used to authenticate against the Crawlera endpoint (mandatory if enabled)

* `CRAWLERA_FETCH_URL` (Type `str`, default `"http://fetch.crawlera.com:8010/fetch/v2/"`)

* `CRAWLERA_FETCH_URL` (Type `str`, default `"http://fetch.crawlera.com:8010/fetch/v2/"`).
The endpoint of a specific Crawlera instance

* `CRAWLERA_FETCH_RAISE_ON_ERROR` (type `bool`, default `True`). Whether or not the middleware will
raise an exception if an error occurs while downloading or decoding a request. If `False`, a
warning will be logged and the raw upstream response will be returned upon encountering an error.
* `CRAWLERA_FETCH_RAISE_ON_ERROR` (type `bool`, default `True`)

Whether or not the middleware will raise an exception if an error occurs while downloading
or decoding a response. If `False`, a warning will be logged and the raw upstream response
will be returned upon encountering an error.

* `CRAWLERA_FETCH_DOWNLOAD_SLOT_POLICY` (type `enum.Enum` - `crawlera_fetch.DownloadSlotPolicy`,
default `DownloadSlotPolicy.Domain`).
default `DownloadSlotPolicy.Domain`)

Possible values are `DownloadSlotPolicy.Domain`, `DownloadSlotPolicy.Single`,
`DownloadSlotPolicydefault` (Scrapy default). If set to `DownloadSlotPolicy.Domain`, please
consider setting `SCHEDULER_PRIORITY_QUEUE="scrapy.pqueues.DownloaderAwarePriorityQueue"` to
make better usage of concurrency options and avoid delays.
make better usage of concurrency options and avoiding delays.

* `CRAWLERA_FETCH_DEFAULT_ARGS` (type `dict`, default `{}`)

Default values to be sent to the Crawlera Fetch API. For instance, set to `{"device": "mobile"}`
to render all requests with a mobile profile.

### Spider attributes

* `crawlera_fetch_enabled` (type `bool`, default `False`). Whether or not the middleware will be enabled.
* `crawlera_fetch_enabled` (type `bool`, default `False`)

Whether or not the middleware will be enabled.
Takes precedence over the `CRAWLERA_FETCH_ENABLED` setting.

### Log formatter
Expand Down

0 comments on commit 1492753

Please sign in to comment.