Add process_spider_output_async() to the spider middleware.

[wRAR]

Fixes #72.

I wonder how can we test the change on Scrapy Cloud before merging this though?

[Gallaecio]

I wonder how can we test the change on Scrapy Cloud before merging this though?

I believe you can install a newer version of the entrypoint through requirements.txt, the only thing that does not update properly is the execution of binaries I think.

[wRAR]

This is for the call in HubstorageDownloaderMiddleware.from_crawler(), added in some older change.

[wRAR]

Check not extracted because we tentatively decided that as a general pattern for such middlewares we prefer to separate "process_request" and "process_item" logic.

[wRAR]

I confirmed this works (if the project has an async middleware, there is a warning normally and no warning if this branch is installed explicitly).

[kmike]

@wRAR it seems Python 3.7 tests are failing. Do you know if these are CI failures, or if the code actually doesn't work?
It seems we should drop Python 3.7 support from the entrypoint, and ensure 3.7 stacks don't get a wrong entrypoint version. But probably it'd be better to do it separately, if the code still works.

[wRAR]

@kmike the Ubuntu 24.04 runner doesn't have 3.7 and earlier, it's a known limitation. I think the suggested way to fix that is pinning 22.04 for such jobs.

[wRAR]

I submitted #92 that includes a fix for the 3.7 CI job.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 11 lines in your changes missing coverage. Please review.

Project coverage is 23.62%. Comparing base (970b45c) to head (0c1cb20).

Files with missing lines	Patch %	Lines
sh_scrapy/middlewares.py	0.00%	11 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master      #91       +/-   ##
===========================================
- Coverage   92.24%   23.62%   -68.62%     
===========================================
  Files          15       15               
  Lines         812      821        +9     
===========================================
- Hits          749      194      -555     
- Misses         63      627      +564     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wRAR]

Hi @elacuesta ! We are close to releasing the new Scrapy version and unless this change is included in the updated stack there will be warnings about this middleware in the spider logs so can we please proceed with reviewing and merging this in the next several weeks? Thank you!