AutoExtractProvider now support the new scrapy-poet cache interface #31
Conversation
Additionaly, the preferred pageType for HTML requests (``AutoExtractProductData``) is now chosen always if listed as dependency instead of just choosing the first dependency ``pageType`` to request the HTML
Codecov Report
@@ Coverage Diff @@
## master #31 +/- ##
==========================================
+ Coverage 85.24% 85.82% +0.57%
==========================================
Files 9 9
Lines 488 515 +27
==========================================
+ Hits 416 442 +26
- Misses 72 73 +1
|
scrapy_autoextract/providers.py
Outdated
| _TASK_MANAGER = "_autoextract_task_manager" | ||
|
|
||
|
|
||
| AEDataType = TypeVar('AEDataType', bound=AutoExtractData, covariant=True) |
There was a problem hiding this comment.
Could you elaborate a little bit, please, why you created a covariant TypeVar here instead of just using AutoExtractData in typing? Is it because it could be product data, article data, and so on, so it won't be an invariant?
There was a problem hiding this comment.
Yep, that's precisely right. Using covariant=True here would cover any subtypes derived from AutoExtractData.
setup.py
Outdated
| 'autoextract-poet>=0.3.0', | ||
| 'zyte-autoextract>=0.7.0', | ||
| 'scrapy-poet>=0.2.0', | ||
| 'scrapy-poet @ git+https://[email protected]/scrapinghub/scrapy-poet@injector_record_replay_native#egg=scrapy-poet', |
There was a problem hiding this comment.
Reminder to remove this after a new release of scrapy-poet to PyPI is done.
This requires this change from
scrapy-poet: scrapinghub/scrapy-poet#55Additionally, the preferred pageType for HTML requests (
AutoExtractProductData)is now chosen always if listed as dependency instead of just choosing
the first dependency
pageTypeto request the HTMLTodo:
scrapy-poetdependency to Pipy once released