WIP: Annotated[Item, PickFields("x", "y")] to decide which fields to populate in callback

[BurnzZ]

A continuation for #88.

Stemming from the idea in scrapinghub/web-poet#77 to use typing.Annotated. See PEP 593.

TODO:

• consider an exclusion list like NotPickFields
  • how does it work when Annotated[PickFields("x", "y"), NotPickFields("y", "z")]?
    • silent? warning? raises ValueError
• avoid returning cached full-items as-is, vice-versa
• handle partial item from the request itself (e.g. meta).
• handle case when field in PickField/NotPickField is not available in PO
• handle the inconsistencies of Annotated in different Python versions
• more tests

When we're happy with the API:

• Changelog
• Docs

[BurnzZ]

There's still some stuff to do but I think this is ready for review to check if we're happy with the API before proceeding any further.

from typing Annotated, Optional

import attrs
import scrapy
from scrapy_poet import ItemPage, PickFields, handle_urls, field

@attrs.define
class BigItem:
    x: str
    y: Optional[str] = None

@handle_urls("example.com")
@attrs.define
class BigPage(ItemPage[BigItem]):
    @field
    def x(self) -> str:
        return "x"

    @field
    def y(self) -> str:
        return "y"

class SomeSpider(scrapy.Spider):
    name = "somespider"

    def start_requests(self):
        yield scrapy.Request("https://example.com", self.parse_item)

    def parse_item(self, response, item: Annotated[BigItem, PickFields("x")]):  #👈
        yield item  # should return BigItem(x="x", y=None)

[Gallaecio]

The API looks good to me.

I think NotPickFields can be a nice API as well. In fact, I imagine it could potentially be used more often than PickFields. And I think we should definitely prevent mixing the two with an error or warning.

[BurnzZ]

Pausing this as we've decided to move most of the functionalities into web-poet. See scrapinghub/web-poet#115.