[1.14 regression] Unpacking an iterator converts the inner type to `Any`

[dscorbett]

Bug Report

Unpacking an iterator into a list produces a list[Any] instead of the more specific type. This worked correctly in mypy 1.13.0 but no longer works in 1.14.0.

To Reproduce

from typing import reveal_type
x = [1, 2]
reveal_type([*reversed(x)])
y = [3, 4]
reveal_type([*map(str, y)])

Expected Behavior

In mypy 1.13.0:

x.py:3: note: Revealed type is "builtins.list[builtins.int]"
x.py:5: note: Revealed type is "builtins.list[builtins.str]"

Actual Behavior

In mypy 1.14.0:

x.py:3: note: Revealed type is "builtins.list[Any]"
x.py:5: note: Revealed type is "builtins.list[Any]"

Your Environment

• Mypy version used: 1.14.0
• Mypy command-line flags: none
• Mypy configuration options from mypy.ini (and other config files): none
• Python version used: 3.13.0

[hauntsaninja]

Bisects to this typeshed sync: #18057

[hauntsaninja]

Looks like this is python/typeshed#12851 ? cc @tungol That change probably makes type checkers a little slower too

[tungol]

I'll take a look

[tungol]

Using the reproduction above, I stepped through. The problem comes when we get into mypy.maptype

Stack starting at the reveal expression:

  mypy/mypy/checkexpr.py(4681)visit_reveal_expr()
-> revealed_type = self.accept(
  mypy/mypy/checkexpr.py(5903)accept()
-> typ = node.accept(self)
  mypy/mypy/nodes.py(2313)accept()
-> return visitor.visit_list_expr(self)
  mypy/mypy/checkexpr.py(4991)visit_list_expr()
-> return self.check_lst_expr(e, "builtins.list", "<list>")
  mypy/mypy/checkexpr.py(5057)check_lst_expr()
-> out = self.check_call(
  mypy/mypy/checkexpr.py(1571)check_call()
-> return self.check_callable_call(
  mypy/mypy/checkexpr.py(1757)check_callable_call()
-> callee = self.infer_function_type_arguments(
  mypy/mypy/checkexpr.py(2088)infer_function_type_arguments()
-> inferred_args, _ = infer_function_type_arguments(
  mypy/mypy/infer.py(56)infer_function_type_arguments()
-> constraints = infer_constraints_for_callable(
  mypy/mypy/constraints.py(243)infer_constraints_for_callable()
-> actual_type = mapper.expand_actual_type(
  mypy/mypy/argmap.py(204)expand_actual_type()
-> return map_instance_to_supertype(
  mypy/mypy/maptype.py(42)map_instance_to_supertype()
-> return map_instance_to_supertypes(instance, superclass)[0]
> mypy/mypy/maptype.py(49)map_instance_to_supertypes()
-> for path in class_derivation_paths(instance.type, supertype):

Where instance = builtins.reversed[builtins.int] and supertype = <TypeInfo typing.Iterable>.

Up the stack a couple places, at argmap.py(204)expand_actual_type(), we successfully pass a check that is_subtype(actual_type, self.context.iterable_type) in both cases, but maptype.map_instance_to_supertype says in its docstring that If 'superclass' is not a nominal superclass of 'instance.type', then all type arguments are mapped to 'Any'.

So we reach maptype.py(49)map_instance_to_supertypes() and when reversed is an explicit subclass of Iterator we run class_derivation_paths(instance.type, supertype) and get [[<TypeInfo typing.Iterator>, <TypeInfo typing.Iterable>]], but when it isn't an explicit subclass of Iterator we hit the same line and get [] instead and end up returning Any.

The iterable builtins are the most prominent and most important, but I'd expect that this issue would come up for any iterator type that doesn't explicitly inherit from Iterator or Iterable, so this situation would ideally be properly handled by mypy without needing to inherit explicitly from those.

[svalentin]

I can make a point release to fix this since it seems pretty bad. If it's easier/faster we could just revert the typeshed change (python/typeshed#12851 -- but maybe the others mentioned in the description have similar bugs?) in mypy for 1.14.1 while we make a proper fix for the 1.15 release.

[tungol]

Yes, there's a similar bug for the others mentioned in the description, along with anything that has an __iter__ method but doesn't explicitly inherit from Iterable. This code reproduces the issue for all of the iterable builtins as well as a couple things from itertools. Some of the other iterables from that series of typeshed MRs need a little more effort to construct and I didn't bother here, but I have no reason to think they're any different:

import itertools
from typing import reveal_type

x = [1, 2]
y = [3, 4]
reveal_type([*enumerate(x)])
reveal_type([*filter(lambda x: True, x)])
reveal_type([*map(str, x)])
reveal_type([*reversed(x)])
reveal_type([*zip(x, y)])

reveal_type([*itertools.cycle(x)])
reveal_type([*itertools.permutations(x)])

1.13.0 says:

test.py:6: note: Revealed type is "builtins.list[tuple[builtins.int, builtins.int]]"
test.py:7: note: Revealed type is "builtins.list[builtins.int]"
test.py:8: note: Revealed type is "builtins.list[builtins.str]"
test.py:9: note: Revealed type is "builtins.list[builtins.int]"
test.py:10: note: Revealed type is "builtins.list[tuple[builtins.int, builtins.int]]"
test.py:12: note: Revealed type is "builtins.list[builtins.int]"
test.py:13: note: Revealed type is "builtins.list[builtins.tuple[builtins.int, ...]]"

1.14.0 says:

test.py:6: note: Revealed type is "builtins.list[Any]"
test.py:7: note: Revealed type is "builtins.list[Any]"
test.py:8: note: Revealed type is "builtins.list[Any]"
test.py:9: note: Revealed type is "builtins.list[Any]"
test.py:10: note: Revealed type is "builtins.list[Any]"
test.py:12: note: Revealed type is "builtins.list[Any]"
test.py:13: note: Revealed type is "builtins.list[Any]"

This code reproduces the issue on any version of mypy, not just 1.14.0:

from typing import reveal_type, Iterable, Iterator, TypeVar, Generic

_T = TypeVar("_T")

class Iter1(Iterable[_T]):
    def __init__(self, value: list[_T]) -> None:
        self.value = value

    def __iter__(self) -> Iterator[_T]:
        return iter(self.value)

class Iter2(Generic[_T]):
    def __init__(self, value: list[_T]) -> None:
        self.value = value

    def __iter__(self) -> Iterator[_T]:
        return iter(self.value)

x = [1, 2]
reveal_type([*Iter1(x)])  # Revealed type is "builtins.list[builtins.int]"
reveal_type([*Iter2(x)])  # Revealed type is "builtins.list[Any]"

If there's an equivalent of map_instance_to_supertype which handles structural super/sub classes already in the codebase somewhere that would be an easy fix. Otherwise I don't think I understand what mypy.argmap and mypy.maptype are doing well enough to take a stab at fixing the underlying cause myself.

Reverting the typeshed change(s) as a quick fix probably makes sense. I'd probably go with at least the builtins and the itertools, but at the high end, really covering everything would mean making sure that everything in typeshed with an __iter__ method has nominal inheritance Iterable, not just structural inheritance.

[tungol]

I looked at everything in typeshed/stdlib that has an __iter__ method on it; here's my analysis. Note that I didn't check the actual behavior. I just looked at the inheritance in the stubs.

_asyncio.Future: Affected
_contextvars.Context: Not affected; inherits from Mapping
_csv.Reader: Affected
_csv._reader: Affected
_ctypes.Array: Affected
_io._IOBase: Affected
_io._TextIOBase: Affected, but inherits from _io._IOBase and will be fixed by that fix.
_typeshed.SupportsIter: Affected, but it's a Protocol class, and also from _typeshed. Should not be changed.
_typeshed.wsgi.InputStream: Affected, but it's a Protocol class, and also from _typeshed. Should not be changed.
_weakrefset.WeakSet: Not affected; inherits from MutableSet
asyncio.locks._ContextManagerMixin: Affected (This looks like a weird class and maybe it should be broken here?)
builtins.str: Not affected; inherits from Sequence
builtins.bytes: Not affected; inherits from Sequence
builtins.bytearray: Not affected; inherits from MutableSequence
builtins.memoryview: Not affected; inherits from Sequence
builtins.tuple: Not affected; inherits from Sequence
builtins.list: Not affected; inherits from MutableSequence
builtins.dict: Not affected; inherits from MutableMapping
builtins.set: Not affected; inherits from MutableSet
builtins.frozenset: Not affected; inherits from AbstractSet
builtins.enumerate: Affected
builtins.range: Not affected; inherits from Sequence
builtins.filter: Affected
builtins.map: Affected
builtins.reversed: Affected
builtins.zip: Affected
cgi.FieldStorage: Affected
codecs.StreamRecoder: Affected but inherits from typing.BinaryIO
codecs.StreamReader: Affected
codecs.StreamReaderWriter: Affected but inherits from typing.TextIO
collections.UserDict: Not affected; inherits from MutableMapping
collections.UserString: Not affected; inherits from Sequence
collections.ChainMap: Not affected; inherits from MutableMapping
configparser.RawConfigParser: Not affected; inherits from MutableMapping
configparser.SectionProxy: Not affected; inherits from MutableMapping
configparser.ConverterMapping: Not affected; inherits from MutableMapping
csv.DictReader: Affected
dbm._Database: Not affected; inherits from MutableMapping
dbm.dumb._Database: Not affected; inherits from MutableMapping
dbm.sqlite3._Database: Not affected; inherits from MutableMapping
dis.Bytecode: Affected
email.message.Message: Affected
enum.EnumMeta: Affected
enum.Flag: Affected
fileinput.FileInput: Affected
http.client.HTTPResponse: Affected but inherits from typing.BinaryIO
http.cookiejar.CookieJar: Affected
importlib.metadata.EntryPoint: Partially affected; It has different inheritance on different python versions
importlib.metadata.Deprecated: Affected
importlib.metadata._meta.PackageMetadata: Affected, but it's a Protocol class
ipaddress._BaseNetwork: Affected
itertools.count: Affected
itertools.cycle: Affected
itertools.repeat: Affected
itertools.accumulate: Affected
itertools.chain: Affected
itertools.compress: Affected
itertools.dropwhile: Affected
itertools.filterfalse: Affected
itertools.groupby: Affected
itertools.isslice: Affected
itertools.starmap: Affected
itertools.takewhile: Affected
itertools.zip_longest: Affected
itertools.product: Affected
itertools.permutations: Affected
itertools.combinations: Affected
itertools.combinations_with_replacement: Affected
itertools.pairwise: Affected
itertools.batched: Affected
mailbox.Mailbox: Affected
mailbox._ProxyFile: Affected
mmap.mmap: Affected
multiprocessing.managers._BaseDictProxy:  Not affected; inherits from MutableMapping
multiprocessing.managers.DictProxy:  Not affected; inherits from MutableMapping
multiprocessing.pool.IMapIterator: Affected
os._Environ:  Not affected; inherits from MutableMapping
os._ScandirIterator: Affected
os._wrap_close: Affected
shelve.Shelf:  Not affected; inherits from MutableMapping
shlex.shlex: Affected
sqlite3.Cursor: Affected
sqlite3.Row: Not affected; inherits from Sequence
tarfile.Tarfile: Affected
tempfile._TemporaryFileWrapper: Affected but inherits from typing.IO
tempfile.SpooledTemporaryFile: Affected but inherits from typing.IO
traceback.FrameSummary: Affected
types.MappingProxyType: Not affected; inherits from Mapping
types.GeneratorType: Not affected; inherits from Generator
typing.TypeVarTuple: Affected
typing.Iterable: Not affected by definition
typing.Iterator: Not affected by definition
typing.Generator: Not affected; inherits from Iterator
typing.Sequence: Not affected; inherits from Reversible and Collection
typing.ItemsView: Not affected; inherits from AbstractSet
typing.KeysView: Not affected; inherits from AbstractSet
typing.ValuesView: Not affected; inherits from Collection
typing.IO: Affected
typing_extensions.TypeVarTuple: Affected
unittest.suite.BaseTestSuite: Affected
weakref.WeakValueDictionary: Not affected; inherits from MutableMapping
weakref.WeakKeyDictionary: Not affected; inherits from MutableMapping
wsgiref.types.InputStream: Affected, but it's a Protocol class
wsgiref.util.FileWrapper: Affected
wsgiref.validate.InputWrapper: Affected
wsgiref.validate.PartialIteratorWrapper: Affected
wsgiref.validate.IteratorWrapper: Affected
xml.dom.pulldom.DOMEventStream: Affected
xml.etree.ElementTree.Element: Affected

[ilevkivskyi]

If there's an equivalent of map_instance_to_supertype which handles structural super/sub classes already in the codebase somewhere that would be an easy fix.

Unfortunately, there is no such thing (commenting since I wanted something like this myself couple times), so I would suggest to aim for a revert. (Implementing such a helper is possible, but not super easy, so I don't want to do it in a hurry.)

[svalentin]

1.14.1 is out and has the typeshed revert #18324
This should be fixed in mypy, but we probably want to revert the changes in https://github.com/python/typeshed as well or we'll have problems next time typeshed gets synced.