Refactor NamedTuple to put it in a stub

[pkch]

Based on @elazarg comment on gitter:

I believe that instead of hard-coding behavior for NamedTuple and other constructs, we should make the building blocks needed to put them in a stub file. For NamedTuple we need a class that is parametrized with TypedDict, and maybe some magical "*Attributes" parameter, or maybe something else. A metaclass can replace some of the methods we've hard-coded. Perhaps we'll still need special treatment inside mypy, but we won't have problems with other mechanisms such as generic types. And the handling inside mypy won't require 100+LOC like it does today.

This was in response to my question about how to make NamedTuple support generics (#2875, #685, #1682).

I can try to do that, but wanted to get feedback first.

Also, @JelleZijlstra:

there's currently code in visit_class_def in semanal.py that treats normal classes and namedtuples in a similar but not quite identical way (I have a pending #3081 that changes it further). it's possible that just changing that code would make generic namedtuples work, but I haven't tried

[gvanrossum]

In theory I would love to get rid of the special-casing for NamedTuple, if possible. (Then again I'd also love to get rid of the special-casing for TypedDict.)

In practice I'm not sure that it's worth anybody's effort -- and if you're volunteering, don't forget the poor reviewers who would have to spend time reviewing your PR, sometimes the review takes more effort than writing the code. It would certainly at this point be a large and subtle change to mypy, so beware of taking on too much (especially if you haven't dug around in the mypy code before).

[JukkaL]

I think that making the type system general enough to represent named tuples or typed dicts without special casing is going to be a very big task and quite possible not even desirable. The named tuple special casing is ugly, yes, but it's mostly implemented already and the code isn't terribly difficult to understand or maintain. I don't really care too much about a bit of ugly code within mypy internals as long as the ugliness is contained within a specific part of the codebase, it's not hard to understand and there's not too much of that. In this case, my preferred improvement would be to move the named tuple specific code to another file that would be called from the semantic analyzer.

I also wrote something about generic tuples elsewhere (#685 (comment)).

[elazarg]

I agree it's not obvious whether making NamedTuple completely without special casing is worth the effort, but at least parts of it are, if not today, then tomorrow. Specifically the following seems useful in general:

1. Generic varargs
2. Some form of *Attributes parameter (I'm not sure how it should work)
3. Some form of literal/singleton/constant types, allowing to parametrize a type based on attribute names (IIUC we all agree we want something like that)
4. Self type (we already have)
5. Metaclasses (we have something)

Each of these can reduce the special-casing inside mypy and allow other libraries to implement similar types of their own without writing plugins. Again, I do not claim it should be top-priority of anyone. I am thinking about it from time to time and if I'll find a way to implement some parts without huge changes I will give it a shot. I do not propose doing anything big in one go.

I think TypedDict should be special-cased. I think it can be the core of TypeInfo or Instance, mimiking dict in Python.

+1 on moving the NamedTuple/TypedDict/NewType parts to a dedicated module; I did something like that in the abandoned ast_matcher, but of course it can be done indpendently.