pickling of Exceptions and kw_only

[arnimarj]

Just stumbled across an interesting issue. The kw_only flag seems to break pickling of objects. E.g.:

import attr
import pickle


@attr.s(auto_exc=True, kw_only=False)
class Positional(Exception):
    field: int = attr.ib()


@attr.s(slots=False, auto_exc=True, kw_only=True)
class KwOnly(Exception):
    field: int = attr.ib()


A = Positional(field=1)
B = KwOnly(field=1)

print('A', pickle.loads(pickle.dumps(A)))
print('B', pickle.loads(pickle.dumps(B)))

raises this error

Traceback (most recent call last):
  File "attrs_kwonly.py", line 19, in <module>
    print('B', pickle.loads(pickle.dumps(B)))
TypeError: __init__() takes 1 positional argument but 2 were given

I couldn't find any mention in previous issues, or current documentation about this behaviour.

[Drino]

Hello!

It seems that this bug appears only for inheritors of Exception:

import attr
import pickle


@attr.s(slots=False, auto_exc=True, kw_only=True)
class KwOnly(object):
    field: int = attr.ib()


B = KwOnly(field=1)
print('B', pickle.loads(pickle.dumps(B)))  # B KwOnly(field=1)

I think the issue here is the way Exception is pickled - it is using __reduce__ to get (cls, args, state) tuple, which works incorrectly in this case:

B.__reduce__()  # (__main__.KwOnly, (1,), {'field': 1})
# If we don't use auto_exc=True
B.__reduce__()  # (__main__.KwOnly, (), {'field': 1})
# Note that pickle is still broken, as __init__ is called

I believe it's a bug in auto_exc and I'm not sure how it should be fixed - it seems that __reduce__ (which is default Exception pickling method and is inherited by child class) is not compatible with kw_only-arguments.

You can probably avoid this issue by providing default to all kw_only-arguments and re-implementing own reduce like:

def __reduce__(self): return (self.__class__, (), self.__dict__)

[dragonpaw]

This seems to actually affect when any field on the exception is marked kw_only:

@define(auto_exc=True)
class CantPickleMeSadly(Exception):
    message: None | str = field(kw_only=False, default=None)
    response: None | requests.Response = field(kw_only=True, default=None)

pickle.loads(pickle.dumps(CantPickleMeSadly('foo')))
TypeError: CantPickleMeSadly.__init__() takes from 1 to 2 positional arguments but 3 were given

[hynek]

I guess the solution is as @Drino suggested: implement a def __reduce__(self): return (self.__class__, (), self.__dict__) if any kind of kw_argsis involved.

[meshy]

I guess the solution is as @Drino suggested: implement a def __reduce__(self): return (self.__class__, (), self.__dict__) if any kind of kw_argsis involved.

Would you consider having this as part of attrs, or do you think this is something that people writing these classes should do themselves?

[meshy]

UPDATE: This doesn't work as I expected. I'll post a follow-up.

---

For reference, I've found that using attrs.asdict means I don't have to add slots=True:

In [22]: @attrs.frozen(kw_only=True)
    ...: class E(Exception):
    ...:     value: int = 1
    ...:     def __reduce__(self):
    ...:         return self.__class__, (), attrs.asdict(self, recurse=False)
    ...:

In [23]: pickle.loads(pickle.dumps(E(value=3)))
Out[23]: E(value=3)

[meshy]

If we subclass a kw_only custom exception without defining kw_only, my example above doesn't work.

import attrs
import inspect
import pickle


@attrs.frozen(kw_only=True)
class Base(Exception):
    has_default: int = 42


@attrs.frozen
class Extension(Base):
    no_default: str

In this example, the init signatures are:

• Base(*, has_default: int = 42)
• Extension(no_default: str, *, has_default: int = 42)

I'm not sure what kind of __reduce__ I can write on Base that will cover this case.

[hynek]

@meshy there's been changes to kw_only-handling in the main branch – mind checking it out?

[meshy]

I've used this example:

# sample.py
import attrs
import inspect
import pickle


@attrs.frozen(kw_only=True)
class Custom(Exception):
    has_default: int = 42


a = Custom(has_default=1337)

print(f"{a=}")
print(f"{inspect.signature(a.__init__)=}")
print(f"{a.__reduce__()=}")
pickle.loads(pickle.dumps(a))

Using the main branch and Python 3.12.3, I get this output from python -m sample:

a=Custom(has_default=1337)
inspect.signature(a.__init__)=<Signature (*, has_default: int = 42) -> None>
a.__reduce__()=(<class '__main__.Custom'>, (1337,))
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/.../sample.py", line 16, in <module>
    pickle.loads(pickle.dumps(a))
TypeError: Custom.__init__() takes 1 positional argument but 2 were given

[hynek]

ugh OK, I guess we should write a dedicated __reduce__ when we detect an exception. Could this be implemented based on __getstate__?

Looking at the same output for data classes, it looks to me like they just do

def __reduce__(self):
    return (self.__class__, (), self.__getstate__())

[meshy]

That doesn't work in the example with a positional arg:

# sample.py
import attrs
import inspect
import pickle


@attrs.frozen(kw_only=True)
class Superclass(Exception):
    has_default: int = 42

    def __reduce__(self):
        return (self.__class__, (), self.__getstate__())

@attrs.frozen
class Subclass(Superclass):
    no_default: str


super_instance = Superclass(has_default=1337)
print(f"{super_instance=}")
print(f"{inspect.signature(super_instance.__init__)=}")
print(f"{super_instance.__reduce__()=}")
print()
pickle.loads(pickle.dumps(super_instance))

subclass_instance = Subclass(no_default="woot")
print(f"{subclass_instance=}")
print(f"{inspect.signature(subclass_instance.__init__)=}")
print(f"{subclass_instance.__reduce__()=}")
print()
pickle.loads(pickle.dumps(subclass_instance))

super_instance=Superclass(has_default=1337)
inspect.signature(super_instance.__init__)=<Signature (*, has_default: int = 42) -> None>
super_instance.__reduce__()=(<class '__main__.Superclass'>, (), {'has_default': 1337})

subclass_instance=Subclass(has_default=42, no_default='woot')
inspect.signature(subclass_instance.__init__)=<Signature (no_default: str, *, has_default: int = 42) -> None>
subclass_instance.__reduce__()=(<class '__main__.Subclass'>, (), {'has_default': 42, 'no_default': 'woot'})

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/charlie/code/oe/kraken-core/example.py", line 31, in <module>
    pickle.loads(pickle.dumps(subclass_instance))
TypeError: Subclass.__init__() missing 1 required positional argument: 'no_default'

Instead, I think we need something more like:

def __reduce__(self):
    return (
        self.__class__,
        tuple(
            getattr(self, arg.name)
            for arg in self.__attrs_attrs__
            if arg.kw_only is False and arg.init is True
        ),
        {
            arg.name: getattr(self, arg.name)
            for arg in self.__attrs_attrs__
            if arg.kw_only is True and arg.init is True
        },
    )

I haven't tested this very robustly though, of course. I remembered that init needs to be accounted for, but I wouldn't be surprised if there are other considerations required, or if the ordering on the positional-args portion needs more thought.