collections.deque should ship with a stdlib json serializer

[acdha]

BPO	20774
Nosy	@rhettinger, @etrepum, @pitrou, @ezio-melotti, @bitdancer, @gareth-rees, @serhiy-storchaka, @lisroach, @Fidget-Spinner
PRs	bpo-20774: Add a JSON serializer to collections.deque #830

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-02-25.21:15:50.560>
labels = ['type-feature', '3.7']
title = 'collections.deque should ship with a stdlib json serializer'
updated_at = <Date 2020-11-05.15:47:24.220>
user = 'https://bugs.python.org/acdha'

bugs.python.org fields:

activity = <Date 2020-11-05.15:47:24.220>
actor = 'kj'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2014-02-25.21:15:50.560>
creator = 'acdha'
dependencies = []
files = []
hgrepos = []
issue_num = 20774
keywords = []
message_count = 15.0
messages = ['212215', '212235', '212264', '212275', '288996', '289000', '289005', '289013', '289022', '289024', '290555', '290561', '290565', '337279', '380417']
nosy_count = 10.0
nosy_names = ['rhettinger', 'bob.ippolito', 'pitrou', 'ezio.melotti', 'r.david.murray', 'gdr@garethrees.org', 'serhiy.storchaka', 'acdha', 'lisroach', 'kj']
pr_nums = ['830']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20774'
versions = ['Python 3.7']

[acdha]

Currently the stdlib json module requires a custom serializer to avoid throwing a TypeError on collections.deque instances:

Python 3.3.4 (default, Feb 12 2014, 09:35:54) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from collections import deque
>>> import json
>>> d = deque(range(0, 10))
>>> json.dumps(d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/__init__.py", line 233, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/encoder.py", line 191, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/encoder.py", line 249, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: deque([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) is not JSON serializable

[bitdancer]

json is only designed to serialize standard data types out of the box. Anything else is an extension. I presume you are asking for this because a deque looks more-or-less like a list. I'm not sure that's reason enough, but we'll see what others think.

[pitrou]

The problem is that it would be deserialized as a list; this breaks the general expectation that serialization formats should round-trip.

(yes, tuple already does this; but I think it is less of a problem for tuples, since the list API is a superset of the tuple API except for hashing)

So, perhaps we could ship an optional serializer (under which form?) accepting any sequence type (and perhaps any mapping type?), but it shouldn't be the default.

[gareth-rees]

The JSON implementation uses these tests to determine how to serialize a Python object:

    isinstance(o, (list, tuple))
    isinstance(o, dict)

So any subclasses of list and tuple are serialized as a list, and any subclass of dict is serialized as an object. For example:

    >>> json.dumps(collections.defaultdict())
    '{}'
    >>> json.dumps(collections.OrderedDict())
    '{}'
    >>> json.dumps(collections.namedtuple('mytuple', ())())
    '[]'

When deserialized, you'll get back a plain dictionary or list, so there's no round-trip property here.

The tests could perhaps be changed to:

    isinstance(o, collections.abc.Sequence)
    isinstance(o, collections.abc.Mapping)

I'm not a JSON expert, so I have no informed opinion on whether this is a good idea or not, but in any case, this change wouldn't help with deques, as a deque is not a Sequence. That's because deques don't have an index method (see bpo-10059 and bpo-12543).

[rhettinger]

See also, the same feature request from Tarek Ziadé, http://bugs.python.org/issue29663 , "collections.deque could be serialized in JSON as a simple array. The only thing we can lose in the process is the maxlen value, but I think it's a decent behaviour to ignore it when encoding and to set it to None when decoding."

+1 from me as well. This isn't really different that how we handle tuples and I can see that it would be useful to be able to dump a deque into JSON. I concur that it is reasonable to ignore maxlen because that is primarily a convenience feature (auto-popping on overflow) rather than something that is intrinsic to the semantics of data itself.

For now, just adding deque support is reasonable. We can't just do all sequences because string/bytearray like objects would need to be excluded.

[serhiy-storchaka]

See bpo-27362 for more general approach.

[rhettinger]

For now, just hardcoding deque support is fine.

Support for a __json__ attribute or JSON array registry is a topic for another day. Even then, I don't think that within the standard library support for JSONification should have its responsibility shifted outside the of json module itself.

[bitdancer]

I disagree, I think a __json__ protocol is sensible. But this is why it needs to be discussed on python-dev or python-ideas first :) In the meantime adding deque support like we added enum support is reasonable, but IMO we shouldn't go to crazy adding support for non-base types before talking about a __json__ protocol.

[rhettinger]

There is a difference. An __json__ attribute would have to convert to a list first. Adding support directly to the json module would allow the deque to be read directly.

I think you all are leaning towards premature generalization and making this harder than it needs to be. Chris and Tarek's proposal is a reasonable and straight-forward, but it is not being pushed towards PEP territory and I think Guido would need to opine on whether to enshrine yet another dunder method that would infest the library and privilege the json serialization format over all formats.

[rhettinger]

FWIW, one of the design goal for deques was to make them easily substitutable for lists when needed. This feature request is a nice-to-have that moves us a little closer.

That said, I think a __json__ attribute is too big of a hammer for this simple proposal.

Also, please add Bob Ippolito to all JSON issues. He has excellent design sensibilities and considerable contact with users of the json module.

[lisroach]

I made PR 830 for this issue, it seems to be a nice feature to have in my opinion.

Let me know if I should add some unit tests :)

[rhettinger]

Thanks Lisa.

[serhiy-storchaka]

Seems there are reference leaks. And I afraid that importing a module for every serialized object can significantly hit the performance. Can you run some benchmarks?

An __json__ attribute would have to convert to a list first. Adding support directly to the json module would allow the deque to be read directly.

With PR 830 the deque is converted to a list by json encoder.