Only READ support for Decimal in json

[andersrundgrennetgmailcom]

BPO	23123
Nosy	@warsaw, @rhettinger, @etrepum, @mdickinson, @ncoghlan, @pitrou

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/etrepum'
closed_at = None
created_at = <Date 2014-12-27.17:10:51.397>
labels = ['extension-modules', 'type-bug']
title = 'Only READ support for Decimal in json'
updated_at = <Date 2015-07-21.07:40:49.271>
user = 'https://bugs.python.org/andersrundgrennetgmailcom'

bugs.python.org fields:

activity = <Date 2015-07-21.07:40:49.271>
actor = 'ethan.furman'
assignee = 'bob.ippolito'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2014-12-27.17:10:51.397>
creator = 'anders.rundgren.net@gmail.com'
dependencies = []
files = []
hgrepos = []
issue_num = 23123
keywords = []
message_count = 24.0
messages = ['233139', '233156', '233157', '233159', '233160', '233161', '233162', '233163', '233165', '233166', '233168', '233169', '233170', '233175', '233183', '233184', '233187', '233188', '233191', '233193', '233195', '233198', '233199', '233238']
nosy_count = 8.0
nosy_names = ['barry', 'rhettinger', 'bob.ippolito', 'mark.dickinson', 'ncoghlan', 'pitrou', 'eli.bendersky', 'anders.rundgren.net@gmail.com']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23123'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[andersrundgrennetgmailcom]

jsonString = '{"t":6,"h":4.50, "g":"text","j":1.40e450}'
jsonObject = json.loads(jsonString, object_pairs_hook=collections.OrderedDict,parse_float=Decimal)
for item in jsonObject:
  print jsonObject[item]
6
4.50
text
1.40E+450

Works as expected.

However, there seems to be no way to get back to the original JSON string as far as I can tell since you have to convert Decimal to str in cls when using json.dumps which adds "" around the arguments

[rhettinger]

It is unfortunate that there doesn't seem to be a way to round-trip Decimals. That would seem to be a fundamental capability that we should expect to support.

I have a vague recollection that you used to be able to trick the encoder by returning a subclass of float with a custom __str__; however, I don't think that hack would work anymore because float subclasses now get coerced back to a regular float in order to make the json module work with Enums (which have a __str__ that is meaningless in JSON).

In Python 3.5, it would be nice to add a hook that affords more control than "cls" currently does. Ideally, it should allow any class to special exactly what it wants written-out.

Another option, for Py2.7, 3.4, and 3.5 is to add direct support for decimal instances (much like the enum support was backported).

[ethanfurman]

Enums (and other numeric subclasses), do not round-trip back to themselves. An IntEnum with the value of 4 is written as 4 and converted back from json as the integer 4 (not Settings.TabSpaces, or whatever).

Given that json is multi-language format (or a javascript format -- but either way not a Python-specific format) I don't know that we can expect much more from it.

[etrepum]

simplejson has had a use_decimal flag for output since 2.1.0 and has been enabled by default since 2.2.0. simplejson 3.2.0 introduced a for_json argument that checks objects for a method of that name for serialization.

https://github.com/simplejson/simplejson/blob/master/CHANGES.txt

[andersrundgrennetgmailcom]

I was actually hoping to implement the final part of this:
https://openkeystore.googlecode.com/svn/resources/trunk/docs/jcs.html#Normalization_and_Signature_Validation

It seems that the current Decimal implementation wouldn't save me anyway since it modifies the input :-(

Anyway, floats in JSON have rather little use so maybe my existing Pyhton (PoC) solution will be "good enough":
https://code.google.com/p/openkeystore/source/browse/python/trunk/src/org/webpki/json/JCSValidator.py

[etrepum]

I'm sure there's some hack that would allow you to preserve the input. I would try using parse_float and have it return some object that preserves the string and will be output in precisely the same way. It may need to be a Decimal subclass. I'm traveling for the next few weeks so I won't have much of a chance to investigate myself.

[andersrundgrennetgmailcom]

It would be great if I could use a sub-classed Decimal during parsing but since it doesn't appear to be a way to serialize the result using the "json" package I'm probably stuck with the current "99%" solution.

I have solved this in Java and JavaScript by writing my own JSON stuff
http://webpki.org/papers/keygen2/doc/org/webpki/json/package-summary.html
but that method obviously doesn't scale and I'm a real n00b when it comes to Python although it was more fun than I had expected :-)

A minor patch addressing serialization of Decimal would probably do fine (after sub-classing) and would be generally useful.

[etrepum]

Subclass Decimal and implement __str__ to return your own representation. Use parse_float to use your Decimal subclass. Should work with simplejson, a similar hack may be possible with the json module.

[rhettinger]

Given that json is multi-language format ... I don't know
that we can expect much more from it.

JSON specifies a textual number format but doesn't dictate whether that format represents a fixed precision binary float point number or a decimal floating point number. It is perfectly reasonable for someone to want to read and write a JSON number format to and from a decimal (we also see this with database formats as well -- such as sqlite).

This bug report isn't a JSON spec issue; rather, it is about how the JSON module API can support (or inhibit) valid use cases.

AFAICT, the patch to make the API better support enums had the side-effect of inhibiting the APIs ability to support number objects that want to control their output via __str__ or __repr__. This seems to block-off decimal support and support for controlling displayed precision.

I think the Enum patch is suspect and could be considered a regression. That said, we could simply add direct support for decimals and leave the enum patch in-place (though it still impairs a user's ability to control the displayed precision).

[andersrundgrennetgmailcom]

I guess my particular requirement/wish is unusual (keeping the original textual representation of a floating point number intact) while using Decimal should be fairly universal.

If these things could be combined in a Decimal support option I would (of course) be extremely happy. They do not appear to be in conflict.

Currently I'm a bit bogged down by the crypto-stuff since it is spread over different and incompatible modules which makes it awkward creating a nice unified RSA/EC solution. I may end-up writing a wrapper...

[etrepum]

I don't think it's reasonable to expect Decimal to always output precisely the same string it was given. It's a waste of complexity and space and the only time you would want this behavior is when you really should've left it accessible as a string in the first place.

It sounds like the spec for that signature may be poorly designed (with regard to portability). Relying on the precise string output of a number is not going to work in any JSON parser I've ever seen. You'd need to work at the tokenizer level and not all of the parsers provide an interface at that layer (since many of them combine tokenization and parsing).

[andersrundgrennetgmailcom]

Bob,
Your'e right, I have put up a requirement for JSON serializing that may be "over the top". OTOH, there are (AFAICT...) only two possible solutions:

1. Outlaw floating point data from the plot
2. Insist that serializers conform to the spec

As a pragmatic I have settled on something in between :-)
https://openkeystore.googlecode.com/svn/resources/trunk/docs/jcs.html#Interoperability

I don't think that the overhead in Decimal would be a problem but I'm not a Python platform maintainer so I leave it to you guys.