TypeError: Decimal is not JSON serializable when submitting job

[craiga]

Decimals aren't handled in client.job.create.

from decimal import Decimal
from zencoder import Zencoder

client = Zencoder('API_KEY')
client.job.create('s3://bucket/key.mp4',
                  outputs=[{'width': Decimal(300)}])

…causes:

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    outputs=[{'width': Decimal(300)}])
  File "…/zencoder/core.py", line 350, in create
    return self.post(self.base_url, body=json.dumps(data))
  File "…/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "…/json/encoder.py", line 192, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "…/json/encoder.py", line 250, in iterencode
    return _iterencode(o, 0)
  File "…/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: Decimal('300') is not JSON serializable

I'm retrieving job settings from DynamoDB with boto and simplejson which seems to like returning numbers as Decimals.

[schworer]

This is an incompatibility between the decimal and json packages, but it turns out that simplejson can serialize decimal correctly.

Is it straightforward for you to convert any instances of Decimal to int before sending to Zencoder? In this case width must be an integer anyway, so there will be no loss in precision.

    from decimal import Decimal
    from zencoder import Zencoder

    client = Zencoder('API_KEY')
    client.job.create('s3://bucket/key.mp4',
                      outputs=[{'width': int(Decimal(300))}])

[schworer]

Also, it looks like boto can be configured to return ints and floats: http://boto.readthedocs.org/en/latest/ref/dynamodb.html#boto.dynamodb.layer2.Layer2.use_decimals

[craiga]

It looks like that configuration will only work when using boto.dynamodb; I'm using boto.dynamodb2. There's also boto3, which might be a different thing altogether.

I can (and probably will) write some code to cast Decimals to ints or floats, but the point of this issue is that I shouldn't have to. Zencoder should make use of simplejson.

[schworer]

Fair enough - back when I was first working on the tool there was a specific reason I picked the json package as the primary module. Note that if json isn't installed we'll attempt to fallback to simplejson. I think the easiest solution in order to preserve compatibility for most folks is to make it configurable.

Here's where we pick the json module to import:
https://github.com/zencoder/zencoder-py/blob/master/zencoder/core.py#L7-L19

[schworer]

Just put together a quick PR that makes this configurable: #30

Please check it out and let me know if it will work for you. Thanks!

[craiga]

Sorry it's taken so long, but this looks like it solves the problem, albeit a little awkwardly. The ZENCODER_PY_JSON_MODULE environment variable needs to be set before importing the zencoder module:

import os
os.environ['ZENCODER_PY_JSON_MODULE'] = 'simplejson'
from zencoder import Zencoder
…
zencoder = Zencoder(api_key)
response = zencoder.job.create(options=job_settings)

It'd be preferable if this environment variable were read when making use of the json module so you could have code like this:

import os
from zencoder import Zencoder
…
zencoder = Zencoder(api_key)
os.environ['ZENCODER_PY_JSON_MODULE'] = 'simplejson'
response = zencoder.job.create(options=job_settings)

…but then you start getting into a dependency injection situation, which as far as I understand it can get tricky pretty quickly in Python.

[schworer]

Gotcha - the way this was intended to be used is that you'd set ZENCODER_PY_JSON_MODULE as an environment variable in your env before launching the python process:
export ZENCODER_PY_JSON_MODULE=simplejson

If you do it that way, there is no need to edit the os.environ dict inside the python process. Does your deploy environment allow you to set environment variables like this?

[craiga]

Yes, we'll be able to set that environment variable.