Store map data in the data store in a compact, encoded form

[jkoshy]

Store data in the data store encoded using protobufs. If done right, this change has the potential to reduce the storage requirements for the back end.

[noblepaul]

Try avro. It has the most frugal encoding.

[jkoshy]

A portability hurdle: building Avro from source requires that the build machine have java, ruby, php, python, c and c++ all installed on it, in addition to Apache ant and mvn. This is a bit much to expect from the typical developer!

I cannot also find pre-built packages for the Linux based OSes that I can point people to.

I had a quick look around in the Avro Python source tree: the build requires ant but this is used only to set the value of the AVRO_VERSION variable.

[noblepaul]

which linux are you looking for? You may shoot a mail to the user mailing
list. I'm sure somebody would have built for that
On 18 May 2011 15:58, "jkoshy" <
[email protected]>
wrote:

[jkoshy]

I searched for prebuilt packages for Ubuntu or Debian; packaged Avro libraries have not reached those repositories yet.

The download page for Avro has source code only; nothing in pre-packaged form.

Anyhow, for now I have forked the Avro tree, let me see if an alternative build scheme is easy to implement...

[jkoshy]

The jkoshy-build-rework branch of my fork of Avro now supports the following features:

• The build.sh script works with a stock /bin/sh.
• The build.sh script supports a -l LANG option, that allows builds to be restricted to language LANG.
• The Python module can be built entirely using the standard installation tools for Python (i.e., distutils), without the need for an intermediate pre-processing step by ant.

[noblepaul]

This looks like a useful feature in general. You may contribute it back to avro itself . Raise an issue here

[jkoshy]

FYI, the following patches have been submitted to the Avro project:

• AVRO-831 Enhance the Python distutils control file setup.py to build Avro/Python without a pre-processing step.
• AVRO-829 Enhance the build.sh script to support building a subset of the available language bindings
• AVRO-827 Use portable shell syntax for build.sh
  (This is a minor patch which I submitted mostly to figure out how to use Apache's issue tracker.)

[jkoshy]

FYI: Google Protocol Buffers vs Apache Avro.

[jkoshy]

Noting another interesting project, for the record: MsgPack.