Mocha Theano integration

[benmoran]

I've started a project to allow Mocha layers to be defined using Theano, via PyCall.
The advantage is being able to define new types of layer on the fly by just writing symbolic expressions, instead of having to implement each of forward(), backward() twice (once for the CPU and CPU).

For instance, you can create a SquareLoss layer from scratch like this:

  layer  = TheanoLossLayer(; bottoms=[:predictions, :labels],
                           eltype=Float32,
                           loss=:( sum((predictions-labels).^2./2size(predictions,-1) ) ) )

It's a shame we don't have a Theano implementation natively in Julia yet!

I was wondering whether you'd want this in the main Mocha repository once it's further developed, or if you prefer to keep it externally? I kept it out because it adds dependencies on PyCall and Theano.

Here it is (WIP - just Loss layers so far, and it's inefficiently copying out the gradients at the moment) - https://github.com/benmoran/MochaTheano.jl

[pluskid]

I agree that it's a shame that we do not have a Theano implementation in Julia yet. We do not even have a feature complete cuArray.jl like package in Julia yet, which is another reason that makes currently writing new layers in Mocha relatively painful.

I think the idea is nice! If you figured out how to do it efficiently without extra copying back and forth, I think it should be a great add-on. I think it could act like the current GPUBackend -- we can use an environment variable to control whether the user want this component or not. So we do not need to introduce extra dependencies like PyCall.jl and Theano by default, unless the user want to try it out.

[jskDr]

@benmoran I am always very exciting to combine Python convenience and Julia power. Please go ahead and let us share progress or ask any help needed.

Sungjin (James) Kim
[email protected]