Wrong logalpha parameterisation for sparsifying variational dropout

[gngdb]

Here:

variational-dropout/varout/layers.py

Lines 334 to 340 in 8d18ec2

	logalpha = T.log(T.nnet.sigmoid(self.logitalpha)).eval()
	# remove the old parameter
	del self.params[self.logitalpha]
	del self.logitalpha
	self.logalpha = theano.shared(
	value=logalpha,
	name='logalpha')

Should depend on w as here: https://github.com/BayesWatch/tf-variational-dropout

[kzhai]

I think your implementation is fine. The reason for including W is due to the combination of variational dropout layer and fully connected layer. Despite the clip/mask operations, in the later computation, the W factor was canceled out. One minor difference though, is that your parameterization is the neuron-level dropout approach, where other implementations are the weight-level drop-connect approach equivalent.