Hyperparameter Optimization

This page is for those wishing to optimize the hyperparameters of the different example scripts in dp.

Svhn

Convolution Neural Network

th examples/convolutionneuralnetwork.lua --dataset Svhn --learningRate 0.1 --maxNormPeriod 1 --accUpdate --cuda --maxOutNorm 1 --batchSize 32

Hyper-parameters	Epoch	Train	Valid	Test
--activation ReLU --hiddenSize '{3000,2000}' --dropout --channelSize '{32,64}' --lecunlcn --normalInit	17	0.9356	0.9263	0.9208
--activation ReLU --hiddenSize '{3000,2000}' --dropout --channelSize '{32,48,64}' --padding '{2,2,2}' --normalInit --lecunlcn	10	0.9168	0.9211	0.9182
--activation ReLU --hiddenSize '{2000}' --dropout --channelSize '{32,64}' --lecunlcn --normalInit	8	0.8858	0.9160	0.9038
--activation ReLU --hiddenSize '{1000}' --channelSize '{32,64}' --lecunlcn --normalInit	27	0.9954	0.9135	0.9018

Deep Inception

Hyper-parameter optimization of the deepinception.lua script for training the Google Street View House Numbers (SVHN) dataset.

base command :

th examples/deepinception.lua --accUpdate --progress --cuda --batchSize 64 --learningRate 0.1 --activation ReLU

The following table contains different inflections of the above command

Hyper-parameters	Epoch	Train	Valid	Test
--hiddenSize '{4000,4000,4000}' --lecunlcn --dropout --normalInit	49	0.9707	0.9752	0.9629

BillionWords

In progress : th examples/recurrentlanguagemodel.lua --batchSize 64 --trainEpochSize 100000000 --validEpochSize 1000000 --softmaxtree --hiddenSize 300 --useDevice 4 --rho 5 --cuda --maxTries 50

th examples/recurrentlanguagemodel.lua --batchSize 64 --trainEpochSize 100000000 --validEpochSize 1000000 --softmaxtree --hiddenSize 500 --useDevice 1 --rho 5 --cuda --maxTries 50