Hyperparameter Optimization

This page is for those wishing to optimize the hyperparameters of the different example scripts in dp.

Svhn

Convolution Neural Network

th examples/convolutionneuralnetwork.lua --dataset Svhn --learningRate 0.1 --maxNormPeriod 1 --accUpdate --cuda --maxOutNorm 1 --batchSize 32

Hyper-parameters	Epoch	Train Accuracy	Valid Accuracy	Test Accuracy
--activation ReLU --hiddenSize '{3000,2000}' --dropout --channelSize '{32,64}' --lecunlcn --normalInit	17	0.9356	0.9263	0.9208
--activation ReLU --hiddenSize '{2000}' --dropout --channelSize '{32,64}' --lecunlcn --normalInit	8	0.8858	0.9160	0.9038
--activation ReLU --hiddenSize '{1000}' --channelSize '{32,64}' --lecunlcn --normalInit	27	0.9954	0.9135	0.9018

Deep Inception

Hyper-parameter optimization of the deepinception.lua script for training the Google Street View House Numbers (SVHN) dataset.

base command :

th examples/deepinception.lua --batchSize 128 --learningRate 0.1 --accUpdate --progress --cuda

The following table contains different inflections of the above command

Hyper-parameters	Epoch	Train Accuracy	Valid Accuracy	Test Accuracy
--batchSize 64 --learningRate 0.1 --activation ReLU --hiddenSize '{4000,4000,4000}' --lecunlcn --dropout --normalInit	106	0.9499	0.9432	0.9435
--batchSize 64 --learningRate 0.01 --activation HardTanh --hiddenSize '{4000,4000}' --lecunlcn --dropout	598	0.8467	0.8948	0.8897
--batchSize 32 --convChannelSize '{16}' --lecunlcn --hiddenSize '{2000}' --maxTries 100 --learningRate 0.01	88	0.9999	0.8961	0.8849
--convChannelSize '{32}' --dropout --lecunlcn	64	0.8108	0.8649	0.8529
--convChannelSize '{32}' --lecunlcn	164	0.9999	0.8600	0.8473

In progress: th examples/deepinception.lua --batchSize 64 --learningRate 0.1 --activation ReLU --hiddenSize '{4000,4000}' --lecunlcn --dropout --accUpdate --normalInit --cuda --useDevice 2 > ../results/deepsvhn6.txt

th examples/deepinception.lua --batchSize 64 --learningRate 0.1 --activation ReLU --convChannelSize '{8,16}' --incepChannelSize '{{24,32},{32,48}}' --hiddenSize '{4000,4000}' --lecunlcn --dropout --accUpdate --normalInit --cuda --useDevice 3 > ../results/deepsvhn7.txt

BillionWords

In progress : th examples/recurrentlanguagemodel.lua --batchSize 64 --trainEpochSize 100000000 --validEpochSize 1000000 --softmaxtree --hiddenSize 300 --useDevice 4 --rho 5 --cuda --maxTries 50

th examples/recurrentlanguagemodel.lua --batchSize 64 --trainEpochSize 100000000 --validEpochSize 1000000 --softmaxtree --hiddenSize 500 --useDevice 1 --rho 5 --cuda --maxTries 50