Hyperparameter Optimization

This page is for those wishing to optimize the hyperparameters of the different example scripts in dp.

Svhn

Convolution Neural Network

th examples/convolutionneuralnetwork.lua --dataset Svhn --learningRate 0.1 --maxNormPeriod 1 --accUpdate --cuda --maxOutNorm 1

Hyper-parameters	Epoch	Train Accuracy	Valid Accuracy	Test Accuracy
--batchSize 32 --channelSize '{32,64}' --activation ReLU --hiddenSize '{1000}' --lecunlcn --normalInit	27	0.9954	0.9135	0.9018

Deep Inception

Hyper-parameter optimization of the deepinception.lua script for training the Google Street View House Numbers (SVHN) dataset.

base command :

th examples/deepinception.lua --batchSize 128 --learningRate 0.1 --accUpdate --progress --cuda

The following table contains different inflections of the above command

Hyper-parameters	Epoch	Train Accuracy	Valid Accuracy	Test Accuracy
--batchSize 64 --learningRate 0.01 --activation HardTanh --hiddenSize '{4000,4000}' --lecunlcn --dropout	598	0.8467	0.8948	0.8897
--batchSize 32 --convChannelSize '{16}' --lecunlcn --hiddenSize '{2000}' --maxTries 100 --learningRate 0.01	88	0.9999	0.8961	0.8849
--convChannelSize '{32}' --dropout --lecunlcn	64	0.8108	0.8649	0.8529
--convChannelSize '{32}' --lecunlcn	164	0.9999	0.8600	0.8473