I am trying to use a convolutional neural network (implemented with keras) to solve a modified version of the MNIST classification problem (I am trying the background variations as described here). I started from this example and played around a bit with the parameters to get better accuracies, but I seem to get stuck at about 90% accuracy on my validation set.
I've read papers who manage to get near-human accuracy on those datasets, but I seem not to be able to improve my network to get over 95% (something I would expect to be possible). Because I have only been guessing the parameters for the network thus far and I don't seem to find anything online, I was wondering whether there are any guidelines to find a good architecture and good parameters for convolutional neural networks.
Would anybody be aware of how to tackle the problem of finding good parameters (including architecture) for a CNN apart from trying?
Best Answer
Would anybody be aware of how to tackle the problem of finding good parameters (including architecture) for a CNN apart from trying?
No. Hence, they are optimized by 'graduate student descent' 🙂
A standard-ish architecture for mnist is lenet-5, and closely related variants, eg Karpathy's convnetjs implementation, which uses the following layers:
type:'input', out_sx:24, out_sy:24, out_depth:1 type:'conv', sx:5, filters:8, stride:1, pad:2, activation:'relu' type:'pool', sx:2, stride:2 type:'conv', sx:5, filters:16, stride:1, pad:2, activation:'relu' type:'pool', sx:3, stride:3 type:'softmax', num_classes:10 (and I think it augments the data by cutting random 24×24 patches out of the original 28×28 images, which is a key part of obtaining higher accuracies on this (tiny) dataset).
There are meta-learning techniques, which is an open research area. For example "Neural architecture search with reinforcement learning", by Barret Zoph and Quoc Le, 2016, uses reinforcement learning to try different architectures, find out what works well. It does this in an automated way, without human intervention. Of course this needs a ton of GPU power…