I am using SVM models to do short term forecasting of air pollutants. To train a new model I need to find appropriate metaparameters for an SVM model (I mean C, gamma and so on).
Libsvm documentation (and many other books I have read) suggests using grid search to find these parameters – so I basically train model for each combination of these parameters from a certain set and choose the best model.
Is there any better way to find optimal (or near optimal) metaparameters? For me it is mainly a matter of computation time – one grid search of this problem takes about two hours (after I did some optimisations).
Pros of grid search:
- It can be easily parallelized – if you have 20 CPUs it will run 20 times faster, parallelizing other methods is harder
- You check big parts of metaparameter space, so if there is a good solution you will find it.
The downside being of grid search being that the runtime grows as fast as the product of the number of options for each parameter.
Here is an entry in Alex Smola’s blog related to your question
Here is a quote:
[…] pick, say 1000 pairs (x,x’) at random from your dataset, compute the distance of all such pairs and take the median, the 0.1 and the 0.9 quantile. Now pick λ to be the inverse any of these three numbers. With a little bit of crossvalidation you will figure out which one of the three is best. In most cases you won’t need to search any further.
I haven’t tried this myself, but it does seem kind of promising.