Is it possible to perform an approximated fully Bayesian (1) selection of hyper-parameters (e.g. covariance scale) with the GPML code, instead of maximizing the marginal likelihood (2) ?
I think using MCMC methods to solve the integrals involving hyper-parameters prior should lead to better results when dealing with overfitting.
Up to my knowledge, the GPML framework doesn’t include these computations, but perhaps there are goods other third party codes.
There is another package for machine learning using Gaussian processes called GPstuff which has it all in my opinion. You can use MCMC, integration on a grid, etc. to marginalise out your hyperparameters.
NB In the documentation they call hyperparameters merely parameters.