Per-dimension constraints on RBFKernel lengthscales

[dmitrii-khizbullin]

I am fitting a GP into a dataset with 379 points in F=7 dimensions. I use BoTorch but the question is in the underlying GPyTorch. I am using the combo of RBFKernel, FixedNoiseGP, ExactMarginalLogLikelihood and fit_gpytorch_mll. After fitting the kernel lengthscales are like this:

lengthscale: [0.17149281 0.44241985 0.55040405 0.09258469 0.00784354 0.50260822 0.40851717]

What is worrysome is that the 5th lengthscale is very low:

lengthscale[4] = 0.00784354

which leads to heavy overfit on the coreesponding feature.

As recommended here I put a LowerThan constraint on my RBFKernel lenghtscale like this:

rbf_kernel = RBFKernel(
    ard_num_dims=feature_pt.shape[-1],
    lengthscale_constraint=Interval(0.1, 0.3))

This regularizes the GP fine, but still feels like I may constrain other dimensions too much.

Q1: Is there a way to create a list of F separate constraints, so I can bound just one dimention's lengthscale?

Also I noticed that the learnt lengthscales are very unreliable, on each refit the values are significantly different.

Q2: How to achieve stable fitting of the GP in the sense that the lengthscales end up about the same on each refit?

[jacobrgardner]

The constraints can take Tensors as lower and/or upper bounds -- just specify a bound for each lengthscale individually when making the LessThan (or Interval) constraint.

Rather than hard constraints, you could also consider just placing priors over the lengthscales that more softly encourage larger lengthscales as well.

In terms of getting the same hypers every time -- the optimization problem is non convex, so your only real recourse here would be to optimize multiple times. That said, if you are seeing stability issues, it's probably not just optimization failing here -- when things are working well, you'll usually end up with a reasonably consistent model. What exactly are you seeing that makes you think multiple solutions is the problem?

[dmitrii-khizbullin]

Dear Jakob, thank you for the extensive reply. Thanks for pointing out that the bounds also accept Tensors. Actually I would love to set both priors and the interval boundaries to the lengthscales. With the priors I tried to make it run with the following code but did not manage to make it work.

rbf_kernel = RBFKernel(
    lengthscale_prior=MultivariateNormalPrior(
        torch.zeros(num_dims),
        torch.tensor(np.diag(empirical_lengthscales))),
...

I will try your solution shortly.

With regards to the stability, the point is that if I provide no priors to the lengthscales and no strict lengthscale constraints, then the following condition leads to overfit on a random feature. Say all features have some variance and the lengthscale constraint is Positive. Then after fit is done, at least one (random) lengthscale is tiny, whereas all others have random madium and high values. In another re-launch, another lengthscale becomes tiny and all other lenghscales have absolutely different values. Thus without any lower bounds on the lengthscales, a GP will overfit on any feature that expresses at least some variance. This is the problem from the engineering perspective.

[dmitrii-khizbullin]

Seems that the bug with tensor lengthscale bounds comes from BoTorch as I've posted here.

[dmitrii-khizbullin]

The bug in setting the constraints in BoTorch has been fixed.

My solution to freeze the lengthscales is this ugly one, which works for me though.

    rbf_kernel = RBFKernel(
        ard_num_dims=num_dims,
        lengthscale_constraint=Interval(
            torch.tensor(empirical_lengthscales, dtype=torch.float32),
            torch.tensor(empirical_lengthscales+0.01, dtype=torch.float32),
            ),
        )
    gp_kernel = ScaleKernel(rbf_kernel)
    gp = FixedNoiseGP(feature_pt, objective_pt, train_Yvar=objective_var_pt,
                      covar_module=gp_kernel)
    mll = ExactMarginalLogLikelihood(gp.likelihood, gp)
    ret = fit_gpytorch_mll(mll)

@jacobrgardner if you could point to a code snippet that showcases use of lengthscale_prior parameter, I would highly appreciate. I did not manage to make it work for me.