model fitting errors for SingleTaskMultiFidelityGP & MultiTaskGP

[ToennisStef]

Hi
During optimization runs using the SingleTaskMultiFidelityGP and the MultiTaskGP i often encounter model fitting errors using the fit_gpytorch_mll function.

I was wondering where these model fitting problems came from. I was also wondering if both models require a strict ordering of the fidelity values? and how do these models handle non-linear inter-task/inter-fidelity dependencies?
For example, I got this test problem with 4 fidelity levels:

Here the SingleTaskMultiFidelityGP had some model fitting issues.
I know that these issues can also arise when there are identical or many close data points, which then leads to an ill-conditioned covariance matrix. But could some problems also arise from the fact that there are non-linear inter-fidelity/inter-task dependencies and/or that the model requires a strict ordering of the fidelity functions, which is not given (meaning target fidelity has lowest function values on the whole domain, second highest fidelity has 2nd lowest function values on the whole domain, etc.)?

To be honest I also set the noise covariance of the models close to zero, which I know can also be troublesome, but I cannot pinpoint the exact problem since the error is not very informative.

Best regards,
Stefan Tönnis

[sdaulton]

SingleTaskMultiFidelityGP does assume that the fidelity of the function is monotonic in the parameter (based on the kernel from https://arxiv.org/abs/1903.04703). You could use a SingletaskGP if the relationship between the fidelity parameter and fidelity of the function is not monotonic. That maybe the source of the model fitting errors. What are the model fitting errors?

[ToennisStef]

A quick followup on this one:

The generall error message when using the fit_gpytorch_mll on the SingleTaskMultiFidelityGP was very uninformative since it was just:
ModelFittingError: All attempts to fit the model have failed.

I wanted to get a general overview of the mll space and therefore plotted the mll for different values of the different hyperparameters of the SingleTaskMultiFidelityGP. However, when setting the hyperparameter values manually I noticed, that i always got a

NotPSDError: Matrix not positive definite after repeatedly adding jitter up to 1.0e-03.

Error message when setting the "power" hyperparameter from the LinearTruncatedFidelityKernel to values below 1. I noticed that the constraint on this hyperparameter is generally a positivity constraint, not a >=1 constraint.
My understanding of the model or rather the LinearTruncatedFidelityKernel is not wellenough to make proper sense of this. I can't tell if its generally problematic if the power hyperparameter of the LinearTruncatedFidelityKernel takes values between 0 and 1, or if its generally possible that this hyperparameter takes values between 0 and 1 and my dataset is causing weird problems.

Also when logging the parameter values during optimization with a callback function. i noticed that the final power hyperparameter value before the ModelFittingError is raised is also below 1. I would therefore assume that this is causing the model fitting error.

Any insight on this is highly appreciated!
Best regards

[ToennisStef]

Now to update again:
I see that the LinearTruncatedFidelityKernel should generally be PSD for power hyperparameter values greater than 0. But when plotting the eigenvalues of the covariance matrix over different power hyperparameter values, i could notice discontinuities:

The orange gradient plot is the gradient of the specific eigenvalue with respect to the raw_power parameter of the model.

My question would be why do these discontinuities occur? Can this be prevented?
I am updating this post as soon as possible with proper code for reproducibility.

[Balandat]

Hmm that is rather interesting indeed. This does look quite clean so it doesn't obviously seem be caused by any numerical issues. Please do share a repro for how you generated the plot.