More options for the interpolation of sparse spectra

[m-lohmann]

I’m opening this issue because I could not find any package working similar to MATLAB’s interp functions. So I started to write my own.
The reason is that spectral data only exists as sparse datasets where interpolation is needed to produce good colorimetric results. As far as I can see, colormatch.jl only provides us with linear interpolation, which I find quite surprising in the context of producing color matching functions.
In colorimetry cubic interpolation is the most common and recommended way to deal with sparse spectra. Usually it is recommended to avoid linear interpolation for obvious reasons.

The interp1d function I wrote is able to interpolate in several ways. I still need to sort out some errors (calling me a “programmer” would be an insult to all of you), but here’s a preview of a few interpolation schemes I have implemented:

http://nbviewer.ipython.org/gist/m-lohmann/e67e67bf4381771d67e8

I’m demonstrating only a handful of schemes because the difference is not visible in the graph showing the full interpolated spectrum.

I’m also going to implement a simple extrapolation function to cope with the problem of missing data at the ends of the spectrum, so you can choose e.g. linear, parabolic or cubic extrapolation to fill in missing data.

Please tell me if someone else already has written a package to do all this, or of someone else is already working on that.
Any thoughts or suggestions?

[timholy]

https://github.com/timholy/Grid.jl

If you go to http://iainnz.github.io/packages.julialang.org/, hit Ctrl-F and type interp you will find it.

[timholy]

Re extrapolation, that's something currently under discussion: timholy/Grid.jl#37

[m-lohmann]

Thanks, @timholy
I noticed the Grid package before, but it’s using the B-spline representation.
I tried to get around the very problem that for irregular grids only nearest neighbor and linear interpolation are available without problems in Grid

My (poor, but working) implementation enables the user to choose from several boundary conditions, like natural (zero 2nd order derivatives), clamped (arbitrary 1st order derivatives), curvature-adjusted (arbitrary 2nd order derivatives) and not-a-knot conditions. Everything is just tailored to the 1-dimensional case of spectral representations. Maybe that is not a smart move on my part.

I’ll try out how your implementation works.

[timholy]

It would just be strange to contribute interpolation algorithms to a package devoted to color, when there is an interpolation package. Why not contribute them to Grid?

[m-lohmann]

@timholy I guess you’re right. I was just playing with this idea because e.g. linear interpolation is implemented at least twice in Color package. Once in colormatch.jl and once in utilities.jl. Linear interpolation is quite simple to implement, of course.
As I said before, I’m far from what you could call a programmer, and I wouldn’t know where to start to implement my interpolation algorithms in your B-spline implementation. I’m using ordinary piecewise spline interpolation, predominantly cubic splines, and my implementation only treats the 1-d case.
I’ll try to accelerate my implementation first, before trying to add them to the Grid package. My cubic splines are still quite slow.

[timholy]

I hadn't noticed it's here. That said, linear interpolation is so trivial, it's not crazy to have local definitions. But once you start adding boundary conditions, higher-order interpolation, etc, it makes more sense to use a common infrastructure.

Regarding implementation, Grid doesn't have to be a B-splines-only package. You may find timholy/Grid.jl#38 inspirational (though complex) if you're looking for really fast implementations.

[timholy]

I'll close this in favor of continuing the conversation over at Grid.

[dcjones]

It would be really nice to do non-linear interpolation in colormatch. Linear interpolation is used now only because I don't do colorimetry and it seemed “good enough” to me. But I agree with Tim that we should try to consolidate non-trivial interpolation functionality in Grid.

[m-lohmann]

Yes, that sounds reasonable.

[glennsweeney]

Sorry I'm so late; I've been traveling this summer. It's worth mentioning that the ASTM outlines very specific interpolation instructions when integrating spectra for tristimulus value calculation (ASTM E2022). This is specifically to try to correct for bandpass error in a wide-band measurement device. However, I'm not sure if this is support we really need or not (in my experience, this correction is usually ignored).

[m-lohmann]

This is for bandpass correction, yes. Most spectra I found already have the correction implemented in the measurement device software if these measurements were taken with high-end spectrophotometers. Some measurements I found were made with more “exotic” devices like tunable liquid crystal filters, glass filters (like the multispectral MER pancam images) and such. I think in academic use the bandpass and other appropriate corrections are already taken into consideration.
It wouldn’t be too hard to write a function to apply necessary corrections for measured spectra which take into account all the variables and bandpass characteristics.
Often you can’t even get any information about the measurement device if you try to find spectra for certain standards, like Munsell, Dupont, ceramic transfer standards (which can be even temperature dependent) and other spectra, or if you have to reconstruct spectra from diagrams because the actual data aren’t public whereas the bandpass is very narrow (2 nm). I know that this isn’t appropriate for high precision work, but in the majority of the cases the precision is more than high enough for “everyday” use.
As far as I know in most modern measurement devices the bandpass correction according to ASTM is already implemented. E.g. in the case of the ceramic transfer standards the measurements were taken according to ASTM and the CIE practices.

[glennsweeney]

Very good points @m-lohmann. I guess I just brought it up because It was on my mind; I had spent the last weeks of class before the summer calibrating an LCTF spectral imaging system, and this came up in conversation. You're completely right that commercial devices should already account for this.