Transforms

Batch transforms for spectral DataLoaders

import torch

Preprocessing for classical models, such as SNV followed by a Savitzky-Golay derivative, can leave spectra with very small values. SNV scales each spectrum to unit variance, and the derivative then shrinks it again. That scale suits PLSR or a random forest. A CNN’s default weight initialization, and fastai’s ActivationStats diagnostics, assume inputs of roughly unit scale.

StandardizeSpectrum rescales each spectrum to zero mean and unit variance across wavenumbers, as it enters the network. It uses no fitted statistics. It gives the same result for a batch of one as for the whole training set, and it leaves the earlier preprocessing and the train/test split unchanged.

---

source

StandardizeSpectrum

def StandardizeSpectrum(
    eps:float=1e-06
):

Standardize each spectrum to zero mean, unit variance across wavenumbers

Spectra with a standard deviation of 0.02 come out with a mean of 0 and a standard deviation of 1:

x = TensorSpectrum(torch.randn(32, 1, 657) * 0.02)
out = StandardizeSpectrum()(x)
assert out.mean(-1).abs().max() < 1e-5
assert (out.std(-1) - 1).abs().max() < 1e-3
out.mean(-1).abs().max(), out.std(-1).min(), out.std(-1).max()

(TensorSpectrum(shape=()), TensorSpectrum(shape=()), TensorSpectrum(shape=()))