Datasets

Toy spectral dataset that ships with the package 
from fastcore.test import test_eq

The toy dataset holds 500 KSSL samples from the Open Soil Spectral Library (OSSL), drawn at random from the 45,126 KSSL samples that have a MIR spectrum and all four soil properties below. The spectra are absorbance from 600 to 4000 cm⁻¹ at a 2 cm⁻¹ step, rounded to four decimals. The properties are:

The dataset is small enough to ship with the package, so examples and tests run without downloading OSSL.

source

load_toy_kssl

def load_toy_kssl()->soilspecdata.types.Spectra:

Load 500 KSSL MIR spectra and their soil properties as a Spectra

load_toy_kssl returns a soilspecdata Spectra, the same type that load_ossl().mir() returns. Its source is a DataFrame of the four properties, indexed by sample ID:

toy = load_toy_kssl()
test_eq(toy.spectra.shape, (500, 1701))
test_eq(toy.wavenumbers[[0, -1]], [600, 4000])
toy.source.head()
k.ext_usda.a725_cmolc.kg cec_usda.a723_cmolc.kg oc_usda.c729_w.pct clay.tot_usda.a334_w.pct
id
8c6794919d33d877f5dc5beffe1d47d3 0.33046 11.80393 0.2116 34.35830
56866b267d88be922b2119bb059a19e4 1.38933 19.81524 0.9000 27.30699
ddff07dda53f7d86ca7480eecbf36d61 0.19019 8.26097 0.1000 22.31749
093b16a099e34ba51ff6103ea9b15620 0.00000 19.61994 0.3000 28.96809
f2ee1db5686b26b96dc9bb030434d092 0.35116 15.30426 1.5500 24.64173

xy gives the spectra, targets and sample IDs for machine learning, as with the full OSSL. It accepts one target or several:

X, y, ids = toy.xy('k.ext_usda.a725_cmolc.kg')
test_eq(y.shape, (500, 1))
X.shape, y.shape, ids[:3]
((500, 1701),
 (500, 1),
 array(['8c6794919d33d877f5dc5beffe1d47d3',
        '56866b267d88be922b2119bb059a19e4',
        'ddff07dda53f7d86ca7480eecbf36d61'], dtype=object))

Building the dataset

This code built soilspecdl/data/toy_kssl_mir.csv.gz from the cached OSSL file. It needs about 12 GB of memory while pandas reads OSSL:

from soilspecdata.all import load_ossl
toy_targets = ['k.ext_usda.a725_cmolc.kg', 'cec_usda.a723_cmolc.kg', 'oc_usda.c729_w.pct', 'clay.tot_usda.a334_w.pct']
ossl = load_ossl()
kssl_mir = ossl[ossl['dataset.code_ascii_txt'] == 'KSSL.SSL'].mir()
X, y, ids = kssl_mir.xy(toy_targets)
idx = np.sort(np.random.default_rng(42).choice(len(X), 500, replace=False))
props = pd.DataFrame(y[idx], index=pd.Index(ids[idx], name='id'), columns=toy_targets)
spectra = pd.DataFrame(X[idx].round(4), index=props.index, columns=kssl_mir.wavenumbers)
pd.concat([props, spectra], axis=1).to_csv('../soilspecdl/data/toy_kssl_mir.csv.gz')