OSSL datasets

Data loading for the OSSL dataset

The OSSL database description describes the OSSL variables and their column names.

The `OSSL` container

OSSL

def OSSL(
    df:pandas.DataFrame, # OSSL data, one row per sample
):

OSSL (Open Soil Spectral Library) data container, indexed by sample ID

OSSL indexes its DataFrame by sample ID (id), using the id.layer_uuid_txt column. The OSSL documentation names this column as the key for joining OSSL tables. Every row has a distinct value.

OSSL passes any attribute it doesn’t define to its DataFrame, through fastcore’s GetAttr. So ossl.loc, ossl.columns and the other DataFrame attributes work directly on ossl. Indexing with a column name or a list of names returns those columns, as in pandas. Indexing with anything else, such as a boolean mask, returns a new OSSL with the selected samples.

The local ID id.layer_local_c is not unique. In OSSL v1.2, 82 local IDs appear in two source datasets. For example, 1002 identifies one sample in CAF.SSL and another in KSSL.SSL. Joining on the local ID pairs the spectra of one sample with the properties of the other. To see the local ID and its source dataset, select them with ossl[['id.layer_local_c', 'dataset.code_ascii_txt']].

source

OSSL._parse_columns

def _parse_columns():

Parse columns into visnir, mir and properties

Loading the data

source

get_cache_path

def get_cache_path(
    dest_dir:str='.soilspecdata', # Name of the cache directory, relative to the home directory
)->pathlib.Path: # Path to the cache directory

Get cache path for OSSL data

The cache directory is ~/.soilspecdata by default:

get_cache_path()

Path('/Users/franckalbinet/.soilspecdata')

load_ossl downloads the whole OSSL v1.2 database as one gzipped CSV file. OSSL publishes it at two data levels:

L0 (the default, ossl_all_L0_v1.2.csv.gz) keeps the lab method of each source dataset. The same property can have one column per method, such as cec_usda.a723_cmolc.kg and cec_iso.11260_cmolc.kg.
L1 (ossl_all_L1_v1.2.csv.gz) converts properties measured with different methods to one target method. It has 24 fewer property columns than L0, for example no cec_iso.11260_cmolc.kg.

Both levels have the same spectra and ID columns. load_ossl caches the file under its own name, in the directory from get_cache_path. Each level or url therefore has its own cache file.

source

load_ossl

def load_ossl(
    level:str='L0', # OSSL data level: 'L0' (original lab methods) or 'L1' (lab methods converted to a target method)
    url:NoneType=None, # OSSL gzipped CSV URL; if None, the whole-database file for `level`
    force_download:bool=False, # if True, download even if the file is cached
)->__main__.OSSL:

Load OSSL data from the cache, downloading it first if needed

Load the default L0 data, from the cache if it is there:

ossl = load_ossl()

Columns

OSSL sorts the columns into VISNIR spectra (visnir_cols), MIR spectra (mir_cols) and all other columns (meta_cols). The other columns hold soil properties, site data and metadata:

ossl.visnir_cols[:2], ossl.mir_cols[:2], ossl.meta_cols[:2]

(['scan_visnir.350_ref', 'scan_visnir.352_ref'],
 ['scan_mir.600_abs', 'scan_mir.602_abs'],
 ['dataset.code_ascii_txt', 'id.layer_local_c'])

meta_cols has 133 columns in v1.2. Soil-lab column names follow the pattern <property>_<method>_<unit>. For example, cec_usda.a723_cmolc.kg is the cation exchange capacity (CEC), measured with USDA method A723, in cmolc/kg. To find the columns for one property, filter on its name:

[c for c in ossl.meta_cols if c.startswith('cec')]

['cec_iso.11260_cmolc.kg', 'cec_usda.a723_cmolc.kg']

The same property can have several methods. The methods come from USDA or ISO standards, aqua regia extraction (aquaregia) or Mehlich-3 extraction (mel3). The table below counts the soil-lab columns of each property by standard. It leaves out the ISO 8601 date columns and the ISO 3166 country column, which use ISO formats rather than lab methods:

lab = pd.Series(ossl.meta_cols).str.extract(r'^(?P<property>[^_]+)_(?P<method>(?:usda|iso)\.\w+|aquaregia|mel3)_').dropna()
lab = lab[~lab.method.isin(['iso.8601', 'iso.3166'])]
pd.crosstab(lab.property, lab.method.str.split('.').str[0])

method	aquaregia	iso	mel3	usda
property
acidity	0	0	0	1
aggstb	0	0	0	1
al.dith	0	0	0	1
al.ext	1	0	0	2
al.ox	0	0	0	1
awc.33.1500kPa	0	0	0	1
b.ext	0	0	1	0
bd	0	1	0	2
c.tot	0	1	0	1
ca.ext	1	0	0	2
caco3	0	1	0	1
cec	0	1	0	1
cf	0	1	0	1
clay.tot	0	1	0	1
cu.ext	0	0	0	1
ec	0	1	0	1
efferv	0	0	0	1
fe.dith	0	0	0	1
fe.ext	1	0	0	1
fe.ox	0	0	0	1
k.ext	1	0	0	2
mg.ext	1	0	0	2
mn.ext	1	0	0	2
n.tot	0	2	0	1
na.ext	1	0	0	2
oc	0	1	0	2
p.ext	1	1	0	4
ph.cacl2	0	1	0	2
ph.h2o	0	1	0	1
s.ext	0	0	1	0
s.tot	0	0	0	1
sand.tot	0	1	0	2
silt.tot	0	1	0	2
wr.10kPa	0	0	0	2
wr.1500kPa	0	0	0	1
wr.33kPa	0	0	0	2
zn.ext	0	0	0	1

Selecting samples

Filter samples with ordinary pandas conditions on the columns of ossl. The result is an OSSL with the selected samples, so every method in this notebook works on it. The KSSL samples only:

kssl = ossl[ossl['dataset.code_ascii_txt'] == 'KSSL.SSL']
test_eq(set(kssl['dataset.code_ascii_txt']), {'KSSL.SSL'})
len(kssl), len(ossl)

(86603, 135651)

Conditions combine with & and |. US samples whose layer starts less than 30 cm below the surface:

us_top = ossl[(ossl['location.country_iso.3166_txt'] == 'USA') & (ossl['layer.upper.depth_usda_cm'] < 30)]
len(us_top)

The metadata also records the instrument that measured each spectrum. Seven MIR instrument models appear in v1.2:

ossl['scan.mir.model.name_utf8_txt'].value_counts()

scan.mir.model.name_utf8_txt
Bruker Vertex 70 with HTS-XT accessory               78391
Bruker Tensor 27 with HTS-XT accessory                5975
Bruker Vertex 70 with with PikeAutoDiff accessory      589
Bruker Tensor 27                                       259
Bruker Invenio-S with HTS-XT                           184
Bruker Alpha 1 ZnSe                                    151
ThermoFisher Nicolet Nexus 670                         135
Name: count, dtype: int64

Spectra

source

OSSL._get_valid_spectra_mask

def _get_valid_spectra_mask(
    spectra_cols:List[str], # Spectra column names
)->pandas.Series: # True for samples with a value in every column of `spectra_cols`

Mask of the samples with complete spectra in spectra_cols

The OSSL file has one row per sample, with metadata, soil properties, VISNIR spectra and MIR spectra as columns. Not every sample was scanned with both instruments. The spectra columns of a missing scan are empty (NaN):

ossl[ossl.mir_cols]

	scan_mir.600_abs	scan_mir.602_abs	scan_mir.604_abs	scan_mir.606_abs	scan_mir.608_abs	scan_mir.610_abs	scan_mir.612_abs	scan_mir.614_abs	scan_mir.616_abs	scan_mir.618_abs	scan_mir.620_abs	scan_mir.622_abs	scan_mir.624_abs	scan_mir.626_abs	scan_mir.628_abs	scan_mir.630_abs	scan_mir.632_abs	scan_mir.634_abs	scan_mir.636_abs	scan_mir.638_abs	scan_mir.640_abs	scan_mir.642_abs	scan_mir.644_abs	scan_mir.646_abs	scan_mir.648_abs	scan_mir.650_abs	scan_mir.652_abs	scan_mir.654_abs	scan_mir.656_abs	scan_mir.658_abs	scan_mir.660_abs	scan_mir.662_abs	scan_mir.664_abs	scan_mir.666_abs	scan_mir.668_abs	scan_mir.670_abs	scan_mir.672_abs	scan_mir.674_abs	scan_mir.676_abs	scan_mir.678_abs	...	scan_mir.3922_abs	scan_mir.3924_abs	scan_mir.3926_abs	scan_mir.3928_abs	scan_mir.3930_abs	scan_mir.3932_abs	scan_mir.3934_abs	scan_mir.3936_abs	scan_mir.3938_abs	scan_mir.3940_abs	scan_mir.3942_abs	scan_mir.3944_abs	scan_mir.3946_abs	scan_mir.3948_abs	scan_mir.3950_abs	scan_mir.3952_abs	scan_mir.3954_abs	scan_mir.3956_abs	scan_mir.3958_abs	scan_mir.3960_abs	scan_mir.3962_abs	scan_mir.3964_abs	scan_mir.3966_abs	scan_mir.3968_abs	scan_mir.3970_abs	scan_mir.3972_abs	scan_mir.3974_abs	scan_mir.3976_abs	scan_mir.3978_abs	scan_mir.3980_abs	scan_mir.3982_abs	scan_mir.3984_abs	scan_mir.3986_abs	scan_mir.3988_abs	scan_mir.3990_abs	scan_mir.3992_abs	scan_mir.3994_abs	scan_mir.3996_abs	scan_mir.3998_abs	scan_mir.4000_abs
id
ded584d70bf0474202d30e575ae24784	1.527853	1.531908	1.532084	1.530892	1.530645	1.531506	1.531582	1.531413	1.532904	1.535459	1.536158	1.534819	1.533536	1.533254	1.533507	1.533371	1.531508	1.528686	1.527023	1.525885	1.523678	1.521059	1.519020	1.517138	1.515718	1.515071	1.514072	1.512213	1.510424	1.508897	1.507223	1.505287	1.502408	1.499718	1.500058	1.501609	1.500349	1.498688	1.497970	1.498123	...	0.364978	0.364960	0.365489	0.364571	0.363776	0.365061	0.366197	0.365321	0.363685	0.362150	0.361905	0.362285	0.361591	0.361145	0.361693	0.361794	0.361289	0.360782	0.360347	0.359986	0.359963	0.359857	0.359324	0.358756	0.358343	0.357963	0.357814	0.357771	0.357349	0.356829	0.356776	0.356642	0.355784	0.354743	0.354104	0.353663	0.353237	0.352923	0.352548	0.352053
7c080e182bc23e19747629f46edbc6c4	1.538449	1.543622	1.545751	1.546997	1.549450	1.553714	1.557981	1.561652	1.566082	1.571555	1.576085	1.579329	1.582497	1.584857	1.586286	1.587778	1.589129	1.589709	1.589914	1.590310	1.591163	1.591589	1.590084	1.587867	1.587066	1.586649	1.586043	1.586114	1.585297	1.582799	1.580890	1.580357	1.579843	1.579577	1.581481	1.582966	1.581305	1.580092	1.580319	1.581344	...	0.370676	0.370564	0.370977	0.369939	0.369043	0.370199	0.371154	0.370077	0.368243	0.366490	0.366035	0.366243	0.365412	0.364868	0.365323	0.365307	0.364670	0.364043	0.363489	0.362972	0.362796	0.362586	0.361964	0.361277	0.360739	0.360235	0.359944	0.359736	0.359177	0.358555	0.358399	0.358142	0.357144	0.355980	0.355242	0.354722	0.354217	0.353825	0.353376	0.352798
03bbd570dfd399bfd866ebcdf860de39	1.619721	1.614226	1.615612	1.620649	1.626406	1.631747	1.636411	1.639527	1.642449	1.646890	1.650462	1.651677	1.651978	1.651377	1.650334	1.650127	1.649949	1.648931	1.647869	1.646961	1.645410	1.643513	1.642263	1.640988	1.639526	1.638149	1.636536	1.634641	1.633043	1.631582	1.629631	1.628164	1.626684	1.624694	1.624018	1.623806	1.621784	1.620112	1.619768	1.619266	...	0.382570	0.382389	0.382765	0.381925	0.381173	0.382303	0.383363	0.382567	0.380999	0.379428	0.378947	0.379114	0.378382	0.377841	0.378239	0.378314	0.377851	0.377363	0.376910	0.376490	0.376373	0.376222	0.375703	0.375130	0.374663	0.374181	0.373902	0.373735	0.373220	0.372628	0.372522	0.372338	0.371425	0.370337	0.369679	0.369245	0.368808	0.368469	0.368084	0.367563
887d316328ef950eeca33293bfe052bf	1.570129	1.567954	1.573055	1.580834	1.586880	1.590397	1.595117	1.600492	1.603847	1.606447	1.609863	1.613982	1.618182	1.621086	1.623261	1.625244	1.626082	1.625795	1.625471	1.625152	1.624547	1.624070	1.623793	1.623076	1.622038	1.620846	1.619842	1.619199	1.618141	1.617494	1.617702	1.616519	1.613644	1.611504	1.611699	1.612149	1.610739	1.609205	1.608250	1.608764	...	0.370839	0.370572	0.370860	0.369891	0.369071	0.370206	0.371199	0.370237	0.368495	0.366777	0.366209	0.366277	0.365402	0.364779	0.365132	0.365128	0.364579	0.364010	0.363474	0.362983	0.362808	0.362590	0.361983	0.361291	0.360698	0.360105	0.359728	0.359469	0.358872	0.358189	0.357992	0.357734	0.356713	0.355480	0.354681	0.354137	0.353619	0.353217	0.352756	0.352158
ca03ce6e439b1909c5a4a771c0afa951	1.484832	1.484367	1.484977	1.486258	1.488400	1.492040	1.495075	1.496595	1.498354	1.501437	1.504405	1.505870	1.505213	1.503459	1.502728	1.503256	1.504089	1.504402	1.503446	1.501327	1.499069	1.497556	1.496766	1.495106	1.492753	1.490479	1.488640	1.487091	1.485670	1.484284	1.482533	1.480965	1.479360	1.476982	1.475857	1.476914	1.477013	1.476642	1.476029	1.474411	...	0.324934	0.324909	0.325455	0.324506	0.323767	0.325287	0.326602	0.325714	0.323969	0.322297	0.321990	0.322363	0.321583	0.321069	0.321660	0.321803	0.321288	0.320743	0.320248	0.319807	0.319741	0.319628	0.319080	0.318486	0.318047	0.317634	0.317442	0.317350	0.316870	0.316302	0.316249	0.316089	0.315098	0.313910	0.313210	0.312758	0.312312	0.311971	0.311568	0.311044
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
13b75cfc186ce496b308579a63521dad	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
ce7853e20ef27df1940eff8c4082a822	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3ea4f0532305d3cccdd2d96a1cfb54f3	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
8c2fb2e1293f915ff469993dc87bd474	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
0f6f8be23cd07a968f36aea9b0524870	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN

135651 rows × 1701 columns

_get_valid_spectra_mask selects the samples with a value in every given column. This example counts the samples with a complete MIR spectrum, and all samples:

mask = ossl._get_valid_spectra_mask(ossl.mir_cols)
mask.sum(), len(mask)

(np.int64(85684), 135651)

source

OSSL._extract_wavenumbers

def _extract_wavenumbers(
    cols:List[str], # Spectral column names
)->numpy.ndarray: # Wavenumbers in cm⁻¹

Wavenumbers of spectral columns, converting VISNIR wavelengths in nm to wavenumbers

MIR column names give the wavenumber in cm⁻¹, as in scan_mir.600_abs. VISNIR column names give the wavelength in nm, as in scan_visnir.350_ref. _extract_wavenumbers converts wavelengths to wavenumbers (10⁷/λ, rounded to the nearest integer), so every method of OSSL works in wavenumbers. The 350 nm VISNIR column becomes 28571 cm⁻¹:

ossl.mir_cols[0], ossl._extract_wavenumbers(ossl.mir_cols)[0], ossl.visnir_cols[0], ossl._extract_wavenumbers(ossl.visnir_cols)[0]

('scan_mir.600_abs', np.int64(600), 'scan_visnir.350_ref', np.int64(28571))

source

OSSL._extract_measurement_type

def _extract_measurement_type(
    cols:List[str], # Spectral column names
)->str: # `abs` (absorbance) or `ref` (reflectance)

Extract measurement type from column names

The suffix of the column names gives the measurement type. OSSL reports VISNIR spectra as reflectance and MIR spectra as absorbance:

ossl._extract_measurement_type(ossl.visnir_cols), ossl._extract_measurement_type(ossl.mir_cols)

('ref', 'abs')

source

OSSL._filter_wavenumbers

def _filter_wavenumbers(
    wavenumbers:numpy.ndarray, # Wavenumbers of `cols`
    cols:List[str], # Spectral column names
    wmin:Optional[int]=None, # Min wavenumber; the smallest in `wavenumbers` if None
    wmax:Optional[int]=None, # Max wavenumber; the largest in `wavenumbers` if None
)->Tuple[numpy.ndarray, List[str]]: # Wavenumbers and column names within the range

Wavenumbers and columns between wmin and wmax, inclusive

The VISNIR columns are in order of increasing wavelength. Their wavenumbers therefore decrease. Between 4000 and 25000 cm⁻¹, the VISNIR columns run from 400 nm (25000 cm⁻¹) to 2500 nm (4000 cm⁻¹):

wavenumbers, cols = ossl._filter_wavenumbers(ossl._extract_wavenumbers(ossl.visnir_cols), ossl.visnir_cols, wmin=4000, wmax=25000)
wavenumbers.min(), wavenumbers.max(), cols[0], cols[-1]

(np.int64(4000),
 np.int64(25000),
 'scan_visnir.400_ref',
 'scan_visnir.2500_ref')

source

OSSL.visnir

def visnir(
    wmin:Optional[int]=4000, # Min wavenumber
    wmax:Optional[int]=25000, # Max wavenumber
)->soilspecdata.types.Spectra: # VISNIR spectra, in increasing wavenumber order

Complete VISNIR spectra between wmin and wmax

visnir and mir return only the samples with a value at every wavenumber in the range. Their wavenumbers increase along the columns, for both VISNIR and MIR. VISNIR spectra between 8000 and 25000 cm⁻¹:

visnir_data = ossl.visnir(wmin=8000, wmax=25000)
test_eq((np.diff(visnir_data.wavenumbers) > 0).all(), True)
visnir_data.spectra.shape, visnir_data.wavenumbers[0], visnir_data.wavenumbers[-1]

((64644, 426), np.int64(8000), np.int64(25000))

source

OSSL.mir

def mir(
    wmin:Optional[int]=600, # Min wavenumber
    wmax:Optional[int]=4000, # Max wavenumber
)->soilspecdata.types.Spectra: # MIR spectra, in increasing wavenumber order

Complete MIR spectra between wmin and wmax

MIR spectra over the default range, 600 to 4000 cm⁻¹:

mir_data = ossl.mir()
mir_data.spectra.shape, mir_data.wavenumbers.min(), mir_data.wavenumbers.max()

((85684, 1701), np.int64(600), np.int64(4000))

Properties and metadata

Select soil properties and metadata as columns of ossl. The result is a DataFrame indexed by sample ID. Machine learning targets need a value for every sample. dropna drops the samples with a missing CEC value:

targets = ossl[['cec_usda.a723_cmolc.kg']].dropna()
targets.shape, targets.head()

((57064, 1),
                                   cec_usda.a723_cmolc.kg
 id                                                      
 3998362dd2659e2252cd7f38b43c9b1f                6.633217
 2bab4dbbac073b8648475ad50d40eb95                3.822628
 29213d2193232be8867d85dec463ec00                3.427324
 8b1ee9146c026faee20a40df86736864                1.906545
 6e8e9d1105e7da7055555cb5d310df5f               13.403203)

Metadata such as coordinates can have missing values (NaN). Without dropna, the selection keeps every sample:

metadata = ossl[['longitude.point_wgs84_dd', 'latitude.point_wgs84_dd']]
metadata.shape, metadata.head()

((135651, 2),
                                   longitude.point_wgs84_dd  latitude.point_wgs84_dd
 id                                                                                 
 ded584d70bf0474202d30e575ae24784                 15.687492                -7.377750
 7c080e182bc23e19747629f46edbc6c4                 15.687492                -7.377750
 03bbd570dfd399bfd866ebcdf860de39                 15.687817                -7.351243
 887d316328ef950eeca33293bfe052bf                 15.687817                -7.351243
 ca03ce6e439b1909c5a4a771c0afa951                 15.687965                -7.331673)

Training data

xy keeps the samples with both a complete spectrum and complete targets. It reads the targets from the OSSL the spectra came from. It returns the spectra X, the targets y and the sample IDs ids, in the same row order. MIR spectra and CEC values:

X, y, ids = mir_data.xy('cec_usda.a723_cmolc.kg')

test_eq(len(set(ids)), len(ids))
X.shape, y.shape, ids.shape

((57062, 1701), (57062, 1), (57062,))

The DataFrame of ossl is indexed by sample ID, so ids finds other columns for the same samples:

ossl.loc[ids, ['longitude.point_wgs84_dd', 'latitude.point_wgs84_dd']].head()

	longitude.point_wgs84_dd	latitude.point_wgs84_dd
id
3998362dd2659e2252cd7f38b43c9b1f	174.42	-36.78
2bab4dbbac073b8648475ad50d40eb95	174.42	-36.78
29213d2193232be8867d85dec463ec00	174.42	-36.78
8b1ee9146c026faee20a40df86736864	174.42	-36.78
6e8e9d1105e7da7055555cb5d310df5f	174.55	-36.67

xy works on any selection of samples. To test how a model transfers between spectral libraries, train on every source dataset except KSSL, and test on KSSL. KSSL holds most of the MIR spectra with a CEC value. The training set is therefore much smaller than the test set:

is_kssl = ossl['dataset.code_ascii_txt'] == 'KSSL.SSL'
X_train, y_train, _ = ossl[~is_kssl].mir().xy('cec_usda.a723_cmolc.kg')
X_test, y_test, _ = ossl[is_kssl].mir().xy('cec_usda.a723_cmolc.kg')
X_train.shape, X_test.shape

((3835, 1701), (53227, 1701))

The OSSL container

OSSL

OSSL._parse_columns

Loading the data

get_cache_path

load_ossl

Columns

Selecting samples

Spectra

OSSL._get_valid_spectra_mask

OSSL._extract_wavenumbers

OSSL._extract_measurement_type

OSSL._filter_wavenumbers

OSSL.visnir

OSSL.mir

Properties and metadata

Training data

The `OSSL` container