Fram Strait 2020-2025

Convert the Fram Strait I-129, U-236 and U-238 seawater records on Zenodo to MARIS NetCDF4

This handler converts the Fram Strait seawater records on Zenodo to MARIS NetCDF4. RECORDS lists six records, from cruises between 2020 and 2025. They hold I-129, U-236 and U-238 measurements from two laboratories: the Laboratory of Ion Beam Physics at ETH Zürich (ETH) and the Vienna Environmental Research Accelerator at the University of Vienna (VERA). Laboratory reconciliation is in progress. The pipeline:

aligns the columns of each record to one schema, and keeps one laboratory’s measurement for each sample and nuclide
renames columns to MARIS names, parses dates and computes depth from pressure
reshapes the value and uncertainty columns into one row per measurement
converts U-238 from ppb to atoms per kg
maps nuclides, units and laboratories to MARIS ids, and marks every value as detected.

encode then adds global attributes and writes the SEAWATER group to one NetCDF file.

from marisco.nc2csv import to_csv
from marisco.configs import get_lut, NC_VARS

Curation summary

Each row is a decision MARIS made about the provider’s data. The last column is the change in the provider’s delivery that would make the decision unnecessary. The linked section has the details. What the pipeline drops lists each column, value and sample that the output leaves out.

What MARIS found	What MARIS did	Request to the provider
Column names encode the laboratory, nuclide and unit, such as `ETH_unc_I129_at_l`.	Parsed the names with `MEAS_PAT` (Reshape to one row per measurement).	Use a tidy layout, with laboratory, nuclide, unit, value and uncertainty in separate columns.
Column names differ between records. FS2020_21 uses `Bottle`, `Salinity_ctd` and `Temperature_degC_ctd`. FS2022S uses `Latitude` and `Longitude`.	Wrote one adapter for FS2020_21 and one for FS2022 to FS2025 (Align record schemas).	Use the same column names in every record.
In FS2020_21, unprefixed columns hold ETH’s I-129 and VERA’s U-236 and U-238.	Added the laboratory prefix by nuclide (Align record schemas).	Prefix every measurement column with its laboratory, or add a laboratory column.
Dates are `dd/mm/yyyy` in FS2020_21, `yyyy-mm-dd` in four records, and separate `Year`, `Month` and `Day` columns in FS2022S.	Converted every date to `yyyy-mm-dd` (Parse time).	Report dates in one format, preferably ISO 8601 (`yyyy-mm-dd`).
ETH and VERA both measured U-236 and U-238 in some FS2020_21 samples.	Kept VERA’s value where VERA reported one, and ETH’s value otherwise, as the provider asked (Laboratory precedence).	None.
FS2020_21 has ETH U-236/U-238 ratio columns.	Dropped them. MARIS derives the ratio from the U-236 and U-238 values.	None.
`Sample_ID` restarts at 1 in each cruise. MARIS has no cruise field.	Kept `Sample_ID` as the provider sample id. It doesn’t identify a sample in the MARIS output. Discussed in #85.	Give each sample an id that is unique across cruises.
ETH’s I-129 and VERA’s U-236 are reported in both atoms per litre and atoms per kg.	Kept both. 517 measurements appear twice in the output, once in each unit (Reshape to one row per measurement).	Say which unit is the measured one.
The records have no detection flag.	Marked every value as detected (Detection limit).	Report a detection flag, and the detection limit for values below it.
U-238 is reported in ppb.	Converted it to atoms per kg with the molar mass of U-238 (Convert U-238 units).	Confirm that ppb refers to the U-238 isotope, not to total uranium.
FS2022 to FS2025 report pressure, not depth.	Computed the depth from `Pressure_dbar` with TEOS-10 for every record (Compute depth from pressure).	None.
FS2020_21 has columns with no MARIS field, such as `Con_Temperature_degC_ctd`, `density_kg_m3` and `Water_Mass_ctd`. It also has `Bottom_Depth_m` and the time of day in `Time`, which MARIS could store but the handler doesn’t map yet.	Dropped them (Raw columns not kept).	None.
Two FS21 samples, at station 372 in Niskin bottles 10 and 11, have no measurement.	Left them out of the output (Samples without a measurement).	Remove such samples from the record, or say why they have no value.
`Cruise`, `Niskin` and `Pressure_dbar` have no MARIS field.	Used `Pressure_dbar` to compute depth. The encoder doesn’t write the three columns (Columns the encoder doesn’t write).	None.

Configuration

status is the handler status that marisco-ingest lists. The cells that download the records from Zenodo are marked eval: false.

Exported source

status = 'Active'

Load data

Each Zenodo record has its own adapter. An adapter takes a downloaded record and returns its columns as RAW_COLS lists them, in that order. load_data then combines the records.

Align record schemas

The FS2020_21 record holds measurements from ETH and VERA. The FS2022 to FS2025 records hold ETH measurements only. FS2020_21 names some metadata columns differently from the other records, and FS2022S names its coordinates Latitude and Longitude.

Each measurement column has a twin with its uncertainty, such as I129_at_l and unc_I129_at_l. Some column names start with the laboratory and some don’t. In FS2020_21, the unprefixed columns hold ETH’s I-129 and VERA’s U-236 and U-238. The FS2022 to FS2025 columns have no prefix and are all ETH’s. _prefix_lab_cols adds the missing prefix to a measurement column and its uncertainty twin:

Exported source

def _prefix_lab_cols(df: pd.DataFrame, lab: str, cols: list) -> pd.DataFrame:
    "Prefix `cols` and their `unc_` twins with `lab_`, recording which lab produced them"
    ren = {c: f'{lab}_{c}' for c in cols} | {f'unc_{c}': f'{lab}_unc_{c}' for c in cols}
    return df.rename(columns=ren)

df = pd.DataFrame(dict(Station=[47], I129_at_l=[1.7e9], unc_I129_at_l=[1.4e8]))
test_eq(_prefix_lab_cols(df, 'ETH', ['I129_at_l']).columns.tolist(), ['Station', 'ETH_I129_at_l', 'ETH_unc_I129_at_l'])

After the prefix, every measurement column follows the pattern {lab}_[unc_]{nuclide}_{unit}. RAW_COLS is the schema that every adapter returns:

Exported source

META_COLS = [  # chkstyle: ignore-node
    'Cruise', 'Station', 'Latitude_degN', 'Longitude_degE', 'Date',
    'Niskin', 'Pressure_dbar', 'PracticalSalinity', 'Temperature_degC', 'Sample_ID']

ETH_COLS = [  # chkstyle: ignore-node
    'ETH_I129_at_l', 'ETH_unc_I129_at_l', 'ETH_I129_at_kg', 'ETH_unc_I129_at_kg',
    'ETH_U236_at_l', 'ETH_unc_U236_at_l', 'ETH_U236_at_kg', 'ETH_unc_U236_at_kg',
    'ETH_U238_ppb', 'ETH_unc_U238_ppb']

VERA_COLS = [  # chkstyle: ignore-node
    'VERA_U236_at_l', 'VERA_unc_U236_at_l', 'VERA_U236_at_kg', 'VERA_unc_U236_at_kg',
    'VERA_U238_ppb', 'VERA_unc_U238_ppb']

RAW_COLS = META_COLS + ETH_COLS + VERA_COLS

Laboratory precedence

MARIS reports one laboratory’s measurement for each sample and nuclide. The data provider set this precedence. ETH measured I-129 in every case. For U-236 and U-238, the handler uses VERA’s value when VERA reported one, and ETH’s value otherwise. _prefer_vera clears ETH’s value and uncertainty wherever VERA reported a value:

Exported source

def _lab_meas(lab: str, stem: str) -> tuple:
    "Value and uncertainty column names for `stem` as measured by `lab`"
    return f'{lab}_{stem}', f'{lab}_unc_{stem}'

Exported source

def _prefer_vera(df: pd.DataFrame, stem: str) -> pd.DataFrame:
    "Where VERA reported `stem`, clear ETH's measurement (value and unc)"
    (v, vu), (e, eu) = _lab_meas('VERA', stem), _lab_meas('ETH', stem)
    df.loc[df[v].notna(), [e, eu]] = np.nan
    return df

VERA reported U-236 for the first sample only. ETH’s value is cleared for that sample and kept for the second:

df = pd.DataFrame(dict(VERA_U236_at_l=[1.5e7, np.nan], VERA_unc_U236_at_l=[1.2e6, np.nan],
    ETH_U236_at_l=[1.4e7, 2.0e7], ETH_unc_U236_at_l=[1.0e6, 1.5e6]))
res = _prefer_vera(df, 'U236_at_l')
test_eq(res.ETH_U236_at_l.isna().tolist(), [True, False])
res

	VERA_U236_at_l	VERA_unc_U236_at_l	ETH_U236_at_l	ETH_unc_U236_at_l
0	15000000.0	1200000.0	NaN	NaN
1	NaN	NaN	20000000.0	1500000.0

Record adapters

_RENAMES maps the metadata column names that differ between records to their RAW_COLS names. _normalize_2020_21 and _normalize_2022_25 apply it and the steps above to their records. _normalize_2020_21 also drops the ETH U-236/U-238 ratio columns, and converts dates from dd/mm/yyyy to yyyy-mm-dd. _normalize_2022_25 builds Date from Year, Month and Day when a record has no Date column.

Exported source

_RENAMES = dict(Bottle='Niskin', Salinity_ctd='PracticalSalinity', Temperature_degC_ctd='Temperature_degC',
    Latitude='Latitude_degN', Longitude='Longitude_degE')

Exported source

def _normalize_2020_21(df: pd.DataFrame) -> pd.DataFrame:
    "Align FS20/FS21 records (ETH + VERA labs) to the canonical Fram Strait raw schema"
    df = df.drop(columns=[c for c in df.columns if 'U236_U238' in c])
    df = df.rename(columns=_RENAMES)
    df = _prefix_lab_cols(df, 'ETH',  ['I129_at_l', 'I129_at_kg'])
    df = _prefix_lab_cols(df, 'VERA', ['U236_at_l', 'U236_at_kg', 'U238_ppb'])
    for stem in ['U236_at_l', 'U236_at_kg', 'U238_ppb']: df = _prefer_vera(df, stem)
    df['Date'] = pd.to_datetime(df['Date'], format='%d/%m/%Y').dt.strftime('%Y-%m-%d')
    return df[RAW_COLS]

Exported source

def _normalize_2022_25(df: pd.DataFrame) -> pd.DataFrame:
    "Align FS22-25 records (ETH lab only) to the canonical Fram Strait raw schema"
    df = df.rename(columns=_RENAMES)
    if 'Date' not in df.columns: df['Date'] = pd.to_datetime(df[['Year', 'Month', 'Day']]).dt.strftime('%Y-%m-%d')
    df = _prefix_lab_cols(df, 'ETH', ['I129_at_l', 'I129_at_kg'])
    return df.reindex(columns=RAW_COLS)

On a row in the FS2022S layout, _normalize_2022_25 builds Date, renames the coordinates and adds the laboratory prefix. Columns that the record lacks, such as VERA’s, are empty:

df = pd.DataFrame(dict(Cruise=['FS2022S'], Station=[1], Latitude=[78.8], Longitude=[-2.0], Year=[2022], Month=[9],
    Day=[10], I129_at_l=[1.7e9], unc_I129_at_l=[1.4e8]))
res = _normalize_2022_25(df)
test_eq(res.columns.tolist(), RAW_COLS)
test_eq(res.Date[0], '2022-09-10')
res[['Cruise', 'Latitude_degN', 'Date', 'ETH_I129_at_l', 'VERA_U236_at_l']]

	Cruise	Latitude_degN	Date	ETH_I129_at_l	VERA_U236_at_l
0	FS2022S	78.8	2022-09-10	1.700000e+09	NaN

RECORDS maps each record name to its Zenodo record id (zenodo), the name of its CSV file (file) and its adapter. A Zenodo record never changes after publication. A newer version of a dataset is a new record, so moving to it means changing its id here:

Exported source

RECORDS = {  # chkstyle: ignore-node
    "FS2020_21": {
        "zenodo": 19387002,
        "file": "FramStrait_2020_2021_radionuclides.csv",
        "adapter": _normalize_2020_21
    },
    "FS2022_i129": {
        "zenodo": 20425174,
        "file": "FS2022_i129.csv",
        "adapter": _normalize_2022_25
    },
    "FS2022S_i129": {
        "zenodo": 20628610,
        "file": "FS2022S_data_i129.csv",
        "adapter": _normalize_2022_25
    },
    "FS2023_i129": {
        "zenodo": 20448047,
        "file": "FS2023_i129.csv",
        "adapter": _normalize_2022_25
    },
    "FS2024_i129": {
        "zenodo": 20761705,
        "file": "FS2024_i129.csv",
        "adapter": _normalize_2022_25
    },
    "FS2025_i129": {
        "zenodo": 20761832,
        "file": "FS2025_i129.csv",
        "adapter": _normalize_2022_25
    },
}

load_data downloads each record, applies its adapter and combines the results into one SEAWATER DataFrame:

source

load_data

def load_data()->dict:

Fetch Fram Strait CSV records and return one combined SEAWATER DataFrame

dfs = load_data()

The seven cruises give 877 samples:

dfs['SEAWATER'].groupby('Cruise').size()

Cruise
FS20       119
FS2022     104
FS2022S    112
FS2023     127
FS2024     119
FS2025     177
FS21       119
dtype: int64

Each cruise’s count, minimum and maximum for the coordinates, CTD values and measurements can be compared with the Zenodo files. The FS2020_21 file holds two cruises, FS20 and FS21:

cols = ['Latitude_degN', 'Longitude_degE', 'Pressure_dbar', 'PracticalSalinity', 'Temperature_degC']
cols += [c for c in ETH_COLS + VERA_COLS if '_unc_' not in c]
dfs['SEAWATER'].groupby('Cruise', sort=False)[cols].agg(['count', 'min', 'max']).T.style.format('{:.6g}', na_rep='')

	Cruise	FS20	FS21	FS2022	FS2022S	FS2023	FS2024	FS2025
Latitude_degN	count	119	119	104	112	127	119	177
	min	78.8285	78.8182	78.8277	78.6735	78.7765	78.8092	78.8322
	max	78.838	78.8385	78.9532	78.8675	78.8393	78.8367	80.4085
Longitude_degE	count	119	119	104	112	127	119	177
	min	-12.0058	-16.9998	-10.0278	-12.2032	-14.0923	-12.0005	-13.4037
	max	8.00167	7.99933	8	7.99667	7.9995	8.00517	8
Pressure_dbar	count	119	119	104	112	127	119	177
	min	4.287	4.712	4.559	1.992	4.325	4.345	4.524
	max	2548.39	1010.46	2703.53	2681.35	1922.25	401.502	2621.69
PracticalSalinity	count	117	119	104	112	127	119	177
	min	29.7932	29.0882	29.2475	31.5173	28.505	28.7045	28.9885
	max	35.0484	35.0333	35.0539	34.9927	35.0559	35.0925	35.1055
Temperature_degC	count	118	119	104	112	127	119	177
	min	-1.7402	-1.7707	-1.6596	-1.7775	-1.7962	-1.7169	-1.815
	max	6.9923	6.6358	7.4289	3.241	7.0967	9.6997	9.0145
ETH_I129_at_l	count	119	114	104	112	127	119	177
	min	1.51096e+08	2.12223e+08	1.46601e+08	1.42494e+08	4.60854e+08	1.72407e+09	1.5022e+08
	max	7.26666e+09	6.87854e+09	6.70593e+09	5.93147e+09	5.70781e+09	7.42241e+09	7.15086e+09
ETH_I129_at_kg	count	119	114	0	112	0	0	0
	min	1.47434e+08	2.06331e+08		1.39041e+08
	max	7.09257e+09	6.7198e+09		5.79559e+09
ETH_U236_at_l	count	14	3	0	0	0	0	0
	min	1.11066e+07	1.16059e+07
	max	1.97496e+07	1.36384e+07
ETH_U236_at_kg	count	14	3	0	0	0	0	0
	min	1.08e+07	1.129e+07
	max	1.921e+07	1.329e+07
ETH_U238_ppb	count	14	3	0	0	0	0	0
	min	2.97	3.2
	max	3.46	3.49
VERA_U236_at_l	count	87	68	0	0	0	0	0
	min	4.73e+06	7.87e+06
	max	3.901e+07	2.621e+07
VERA_U236_at_kg	count	87	68	0	0	0	0	0
	min	4.54832e+06	7.61964e+06
	max	3.79933e+07	2.54981e+07
VERA_U238_ppb	count	87	68	0	0	0	0	0
	min	2.64	2.9
	max	3.52	3.6

Pipeline steps

Rename columns

The Fram Strait records are seawater only. RenameColsCB maps the provider’s metadata columns to MARIS names:

source

RenameColsCB

def RenameColsCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Map Fram Strait provider columns to MARIS standard names

df = pd.DataFrame(dict(Station=[341], Latitude_degN=[78.83], Longitude_degE=[-2.0], PracticalSalinity=[34.9],
    Temperature_degC=[0.05], Sample_ID=[1]))
tfm = Transformer({'SEAWATER': df}, cbs=[RenameColsCB()])
tfm()
test_eq(tfm.dfs['SEAWATER'].columns.tolist(), ['STATION', 'LAT', 'LON', 'SAL', 'TEMP', 'SMP_ID_PROVIDER'])

tfm = Transformer(dfs, cbs=[RenameColsCB()])
tfm()
tfm.dfs['SEAWATER'][['LAT', 'LON', 'STATION', 'SAL', 'TEMP', 'SMP_ID_PROVIDER']].head(2)

	LAT	LON	STATION	SAL	TEMP	SMP_ID_PROVIDER
0	78.833167	-0.003667	47	34.9311	1.7576	1
1	78.833167	-0.003667	47	34.9247	2.4914	2

Parse time

The adapters write every date as yyyy-mm-dd in Date. ParseDateTimeCB parses it into a UTC TIME and drops Date. TIME is midnight UTC on the sampling date. The FS2020_21 adapter drops the time of day that FS2020_21 reports in its Time column:

source

ParseDateTimeCB

def ParseDateTimeCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Parse FS collection date into a UTC TIME value

tfm = Transformer({'SEAWATER': pd.DataFrame(dict(Date=['2025-07-30']))}, cbs=[ParseDateTimeCB()])
tfm()
test_eq(tfm.dfs['SEAWATER'].columns.tolist(), ['TIME'])
test_eq(tfm.dfs['SEAWATER'].TIME[0], pd.Timestamp('2025-07-30', tz='UTC'))

tfm = Transformer(dfs, cbs=[  # chkstyle: ignore-node
    RenameColsCB(),
    ParseDateTimeCB()
])
tfm()
tfm.dfs['SEAWATER'].TIME.agg(['min', 'max'])

min   2020-08-27 00:00:00+00:00
max   2025-08-13 00:00:00+00:00
Name: TIME, dtype: datetime64[us, UTC]

Compute depth from pressure

The FS2022 to FS2025 records report pressure, not depth. AddDepthCB computes the sampling depth in metres from Pressure_dbar with gsw.z_from_p, for every record. The gsw package implements the Thermodynamic Equation of Seawater 2010 (TEOS-10). The reported latitude sets the gravity correction. Depths are rounded to one decimal place. FS2020_21 also reports Depth_m, and its values agree with the computed depths to within 0.05 m:

source

AddDepthCB

def AddDepthCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Compute sampling depth using Thermodynamic Equation of SeaWater 2010 (TEOS-10)

The first FS2020_21 sample, at 400.374 dbar and 78.83° N, is at 395.8 m:

tfm = Transformer({'SEAWATER': pd.DataFrame(dict(Pressure_dbar=[400.374], LAT=[78.833167]))}, cbs=[AddDepthCB()])
tfm()
test_eq(tfm.dfs['SEAWATER'].SMP_DEPTH[0], 395.8)

tfm = Transformer(dfs, cbs=[  # chkstyle: ignore-node
    RenameColsCB(),
    ParseDateTimeCB(),
    AddDepthCB()
])
tfm()
tfm.dfs['SEAWATER'].SMP_DEPTH.describe()

count     877.000000
mean      183.726682
std       304.633340
min         2.000000
25%        25.400000
50%        99.300000
75%       198.200000
max      2658.000000
Name: SMP_DEPTH, dtype: float64

Reshape to one row per measurement

Each raw row holds one sample. Each of its measurements sits in its own column, named for the laboratory, nuclide and unit, with a twin column for the uncertainty. MARIS stores one measurement per row, with the laboratory, nuclide, unit, value and uncertainty in separate columns.

FEEDBACK TO DATA PROVIDER

Column headers in the Fram Strait dataset encode multiple pieces of information (lab, nuclide, unit) into a single string. This adds friction to data ingestion, since every new dataset with a different naming convention requires custom parsing logic. MARIS prefers a tidy data layout (Wickham 2014, doi:10.18637/jss.v059.i10), where each column holds a single variable, and metadata like nuclide, unit, and method are stored as separate columns, not baked into the header.

MEAS_PAT splits a measurement column name into the laboratory, the uncertainty marker, the nuclide and the unit:

Exported source

MEAS_PAT = re.compile(r'^(?P<LAB>[A-Z]+)_(?:(?P<UNC>unc)_)?(?P<NUCLIDE>[A-Za-z]+\d+)_(?P<UNIT>.+)$')

ETH_unc_I129_at_l is ETH’s uncertainty for I-129 in atoms per litre. A value column has no unc part:

test_eq(MEAS_PAT.fullmatch('ETH_unc_I129_at_l').groupdict(), dict(LAB='ETH', UNC='unc', NUCLIDE='I129', UNIT='at_l'))
MEAS_PAT.fullmatch('VERA_U238_ppb').groupdict()

{'LAB': 'VERA', 'UNC': None, 'NUCLIDE': 'U238', 'UNIT': 'ppb'}

MeltFramStraitCB melts every column that MEAS_PAT matches, pairs each value with its uncertainty, and drops rows without a value:

source

MeltFramStraitCB

def MeltFramStraitCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Reshape wide Fram Strait value and uncertainty columns into one row per measurement

In this example, station 101 has three measurements from two laboratories. Station 102 has one, and its empty at_kg and VERA columns give no rows:

df = pd.DataFrame(dict(  # chkstyle: ignore-node
    STATION=[101, 102],
    ETH_I129_at_l=[1.2e9, 2.4e9], ETH_unc_I129_at_l=[1.0e8, 2.0e8],
    ETH_I129_at_kg=[3.4e9, np.nan], ETH_unc_I129_at_kg=[3.0e8, np.nan],
    VERA_U236_at_l=[1.5e7, np.nan], VERA_unc_U236_at_l=[1.2e6, np.nan]))
tfm = Transformer({'SEAWATER': df}, cbs=[MeltFramStraitCB()])
tfm()
out = tfm.dfs['SEAWATER'].sort_values(['STATION', 'LAB', 'UNIT']).reset_index(drop=True)
test_eq(out.VALUE.tolist(), [3.4e9, 1.2e9, 1.5e7, 2.4e9])
test_eq(out.UNC.tolist(), [3.0e8, 1.0e8, 1.2e6, 2.0e8])
out

	STATION	LAB	NUCLIDE	UNIT	UNC	VALUE
0	101	ETH	I129	at_kg	300000000.0	3.400000e+09
1	101	ETH	I129	at_l	100000000.0	1.200000e+09
2	101	VERA	U236	at_l	1200000.0	1.500000e+07
3	102	ETH	I129	at_l	200000000.0	2.400000e+09

On the Fram Strait data, the reshape gives one row for each laboratory, nuclide and unit that a sample reports:

tfm = Transformer(dfs, cbs=[  # chkstyle: ignore-node
    RenameColsCB(),
    ParseDateTimeCB(),
    AddDepthCB(),
    MeltFramStraitCB()
])
tfm()
out = tfm.dfs['SEAWATER']
out.groupby(['LAB', 'NUCLIDE', 'UNIT']).size()

LAB   NUCLIDE  UNIT 
ETH   I129     at_kg    345
               at_l     872
      U236     at_kg     17
               at_l      17
      U238     ppb       17
VERA  U236     at_kg    155
               at_l     155
      U238     ppb      155
dtype: int64

FEEDBACK TO DATA PROVIDER

ETH’s I-129 and VERA’s U-236 are reported both in atoms per litre (at_l) and in atoms per kg (at_kg). Each at_kg value equals the at_l value divided by a seawater density between 1020 and 1029 kg/m³. For U-236 in FS2020_21, this is the record’s density_kg_m3 column. MARIS keeps both units for now: 517 measurements appear twice in the output, once in each unit. FS2022, FS2023, FS2024 and FS2025 report I-129 in at_l only.

Should MARIS keep both units, or only the unit you measured? If only one, which unit is it?

Each value has an uncertainty. The output therefore has half as many rows as there are non-empty measurement cells in the raw data:

meas = [c for c in dfs['SEAWATER'].columns if MEAS_PAT.fullmatch(c)]
test_eq(len(out) * 2, dfs['SEAWATER'][meas].notna().sum().sum())

Convert U-238 units

The provider reports U-238 in parts per billion (ppb), the mass of uranium per mass of seawater. MARIS stores it in atoms per kg. ConvertU238CB converts the VALUE and UNC of U-238 rows and sets their UNIT to at_kg:

\[\text{atoms/kg} = C_{\text{ppb}} \times 10^{-9} \times 1000 \times \frac{N_A}{M_{238}}\]

(N_A = 6.02214076 ^{23}{-1}) is the Avogadro constant and (M_{238} = 238.05 ^{-1}) is the molar mass of U-238. The conversion factor is (2.530 ^{15}{-1},^{-1}). Rows for other nuclides are unchanged.

source

ConvertU238CB

def ConvertU238CB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Convert U-238 VALUE from ppb to atoms/kg

Exported source

U238_PPB_TO_AT_KG = 2.529_780e15

df = pd.DataFrame(dict(NUCLIDE=['I129', 'U238'], UNIT=['at_kg', 'ppb'], VALUE=[1.0, 2.0], UNC=[0.1, 0.2]))
tfm = Transformer({'SEAWATER': df}, cbs=[ConvertU238CB()])
tfm()
out = tfm.dfs['SEAWATER']
test_eq(out.VALUE.tolist(), [1.0, 2.0 * U238_PPB_TO_AT_KG])
test_eq(out.UNC.tolist(), [0.1, 0.2 * U238_PPB_TO_AT_KG])
test_eq(out.UNIT.tolist(), ['at_kg', 'at_kg'])
out

	NUCLIDE	UNIT	VALUE	UNC
0	I129	at_kg	1.000000e+00	1.000000e-01
1	U238	at_kg	5.059560e+15	5.059560e+14

tfm = Transformer(dfs, cbs=[  # chkstyle: ignore-node
    RenameColsCB(),
    ParseDateTimeCB(),
    AddDepthCB(),
    MeltFramStraitCB(),
    ConvertU238CB()
])
tfm()
out = tfm.dfs['SEAWATER']
out[out.NUCLIDE == 'U238'].groupby(['LAB', 'UNIT']).VALUE.describe()

		count	mean	std	min	25%	50%	75%	max
LAB	UNIT
ETH	at_kg	17.0	8.364643e+15	3.876987e+14	7.513447e+15	8.196487e+15	8.500061e+15	8.651848e+15	8.828932e+15
VERA	at_kg	155.0	8.198446e+15	5.359231e+14	6.678619e+15	7.817020e+15	8.145892e+15	8.639199e+15	9.107208e+15

Remap nomenclatures

After the reshape, NUCLIDE, UNIT and LAB hold the provider’s strings, such as I129, at_l and ETH. MARIS stores integer ids from its lookup tables. NUCLIDE_LUT, UNIT_LUT and LAB_LUT map each string to its id:

Exported source

NUCLIDE_LUT = dict(I129=28, U236=108, U238=64)
UNIT_LUT = dict(at_kg=9, at_l=12)
LAB_LUT = dict(ETH=345, VERA=504)

The ids come from the MARIS lookup tables that get_lut reads. Each id resolves to the MARIS name below:

def maris_names(lut, name):
    "MARIS name of each id in `lut`, from the `name` lookup table"
    names = {v: k for k, v in get_lut(name).items()}
    return {k: names[i] for k, i in lut.items()}

test_eq(maris_names(NUCLIDE_LUT, 'NUCLIDE'), dict(I129='i129', U236='u236', U238='u238'))
test_eq(maris_names(UNIT_LUT, 'UNIT'), dict(at_kg='atom per kg', at_l='atom per l'))
maris_names(LAB_LUT, 'LAB')

{'ETH': 'Laboratory of Ion Beam Physics _LIP_, ETZ Zürich, Switzerland',
 'VERA': 'Vienna Environmental Research Accelerator facility at the University of Vienna, Austria'}

RemapCB replaces each string with its id:

df = pd.DataFrame(dict(NUCLIDE=['I129', 'U236'], UNIT=['at_kg', 'at_l'], LAB=['ETH', 'VERA']))
tfm = Transformer({'SEAWATER': df}, cbs=[  # chkstyle: ignore-node
    RemapCB(lut=NUCLIDE_LUT, col_remap='NUCLIDE', col_src='NUCLIDE'),
    RemapCB(lut=UNIT_LUT, col_remap='UNIT', col_src='UNIT'),
    RemapCB(lut=LAB_LUT, col_remap='LAB', col_src='LAB')
])
tfm()
test_eq(tfm.dfs['SEAWATER'].to_dict('list'), dict(NUCLIDE=[28, 108], UNIT=[9, 12], LAB=[345, 504]))

tfm = Transformer(dfs, cbs=[  # chkstyle: ignore-node
    RenameColsCB(),
    ParseDateTimeCB(),
    AddDepthCB(),
    MeltFramStraitCB(),
    ConvertU238CB(),
    RemapCB(lut=NUCLIDE_LUT, col_remap='NUCLIDE', col_src='NUCLIDE'),
    RemapCB(lut=UNIT_LUT, col_remap='UNIT', col_src='UNIT'),
    RemapCB(lut=LAB_LUT, col_remap='LAB', col_src='LAB')
])
tfm()
tfm.dfs['SEAWATER'].groupby(['LAB', 'NUCLIDE', 'UNIT']).size()

LAB  NUCLIDE  UNIT
345  28       9       345
              12      872
     64       9        17
     108      9        17
              12       17
504  64       9       155
     108      9       155
              12      155
dtype: int64

Detection limit

The records have no detection flag. MARIS requires one of these categories:

get_lut('DL')

{'Not applicable': -1,
 'Not available': 0,
 'Detected value': 1,
 'Detection limit': 2,
 'Not detected': 3,
 'Derived': 4}

AddDetectionLimitCB sets DL to 1 (Detected value) for every row.

source

AddDetectionLimitCB

def AddDetectionLimitCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Assign missing Detection Limit column to MARIS ‘Detected value: 1’ category

Coordinates, time encoding and sample ids

The shared SanitizeLonLatCB, EncodeTimeCB and AddSampleIDCB from marisco.callbacks finish the pipeline. SanitizeLonLatCB replaces decimal commas in coordinates, and drops rows with out-of-range coordinates or with (0, 0). EncodeTimeCB converts TIME to seconds since 1970-01-01. AddSampleIDCB numbers the rows in SMP_ID and casts SMP_ID_PROVIDER to a string.

STATION as string

pandas reads the numeric Station column as int64. The NetCDF template declares station as a string. FormatStationCB casts STATION to str before encoding:

source

FormatStationCB

def FormatStationCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Cast STATION to str for the NetCDF4 string-typed station variable

tfm = Transformer({'SEAWATER': pd.DataFrame(dict(STATION=[341, 415]))}, cbs=[FormatStationCB()])
tfm()
test_eq(tfm.dfs['SEAWATER'].STATION.tolist(), ['341', '415'])

Encode to NetCDF

get_cbs returns new instances of the pipeline’s callbacks, in order. encode and the example below both call it.

source

get_cbs

def get_cbs()->list:

Callbacks, in pipeline order, that turn raw Fram Strait data into MARIS-standard DataFrames

Exported source

def get_cbs() -> list:
    "Callbacks, in pipeline order, that turn raw Fram Strait data into MARIS-standard DataFrames"
    return [  # chkstyle: ignore-node
        RenameColsCB(),
        ParseDateTimeCB(),
        AddDepthCB(),
        MeltFramStraitCB(),
        ConvertU238CB(),
        RemapCB(lut=NUCLIDE_LUT, col_remap='NUCLIDE', col_src='NUCLIDE'),
        RemapCB(lut=UNIT_LUT, col_remap='UNIT', col_src='UNIT'),
        RemapCB(lut=LAB_LUT, col_remap='LAB', col_src='LAB'),
        AddDetectionLimitCB(),
        SanitizeLonLatCB(),
        EncodeTimeCB(),
        AddSampleIDCB(col_provider="SMP_ID_PROVIDER"),
        FormatStationCB()]

tfm = Transformer(dfs, cbs=get_cbs())
tfm()
out = tfm.dfs['SEAWATER']
out[['SMP_ID', 'SMP_ID_PROVIDER', 'STATION', 'TIME', 'SMP_DEPTH', 'LAB', 'NUCLIDE', 'UNIT', 'VALUE', 'UNC', 'DL']].head()

	SMP_ID	SMP_ID_PROVIDER	STATION	TIME	SMP_DEPTH	LAB	NUCLIDE	UNIT	VALUE	UNC	DL
0	1	1	47	1598486400	395.8	345	28	9	1.702534e+09	1.341982e+08	1
1	2	1	47	1598486400	395.8	345	28	12	1.744666e+09	1.375192e+08	1
2	3	1	47	1598486400	395.8	504	108	9	1.317705e+07	1.087568e+06	1
3	4	1	47	1598486400	395.8	504	108	12	1.357000e+07	1.120000e+06	1
4	5	1	47	1598486400	395.8	504	64	9	8.474763e+15	3.794670e+14	1

The ranges of the MARIS columns in the final data:

out[[c for c in out.columns if c.isupper()]].describe()

	LAT	LON	SAL	TEMP	TIME	SMP_DEPTH	LAB	NUCLIDE	UNIT	UNC	VALUE	DL	SMP_ID
count	1733.000000	1733.000000	1723.000000	1728.000000	1.733000e+03	1733.000000	1733.000000	1733.000000	1733.000000	1.733000e+03	1.733000e+03	1733.0	1733.000000
mean	78.846159	-3.794303	33.565289	0.622876	1.648324e+09	165.295788	387.663012	47.452972	10.807271	2.694743e+13	8.153272e+14	1.0	867.000000
std	0.139003	5.595021	1.598968	2.359036	5.075129e+07	277.559678	70.470930	31.956175	1.468615	9.420889e+13	2.462455e+15	0.0	500.418325
min	78.673500	-16.999833	28.504999	-1.815000	1.598486e+09	2.000000	345.000000	28.000000	9.000000	3.700000e+05	4.548323e+06	1.0	1.000000
25%	78.833000	-7.999333	32.363700	-1.419200	1.598918e+09	25.300000	345.000000	28.000000	9.000000	3.906112e+07	1.808362e+09	1.0	434.000000
50%	78.833333	-4.005833	34.243200	-0.106200	1.628208e+09	99.100000	345.000000	28.000000	12.000000	7.906752e+07	2.989910e+09	1.0	867.000000
75%	78.834167	-1.000000	34.902600	2.294700	1.663373e+09	197.800000	504.000000	64.000000	12.000000	1.896498e+08	4.289536e+09	1.0	1300.000000
max	80.408500	8.005167	35.105500	9.699700	1.755043e+09	2658.000000	504.000000	108.000000	12.000000	6.830406e+14	9.107208e+15	1.0	1733.000000

get_attrs builds the global attributes from the transformed data: the bounding box, depth range, time range, keywords and the processing log. The callback docstrings make up the processing log.

source

get_attrs

def get_attrs(
    tfm
):

Retrieve global attributes for Fram Strait

Exported source

FS_KEYWORDS = ["Fram Strait", "Greenland Sea", "I-129", "U-236", "U-238", "radionuclides", "seawater", "Arctic Ocean"]

def get_attrs(tfm):
    "Retrieve global attributes for Fram Strait"
    return GlobAttrsFeeder(tfm.dfs, cbs=[  # chkstyle: ignore-node
        BboxCB(),
        DepthRangeCB(),
        TimeRangeCB(),
        KeyValuePairCB("keywords", ", ".join(FS_KEYWORDS)),
        KeyValuePairCB("publisher_postprocess_logs", ", ".join(tfm.logs)),
    ])()

encode downloads the records, runs the pipeline and writes the SEAWATER group to dest:

source

encode

def encode(
    dest:NoneType=None, # Output NetCDF file path
    src:NoneType=None, # Unused; Fram Strait downloads its data from `RECORDS`
    **kwargs
):

Fram Strait 2020-2025 I-129, U-236, U-238 seawater radionuclide data

Exported source

def encode(
    dest=None, # Output NetCDF file path
    src=None, # Unused; Fram Strait downloads its data from `RECORDS`
    **kwargs # Unused
):
    "Fram Strait 2020-2025 I-129, U-236, U-238 seawater radionuclide data"
    tfm = Transformer(load_data(), cbs=get_cbs())
    tfm()
    NetCDFEncoder(tfm.dfs, dest_fname=dest, global_attrs=get_attrs(tfm)).encode()

encode(dest='../../_data/output/fram_strait_2020_2025.nc')

to_csv converts the NetCDF file to MARIS CSV files, one for each group:

to_csv('../../_data/output/fram_strait_2020_2025.nc')

[Path('../../_data/output/fram_strait_2020_2025_SEAWATER.csv')]

The CSV file uses MARIS column names and lookup-table ids:

csv = pd.read_csv('../../_data/output/fram_strait_2020_2025_SEAWATER.csv')
csv.head()

	detection	lab_id	latitude	longitude	nuclide_id	salinity	sampdepth	samplabcode	station	temperatur	begperiod	uncertaint	unit_id	activity	samptype_id
0	=	345	78.83317	-0.003667	28	34.9311	395.8	1	47	1.7576	2020-08-27	1.341982e+08	9	1.702534e+09	1
1	=	345	78.83317	-0.003667	28	34.9311	395.8	1	47	1.7576	2020-08-27	1.375192e+08	12	1.744666e+09	1
2	=	504	78.83317	-0.003667	108	34.9311	395.8	1	47	1.7576	2020-08-27	1.087568e+06	9	1.317705e+07	1
3	=	504	78.83317	-0.003667	108	34.9311	395.8	1	47	1.7576	2020-08-27	1.120000e+06	12	1.357000e+07	1
4	=	504	78.83317	-0.003667	64	34.9311	395.8	1	47	1.7576	2020-08-27	3.794670e+14	9	8.474763e+15	1

What the pipeline drops

The pipeline leaves data out in four places: raw columns that the adapters don’t select, ETH values that VERA’s replace, samples without a measurement, and columns that the encoder doesn’t write. Each subsection computes one of them from the Zenodo records. The row counts after each step show that no other step removes a row.

Raw columns not kept

is_kept says whether a raw column survives the adapter, under its own name or after _RENAMES, the laboratory prefix or the date assembly:

def is_kept(col):
    "Whether the adapters keep raw column `col` in `RAW_COLS`, renamed or not"
    if col in ('Year', 'Month', 'Day'): return True
    return any(c in RAW_COLS for c in [_RENAMES.get(col, col), f'ETH_{col}', f'VERA_{col}'])

test_eq([is_kept(c) for c in ['Bottle', 'Year', 'U236_at_l', 'Water_Mass_ctd']], [True, True, True, False])

DROP_REASONS records why each dropped column is dropped. The test fails when a record has a dropped column without a reason, such as a column the provider adds later:

DROP_REASONS = dict(  # chkstyle: ignore-node
    Time="Time of day. Not mapped yet. `TIME` holds the sampling date only.",
    Bottom_Depth_m="Not mapped yet. MARIS has a total depth field, `TOT_DEPTH`.",
    Depth_m="`SMP_DEPTH` is computed from `Pressure_dbar` for every record. The two agree to within 0.05 m.",
    Con_Temperature_degC_ctd="No MARIS field.",
    Abs_Salinity_ctd="No MARIS field.",
    T_pot_degC_ctd="No MARIS field.",
    density_kg_m3="No MARIS field.",
    Sigma_Theta_0_ctd="No MARIS field.",
    Water_Mass_ctd="No MARIS field.",
    ETH_U236_U238="MARIS derives the ratio from the U-236 and U-238 values.",
    ETH_unc_U236_U238="MARIS derives the ratio from the U-236 and U-238 values.")

raws = {name: _fetch(rec) for name, rec in RECORDS.items()}
dropped = [(name, c, DROP_REASONS.get(c)) for name, raw in raws.items() for c in raw.columns if not is_kept(c)]
dropped = pd.DataFrame(dropped, columns=['Record', 'Column', 'Reason'])
test_eq(dropped.Reason.isna().sum(), 0)
dropped

	Record	Column	Reason
0	FS2020_21	Time	Time of day. Not mapped yet. `TIME` holds the ...
1	FS2020_21	Bottom_Depth_m	Not mapped yet. MARIS has a total depth field,...
2	FS2020_21	Depth_m	`SMP_DEPTH` is computed from `Pressure_dbar` f...
3	FS2020_21	Con_Temperature_degC_ctd	No MARIS field.
4	FS2020_21	Abs_Salinity_ctd	No MARIS field.
5	FS2020_21	T_pot_degC_ctd	No MARIS field.
6	FS2020_21	density_kg_m3	No MARIS field.
7	FS2020_21	Sigma_Theta_0_ctd	No MARIS field.
8	FS2020_21	ETH_U236_U238	MARIS derives the ratio from the U-236 and U-2...
9	FS2020_21	ETH_unc_U236_U238	MARIS derives the ratio from the U-236 and U-2...
10	FS2020_21	Water_Mass_ctd	No MARIS field.

ETH values replaced by VERA’s

In FS2020_21, the unprefixed U-236 and U-238 columns are VERA’s. Where VERA and ETH both report a value, _prefer_vera clears ETH’s value and its uncertainty. The number of ETH values cleared in each column:

fs20_21 = raws['FS2020_21']
pd.Series({stem: (fs20_21[stem].notna() & fs20_21[f'ETH_{stem}'].notna()).sum() for stem in ['U236_at_l', 'U236_at_kg', 'U238_ppb']})

U236_at_l     16
U236_at_kg    16
U238_ppb      16
dtype: int64

Samples without a measurement

MeltFramStraitCB drops empty measurement cells. A sample with no value in any measurement column leaves no row in the output. Two FS21 samples have none:

vals = [c for c in RAW_COLS if MEAS_PAT.fullmatch(c) and '_unc_' not in c]
seawater = dfs['SEAWATER']
seawater[seawater[vals].isna().all(axis=1)][['Cruise', 'Station', 'Niskin', 'Sample_ID', 'Pressure_dbar']]

	Cruise	Station	Niskin	Sample_ID	Pressure_dbar
164	FS21	372	10	46	75.085
165	FS21	372	11	47	49.298

Rows after each step

step_counts runs the pipeline one callback at a time and counts the rows of each cruise after each one:

def step_counts(dfs, cbs, by='Cruise'):
    "Rows of each `by` value in the SEAWATER group, after each callback in `cbs`"
    rows = {'Loaded': dfs['SEAWATER'][by].value_counts()}
    for i, cb in enumerate(cbs, 1):
        tfm = Transformer({k: v.copy() for k, v in dfs.items()}, cbs=cbs[:i])
        tfm()
        rows[f'{i}. {type(cb).__name__}'] = tfm.dfs['SEAWATER'][by].value_counts()
    return pd.DataFrame(rows).T[dfs['SEAWATER'][by].unique()]

Only the reshape changes the counts. Before it, a row is a sample. After it, a row is a measurement. No later step removes a row:

counts = step_counts(dfs, get_cbs())
test_eq(counts.iloc[-1].sum(), len(out))
counts

Cruise	FS20	FS21	FS2022	FS2022S	FS2023	FS2024	FS2025
Loaded	119	119	104	112	127	119	177
1. RenameColsCB	119	119	104	112	127	119	177
2. ParseDateTimeCB	119	119	104	112	127	119	177
3. AddDepthCB	119	119	104	112	127	119	177
4. MeltFramStraitCB	541	441	104	224	127	119	177
5. ConvertU238CB	541	441	104	224	127	119	177
6. RemapCB	541	441	104	224	127	119	177
7. RemapCB	541	441	104	224	127	119	177
8. RemapCB	541	441	104	224	127	119	177
9. AddDetectionLimitCB	541	441	104	224	127	119	177
10. SanitizeLonLatCB	541	441	104	224	127	119	177
11. EncodeTimeCB	541	441	104	224	127	119	177
12. AddSampleIDCB	541	441	104	224	127	119	177
13. FormatStationCB	541	441	104	224	127	119	177

Columns the encoder doesn’t write

NetCDFEncoder writes only the columns that have a MARIS NetCDF variable in NC_VARS. It skips the other columns without a message. Three columns reach the encoder and are not written:

[c for c in out.columns if c not in NC_VARS]

['Cruise', 'Niskin', 'Pressure_dbar']