This handler ingests OSPAR (Convention for the Protection of the Marine Environment of the North-East Atlantic) radioactive substances monitoring data for seawater and biota, as submitted by the contracting parties to ODIMS. It transforms the data into the MARIS NetCDF format through a pipeline that standardises nomenclatures, parses time and coordinates, and converts values, units, uncertainties and detection-limit flags to MARIS conventions.

For detailed guidance on the reconciliation workflow used throughout this handler, see the writing-a-handler and reconcile-nomenclature how-to guides.

For the MARIS data model and field conventions, see the reference guide and field definitions.

Configuration & file paths

  • src_dir: path or URL to the maris-crawlers folder containing the OSPAR data in CSV format.

  • Zotero key: used to retrieve attributes related to the dataset from Zotero. The MARIS datasets include a library available on Zotero.

Exported source 
src_dir = 'https://raw.githubusercontent.com/franckalbinet/maris-crawlers/refs/heads/main/data/processed/OSPAR'
zotero_key ='LQRA4MMK' # OSPAR MORS zotero key
status = 'Under review'

Load data

OSPAR data is provided as a zipped Microsoft Access database. We automatically fetch and convert this dataset with database tables exported as .csv files using a Github action here: maris-crawlers.

The dataset is then accessible in an amenable format for the marisco data pipeline, as one file per sample type: Seawater data.csv and Biota data.csv.

source

load_data

def load_data(
    fname_in, # Path or URL to the folder of raw OSPAR csv files
):

Load OSPAR data; returns dict of DataFrames keyed by sample type

Exported source 
default_smp_types = {
    'Biota': 'BIOTA',
    'Seawater': 'SEAWATER',
}

dfs is a dictionary of dataframes created from the OSPAR dataset located at the path src_dir, with one entry per sample type. Column names are lowercased on load.

dfs = load_data(src_dir)

# or with a subset
# dfs = {k: v.sample(10, random_state=42) for k, v in dfs.items()}

test_eq(list(dfs.keys()), ['BIOTA', 'SEAWATER'])
for k,v in dfs.items():
    print(f"{k}: {v.shape[0]} rows, {v.shape[1]} cols")
    print(v.columns.tolist(), '\n')
BIOTA: 15951 rows, 27 cols
['id', 'contracting party', 'rsc sub-division', 'station id', 'sample id', 'latd', 'latm', 'lats', 'latdir', 'longd', 'longm', 'longs', 'longdir', 'sample type', 'biological group', 'species', 'body part', 'sampling date', 'nuclide', 'value type', 'activity or mda', 'uncertainty', 'unit', 'data provider', 'measurement comment', 'sample comment', 'reference comment'] 

SEAWATER: 19193 rows, 25 cols
['id', 'contracting party', 'rsc sub-division', 'station id', 'sample id', 'latd', 'latm', 'lats', 'latdir', 'longd', 'longm', 'longs', 'longdir', 'sample type', 'sampling depth', 'sampling date', 'nuclide', 'value type', 'activity or mda', 'uncertainty', 'unit', 'data provider', 'measurement comment', 'sample comment', 'reference comment']

Remove missing values

Some OSPAR seawater records have no sampling date and no measurement value. MARIS cannot use a record without a date or a value. The handler drops these records.

ImportantFEEDBACK TO DATA PROVIDER

10 SEAWATER records have no sampling date and no activity or mda value. Eight of them are Irish records (IDs 120363 to 120370) with no nuclide, value type, unit or sample ID either. Only their station is filled in. The other two are Swedish tritium (3H) records (IDs 97948 and 97952) with no date, value or value type. These records should be completed or removed at source.

sw = dfs['SEAWATER']
missing = sw[sw['sampling date'].isna() | sw['activity or mda'].isna()]
print(missing[['id', 'contracting party', 'station id', 'nuclide', 'sampling date', 'activity or mda', 'unit']].to_string())
           id contracting party      station id nuclide sampling date  activity or mda  unit
14776   97948            Sweden             SW7      3H           NaN              NaN  Bq/l
14780   97952            Sweden  Ringhals (R35)      3H           NaN              NaN  Bq/l
16161  120369           Ireland        Salthill     NaN           NaN              NaN   NaN
16162  120370           Ireland       Woodstown     NaN           NaN              NaN   NaN
16586  120363           Ireland              N1     NaN           NaN              NaN   NaN
19188  120364           Ireland              N2     NaN           NaN              NaN   NaN
19189  120365           Ireland              N3     NaN           NaN              NaN   NaN
19190  120366           Ireland              N8     NaN           NaN              NaN   NaN
19191  120367           Ireland              N9     NaN           NaN              NaN   NaN
19192  120368           Ireland             N10     NaN           NaN              NaN   NaN

RemoveAllNAValuesCB drops the rows where every column listed in nan_cols_to_check is empty. In the current data, every record without a sampling date also has no value, and every record without a value also has no sampling date.

Exported source 
nan_cols_to_check = ['sampling date', 'activity or mda']
tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check)])
dfs_out = tfm()
for grp in dfs: print(f"{grp}: {len(dfs[grp])} -> {len(dfs_out[grp])} rows")
test_eq(len(dfs['SEAWATER']) - len(dfs_out['SEAWATER']), 10)
BIOTA: 15951 -> 15951 rows
SEAWATER: 19193 -> 19183 rows

Normalize nuclide names

Fix case and trailing spaces

ImportantFEEDBACK TO DATA PROVIDER

Nuclide names are written in several formats. Caesium-137 appears as 137Cs, Cs-137 and CS-137. Plutonium-239+240 appears as 239, 240 Pu and 239,240Pu. 599 rows have trailing spaces, for example 137Cs and 99Tc. A standardised nuclide pick-list at the point of entry would prevent these issues.

raw = pd.concat(dfs.values(), ignore_index=True)['nuclide'].dropna()
print(f"{(raw != raw.str.strip()).sum()} rows with trailing spaces. Distinct raw names:\n")
print(sorted(raw.unique(), key=str.lower))
599 rows with trailing spaces. Distinct raw names:

['137Cs', '137Cs  ', '210Pb', '210Po', '210Po  ', '226Ra', '228Ra', '238Pu', '239, 240 Pu', '239,240Pu', '241Am', '3H', '99Tc', '99Tc  ', '99Tc   ', 'Cs-137', 'CS-137']

LowerStripNameCB lowercases and strips them into a standardised NUCLIDE column.

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    LowerStripNameCB(col_src='nuclide', col_dst='NUCLIDE')
    ])
dfs_out = tfm()

for df in dfs_out.values():
    test_eq(df['NUCLIDE'], df['NUCLIDE'].str.lower().str.strip())
print(f"All nuclide names normalised across {len(dfs_out)} sample groups.")
All nuclide names normalised across 2 sample groups.

Align nuclide names with MARIS

OSPAR writes most nuclide names with the mass number first, as in 137cs and 241am. MARIS writes the element first, as in cs137 and am241. Fuzzy matching cannot recover from this reordering. We reconcile the names with the semi-automated workflow used across marisco handlers: try an automatic mapping, fix what it got wrong, and check the result.

Try an automatic mapping

Derive unique provider values and fuzzy-match against MARIS reference.

provider_lut = lut_from(dfs_out, 'NUCLIDE')
maris_ref = get_lut('NUCLIDE', as_df=True)

print("provider_lut:", provider_lut.columns.tolist())
print("maris_ref:   ", maris_ref.columns.tolist())

merged = fuzzy_merge(provider_lut, maris_ref, left_on='value', right_on='nc_name')
provider_lut: ['value']
maris_ref:    ['nuclide_id', 'nc_name']

Inspect the borderline matches

Review non-exact matches to identify cases the fuzzy matcher could not resolve.

# Entries with score > 0 need human review
non_exact = merged[merged.score > 0].sort_values('score', ascending=False)
print(non_exact)
          value  nuclide_id nc_name  score
6   239, 240 pu          69   pu240      8
7     239,240pu          69   pu240      6
0         137cs           1      h3      4
1         210pb           3     c14      4
2         210po           3     c14      4
3         226ra           9    co60      4
4         228ra          11    sr89      4
8         241am           3     c14      4
5         238pu          64    u238      3
10         99tc         137      tu      3
9            3h           1      h3      2
11       cs-137          33   cs137      1

Only cs-137 matches the right nuclide, with a score of 1 for the hyphen. It needs no override. Every other name matches a wrong nuclide: 137cs matches h3, and 239,240pu matches pu240. OSPAR reports plutonium-239 and plutonium-240 as one combined value, which maps to the MARIS total pu239_240_tot.

Fix what it got wrong

Apply expert overrides for cases the fuzzy match could not resolve correctly.

The dictionary below records our expert decisions for every case the fuzzy matcher got wrong. Each entry maps a provider nuclide value to its correct MARIS nc_name. The fix_lut function applies these overrides and resets the score to 0.

Exported source 
fixes_nuclide_names = {
    '137cs': 'cs137',
    '210pb': 'pb210',
    '210po': 'po210',
    '226ra': 'ra226',
    '228ra': 'ra228',
    '238pu': 'pu238',
    '239, 240 pu': 'pu239_240_tot',
    '239,240pu': 'pu239_240_tot',
    '241am': 'am241',
    '3h': 'h3',
    '99tc': 'tc99'
    }
fixed = fix_lut(merged, fixes_nuclide_names, maris_ref,
                left_on='value', right_on='nc_name', id_col='nuclide_id')

# Verify: no unresolved matches remain apart from cs-137, which is correct
unresolved = fixed[(fixed['score'] > 0) & (fixed['value'] != 'cs-137')]
test_eq(len(unresolved), 0)
print(fixed.sort_values('value').to_string(index=False))
      value  nuclide_id       nc_name  score
      137cs          33         cs137      0
      210pb          41         pb210      0
      210po          47         po210      0
      226ra          53         ra226      0
      228ra          54         ra228      0
      238pu          67         pu238      0
239, 240 pu          77 pu239_240_tot      0
  239,240pu          77 pu239_240_tot      0
      241am          72         am241      0
         3h           1            h3      0
       99tc          15          tc99      0
     cs-137          33         cs137      1

Assemble the final mapping

The steps above (unique values, fuzzy match, expert overrides, verification) told us what the correct MARIS translations are. The make_lut function packages the expert fixes and the MARIS reference table into a single function. The Transformer calls it later, when it processes the data through the pipeline.

Exported source 
# Resolved nuclide lookup table (provider → MARIS nuclide_id); lazy, resolves at Transformer time
nuclide_lut = make_lut('NUCLIDE', fixes=fixes_nuclide_names)

The nuclide_lut lookup table is passed to the generic RemapCB callback, which looks up the MARIS nuclide reference table when the Transformer runs. It translates NUCLIDE from the provider string (after lowercasing and stripping) into the MARIS integer nuclide_id, across all sample-type groups.

Let’s verify the full pipeline works:

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    LowerStripNameCB(col_src='nuclide', col_dst='NUCLIDE'),
    RemapCB(lut=nuclide_lut, col_remap='NUCLIDE', col_src='NUCLIDE')
    ])
dfs_out = tfm()

for grp, df in dfs_out.items():
    test_eq(df['NUCLIDE'].dtype, 'int64')
    test_eq((df['NUCLIDE'] == 0).sum(), 0)
print(f"NUCLIDE is a MARIS integer ID in every row of all {len(dfs_out)} groups.")
NUCLIDE is a MARIS integer ID in every row of all 2 groups.

Rows per MARIS nuclide, to check that every provider name reached the expected nuclide:

names = get_lut('NUCLIDE', as_df=True).set_index('nuclide_id')['nc_name']
for grp, df in dfs_out.items():
    print(grp, df['NUCLIDE'].map(names).value_counts().to_dict())
BIOTA {'cs137': 8822, 'tc99': 2592, 'pu239_240_tot': 2028, 'ra226': 983, 'ra228': 729, 'po210': 345, 'pb210': 331, 'h3': 101, 'pu238': 10, 'am241': 10}
SEAWATER {'cs137': 7572, 'h3': 6109, 'pu239_240_tot': 1876, 'ra226': 1420, 'tc99': 1398, 'ra228': 748, 'po210': 57, 'pb210': 3}

Standardize time

OSPAR provides the sampling date in a sampling date column, in the format MM/DD/YY HH:MM:SS. Most records have a time of 00:00:00, meaning that no time of day was recorded. 2,890 records have an actual sampling time, mostly from the Netherlands, Ireland and Belgium. ParseTimeCB keeps the time of day.

Years have two digits. pandas reads 69 to 99 as 1969 to 1999, and 00 to 68 as 2000 to 2068. The OSPAR data covers 1995 to 2022, which falls inside that window.

ImportantFEEDBACK TO DATA PROVIDER

Sampling dates use two-digit years (MM/DD/YY). A two-digit year is ambiguous once a dataset spans more than a century of possible dates. ISO 8601 dates (YYYY-MM-DD) would remove the ambiguity.

Most records give 00:00:00 as the time of day. This value cannot be told apart from a sample taken at midnight. Leaving the time empty when it was not recorded would make the distinction clear.

dates = pd.concat(dfs.values(), ignore_index=True)['sampling date'].dropna()
print(dates.sample(3, random_state=42).to_list())
print(f"Time of day other than 00:00:00: {(~dates.str.endswith('00:00:00')).sum()} rows")
['11/17/21 00:00:00', '10/10/14 00:00:00', '10/15/20 00:00:00']
Time of day other than 00:00:00: 2890 rows

ParseTimeCB parses sampling date into a TIME column. A date it cannot parse becomes NaT.

source

ParseTimeCB

def ParseTimeCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Parse OSPAR sampling date (MM/DD/YY HH:MM:SS) into TIME

Verify ParseTimeCB on mock data. The third row has no date and becomes NaT:

dfs_mock = {
    'SEAWATER': pd.DataFrame({'sampling date': ['06/13/95 00:00:00', '12/31/22 00:00:00', None]}),
    'BIOTA':    pd.DataFrame({'sampling date': ['01/03/05 00:00:00']}),
}
tfm = Transformer(dfs_mock, cbs=[ParseTimeCB()])
tfm()
test_eq(tfm.dfs['SEAWATER']['TIME'].iloc[:2].to_list(), [pd.Timestamp('1995-06-13'), pd.Timestamp('2022-12-31')])
test_eq(tfm.dfs['SEAWATER']['TIME'].isna().to_list(), [False, False, True])
test_eq(tfm.dfs['BIOTA']['TIME'].to_list(), [pd.Timestamp('2005-01-03')])

Applying ParseTimeCB to the real data, after dropping the records with no date:

tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check), ParseTimeCB()])
dfs_out = tfm()

for grp, df in dfs_out.items():
    test_eq(df['TIME'].isna().sum(), 0)
    print(f"{grp}: {df['TIME'].min():%Y-%m-%d} to {df['TIME'].max():%Y-%m-%d}")
BIOTA: 1995-01-01 to 2022-12-31
SEAWATER: 1995-01-03 to 2022-12-25

NetCDF stores time as seconds since 1970-01-01, as defined in the template’s CDL. EncodeTimeCB converts the parsed TIME column to this integer format. It drops any row without a TIME. No such row remains after RemoveAllNAValuesCB.

Applying ParseTimeCB and EncodeTimeCB together:

tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check), ParseTimeCB(), EncodeTimeCB()])
dfs_out = tfm()

for grp, df in dfs_out.items():
    test_eq(df['TIME'].dtype, 'int64')
    test_eq(len(df), len(dfs[grp].dropna(subset=nan_cols_to_check, how='all')))
print(f"TIME encoded as int64 in all {len(dfs_out)} groups, with no row dropped.")
TIME encoded as int64 in all 2 groups, with no row dropped.

Sanitize value

OSPAR reports each measurement in an activity or mda column, in both sample types. The column holds a measured activity when value type is =, and a minimum detectable activity (MDA) when value type is <. SanitizeValueCB copies the column into the MARIS VALUE column and drops any row without a value. No such row remains after RemoveAllNAValuesCB.

ImportantFEEDBACK TO DATA PROVIDER

Three Norwegian 99Tc BIOTA records (IDs 89894 to 89896) report a measured activity (value type =) of exactly 0, with an uncertainty of 0. A measured activity of 0 with no uncertainty has no physical meaning. If the activity was below detection, the record should give value type < and the MDA.

biota = dfs['BIOTA']
print(biota.loc[biota['activity or mda'] == 0,
                ['id', 'contracting party', 'nuclide', 'species', 'value type', 'activity or mda', 'uncertainty']].to_string())
          id contracting party nuclide              species value type  activity or mda  uncertainty
13860  89894            Norway    99Tc    Fucus vesiculosus          =              0.0          0.0
13861  89895            Norway    99Tc    Fucus vesiculosus          =              0.0          0.0
13862  89896            Norway    99Tc  Ascophyllum nodosum          =              0.0          0.0
NoteFEEDBACK TO MARIS DATA TEAM

The three Norwegian 99Tc BIOTA records with a measured activity of 0 and an uncertainty of 0 (IDs 89894 to 89896) are kept as reported. Please decide whether to keep them, drop them, or record them as below the MDA (DL 2).

source

SanitizeValueCB

def SanitizeValueCB(
    coi:Dict[str, Dict[str, str]], # Columns of interest. Format: {group_name: {'VALUE': 'column_name'}}
):

Sanitize measurement values by removing blanks and standardizing to use the VALUE column

Exported source 
coi_val = {'SEAWATER': {'VALUE': 'activity or mda'},
           'BIOTA':    {'VALUE': 'activity or mda'}}

Verify SanitizeValueCB on mock data. The SEAWATER row without a value is dropped:

dfs_mock = {
    'SEAWATER': pd.DataFrame({'activity or mda': [1.5, None, 0.02]}),
    'BIOTA':    pd.DataFrame({'activity or mda': [0.2]}),
}
tfm = Transformer(dfs_mock, cbs=[SanitizeValueCB(coi_val)])
tfm()
test_eq(tfm.dfs['SEAWATER']['VALUE'].to_list(), [1.5, 0.02])
test_eq(tfm.dfs['BIOTA']['VALUE'].to_list(), [0.2])

Applying SanitizeValueCB to the real data:

tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check), SanitizeValueCB(coi_val)])
dfs_out = tfm()

for grp, df in dfs_out.items():
    test_eq(df['VALUE'].isna().sum(), 0)
    test_eq(len(df), len(dfs[grp].dropna(subset=nan_cols_to_check, how='all')))
print(f"VALUE column created across all {len(dfs_out)} groups, with no row dropped.")
VALUE column created across all 2 groups, with no row dropped.

Normalize uncertainty

OSPAR reports the uncertainty of each measurement as an expanded uncertainty with a coverage factor of 2 (k=2), in the uncertainty column. See the OSPAR reporting guidelines. MARIS stores the standard uncertainty (k=1). NormalizeUncCB divides the reported uncertainty by the coverage factor and writes the result to the UNC column.

Records below the MDA (value type <) carry no uncertainty. Their UNC stays empty.

source

NormalizeUncCB

def NormalizeUncCB(
    coi:dict={'SEAWATER': 'uncertainty', 'BIOTA': 'uncertainty'}, # {group: uncertainty column}
    k:float=2, # Coverage factor of the reported uncertainty
):

Convert expanded uncertainty (coverage factor k) to standard uncertainty UNC

Exported source 
coi_unc = {'SEAWATER': 'uncertainty',
           'BIOTA':    'uncertainty'}

Verify NormalizeUncCB on mock data. A missing uncertainty stays missing:

dfs_mock = {
    'SEAWATER': pd.DataFrame({'VALUE': [5.3, 0.02], 'uncertainty': [0.6, None]}),
    'BIOTA':    pd.DataFrame({'VALUE': [135.3],     'uncertainty': [9.0]}),
}
tfm = Transformer(dfs_mock, cbs=[NormalizeUncCB()])
tfm()
test_eq(tfm.dfs['SEAWATER']['UNC'].iloc[0], 0.3)
test_eq(tfm.dfs['SEAWATER']['UNC'].isna().to_list(), [False, True])
test_eq(tfm.dfs['BIOTA']['UNC'].to_list(), [4.5])

Applying NormalizeUncCB to the real data:

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    SanitizeValueCB(coi_val),
    NormalizeUncCB()
    ])
dfs_out = tfm()

for grp, df in dfs_out.items():
    pd.testing.assert_series_equal(df['UNC'], df['uncertainty'] / 2, check_names=False)
    no_unc = df['UNC'].isna()
    print(f"{grp}: {no_unc.sum()} rows without UNC, of which {(df.loc[no_unc, 'value type'] == '<').sum()} are below the MDA")
BIOTA: 4881 rows without UNC, of which 4873 are below the MDA
SEAWATER: 4820 rows without UNC, of which 4818 are below the MDA
ImportantFEEDBACK TO DATA PROVIDER

99 measured BIOTA records and 95 measured SEAWATER records report an uncertainty larger than the value itself. In 67 of them the uncertainty is more than 10 times the value. For example, a Norwegian plutonium-239+240 seawater record (ID 15018) reports an activity of 2.4e-12 Bq/L with an uncertainty of 4e-7 Bq/L. Ratios this large suggest that the value or the uncertainty was reported in a different unit, or that the uncertainty was reported as a percentage. These records should be checked at source.

Also, 10 measured records have no uncertainty, and 13 records below the MDA report one.

The handler keeps these values as reported. Records with an uncertainty more than 10 times the value, largest ratio first:

cols = ['id', 'contracting party', 'nuclide', 'activity or mda', 'uncertainty', 'unit']
for grp, df in dfs_out.items():
    ratio = (df['uncertainty'] / df['activity or mda']).round(0)
    big = df[(df['value type'] == '=') & (ratio > 10)].assign(ratio=ratio)
    print(f"{grp}: {len(big)} records")
    print(big.sort_values('ratio', ascending=False)[cols + ['ratio']].head(10).to_string(), '\n')
BIOTA: 3 records
         id contracting party nuclide  activity or mda  uncertainty        unit  ratio
3027  35011           Belgium   137Cs           0.1619         66.0  Bq/kg f.w.  408.0
2693  29984           Belgium   137Cs           0.1690         27.0  Bq/kg f.w.  160.0
2338  23895           Belgium   226Ra           1.4000        118.0  Bq/kg f.w.   84.0 

SEAWATER: 64 records
           id contracting party    nuclide  activity or mda   uncertainty  unit     ratio
1379    15018            Norway  239,240Pu     2.400000e-12  4.000000e-07  Bq/l  166667.0
6462    55488    United Kingdom         3H     1.110910e+01  9.716400e+04  Bq/l    8746.0
14237   94695    United Kingdom         3H     1.554393e-01  5.528771e+02  Bq/l    3557.0
10230   75829            Norway       99Tc     1.000000e-04  6.000000e-02  Bq/l     600.0
10231   75830            Norway       99Tc     8.000000e-05  4.000000e-02  Bq/l     500.0
7739    61137            Norway       99Tc     5.121716e-05  2.262377e-02  Bq/l     442.0
7740    61138            Norway       99Tc     5.266393e-05  2.183357e-02  Bq/l     415.0
16555  121584    United Kingdom      137Cs     1.690000e-03  5.120700e-01  Bq/l     303.0
10229   75828            Norway       99Tc     1.400000e-04  4.000000e-02  Bq/l     286.0
7738    61136            Norway       99Tc     6.402726e-05  1.747559e-02  Bq/l     273.0
NoteFEEDBACK TO MARIS DATA TEAM

Two decisions need review:

  • The records with an uncertainty more than 10 times the value are kept as reported. Please decide whether to keep them, drop them, or ask the provider for corrections first.
  • The handler divides every uncertainty by 2, following the OSPAR guideline that uncertainties are reported with k=2. Some ratios above suggest that a few contracting parties report k=1 or a relative uncertainty. Please confirm that k=2 applies to all contracting parties.

Remap units

OSPAR reports seawater activities in Bq/L and biota activities in Bq/kg fresh weight (Bq/kg f.w.). MARIS stores seawater in Bq/m³. RemapUnitCB sets the MARIS UNIT column and multiplies seawater VALUE and UNC by 1000. For records below the MDA, VALUE holds the MDA, which is converted the same way. Biota values need no conversion. MARIS unit 5 (Bq per kgw, wet weight) covers fresh weight.

ImportantFEEDBACK TO DATA PROVIDER

The seawater unit is spelled three ways: Bq/l (18,761 records), Bq/L (376) and BQ/L (48). A single spelling would simplify processing. The 8 records with no unit are among the empty records removed in “Remove missing values”.

for grp, df in dfs.items(): print(grp, df['unit'].value_counts(dropna=False).to_dict())
BIOTA {'Bq/kg f.w.': 15951}
SEAWATER {'Bq/l': 18761, 'Bq/L': 376, 'BQ/L': 48, nan: 8}

The MARIS units:

print(get_lut('UNIT', as_df=True).head(7))
   unit_id  unit_sanitized
0       -1  Not applicable
1        0   NOT AVAILABLE
2        1       Bq per m3
3        2       Bq per m2
4        3       Bq per kg
5        4      Bq per kgd
6        5      Bq per kgw

lut_units maps each OSPAR unit, lowercased, to its MARIS unit ID and the factor that converts the value to that unit. A unit missing from lut_units gets UNIT 0 (not available), and its values stay as reported.

source

RemapUnitCB

def RemapUnitCB(
    lut:dict={'bq/l': (1, 1000), 'bq/kg f.w.': (5, 1)}, # {lowercased provider unit: (MARIS unit ID, conversion factor)}
):

Set UNIT from the reported unit and convert VALUE and UNC to it (seawater Bq/L to Bq/m³)

Exported source 
lut_units = {
    'bq/l':       (1, 1000), # Bq/L to Bq/m³ (MARIS unit 1)
    'bq/kg f.w.': (5, 1),    # Bq/kg fresh weight, stored as Bq/kgw (MARIS unit 5)
}

Verify RemapUnitCB on mock data. The three seawater spellings convert to Bq/m³, and an unknown unit gets UNIT 0 with its value unchanged:

dfs_mock = {
    'SEAWATER': pd.DataFrame({'unit': ['Bq/l', 'Bq/L', 'BQ/L', 'pCi/l'],
                              'VALUE': [0.005, 2.8, 0.01, 7.0], 'UNC': [0.0005, None, 0.001, 1.0]}),
    'BIOTA':    pd.DataFrame({'unit': ['Bq/kg f.w.'], 'VALUE': [0.2], 'UNC': [0.02]}),
}
tfm = Transformer(dfs_mock, cbs=[RemapUnitCB()])
tfm()
sw = tfm.dfs['SEAWATER']
test_eq(sw['UNIT'].to_list(), [1, 1, 1, 0])
test_close(sw['VALUE'].to_list(), [5.0, 2800.0, 10.0, 7.0])
test_close(sw['UNC'].iloc[[0, 2, 3]].to_list(), [0.5, 1.0, 1.0])
test_eq(sw['UNC'].isna().to_list(), [False, True, False, False])
test_eq(tfm.dfs['BIOTA'][['UNIT', 'VALUE', 'UNC']].values.tolist(), [[5, 0.2, 0.02]])

Applying RemapUnitCB to the real data:

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    SanitizeValueCB(coi_val),
    NormalizeUncCB(),
    RemapUnitCB()
    ])
dfs_out = tfm()

for grp, df in dfs_out.items(): print(f"{grp}: UNIT values = {df['UNIT'].unique()}")
test_eq(set(dfs_out['SEAWATER']['UNIT']), {1})
test_eq(set(dfs_out['BIOTA']['UNIT']), {5})

sw = dfs_out['SEAWATER']
test_close(sw['VALUE'], sw['activity or mda'] * 1000)
print(sw[['nuclide', 'activity or mda', 'unit', 'VALUE', 'UNC', 'UNIT']].head(3).to_string())
BIOTA: UNIT values = [5]
SEAWATER: UNIT values = [1]
  nuclide  activity or mda  unit  VALUE  UNC  UNIT
0   137Cs             0.20  Bq/l  200.0  NaN     1
1   137Cs             0.27  Bq/l  270.0  NaN     1
2   137Cs             0.26  Bq/l  260.0  NaN     1
NoteFEEDBACK TO MARIS DATA TEAM

Biota activities reported per kg fresh weight (Bq/kg f.w.) map to MARIS unit 5, Bq per kgw (wet weight), as in the previous version of this handler. Please confirm that fresh weight and wet weight are equivalent for MARIS.

Remap detection limit

OSPAR records whether a value is a measured activity or a minimum detectable activity (MDA) in the value type column: = for a measured activity, < for an MDA. MARIS uses the following integer codes for this distinction:

print(get_lut('DL', as_df=True))
   id   name_sanitized
0  -1   Not applicable
1   0    Not available
2   1   Detected value
3   2  Detection limit
4   3     Not detected
5   4          Derived
ImportantFEEDBACK TO DATA PROVIDER

77 records have no value type: 23 Belgian BIOTA records, and 52 UK and 2 Belgian SEAWATER records, all sampled in 2020 and 2021. Every one of them reports an uncertainty, which suggests a measured activity. The value type should be filled in at source.

cols = ['id', 'contracting party', 'nuclide', 'sampling date', 'activity or mda', 'uncertainty']
for grp, df in dfs.items():
    na = df[df['value type'].isna() & df['activity or mda'].notna()]
    print(f"{grp}: {len(na)} records without value type, {na['uncertainty'].notna().sum()} of them with an uncertainty")
    print(na[cols].head(3).to_string(), '\n')
BIOTA: 23 records without value type, 23 of them with an uncertainty
          id contracting party nuclide      sampling date  activity or mda  uncertainty
15508  95221           Belgium   226Ra  11/22/21 00:00:00         0.509549     0.162129
15583  95278           Belgium    99Tc  11/26/21 00:00:00         2.527464     0.229769
15587  95282           Belgium   226Ra  03/03/21 00:00:00         0.477137     0.458052 

SEAWATER: 54 records without value type, 54 of them with an uncertainty
           id contracting party nuclide      sampling date  activity or mda  uncertainty
2880   119105           Belgium   137Cs  01/27/21 00:00:00          0.00130     0.000300
3245   119147           Belgium   228Ra  03/24/21 00:00:00          0.29000     0.190000
16005  118947    United Kingdom      3H  09/23/20 00:00:00          2.37474     1.099254
NoteFEEDBACK TO MARIS DATA TEAM

The 77 records without a value type are treated as measured activities (DL 1), because each reports an uncertainty. This is the rule of the previous version of this handler. Please confirm it, or ask the provider for the missing values first.

lut_dl maps each value type to its MARIS code. RemapDetectionLimitCB writes the code to the DL column. A record with no value type but with an uncertainty is treated as a measured activity (code 1). Any other record with no value type gets code 0 (not available).

source

RemapDetectionLimitCB

def RemapDetectionLimitCB(
    coi:dict={'SEAWATER': {'DL': 'value type'}, 'BIOTA': {'DL': 'value type'}}, # {group: {'DL': column holding the value type}}
    lut:dict={'=': 1, '<': 2}, # {value type: MARIS DL code}
):

Map OSPAR value type (= / <) to MARIS detection-limit codes, treating a missing type with an uncertainty as detected

Exported source 
coi_dl = {'SEAWATER': {'DL': 'value type'},
          'BIOTA':    {'DL': 'value type'}}

lut_dl = {'=': 1, # Detected value
          '<': 2} # Detection limit

Verify RemapDetectionLimitCB on mock data. The third row has no value type but an uncertainty, and the fourth has neither:

dfs_mock = {
    'SEAWATER': pd.DataFrame({'value type': ['=', '<', None, None], 'UNC': [0.1, None, 0.2, None]}),
    'BIOTA':    pd.DataFrame({'value type': ['<', '='], 'UNC': [None, 0.5]}),
}
tfm = Transformer(dfs_mock, cbs=[RemapDetectionLimitCB()])
tfm()
test_eq(tfm.dfs['SEAWATER']['DL'].to_list(), [1, 2, 1, 0])
test_eq(tfm.dfs['BIOTA']['DL'].to_list(), [2, 1])

Applying RemapDetectionLimitCB to the real data:

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    SanitizeValueCB(coi_val),
    NormalizeUncCB(),
    RemapUnitCB(),
    RemapDetectionLimitCB()
    ])
dfs_out = tfm()

for grp, df in dfs_out.items():
    print(f"{grp}: DL counts = {df['DL'].value_counts().to_dict()}")
    test_eq(set(df['DL']), {1, 2})
BIOTA: DL counts = {1: 11073, 2: 4878}
SEAWATER: DL counts = {1: 14357, 2: 4826}

Remap Biota species

OSPAR records each biota sample’s species by scientific name, in the species column. OSPAR supplies no species lookup table. We derive the unique names from the data and reconcile them with the MARIS species reference, following the “no provider lookup table” route of the reconcile-nomenclature how-to.

ImportantFEEDBACK TO DATA PROVIDER

Species names are not standardised:

  • 2,198 Belgian BIOTA records, all in the Fish biological group, have no species.
  • 193 records have trailing spaces, for example Gadus morhua and MERLUCCIUS MERLUCCIUS.
  • The same species is written in different cases, for example Solea solea (S.vulgaris) and SOLEA SOLEA (S.VULGARIS). After lowercasing and trimming, 166 distinct names reduce to 112.
  • Some names include a synonym in brackets, for example Cerastoderma (Cardium) edule and Dicentrarchus (Morone) labrax.
  • Some names are misspelt: PLUERONECTES PLATESSA, MERLANGUIS MERLANGUIS, ASCOPHYLLUN NODOSUM, Sebastes vivipares, Trisopterus esmarki and Gaidropsarus argenteus.
  • Monodonta lineata is an outdated name. The accepted name is Phorcus lineatus.
  • The species column also holds placeholders (Unknown), common names (Flatfish) and mixtures (Mixture of green, red and brown algae).

Reporting species with their WoRMS accepted name, or their AphiaID, would remove these issues.

species = dfs['BIOTA']['species']
print(f"Records without species: {species.isna().sum()}")
print(f"Records with leading or trailing spaces: {(species.dropna() != species.dropna().str.strip()).sum()}")
print(f"Distinct names: {species.nunique()} raw, {species.str.strip().str.lower().nunique()} after lowercasing and trimming")
Records without species: 2198
Records with leading or trailing spaces: 193
Distinct names: 166 raw, 112 after lowercasing and trimming

LowerStripNameCB lowercases and trims the BIOTA names into a SPECIES column. The fuzzy matching ignores case, and lowercasing merges the case variants of the same name into one entry. grps=['BIOTA'] restricts it to BIOTA, the only group with a species column.

Try an automatic mapping

Derive unique provider values and fuzzy-match against MARIS reference.

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    LowerStripNameCB(col_src='species', col_dst='SPECIES', grps=['BIOTA'])
    ])
dfs_out = tfm()

# lut_from skips records without species. The next section fills them in.
provider_lut = lut_from(dfs_out, 'SPECIES')
maris_ref = get_lut('SPECIES', as_df=True)
merged = fuzzy_merge(provider_lut, maris_ref, left_on='value', right_on='species')
print(f"{len(merged)} distinct names: {(merged.score == 0).sum()} exact matches, {(merged.score > 0).sum()} to review")
112 distinct names: 83 exact matches, 29 to review

Inspect the borderline matches

Review non-exact matches to identify cases the fuzzy matcher could not resolve.

non_exact = merged[merged.score > 0].sort_values('score', ascending=False)
print(non_exact[['value', 'species', 'species_id', 'score']].to_string(index=False))
                                       value                 species  species_id  score
rhodymenia pseudopalamata & palmaria palmata     Hoplobrotula armata         137     31
       mixture of green, red and brown algae Strongylura anastomella         541     26
                    solea solea (s.vulgaris)         Loligo vulgaris         161     12
                cerastoderma (cardium) edule      Cerastoderma edule         274     10
               dicentrarchus (morone) labrax    Dicentrarchus labrax         424      9
                            rajidae/batoidea                Batoidea         992      8
                   pleuronectiformes [order]       Pleuronectiformes         411      8
                            palmaria palmata        Alaria marginata        1104      7
                         gadiculus argenteus        Pampus argenteus         223      6
                           monodonta lineata         Monodonta labio        1684      6
                         raja dipturus batis          Dipturus batis         436      5
                                     unknown                Plankton         280      5
                             rhodymenia spp.              Rhodymenia         396      5
                                  fucus spp.                   Fucus         395      5
                                    flatfish                  Lambia         148      5
                                  sepia spp.                   Sepia         408      5
                                   tapes sp.                   Tapes         431      4
                                 thunnus sp.                 Thunnus         442      4
                              rhodymenia spp              Rhodymenia         396      4
                                 patella sp.                 Patella         413      4
                                   gadus sp.                   Gadus         409      4
                                   fucus spp                   Fucus         395      4
                                   fucus sp.                   Fucus         395      4
                       merlanguis merlanguis    Merlangius merlangus         139      3
                       plueronectes platessa   Pleuronectes platessa         192      2
                      gaidropsarus argenteus Gaidropsarus argentatus         444      2
                          sebastes vivipares      Sebastes viviparus         390      1
                         trisopterus esmarki    Trisopterus esmarkii         385      1
                         ascophyllun nodosum     Ascophyllum nodosum         401      1

Most of the non-exact matches are right. The fuzzy matcher recovers misspellings (plueronectes platessa), names with a bracketed synonym (cerastoderma (cardium) edule), and genus-level names (fucus spp. to Fucus, thunnus sp. to Thunnus). The matches it gets wrong are overridden below.

Fix what it got wrong

Apply expert overrides for cases the fuzzy match could not resolve correctly.

NoteFEEDBACK TO MARIS DATA TEAM

The overrides below include judgement calls. Please review them.

  • solea solea (s.vulgaris): the fuzzy matcher picked the squid Loligo vulgaris from the bracketed synonym. The fish is Solea solea.
  • monodonta lineata: the fuzzy matcher picked Monodonta labio, a different species. Monodonta lineata is a synonym of Phorcus lineatus, which is in the MARIS reference.
  • flatfish: the common name for the order Pleuronectiformes.
  • unknown: no species information. It maps to NOT AVAILABLE, and the next section fills it in from the biological group.
  • palmaria palmata (dulse): a red alga with no entry in the MARIS reference. It maps to the phylum Rhodophyta, the most specific MARIS entry that contains it. The fuzzy matcher picked the brown alga Alaria marginata. Please consider adding Palmaria palmata to the MARIS species reference.
  • rhodymenia pseudopalamata & palmaria palmata: a mixture of two red algae. It maps to Rhodophyta, which contains both.
  • mixture of green, red and brown algae: a mixture spanning three algal groups. It maps to Marine algae, the MARIS entry that covers all three.
  • gadiculus argenteus: the fuzzy matcher picked the pomfret Pampus argenteus. The MARIS reference has only Gadiculus argenteus thori, the North-East Atlantic form, which matches the OSPAR area.

Two non-exact matches are kept without an override:

  • rajidae/batoidea matches Batoidea. The record does not say which of the two groups applies, and Batoidea contains Rajidae.
  • gadus sp. matches the genus Gadus. The previous version of this handler mapped it to not available, before the genus was in the MARIS reference.
Exported source 
fixes_species = {
    'solea solea (s.vulgaris)': 'Solea solea', # Fuzzy match picked Loligo vulgaris from the synonym
    'monodonta lineata': 'Phorcus lineatus', # Accepted name of the same species
    'flatfish': 'Pleuronectiformes', # Common name of the order
    'unknown': 'NOT AVAILABLE', # Filled in later from the biological group
    'palmaria palmata': 'Rhodophyta', # Not in the MARIS reference, mapped to its phylum
    'rhodymenia pseudopalamata & palmaria palmata': 'Rhodophyta', # Mixture of two red algae
    'mixture of green, red and brown algae': 'Marine algae', # Mixture of algal groups
    'gadiculus argenteus': 'Gadiculus argenteus thori', # North-East Atlantic form
}
fixed = fix_lut(merged, fixes_species, maris_ref,
                left_on='value', right_on='species', id_col='species_id')

# The remaining non-exact matches are the ones reviewed above and accepted as correct
remaining = fixed[fixed.score > 0].sort_values('value')
print(remaining[['value', 'species', 'score']].to_string(index=False))
test_eq(len(remaining), 21)
                        value                 species  score
          ascophyllun nodosum     Ascophyllum nodosum      1
 cerastoderma (cardium) edule      Cerastoderma edule     10
dicentrarchus (morone) labrax    Dicentrarchus labrax      9
                    fucus sp.                   Fucus      4
                    fucus spp                   Fucus      4
                   fucus spp.                   Fucus      5
                    gadus sp.                   Gadus      4
       gaidropsarus argenteus Gaidropsarus argentatus      2
        merlanguis merlanguis    Merlangius merlangus      3
                  patella sp.                 Patella      4
    pleuronectiformes [order]       Pleuronectiformes      8
        plueronectes platessa   Pleuronectes platessa      2
          raja dipturus batis          Dipturus batis      5
             rajidae/batoidea                Batoidea      8
               rhodymenia spp              Rhodymenia      4
              rhodymenia spp.              Rhodymenia      5
           sebastes vivipares      Sebastes viviparus      1
                   sepia spp.                   Sepia      5
                    tapes sp.                   Tapes      4
                  thunnus sp.                 Thunnus      4
          trisopterus esmarki    Trisopterus esmarkii      1

Assemble the final mapping

The steps above (unique values, fuzzy match, expert overrides, verification) told us what the correct MARIS translations are. make_lut packages the expert fixes and the MARIS reference table into a single function. The Transformer calls it later, when it processes the data through the pipeline. Records without species are left out of the matching, and RemapCB gives them SPECIES 0.

Exported source 
species_lut = make_lut('SPECIES', fixes=fixes_species)

Verify the species lookup on mock data. Case, spaces and a bracketed synonym are resolved. A missing species and Unknown both get SPECIES 0:

dfs_mock = {'BIOTA': pd.DataFrame({'species': ['SOLEA SOLEA (S.VULGARIS) ', 'Fucus vesiculosus', None, 'Unknown']})}
tfm = Transformer(dfs_mock, cbs=[
    LowerStripNameCB(col_src='species', col_dst='SPECIES', grps=['BIOTA']),
    RemapCB(lut=species_lut, col_remap='SPECIES', col_src='SPECIES', grps=['BIOTA'])
    ])
tfm()
test_eq(tfm.dfs['BIOTA']['SPECIES'].to_list(), [397, 96, 0, 0])

Map species on the real OSPAR BIOTA data:

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    LowerStripNameCB(col_src='species', col_dst='SPECIES', grps=['BIOTA']),
    RemapCB(lut=species_lut, col_remap='SPECIES', col_src='SPECIES', grps=['BIOTA'])
    ])
dfs_out = tfm()

biota = dfs_out['BIOTA']
no_species = biota['species'].isna() | (biota['species'].str.strip().str.lower() == 'unknown')
print(f"SPECIES 0 (not available): {(biota['SPECIES'] == 0).sum()} rows, all without species information")
test_eq(biota.loc[biota['SPECIES'] == 0].index.tolist(), biota[no_species].index.tolist())
SPECIES 0 (not available): 2234 rows, all without species information

Fill in species from the biological group

After the species remap, 2,234 BIOTA records have SPECIES 0: the 2,198 records without species and the 36 marked unknown. OSPAR also records a broad biological group for every biota record: fish, molluscs or seaweed. EnhanceSpeciesCB uses it to give these records a broad MARIS species entry instead of 0. Records that already have a species keep it.

ImportantFEEDBACK TO DATA PROVIDER

The biological group column spells three groups in ten ways: Fish, FISH and fish; Molluscs, MOLLUSCS and molluscs; Seaweed, SEAWEED, seaweed and Seaweeds. A fixed pick-list would prevent this.

biota = dfs['BIOTA']
print(biota['biological group'].value_counts(dropna=False).to_dict())
no_species = biota['species'].isna() | (biota['species'].str.strip().str.lower() == 'unknown')
print(f"Records without species, by biological group: {biota.loc[no_species, 'biological group'].value_counts().to_dict()}")
{'Seaweed': 5705, 'Fish': 5000, 'Molluscs': 3791, 'SEAWEED': 831, 'FISH': 459, 'Seaweeds': 48, 'MOLLUSCS': 46, 'fish': 28, 'molluscs': 23, 'seaweed': 20}
Records without species, by biological group: {'Fish': 2208, 'Seaweed': 26}

With three groups to map, a plain dict is enough. lut_biogroup_species maps each lowercased, trimmed group to the broad MARIS species entry that covers it:

Exported source 
lut_biogroup_species = {
    'fish':     712,  # Pisces
    'molluscs': 873,  # Mollusca
    'seaweed':  1059, # Seaweed
    'seaweeds': 1059, # Seaweed
}
names = get_lut('SPECIES', as_df=True).set_index('species_id')['species']
print({grp: names[sid] for grp, sid in lut_biogroup_species.items()})
test_eq([names[sid] for sid in lut_biogroup_species.values()], ['Pisces', 'Mollusca', 'Seaweed', 'Seaweed'])
{'fish': 'Pisces', 'molluscs': 'Mollusca', 'seaweed': 'Seaweed', 'seaweeds': 'Seaweed'}
NoteFEEDBACK TO MARIS DATA TEAM

Seaweed records without a species map to the MARIS entry Seaweed (1059). Marine algae (734) is an alternative. Please confirm the choice.

source

EnhanceSpeciesCB

def EnhanceSpeciesCB(
    lut:dict={'fish': 712, 'molluscs': 873, 'seaweed': 1059, 'seaweeds': 1059}, # {lowercased biological group: MARIS species_id}
):

Replace SPECIES 0 (not available) with the broad MARIS entry of the record’s biological group

Verify EnhanceSpeciesCB on mock data. Only the records with SPECIES 0 change, and an unknown group leaves SPECIES at 0:

dfs_mock = {'BIOTA': pd.DataFrame({'SPECIES': [0, 0, 99, 0],
                                   'biological group': ['FISH', 'Seaweeds ', 'Fish', 'Plankton']})}
tfm = Transformer(dfs_mock, cbs=[EnhanceSpeciesCB()])
tfm()
test_eq(tfm.dfs['BIOTA']['SPECIES'].to_list(), [712, 1059, 99, 0])

Apply the species remap and EnhanceSpeciesCB to the real OSPAR BIOTA data:

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    LowerStripNameCB(col_src='species', col_dst='SPECIES', grps=['BIOTA']),
    RemapCB(lut=species_lut, col_remap='SPECIES', col_src='SPECIES', grps=['BIOTA']),
    EnhanceSpeciesCB()
    ])
biota = tfm()['BIOTA']

test_eq((biota['SPECIES'] == 0).sum(), 0)
print(f"No BIOTA record has SPECIES 0. Records filled from the biological group: "
      f"{biota.loc[no_species, 'SPECIES'].map(names).value_counts().to_dict()}")
No BIOTA record has SPECIES 0. Records filled from the biological group: {'Pisces': 2208, 'Seaweed': 26}

Remap Biota body part

OSPAR records the analysed tissue of each biota sample in the body part column. OSPAR supplies no body-part lookup table. We derive the unique values from the data and reconcile them with the MARIS body-part reference, following the same workflow as for species.

ImportantFEEDBACK TO DATA PROVIDER

Body-part values are not standardised:

  • 34 records have trailing spaces, and the same value is written in different cases, for example WHOLE FISH and Whole fish. After lowercasing and trimming, 29 distinct values reduce to 19.
  • Whole fisk is a typo for Whole fish, and FLESH WITHOUT BONE a variant of FLESH WITHOUT BONES.
  • WHOLE and Whole do not say whether the sample is a whole animal or a whole plant. We resolve them from the biological group.
  • FLESH and Flesh (226 Danish and German fish records) do not say whether bones were included.
  • UNKNOWN gives no information.
  • The Norwegian lobster (Homarus gammarus) records, with body part Tail and claws, are in the Fish biological group.
  • 92 eel (Anguilla anguilla) records report Soft parts, a term used for molluscs.
  • One kelp (Laminaria digitata) record and 4 mussel (Mytilus edulis) records report FLESH WITHOUT BONES.
bp = dfs['BIOTA']['body part']
print(f"Records with leading or trailing spaces: {(bp != bp.str.strip()).sum()}")
print(f"Distinct values: {bp.str.strip().nunique()} trimmed, {bp.str.strip().str.lower().nunique()} after lowercasing")
Records with leading or trailing spaces: 34
Distinct values: 29 trimmed, 19 after lowercasing

NormalizeBodyPartCB lowercases and trims body part into a BODY_PART column. It also resolves a plain whole: to whole plant for seaweed, and to whole animal for fish and molluscs.

source

NormalizeBodyPartCB

def NormalizeBodyPartCB(
    plant_grps:list=['seaweed', 'seaweeds'], # Lowercased biological groups whose `whole` means whole plant
):

Lowercase and trim body part into BODY_PART, resolving plain whole to whole animal or whole plant from the biological group

dfs_mock = {'BIOTA': pd.DataFrame({'body part': ['WHOLE ', 'Whole', 'Flesh without bones', 'whole plant'],
                                   'biological group': ['Fish', 'SEAWEED', 'Fish', 'Seaweeds']})}
tfm = Transformer(dfs_mock, cbs=[NormalizeBodyPartCB()])
tfm()
test_eq(tfm.dfs['BIOTA']['BODY_PART'].to_list(), ['whole animal', 'whole plant', 'flesh without bones', 'whole plant'])

Try an automatic mapping

Derive unique provider values and fuzzy-match against MARIS reference.

tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check), NormalizeBodyPartCB()])
dfs_out = tfm()

provider_lut = lut_from(dfs_out, 'BODY_PART')
maris_ref = get_lut('BODY_PART', as_df=True)
merged = fuzzy_merge(provider_lut, maris_ref, left_on='value', right_on='bodypar')
print(f"{len(merged)} distinct values: {(merged.score == 0).sum()} exact matches, {(merged.score > 0).sum()} to review")
18 distinct values: 9 exact matches, 9 to review

Inspect the borderline matches

Review non-exact matches to identify cases the fuzzy matcher could not resolve.

non_exact = merged[merged.score > 0].sort_values('score', ascending=False)
print(non_exact[['value', 'bodypar', 'bodypar_id', 'score']].to_string(index=False))
                                     value             bodypar  bodypar_id  score
mix of muscle and whole fish without liver Flesh without bones          52     27
                            tail and claws        Whole animal           1     10
                        whole without head    Whole macro alga          49     10
                             cod medallion            Skeleton           6      9
                                   unknown                Skin          12      5
                                whole fish        Whole animal           1      5
                                whole fisk        Whole animal           1      5
                                     flesh                Fins          16      3
                        flesh without bone Flesh without bones          52      1

Three non-exact matches are right: whole fish and the typo whole fisk match Whole animal, and flesh without bone matches Flesh without bones. The other six are wrong. For example, cod medallion matches Skeleton and flesh matches Fins. They are overridden below.

Fix what it got wrong

Apply expert overrides for cases the fuzzy match could not resolve correctly.

NoteFEEDBACK TO MARIS DATA TEAM

The overrides below include judgement calls. Please review them.

  • flesh (226 fish records): the provider does not say whether bones were included. It maps to Flesh with bones, the choice made by the previous version of this handler.
  • cod medallion (4 Norwegian cod records): a medallion is a boneless cut of fillet. It maps to Flesh without bones.
  • tail and claws (8 Norwegian lobster records): the edible meat of the tail and claws of a crustacean, which has no bones. It maps to Muscle.
  • whole without head (5 Norwegian fish records): MARIS has Whole animal eviscerated without head, but these fish are not reported as eviscerated. It maps to (Not available) rather than to a body part that states a different preparation. Please consider adding a Whole animal without head entry to the MARIS body-part reference.
  • mix of muscle and whole fish without liver (1 record): a mixture of two preparations with no MARIS equivalent. It maps to (Not available).
  • unknown (5 records): no information. It maps to (Not available).
  • A plain whole (43 records) is resolved from the biological group before matching: Whole plant for seaweed, Whole animal for fish.
Exported source 
fixes_body_part = {
    'flesh': 'Flesh with bones', # Bones not specified; choice of the previous handler version
    'cod medallion': 'Flesh without bones', # Boneless cut of fillet
    'tail and claws': 'Muscle', # Lobster meat; crustaceans have no bones
    'whole without head': '(Not available)', # No MARIS entry for a whole animal without head only
    'mix of muscle and whole fish without liver': '(Not available)', # Mixture of preparations
    'unknown': '(Not available)',
}
fixed = fix_lut(merged, fixes_body_part, maris_ref,
                left_on='value', right_on='bodypar', id_col='bodypar_id')

# The remaining non-exact matches are the ones reviewed above and accepted as correct
remaining = fixed[fixed.score > 0].sort_values('value')
print(remaining[['value', 'bodypar', 'score']].to_string(index=False))
test_eq(remaining['value'].to_list(), ['flesh without bone', 'whole fish', 'whole fisk'])

Assemble the final mapping

make_lut packages the expert fixes and the MARIS reference table into a single function. The Transformer calls it later, when it processes the data through the pipeline.

Exported source 
body_part_lut = make_lut('BODY_PART', fixes=fixes_body_part)

Verify the body-part lookup on mock data:

dfs_mock = {'BIOTA': pd.DataFrame({'body part': ['WHOLE FISH', 'Whole', 'Tail and claws', 'UNKNOWN'],
                                   'biological group': ['Fish', 'SEAWEED', 'Fish', 'Fish']})}
tfm = Transformer(dfs_mock, cbs=[
    NormalizeBodyPartCB(),
    RemapCB(lut=body_part_lut, col_remap='BODY_PART', col_src='BODY_PART', grps=['BIOTA'])
    ])
tfm()
test_eq(tfm.dfs['BIOTA']['BODY_PART'].to_list(), [1, 40, 34, 0])

Map body parts on the real OSPAR BIOTA data:

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    NormalizeBodyPartCB(),
    RemapCB(lut=body_part_lut, col_remap='BODY_PART', col_src='BODY_PART', grps=['BIOTA'])
    ])
biota = tfm()['BIOTA']

names = get_lut('BODY_PART', as_df=True).set_index('bodypar_id')['bodypar']
print(biota['BODY_PART'].map(names).value_counts().to_string())
test_eq((biota['BODY_PART'] == 0).sum(), 11)
BODY_PART
Whole plant            5123
Whole animal           3397
Soft parts             3235
Flesh without bones    1976
Growing tips           1480
Muscle                  457
Flesh with bones        226
Flesh with scales        23
Liver                    12
Head                     11
(Not available)          11

Remap biological group

The MARIS SPECIES lookup table includes a biogroup_id column. Every OSPAR BIOTA record now has a MARIS SPECIES ID, so the biological group is a direct lookup. No fuzzy matching is needed.

Exported source 
lut_biogroup = get_lut('SPECIES', key='species_id', value='biogroup_id')

Verify the biological-group lookup on mock species IDs: Pisces, Mollusca and Seaweed, the three entries used to fill in missing species:

dfs_mock = {'BIOTA': pd.DataFrame({'SPECIES': [712, 873, 1059]})}
tfm = Transformer(dfs_mock, cbs=[RemapCB(lut=lut_biogroup, col_remap='BIO_GROUP', col_src='SPECIES', grps=['BIOTA'])])
tfm()
test_eq(tfm.dfs['BIOTA']['BIO_GROUP'].to_list(), [4, 6, 11])

Apply to the real OSPAR BIOTA data, chained after the species remap:

tfm = Transformer(dfs, cbs=[
    RemoveAllNAValuesCB(nan_cols_to_check),
    LowerStripNameCB(col_src='species', col_dst='SPECIES', grps=['BIOTA']),
    RemapCB(lut=species_lut, col_remap='SPECIES', col_src='SPECIES', grps=['BIOTA']),
    EnhanceSpeciesCB(),
    RemapCB(lut=lut_biogroup, col_remap='BIO_GROUP', col_src='SPECIES', grps=['BIOTA'])
    ])
biota = tfm()['BIOTA']
test_eq((biota['BIO_GROUP'] == 0).sum(), 0)
print(f"BIO_GROUP counts: {biota['BIO_GROUP'].value_counts().to_dict()}")
BIO_GROUP counts: {11: 6604, 4: 5478, 14: 2036, 13: 1788, 2: 43, 12: 1, 5: 1}

Add sample ID

MARIS requires two identifier columns:

  • SMP_ID is an internal unique identifier for each record
  • SMP_ID_PROVIDER is the sample identifier given by the data provider

OSPAR gives the sample identifier in the sample id column. Several measurements of the same sample share it. AddSampleIDCB generates the sequential SMP_ID and copies sample id into SMP_ID_PROVIDER. A missing sample id becomes an empty string.

ImportantFEEDBACK TO DATA PROVIDER

sample id is missing for 5,578 BIOTA records and 6,809 SEAWATER records, about 35% of each group. Every contracting party has records without it. Without a sample identifier, measurements from the same sample cannot be linked.

for grp, df in dfs.items():
    missing = df['sample id'].isna() & df['activity or mda'].notna()
    print(f"{grp}: {missing.sum()} of {df['activity or mda'].notna().sum()} records without sample id")
BIOTA: 5578 of 15951 records without sample id
SEAWATER: 6809 of 19183 records without sample id
NoteFEEDBACK TO MARIS DATA TEAM

SMP_ID_PROVIDER holds OSPAR’s sample id, which traces a MARIS record back to the provider’s sample when it is given. For the 35% of records without one, there are two options for a future version:

  • Fall back to OSPAR’s record id when sample id is missing. id is present and unique for every record, but it identifies a measurement, not a sample.
  • Store OSPAR’s record id in a separate provider field, if MARIS adds one, and keep SMP_ID_PROVIDER for the sample identifier.

source

AddSampleIDCB

def AddSampleIDCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Assign internal sequential SMP_ID and preserve OSPAR sample id as SMP_ID_PROVIDER

# Verify sample IDs on mock data
dfs_mock = {'BIOTA': pd.DataFrame({'sample id': ['FNZ14007', None, 'FNZ14007']})}
tfm = Transformer(dfs_mock, cbs=[AddSampleIDCB()])
tfm()
test_eq(tfm.dfs['BIOTA']['SMP_ID'].to_list(), [1, 2, 3])
test_eq(tfm.dfs['BIOTA']['SMP_ID_PROVIDER'].to_list(), ['FNZ14007', '', 'FNZ14007'])
tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check), AddSampleIDCB()])
dfs_out = tfm()
for grp, df in dfs_out.items(): test_eq(df['SMP_ID'].is_unique, True)
print(dfs_out['SEAWATER'][['SMP_ID', 'SMP_ID_PROVIDER']].head())
   SMP_ID SMP_ID_PROVIDER
0       1          WNZ 01
1       2          WNZ 02
2       3          WNZ 03
3       4          WNZ 04
4       5          WNZ 05

Add depth

OSPAR gives the sampling depth, in metres, in a sampling depth column for SEAWATER only. AddDepthCB copies it into the MARIS SMP_DEPTH column as a float. 43 SEAWATER records have no sampling depth. Their SMP_DEPTH stays empty.

source

AddDepthCB

def AddDepthCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Copy OSPAR sampling depth (SEAWATER) to MARIS-standard SMP_DEPTH as float

dfs_mock = {
    'SEAWATER': pd.DataFrame({'sampling depth': [5, None]}),
    'BIOTA':    pd.DataFrame({'species': ['Fucus vesiculosus']}),
}
tfm = Transformer(dfs_mock, cbs=[AddDepthCB()])
tfm()
test_eq(tfm.dfs['SEAWATER']['SMP_DEPTH'].iloc[0], 5.0)
test_eq(tfm.dfs['SEAWATER']['SMP_DEPTH'].isna().to_list(), [False, True])
test_eq('SMP_DEPTH' in tfm.dfs['BIOTA'].columns, False)
tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check), AddDepthCB()])
depth = tfm()['SEAWATER']['SMP_DEPTH']
print(f"SMP_DEPTH: {depth.min()} to {depth.max()} m, {depth.isna().sum()} records without depth")
SMP_DEPTH: 0.0 to 1850.0 m, 43 records without depth
NoteFEEDBACK TO MARIS DATA TEAM

The 43 SEAWATER records without a sampling depth keep an empty SMP_DEPTH, as in the HELCOM handler. The curation rules in the writing-a-handler guide ask for -1 instead. Please decide which convention applies, for all handlers.

Standardize coordinates

OSPAR gives coordinates in degrees, minutes and seconds (latd, latm, lats, longd, longm, longs), with the hemisphere in latdir (N or S) and longdir (E or W). MARIS stores decimal degrees in LAT and LON. dms_to_dd converts them, and ParseCoordinatesCB applies it to every group. Some records break the format. The conversion handles them as follows:

  • A seconds value of -1 means that seconds were not recorded. It counts as 0.
  • A negative degree, minute or second, other than the -1 above, places the point west or south, whatever the hemisphere column says. For example, Jan Mayen is recorded as -9° 8' 3" E, and lies at about 9° W.
  • A minute or second value of 60 carries over into the next unit.
  • A minute value above 60 cannot be interpreted. The coordinate becomes empty, and ParseCoordinatesCB drops the record.
  • A second value between 61 and 99 is converted as seconds. The error is under 40 seconds of arc, about 1 km.
ImportantFEEDBACK TO DATA PROVIDER

Some coordinates fall outside the degree-minute-second format:

  • 11 Norwegian records use -1 for missing seconds.
  • 6 SEAWATER records give a negative degree or minute. Three Norwegian records at Jan Mayen give -9° or -8° with hemisphere E, for a position at about 8° to 9° W. Two Norwegian records at station St.685 OF give 0° -29' -56" with hemisphere E. One UK record at Sellafield gives -3°.
  • 97 records, mostly from the United Kingdom and Norway, have seconds between 61 and 99. These may be hundredths of a minute rather than seconds.
  • 1 UK SEAWATER record at Sellafield (ID 121611) gives 441 minutes of latitude. It is dropped.
  • 2 SEAWATER records have a trailing space in latdir (N).

Reporting coordinates in decimal degrees, with a sign for the hemisphere, would remove these issues.

cols = ['id', 'contracting party', 'station id', 'latd', 'latm', 'lats', 'latdir', 'longd', 'longm', 'longs', 'longdir']
for grp, df in dfs.items():
    parts = df[['latd', 'latm', 'lats', 'longd', 'longm', 'longs']]
    odd = (parts < 0).any(axis=1) | (parts[['latm', 'lats', 'longm', 'longs']] > 60).any(axis=1)
    print(f"{grp}: {odd.sum()} records with negative parts, or minutes or seconds above 60")
    print(df.loc[odd, cols].head(5).to_string(), '\n')
BIOTA: 28 records with negative parts, or minutes or seconds above 60
          id contracting party   station id  latd  latm  lats latdir  longd  longm  longs longdir
9302   72711            Norway  Utsira Nord    60  19.0  47.0      N      2   39.0   -1.0       E
10491  80487            Norway   Eggakanten    68  30.0   0.0      N     11   23.0   -1.0       E
10494  80490            Norway   Eggakanten    66  19.0  59.0      N      6   17.0   -1.0       E
10502  80498            Norway   Eggakanten    66  34.0  59.0      N      6   49.0   -1.0       E
10505  80501            Norway   Eggakanten    66  40.0   0.0      N      5   35.0   -1.0       E 

SEAWATER: 87 records with negative parts, or minutes or seconds above 60
          id contracting party                          station id  latd  latm  lats latdir  longd  longm  longs longdir
3255   99103            Faroes                             Kirkeby    61  57.0  18.0      N      6   47.0   66.0       W
3256   99104            Faroes                             Kirkeby    61  57.0  18.0      N      6   47.0   66.0       W
3257   99105            Faroes                             Kirkeby    61  57.0  18.0      N      6   47.0   66.0       W
7735   61133            Norway                           Jan Mayen    70  50.0  48.0      N     -9    8.0    3.0       E
11354  85454            Norway  Oksø-Hanstholm (56 n.m.) (St. 106)    57  10.0  59.0      N      8   34.0   -1.0       E
NoteFEEDBACK TO MARIS DATA TEAM

The coordinate rules above include judgement calls. Please review them:

  • Seconds between 61 and 99 (97 records) are converted as seconds. If the provider meant hundredths of a minute, the positions are off by up to about 1 km.
  • A negative degree or minute places the point west or south, whatever the hemisphere column says. This matches the known positions of Jan Mayen.
  • The Sellafield record with 441 minutes of latitude (ID 121611) is dropped. The provider may be able to give the correct position.

source

dms_to_dd

def dms_to_dd(
    d:pandas.Series, # Degrees
    m:pandas.Series, # Minutes
    s:pandas.Series, # Seconds; -1 means not recorded
    hemi:pandas.Series, # Hemisphere: N, S, E or W
)->pandas.Series: # Decimal degrees; NaN when minutes exceed 60

Convert OSPAR degree, minute, second and hemisphere columns to signed decimal degrees

source

ParseCoordinatesCB

def ParseCoordinatesCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Convert OSPAR degree-minute-second coordinates to decimal LAT and LON, dropping records whose coordinates cannot be interpreted

Verify ParseCoordinatesCB on mock data. The rows cover a plain position, -1 seconds, a negative degree with hemisphere E, a trailing space in the hemisphere, and 441 minutes:

dfs_mock = {'SEAWATER': pd.DataFrame({
    'latd': [54, 60, 70, 55, 54], 'latm': [30., 19., 50., 60., 441.], 'lats': [0., 47., 48., 0., 42.], 'latdir': ['N', 'N', 'N', 'N ', 'N'],
    'longd': [3, 2, -9, 2, 4], 'longm': [15., 39., 8., 59., 21.], 'longs': [0., -1., 3., 59., 34.], 'longdir': ['W', 'E', 'E', 'E', 'W'],
})}
tfm = Transformer(dfs_mock, cbs=[ParseCoordinatesCB()])
tfm()
sw = tfm.dfs['SEAWATER']
test_eq(len(sw), 4)
test_close(sw['LAT'].to_list(), [54.5, 60 + 19/60 + 47/3600, 70 + 50/60 + 48/3600, 56.0])
test_close(sw['LON'].to_list(), [-3.25, 2 + 39/60, -(9 + 8/60 + 3/3600), 2 + 59/60 + 59/3600])

Apply ParseCoordinatesCB and the generic SanitizeLonLatCB to the real data. SanitizeLonLatCB drops records at (0, 0) or outside the valid range:

tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check), ParseCoordinatesCB(), SanitizeLonLatCB()])
dfs_out = tfm()
for grp, df in dfs_out.items():
    n_in = len(dfs[grp].dropna(subset=nan_cols_to_check, how='all'))
    print(f"{grp}: {n_in - len(df)} records dropped; LAT {df['LAT'].min():.2f} to {df['LAT'].max():.2f}, LON {df['LON'].min():.2f} to {df['LON'].max():.2f}")
BIOTA: 0 records dropped; LAT 43.46 to 79.93, LON -39.63 to 49.43
SEAWATER: 1 records dropped; LAT 36.18 to 81.27, LON -58.23 to 43.03

Add station

OSPAR identifies each sampling location in a station id column. AddStationCB copies it, trimmed, into the MARIS STATION column. 110 records have trailing spaces, and a missing station becomes an empty string.

source

AddStationCB

def AddStationCB(
    grps:list=None, # Groups to process; None = all groups in `tfm.dfs`
):

Copy OSPAR station id, trimmed, to MARIS-standard STATION

dfs_mock = {
    'SEAWATER': pd.DataFrame({'station id': ['Sellafield ', None]}),
    'BIOTA':    pd.DataFrame({'station id': ['Utsira Nord']}),
}
tfm = Transformer(dfs_mock, cbs=[AddStationCB()])
tfm()
test_eq(tfm.dfs['SEAWATER']['STATION'].to_list(), ['Sellafield', ''])
test_eq(tfm.dfs['BIOTA']['STATION'].to_list(), ['Utsira Nord'])
tfm = Transformer(dfs, cbs=[RemoveAllNAValuesCB(nan_cols_to_check), AddStationCB()])
for grp, df in tfm().items():
    print(f"{grp}: {df['STATION'].nunique()} stations, {(df['STATION'] == '').sum()} records without station")
BIOTA: 572 stations, 398 records without station
SEAWATER: 1315 stations, 112 records without station

NetCDF encoder

get_cbs returns the callbacks of the pipeline, in order. The cell below and encode both use it, so the pipeline is defined once. It returns new instances on each call, so no state carries over between runs. AddSampleIDCB runs last, after every step that can drop a record, so that SMP_ID numbers the records without gaps.

source

get_cbs

def get_cbs()->list:

Callbacks, in pipeline order, that turn raw OSPAR data into MARIS-standard DataFrames

Exported source 
def get_cbs() -> list:
    "Callbacks, in pipeline order, that turn raw OSPAR data into MARIS-standard DataFrames"
    return [
        RemoveAllNAValuesCB(nan_cols_to_check),
        # Nuclide normalisation and mapping
        LowerStripNameCB(col_src='nuclide', col_dst='NUCLIDE'),
        RemapCB(lut=nuclide_lut, col_remap='NUCLIDE', col_src='NUCLIDE'),
        # Time
        ParseTimeCB(),
        EncodeTimeCB(),
        # Value, uncertainty, unit and detection limit
        SanitizeValueCB(coi_val),
        NormalizeUncCB(),
        RemapUnitCB(),
        RemapDetectionLimitCB(),
        # BIOTA lookups: species, biological group, body part
        LowerStripNameCB(col_src='species', col_dst='SPECIES', grps=['BIOTA']),
        RemapCB(lut=species_lut, col_remap='SPECIES', col_src='SPECIES', grps=['BIOTA']),
        EnhanceSpeciesCB(),
        RemapCB(lut=lut_biogroup, col_remap='BIO_GROUP', col_src='SPECIES', grps=['BIOTA']),
        NormalizeBodyPartCB(),
        RemapCB(lut=body_part_lut, col_remap='BODY_PART', col_src='BODY_PART', grps=['BIOTA']),
        # Depth, coordinates, station
        AddDepthCB(),
        ParseCoordinatesCB(),
        SanitizeLonLatCB(),
        AddStationCB(),
        # Sample identifiers, once all record drops are done
        AddSampleIDCB()
    ]
tfm = Transformer(dfs, cbs=get_cbs())
dfs_out = tfm()

for grp, df in dfs_out.items():
    print(f"{grp}: {len(df)} records")
    print(df[[c for c in df.columns if c.isupper()]].head(3).to_string(), '\n')
BIOTA: 15951 records
   NUCLIDE        TIME     VALUE  UNC  UNIT  DL  SPECIES  BIO_GROUP  BODY_PART        LAT       LON                STATION  SMP_ID SMP_ID_PROVIDER
0       33  1267574400  0.326416  NaN     5   2      377         14          1  51.393333  4.031111  Kloosterzande-Schelde       1        DA 17531
1       33  1276473600  0.442704  NaN     5   2      377         14          1  51.393333  4.031111  Kloosterzande-Schelde       2        DA 17534
2       33  1285545600  0.412989  NaN     5   2      377         14          1  51.393333  4.031111  Kloosterzande-Schelde       3        DA 17537 

SEAWATER: 19182 records
   NUCLIDE        TIME  VALUE  UNC  UNIT  DL  SMP_DEPTH        LAT       LON      STATION  SMP_ID SMP_ID_PROVIDER
0       33  1264550400  200.0  NaN     1   2        3.0  51.375278  3.188056  Belgica-W01       1          WNZ 01
1       33  1264550400  270.0  NaN     1   2        3.0  51.223611  2.859444  Belgica-W02       2          WNZ 02
2       33  1264550400  260.0  NaN     1   2        3.0  51.184444  2.713611  Belgica-W03       3          WNZ 03

Example change logs

tfm.logs
['Remove rows with all NA values in specified columns',
 "Convert 'nuclide' column values to lowercase, strip spaces, and store in 'NUCLIDE' column.",
 "Remap values from 'NUCLIDE' to 'NUCLIDE' for groups: all.",
 'Parse OSPAR `sampling date` (MM/DD/YY HH:MM:SS) into TIME',
 'Encode time as seconds since epoch',
 'Sanitize measurement values by removing blanks and standardizing to use the `VALUE` column',
 'Convert expanded uncertainty (coverage factor k) to standard uncertainty UNC',
 'Set UNIT from the reported unit and convert VALUE and UNC to it (seawater Bq/L to Bq/m³)',
 'Map OSPAR `value type` (`=` / `<`) to MARIS detection-limit codes, treating a missing type with an uncertainty as detected',
 "Convert 'species' column values to lowercase, strip spaces, and store in 'SPECIES' column.",
 "Remap values from 'SPECIES' to 'SPECIES' for groups: BIOTA.",
 "Replace SPECIES 0 (not available) with the broad MARIS entry of the record's biological group",
 "Remap values from 'SPECIES' to 'BIO_GROUP' for groups: BIOTA.",
 'Lowercase and trim `body part` into BODY_PART, resolving plain `whole` to whole animal or whole plant from the biological group',
 "Remap values from 'BODY_PART' to 'BODY_PART' for groups: BIOTA.",
 'Copy OSPAR `sampling depth` (SEAWATER) to MARIS-standard SMP_DEPTH as float',
 'Convert OSPAR degree-minute-second coordinates to decimal LAT and LON, dropping records whose coordinates cannot be interpreted',
 'Drop rows with invalid longitude & latitude values. Convert `,` separator to `.` separator',
 'Copy OSPAR `station id`, trimmed, to MARIS-standard STATION',
 'Assign internal sequential SMP_ID and preserve OSPAR `sample id` as SMP_ID_PROVIDER']

Feed global attributes

source

get_attrs

def get_attrs(
    tfm:marisco.callbacks.Transformer, # Transformer object
    zotero_key:str, # Zotero dataset record key
    kw:list=['oceanography', 'Earth Science > Oceans > Ocean Chemistry> Radionuclides', 'Earth Science > Human Dimensions > Environmental Impacts > Nuclear Radiation Exposure', 'Earth Science > Oceans > Ocean Chemistry > Ocean Tracers', 'Earth Science > Oceans > Ocean Chemistry, Earth Science > Oceans > Sea Ice > Isotopes', 'Earth Science > Oceans > Water Quality > Ocean Contaminants', 'Earth Science > Biological Classification > Animals/Vertebrates > Fish', 'Earth Science > Biosphere > Ecosystems > Marine Ecosystems', 'Earth Science > Biological Classification > Animals/Invertebrates > Mollusks', 'Earth Science > Biological Classification > Animals/Invertebrates > Arthropods > Crustaceans', 'Earth Science > Biological Classification > Plants > Macroalgae (Seaweeds)'], # List of keywords
)->dict: # Global attributes

Retrieve all global attributes

Exported source 
def get_attrs(
    tfm: Transformer, # Transformer object
    zotero_key: str, # Zotero dataset record key
    kw: list = kw # List of keywords
    ) -> dict: # Global attributes
    "Retrieve all global attributes"
    return GlobAttrsFeeder(tfm.dfs, cbs=[
        BboxCB(),
        DepthRangeCB(),
        TimeRangeCB(),
        ZoteroCB(zotero_key),
        KeyValuePairCB('keywords', ', '.join(kw)),
        KeyValuePairCB('publisher_postprocess_logs', ', '.join(tfm.logs))
        ])()
get_attrs(tfm, zotero_key=zotero_key, kw=kw)
{'geospatial_lat_min': '36.181666666666665',
 'geospatial_lat_max': '81.26805555555555',
 'geospatial_lon_min': '-58.23166666666667',
 'geospatial_lon_max': '49.43222222222222',
 'geospatial_bounds': 'POLYGON ((-58.23166666666667 36.181666666666665, 49.43222222222222 36.181666666666665, 49.43222222222222 81.26805555555555, -58.23166666666667 81.26805555555555, -58.23166666666667 36.181666666666665))',
 'geospatial_vertical_max': '1850.0',
 'geospatial_vertical_min': '0.0',
 'time_coverage_start': '1995-01-01T00:00:00',
 'time_coverage_end': '2022-12-31T00:00:00',
 'id': 'LQRA4MMK',
 'title': 'OSPAR Environmental Monitoring of Radioactive Substances',
 'summary': '',
 'creator_name': '[{"creatorType": "author", "firstName": "", "lastName": "OSPAR Comission\'s Radioactive Substances Committee (RSC)"}]',
 'keywords': 'oceanography, Earth Science > Oceans > Ocean Chemistry> Radionuclides, Earth Science > Human Dimensions > Environmental Impacts > Nuclear Radiation Exposure, Earth Science > Oceans > Ocean Chemistry > Ocean Tracers, Earth Science > Oceans > Ocean Chemistry, Earth Science > Oceans > Sea Ice > Isotopes, Earth Science > Oceans > Water Quality > Ocean Contaminants, Earth Science > Biological Classification > Animals/Vertebrates > Fish, Earth Science > Biosphere > Ecosystems > Marine Ecosystems, Earth Science > Biological Classification > Animals/Invertebrates > Mollusks, Earth Science > Biological Classification > Animals/Invertebrates > Arthropods > Crustaceans, Earth Science > Biological Classification > Plants > Macroalgae (Seaweeds)',
 'publisher_postprocess_logs': "Remove rows with all NA values in specified columns, Convert 'nuclide' column values to lowercase, strip spaces, and store in 'NUCLIDE' column., Remap values from 'NUCLIDE' to 'NUCLIDE' for groups: all., Parse OSPAR `sampling date` (MM/DD/YY HH:MM:SS) into TIME, Encode time as seconds since epoch, Sanitize measurement values by removing blanks and standardizing to use the `VALUE` column, Convert expanded uncertainty (coverage factor k) to standard uncertainty UNC, Set UNIT from the reported unit and convert VALUE and UNC to it (seawater Bq/L to Bq/m³), Map OSPAR `value type` (`=` / `<`) to MARIS detection-limit codes, treating a missing type with an uncertainty as detected, Convert 'species' column values to lowercase, strip spaces, and store in 'SPECIES' column., Remap values from 'SPECIES' to 'SPECIES' for groups: BIOTA., Replace SPECIES 0 (not available) with the broad MARIS entry of the record's biological group, Remap values from 'SPECIES' to 'BIO_GROUP' for groups: BIOTA., Lowercase and trim `body part` into BODY_PART, resolving plain `whole` to whole animal or whole plant from the biological group, Remap values from 'BODY_PART' to 'BODY_PART' for groups: BIOTA., Copy OSPAR `sampling depth` (SEAWATER) to MARIS-standard SMP_DEPTH as float, Convert OSPAR degree-minute-second coordinates to decimal LAT and LON, dropping records whose coordinates cannot be interpreted, Drop rows with invalid longitude & latitude values. Convert `,` separator to `.` separator, Copy OSPAR `station id`, trimmed, to MARIS-standard STATION, Assign internal sequential SMP_ID and preserve OSPAR `sample id` as SMP_ID_PROVIDER"}

Encoding

source

encode

def encode(
    dest:str, # Output file name
    src:str=None, # Path to raw OSPAR data; defaults to `src_dir`
    **kwargs
)->None: # Additional arguments

OSPAR radioactive substances monitoring data (seawater and biota)

Exported source 
def encode(
    dest: str, # Output file name
    src: str = None, # Path to raw OSPAR data; defaults to `src_dir`
    **kwargs # Additional arguments
    ) -> None:
    "OSPAR radioactive substances monitoring data (seawater and biota)"
    dfs = load_data(src or src_dir)
    # For a quick test file, keep 10 random records per sample type:
    # dfs = {k: v.sample(10, random_state=42) for k, v in dfs.items()}
    tfm = Transformer(dfs, cbs=get_cbs())
    tfm()
    encoder = NetCDFEncoder(tfm.dfs,
                            dest_fname=dest,
                            global_attrs=get_attrs(tfm, zotero_key=zotero_key, kw=kw),
                            verbose=kwargs.get('verbose', False),
                           )
    encoder.encode()
fname_out = '../../_data/output/191-OSPAR-2024.nc'
encode(dest=fname_out, verbose=False)

NetCDF → CSV (MARIS DB import)

to_csv converts the MARIS NetCDF file into one CSV file per sample type, with the column names the MARIS master database import expects. The marisco-export command does the same from the command line.

to_csv(fname_out)