Callback inventory

Where each pipeline callback is defined, and which names are defined in more than one place

A handler builds its pipeline from shared callbacks in marisco.callbacks and marisco.metadata, and from callbacks it defines itself. Each handler notebook contains every callback it doesn’t import from those two modules, so it can be read on its own. Two handlers can therefore define the same callback. The rules are in CRAFTs/handler-doc-style.ipynb.

This page lists every callback and the module that defines it, and it shows which names are defined more than once. The tables are generated from the installed modules: run the notebook again to update them.

import importlib, inspect, pkgutil, pandas as pd
import marisco.callbacks, marisco.metadata, marisco.handlers
from marisco.callbacks import Callback
from fastcore.test import test_eq

Modules scanned

The scan covers the two shared modules and every module in marisco.handlers. A handler added later is included without a change to this page.

mods = {m.__name__.removeprefix('marisco.'): m for m in [marisco.callbacks, marisco.metadata] + [
    importlib.import_module(f'marisco.handlers.{i.name}') for i in pkgutil.iter_modules(marisco.handlers.__path__)]}
list(mods)

['callbacks',
 'metadata',
 'handlers.fram_strait',
 'handlers.geotraces',
 'handlers.helcom',
 'handlers.jois',
 'handlers.maris_legacy',
 'handlers.ospar',
 'handlers.tepco']

A callback belongs to the module that defines it. Names that a module only imports are skipped, so a handler that imports a shared callback doesn’t list it again:

def module_cbs(m):
    "Callback classes defined in module `m`, by name"
    return {k:v for k,v in vars(m).items()
        if inspect.isclass(v) and issubclass(v, Callback) and v is not Callback and v.__module__==m.__name__}

cbs = pd.DataFrame([
    dict(module=n, callback=k, base=v.__bases__[0].__name__, doc=inspect.cleandoc(v.__doc__ or '').split('\n')[0], cls=v)
    for n,m in mods.items() for k,v in module_cbs(m).items()])

Callbacks per module

handlers.tepco exports no callbacks yet: its notebook defines them in cells that aren’t exported.

cbs.module.value_counts(sort=False).reindex(list(mods), fill_value=0)

module
callbacks                13
metadata                  6
handlers.fram_strait      7
handlers.geotraces       11
handlers.helcom          17
handlers.jois             7
handlers.maris_legacy     3
handlers.ospar           11
handlers.tepco            0
Name: count, dtype: int64

All callbacks

Each row gives the base class and the first line of the docstring. The docstring is also the text a callback writes to the NetCDF processing log.

cbs.drop(columns='cls').set_index(['module', 'callback'])

		base	doc
module	callback
callbacks	PerGroupCB	Callback	Calls `each_grp` for each group in `tfm.dfs`; set `grps` to restrict to specific groups
	SanitizeLonLatCB	PerGroupCB	Drop rows with invalid longitude & latitude values. Convert `,` separator to `.` separator
	RemoveAllNAValuesCB	Callback	Remove rows with all NA values in specified columns
	RemapCB	PerGroupCB	Remap source values to MARIS standard identifiers using a lookup table
	LowerStripNameCB	PerGroupCB	Convert values to lowercase and strip any trailing spaces
	AddSampleTypeIdColumnCB	PerGroupCB	Add a column with the sample type as defined in the CDL
	RenameColumnsCB	PerGroupCB	Rename variables to MARIS standard names, keeping only renamed columns
	MeltWideNuclidesCB	Callback	Reshape wide nuclide columns to long format using a named-dict spec
	AddSampleIDCB	PerGroupCB	Assign 1-based sequential SMP_ID; optionally cast a provider ID column to str for NetCDF VLEN compatibility
	UniqueIndexCB	PerGroupCB	Set unique index for each group
	ParseTimeCB	PerGroupCB	Parse time column from ISO8601 string to datetime
	EncodeTimeCB	PerGroupCB	Encode time as seconds since epoch
	DecodeTimeCB	PerGroupCB	Decode time from seconds since epoch to datetime format
metadata	KeyValuePairCB	Callback	Add a single key-value pair as a NetCDF global attribute
	BboxCB	Callback	Compute dataset geographical bounding box
	DepthRangeCB	Callback	Compute minimum and maximum depth values
	TimeRangeCB	Callback	Decode the min/max of the NetCDF-encoded TIME column into ISO date strings for global attributes
	ZoteroCB	Callback	Populate global attributes from Zotero bibliographic metadata
	InisCB	Callback	Populate global attributes from INIS metadata
handlers.fram_strait	RenameColsCB	PerGroupCB	Map Fram Strait provider columns to MARIS standard names
	ParseDateTimeCB	PerGroupCB	Parse FS collection date into a UTC TIME value
	AddDepthCB	PerGroupCB	Compute sampling depth using Thermodynamic Equation of SeaWater 2010 (TEOS-10)
	MeltFramStraitCB	PerGroupCB	Reshape wide Fram Strait value and uncertainty columns into one row per measurement
	ConvertU238CB	PerGroupCB	Convert U-238 VALUE from ppb to atoms/kg
	AddDetectionLimitCB	PerGroupCB	Assign missing Detection Limit column to MARIS 'Detected value: 1' category
	FormatStationCB	PerGroupCB	Cast STATION to str for the NetCDF4 string-typed station variable
handlers.geotraces	SelectColsOfInterestCB	Callback	Select columns of interest from the wide Geotraces dataframe
	WideToLongCB	Callback	Reshape wide nuclide columns to long format so unit, method, and filter status can be extracted from column names
	ExtractUnitCB	Callback	Extract measurement unit from nuclide column names (e.g. 'Cs_137_D_CONC_BOTTLE [uBq/kg]' → 'uBq/kg')
	ExtractFilteringStatusCB	Callback	Extract filtering status and sample-type group from nuclide column names using phase code (e.g. _D_, _T_, _TP_)
	ExtractSamplingMethodCB	Callback	Extract sampling method from nuclide names
	RenameNuclideCB	Callback	Remap nuclides name to MARIS standard
	StandardizeUnitCB	Callback	Remap Geotraces unit strings to MARIS unit IDs, rescaling measurement values by the appropriate conversion factor where units share a common MARIS unit ID (e.g. uBq/kg and mBq/kg both map to ID 3 but differ 1000x)
	RenameColumnCB	Callback	Remap Geotraces-specific coordinate, depth, and sample-ID column names to MARIS standard nomenclature
	UnshiftLongitudeCB	Callback	Shift longitudes from Geotraces [0, 360] convention to MARIS [-180, 180] by subtracting 180
	DispatchToGroupCB	Callback	Split flat dataframe into per-group dict keyed by sample type (SEAWATER, SUSPENDED_MATTER, …)
	AddGeotracesSampleIDCB	PerGroupCB	Assign a sequential SMP_ID per sample-type group; cast SMP_ID_PROVIDER (BODC Bottle Number) to string for NetCDF VLEN compatibility
handlers.helcom	ParseHelcomTimeCB	PerGroupCB	Parse HELCOM DATE (MM/DD/YY HH:MM:SS) with fallback to YEAR/MONTH/DAY
	MeltSedimentValuesCB	PerGroupCB	Melt HELCOM dual-value sediment rows into separate rows per measurement type (Bq/kg, Bq/m²)
	SanitizeValueCB	PerGroupCB	Sanitize measurement values by removing blanks and standardizing to use the `VALUE` column
	NormalizeUncCB	PerGroupCB	Convert relative uncertainty (percent) to absolute (standard) uncertainty per group
	RemapUnitCB	PerGroupCB	Set the MARIS-standard UNIT column from per-sample-type conventions (column name, basis column, or melt result)
	RemapDetectionLimitCB	PerGroupCB	Map HELCOM `<` / detected-value conventions to MARIS detection-limit integer codes (2 for DL, 1 for detected)
	CleanSedimentCodesCB	PerGroupCB	Replace invalid HELCOM SEDI codes with -99 sentinel before nomenclature lookup
	AddHelcomSampleIDCB	PerGroupCB	Assign internal sequential SMP_ID and preserve provider KEY as SMP_ID_PROVIDER
	AddDepthCB	PerGroupCB	Rename HELCOM sdepth/tdepth columns to MARIS-standard SMP_DEPTH/TOT_DEPTH and cast as float
	AddSalinityCB	PerGroupCB	Add salinity (SAL) from HELCOM salin column where present
	AddStationCB	PerGroupCB	Add station to all DataFrames
	AddTemperatureCB	PerGroupCB	Add temperature (TEMP) from HELCOM ttemp column
	RemapSedSliceTopBottomCB	PerGroupCB	Remap Sediment slice top and bottom to MARIS format
	CleanBasisCB	PerGroupCB	Map basis F to W (BIOTA)
	PercentWeightCB	PerGroupCB	Compute PERCENTWT = dw% / 100 (SEDIMENT)
	WeightCB	PerGroupCB	Compute DRYWT / WETWT from weight + basis (BIOTA)
	ParseCoordinatesCB	PerGroupCB	Parse lat/lon from decimal-degree or degree-minute columns, preferring decimal
handlers.jois	RenameNucColsCB	PerGroupCB	Rename U238_ppb and U236_U238 to the {Nuc}_at_{Unit} convention used by other datasets
	RenameColsCB	PerGroupCB	Map JOIS provider CTD and sample columns to MARIS standard names
	ParseDateTimeCB	PerGroupCB	Combine JOIS Date and Time columns into a single TIME column
	MeltJOISCB	PerGroupCB	Reshape JOIS wide nuclide columns to long format with NUCLIDE, UNIT, VALUE, UNC columns
	ConvertU238CB	PerGroupCB	Convert U-238 VALUE from ppb to atoms/kg
	AddLabCB	PerGroupCB	Assing new `LAB` column to ETH's MARIS laboratory id: 345
	AddDetectionLimitCB	PerGroupCB	Assign missing Detection Limit column to MARIS 'Detected value: 1' category
handlers.maris_legacy	CastStationToStringCB	PerGroupCB	Convert STATION column to string type, filling any missing values with empty string
	DropNAColumnsCB	PerGroupCB	Drop columns containing only NaN or 'Not available' (id=0 in MARIS lookup tables), except those in `keep`
	CastSampleIDCB	PerGroupCB	Cast SMP_ID to int and SMP_ID_PROVIDER to string (renamed from samplabcode in the pipeline)
handlers.ospar	ParseOsparTimeCB	PerGroupCB	Parse OSPAR `sampling date` (MM/DD/YY HH:MM:SS) into TIME
	SanitizeValueCB	PerGroupCB	Sanitize measurement values by removing blanks and standardizing to use the `VALUE` column
	NormalizeUncCB	PerGroupCB	Convert expanded uncertainty (coverage factor k) to standard uncertainty UNC
	RemapUnitCB	PerGroupCB	Set UNIT from the reported unit and convert VALUE and UNC to it (seawater Bq/L to Bq/m³)
	RemapDetectionLimitCB	PerGroupCB	Map OSPAR `value type` (`=` / `<`) to MARIS detection-limit codes, treating a missing type with an uncertainty as detected
	EnhanceSpeciesCB	PerGroupCB	Replace SPECIES 0 (not available) with the broad MARIS entry of the record's biological group
	NormalizeBodyPartCB	PerGroupCB	Lowercase and trim `body part` into BODY_PART, resolving plain `whole` to whole animal or whole plant from the biological group
	AddOsparSampleIDCB	PerGroupCB	Assign internal sequential SMP_ID and preserve OSPAR `sample id` as SMP_ID_PROVIDER
	AddDepthCB	PerGroupCB	Copy OSPAR `sampling depth` (SEAWATER) to MARIS-standard SMP_DEPTH as float
	ParseCoordinatesCB	PerGroupCB	Convert OSPAR degree-minute-second coordinates to decimal LAT and LON, dropping records whose coordinates cannot be interpreted
	AddStationCB	PerGroupCB	Copy OSPAR `station id`, trimmed, to MARIS-standard STATION

Names defined more than once

Handlers can repeat each other’s names. A handler callback must not reuse a name from marisco.callbacks or marisco.metadata, because a reader would take it for the shared callback:

shared = cbs.module.isin(['callbacks', 'metadata'])
test_eq(set(cbs[~shared].callback) & set(cbs[shared].callback), set())

The table lists every name defined in more than one module. implementations counts the different source texts among the definitions of a name, so a count of 1 means every definition is an exact copy:

dups = cbs[cbs.callback.duplicated(keep=False)].groupby('callback').agg(
    modules=('module', ', '.join), implementations=('cls', lambda s: s.map(inspect.getsource).nunique()))
dups

	modules	implementations
callback
AddDepthCB	handlers.fram_strait, handlers.helcom, handlers.ospar	3
AddDetectionLimitCB	handlers.fram_strait, handlers.jois	1
AddStationCB	handlers.helcom, handlers.ospar	2
ConvertU238CB	handlers.fram_strait, handlers.jois	1
NormalizeUncCB	handlers.helcom, handlers.ospar	2
ParseCoordinatesCB	handlers.helcom, handlers.ospar	2
ParseDateTimeCB	handlers.fram_strait, handlers.jois	2
RemapDetectionLimitCB	handlers.helcom, handlers.ospar	2
RemapUnitCB	handlers.helcom, handlers.ospar	2
RenameColsCB	handlers.fram_strait, handlers.jois	2
SanitizeValueCB	handlers.helcom, handlers.ospar	1