Callbacks to populate NetCDF global attributes.

The callbacks cover four concerns, in the order of this page:

from marisco.configs import INIS_QA_API
from nbdev.showdoc import show_doc

Global attributes feeder

GlobAttrsFeeder follows the same callback pattern as marisco.callbacks.Transformer: it takes a list of Callback objects, runs them in order, and collects their results into an attrs dict. Each callback below contributes one piece of the NetCDF global metadata.

source

GlobAttrsFeeder

def GlobAttrsFeeder(
    dfs:dict[str, pandas.DataFrame], # Dictionary of NetCDF group DataFrames
    cbs:list[marisco.callbacks.Callback]=None, # Callbacks
    logs:list[str]=None, # List of preprocessing steps taken
)->None:

Produce NetCDF global attributes as specified by the callbacks

source

KeyValuePairCB

def KeyValuePairCB(
    k, # NetCDF global attribute key name
    v, # NetCDF global attribute value
):

Add a single key-value pair as a NetCDF global attribute

KeyValuePairCB is the simplest callback. It sets one global attribute to a fixed value, for attributes that don’t come from the data, such as keywords or publisher_name:

attrs = GlobAttrsFeeder(None, cbs=[KeyValuePairCB('publisher_name', 'IAEA'), KeyValuePairCB('license', 'CC-BY-4.0')])()
test_eq(attrs, {'publisher_name': 'IAEA', 'license': 'CC-BY-4.0'})
attrs

GlobAttrsFeeder accepts only the attribute names listed in NC_GLOBAL_ATTRS (in marisco.configs). A misspelt or unknown name raises KeyError when the handler runs, before anything reaches the NetCDF file:

with expect_fail(KeyError, contains='publisher'): GlobAttrsFeeder(None, cbs=[KeyValuePairCB('publisher', 'IAEA')])()

Spatial and temporal coverage

source

BboxCB

def BboxCB():

Compute dataset geographical bounding box

BboxCB sets the longitude and latitude limits of all groups as strings, and the bounding box as a WKT polygon:

mock_dfs = {'SEAWATER': pd.DataFrame({'LON': [10.0, 20.0], 'LAT': [30.0, 40.0]})}
feed = GlobAttrsFeeder(mock_dfs, cbs=[BboxCB()])
attrs = feed()

test_eq(attrs['geospatial_lat_min'], '30.0')
test_eq(attrs['geospatial_lat_max'], '40.0')
test_eq(attrs['geospatial_lon_min'], '10.0')
test_eq(attrs['geospatial_lon_max'], '20.0')
test_eq(attrs['geospatial_bounds'], 'POLYGON ((10 30, 20 30, 20 40, 10 40, 10 30))')
attrs

source

DepthRangeCB

def DepthRangeCB(
    depth_col:str='SMP_DEPTH', # Column name for sampling depth values
):

Compute minimum and maximum depth values

DepthRangeCB sets the minimum and maximum sampling depth. It sets nothing when no group has the depth column:

mock_dfs = {'SEDIMENT': pd.DataFrame({'SMP_DEPTH': [5.0, 200.0]})}
feed = GlobAttrsFeeder(mock_dfs, cbs=[DepthRangeCB()])
test_eq(feed()['geospatial_vertical_min'], '5.0')
test_eq(feed()['geospatial_vertical_max'], '200.0')
test_eq(GlobAttrsFeeder({'SEAWATER': pd.DataFrame({'LON': [10.0]})}, cbs=[DepthRangeCB()])(), {})

source

TimeRangeCB

def TimeRangeCB(
    time_col:str='TIME', # Column name for time values
    fn_time_unit:Callable=get_time_units, # Function returning the NetCDF time unit string
):

Decode the min/max of the NetCDF-encoded TIME column into ISO date strings for global attributes

TimeRangeCB decodes the earliest and latest TIME values with the unit from get_time_units, which is seconds since 1970-01-01 00:00:00.0. Here 1.0 and 365.0 are 1 s and 365 s after the Unix epoch:

mock_dfs = {'SEAWATER': pd.DataFrame({'TIME': [1.0, 365.0]})}
feed = GlobAttrsFeeder(mock_dfs, cbs=[TimeRangeCB()])
attrs = feed()
test_eq(attrs['time_coverage_start'], '1970-01-01T00:00:01')
test_eq(attrs['time_coverage_end'], '1970-01-01T00:06:05')
attrs

Bibliographic metadata

Every curated dataset in MARIS needs bibliographic metadata (title, abstract, creators, DOI, …) stored as global attributes in the NetCDF file produced by each dataset handler (e.g. the Geotraces handler). MARIS previously managed this metadata through the IAEA’s Zotero group library, but the IAEA’s own INIS bibliographic database is now the target single source of truth; migration is ongoing.

ZoteroCB and InisCB both set id and the attributes listed in BIB_ATTRS. A handler can use either one in its callback pipeline. InisCB also sets references (the DOI) and metadata_link (the record URL). To add a bibliographic attribute, add its name to BIB_ATTRS and add a property with that name to both ZoteroClient and INISClient.

Exported source 
BIB_ATTRS = ['title', 'summary', 'creator_name']

Zotero

Bibliographic metadata for each dataset is managed in the IAEA’s Zotero group library. fetch_zotero fetches one item. ZoteroClient reads its bibliographic fields, and ZoteroCB writes them as global attributes.

source

fetch_zotero

def fetch_zotero(
    item_id:str, # Zotero item key to retrieve
    lib_id:str, # Zotero library ID
    api_key:str, # Zotero API key
)->dict | None: # Zotero item, or None if the item does not exist

Fetch an item from a Zotero group library

source

ZoteroClient

def ZoteroClient(
    item_id:str, # Zotero item key to retrieve
    lib_id:str, # Zotero library ID
    api_key:str, # Zotero API key
)->None:

Zotero API client to fetch a bibliographic record

ZoteroClient has these read-only properties, taken from the fetched Zotero record: title, summary and creator_name (a JSON-encoded list). Each property returns a str.

When the item does not exist, exist() returns False. Every other failure raises an error, for example an invalid ZOTERO_API_KEY or a network failure.

item = ZoteroClient('26VMZZ2Q', ZOTERO_LIB_ID, os.getenv('ZOTERO_API_KEY'))
test_eq(item.title, 'Environmental database - Helsinki Commission Monitoring of Radioactive Substances')
test_eq(item.summary[:30], 'MORS Environment database has ')
creators = json.loads(item.creator_name)
test_eq(len(creators), 1)
test_eq(creators[0]['creatorType'], 'author')

source

ZoteroCB

def ZoteroCB(
    item_id:str, # Zotero item key to retrieve
):

Populate global attributes from Zotero bibliographic metadata

ZoteroCB sets id to the Zotero item key, and the BIB_ATTRS attributes from the item:

attrs = GlobAttrsFeeder(None, cbs=[ZoteroCB('26VMZZ2Q')])()
test_eq(attrs['id'], '26VMZZ2Q')
test_eq(attrs['title'], 'Environmental database - Helsinki Commission Monitoring of Radioactive Substances')

When the item does not exist, ZoteroCB sets no attributes:

attrs = GlobAttrsFeeder(None, cbs=[ZoteroCB('x')])()
test_eq(attrs, {})

INIS

Bibliographic metadata can also be fetched from the IAEA’s InvenioRDM-based INIS API. fetch_inis fetches one record. INISClient reads its bibliographic fields, and InisCB writes them as global attributes, together with the record’s DOI and web URL.

INIS support is in development. The production INIS API currently blocks scripted access. The INIS examples below will run again once IAEA restores access.

source

fetch_inis

def fetch_inis(
    inis_id:str, # INIS record identifier (e.g. 'vq0ha-86k24')
    base_url:str='https://inis.iaea.org/api', # API root
)->dict | None: # Raw INIS record payload, or None if the record does not exist

Fetch an INIS record

fetch_inis returns the record as InvenioRDM JSON. INISClient below reads its metadata, pids and links fields:

rec = fetch_inis('5smfm-0a377')
test_eq(rec['id'], '5smfm-0a377')
list(rec)

source

INISClient

def INISClient(
    inis_id:str, # INIS record identifier (e.g. 'vq0ha-86k24')
    base_url:str='https://inis.iaea.org/api', # API base URL
)->None:

Retrieve INIS metadata from the InvenioRDM API

INISClient has these read-only properties, taken from the fetched INIS record: title, summary and creator_name (a JSON-encoded list). Each property returns a str. The doi and url properties are described below.

When INIS returns HTTP 404, exist() returns False and every property returns ''. Every other failure raises an error. For example, fetch_inis raises requests.HTTPError, a subclass of OSError, when INIS answers HTTP 403 to block the request.

inis = INISClient('5smfm-0a377')
test_eq(inis.title, 'The GEOTRACES Intermediate Data Product 2017')
test_eq(inis.exist(), True)
test_eq(inis.summary[:30], 'The GEOTRACES Intermediate Dat')
creators = json.loads(inis.creator_name)
test_eq(len(creators), 286)
test_eq(creators[0]['person_or_org']['family_name'], 'Schlitzer')

A record that does not exist on INIS gives empty properties:

nonexistent = INISClient('this-does-not-exist')
test_eq(nonexistent.exist(), False)
test_eq(nonexistent.title, '')

source

INISClient.doi

def doi()->str:

DOI of the record, or ’’ if it has none

doi reads the record’s pids.doi. When that is empty, it uses the first doi entry in metadata.identifiers. A record that does not exist gives '':

test_eq(inis.doi, '10.1016/j.chemgeo.2018.05.040')
test_eq(nonexistent.doi, '')

The QA record g7wwp-fcc77 has a DOI in neither place:

no_doi = INISClient('g7wwp-fcc77', base_url=INIS_QA_API)
test_eq(no_doi.doi, '')
test_eq(no_doi.exist(), True)

source

INISClient.url

def url()->str:

Web page of the record, or ’’ if it has none

url uses the first of the record’s links self_html, latest_html and self that is present:

test_eq(inis.url, 'https://inis.iaea.org/records/5smfm-0a377')

source

InisCB

def InisCB(
    inis_id:str, # INIS record identifier to retrieve
    base_url:str='https://inis.iaea.org/api', # API base URL
):

Populate global attributes from INIS metadata

InisCB sets id to the INIS record identifier, and the BIB_ATTRS attributes from the record. It also sets references to the DOI and metadata_link to the URL, when the record has them:

attrs = GlobAttrsFeeder(None, cbs=[InisCB('5smfm-0a377')])()
test_eq(set(attrs), {'id', 'title', 'summary', 'creator_name', 'references', 'metadata_link'})
test_eq(attrs['id'], '5smfm-0a377')
test_eq(attrs['references'], '10.1016/j.chemgeo.2018.05.040')

When the record does not exist, InisCB sets no attributes:

test_eq(GlobAttrsFeeder(None, cbs=[InisCB('this-does-not-exist')])(), {})

Usage

dfs = pd.read_pickle('../files/pkl/dfs_test.pkl')
kw = ['oceanography', 'Earth Science > Oceans > Ocean Chemistry> Radionuclides',
    'Earth Science > Human Dimensions > Environmental Impacts > Nuclear Radiation Exposure',
    'Earth Science > Oceans > Ocean Chemistry > Ocean Tracers, Earth Science > Oceans > Marine Sediments',
    'Earth Science > Oceans > Ocean Chemistry, Earth Science > Oceans > Sea Ice > Isotopes',
    'Earth Science > Oceans > Water Quality > Ocean Contaminants',
    'Earth Science > Biological Classification > Animals/Vertebrates > Fish',
    'Earth Science > Biosphere > Ecosystems > Marine Ecosystems',
    'Earth Science > Biological Classification > Animals/Invertebrates > Mollusks',
    'Earth Science > Biological Classification > Animals/Invertebrates > Arthropods > Crustaceans',
    'Earth Science > Biological Classification > Plants > Macroalgae (Seaweeds)']
feed = GlobAttrsFeeder(dfs, cbs=[BboxCB(), DepthRangeCB(), TimeRangeCB(), ZoteroCB('26VMZZ2Q'), KeyValuePairCB('keywords', ', '.join(kw))])
attrs = feed()
attrs
{'geospatial_lat_min': '179.9986',
 'geospatial_lat_max': '89.9905',
 'geospatial_lon_min': '-180.0',
 'geospatial_lon_max': '-70.5744',
 'geospatial_bounds': 'POLYGON ((-180 -70.5744, 179.9986 -70.5744, 179.9986 89.9905, -180 89.9905, -180 -70.5744))',
 'geospatial_vertical_max': '5815.3',
 'geospatial_vertical_min': '0.5',
 'time_coverage_start': '2007-07-30T10:37:19',
 'time_coverage_end': '2018-11-22T07:33:10',
 'id': '26VMZZ2Q',
 'title': 'Environmental database - Helsinki Commission Monitoring of Radioactive Substances',
 'summary': 'MORS Environment database has been used to collate data resulting from monitoring of environmental radioactivity in the Baltic Sea based on HELCOM Recommendation 26/3.\n\nThe database is structured according to HELCOM Guidelines on Monitoring of Radioactive Substances (https://www.helcom.fi/wp-content/uploads/2019/08/Guidelines-for-Monitoring-of-Radioactive-Substances.pdf), which specifies reporting format, database structure, data types and obligatory parameters used for reporting data under Recommendation 26/3.\n\nThe database is updated and quality assured annually by HELCOM MORS EG.',
 'creator_name': '[{"creatorType": "author", "name": "HELCOM MORS"}]',
 'keywords': 'oceanography, Earth Science > Oceans > Ocean Chemistry> Radionuclides, Earth Science > Human Dimensions > Environmental Impacts > Nuclear Radiation Exposure, Earth Science > Oceans > Ocean Chemistry > Ocean Tracers, Earth Science > Oceans > Marine Sediments, Earth Science > Oceans > Ocean Chemistry, Earth Science > Oceans > Sea Ice > Isotopes, Earth Science > Oceans > Water Quality > Ocean Contaminants, Earth Science > Biological Classification > Animals/Vertebrates > Fish, Earth Science > Biosphere > Ecosystems > Marine Ecosystems, Earth Science > Biological Classification > Animals/Invertebrates > Mollusks, Earth Science > Biological Classification > Animals/Invertebrates > Arthropods > Crustaceans, Earth Science > Biological Classification > Plants > Macroalgae (Seaweeds)'}