MARISCO

IAEA | MARIS data NetCDF encoders

The IAEA Marine Radioactivity Information System (MARIS) provides open access to radioactivity measurements in marine environments. Developed by the IAEA Marine Environmental Laboratories in Monaco, MARIS offers data on seawater, biota, sediment, and suspended matter.

This Python package includes command-line tools to convert MARIS datasets into NetCDF or .csv formats, enhancing compatibility with various scientific and data analysis software.

Core Concept: Handlers

marisco is built around the concept of handlers - specialized modules designed to convert MARIS datasets into NetCDF format. Each handler is tailored to a specific data provider and implemented as a dedicated Jupyter notebook.

Literate Programming Approach

We’ve adopted a Literate Programming approach, which means:

Documentation: Each handler serves as comprehensive documentation.
Code Reference: The notebooks contain the actual implementation code.
Communication Tool: They facilitate discussions with data providers about discrepancies or inconsistencies.

Powered by nbdev

To achieve this, we leverage nbdev, a powerful tool that allows us to:

Write code within Jupyter notebooks
Automatically export relevant parts as dedicated Python modules

This approach bridges the gap between documentation and implementation, ensuring they remain in sync.

See It in Action

For a concrete example of this approach, check out our OSPAR dataset handler implementation.

List of currently available handlers

MARISCO includes a suite of specialized data handlers designed to:

Convert provider-specific data formats into standardized MARIS NetCDF files
Ensure data quality and consistency across providers
Facilitate integration with the MARIS marine radioactivity database
Support automated data processing workflows

The following handlers are currently implemented:

Handler	Description	Link to Data Source
MARIS Legacy	All legacy MARIS datasets from the MARIS Master Database	-
HELCOM	HELCOM marine environment protection datasets	HELCOM
OSPAR	OSPAR marine environment datasets	ODIMS OSPAR
TEPCO	TEPCO Fukushima monitoring data	TEPCO Monitoring
GEOTRACES	BODC GEOTRACES oceanographic data	GEOTRACES IDP2021

Install

Now, to install marisco simply run

pip install marisco

Once successfully installed, run the following command:

maris_init

This command:

creates a .marisco/ directory containing various configuration/configurable files ((below)) in your /home directory;
creates a configs.toml file containing default but configurable settings (default paths, …);
downloads several MARIS DB nomenclature/lookup table into .marisco/lut/ directory;
downloads maris-template.nc, the MARIS NetCDF4 template.

Zotero API key

Upon conversion, marisco will automatically retrieve the bibliographic metadata of each MARIS dataset from Zotero. To do so, you need to define the following environment variable ZOTERO_API_KEY containing the MARIS Zotero API key. Please contact the MARIS team to get your API key.

Getting started

Command line utilities

All commands accept a -h argument to get access to its documentation.

`maris_init`

Download configuration file, NetCDF MARIS template and required lookup tables (nomenclatures).

`maris_to_nc`

Convert helcom, geotraces, tepco or ospar marine radioactivity datasets to MARIS NetCDF4 format.

usage: maris_to_nc [-h] [--src SRC] ds dest

positional arguments:
  ds          Name of the dataset to encode as NetCDF4
  dest        Output path for NetCDF file

options:
  -h, --help  show this help message and exit
  --src SRC   Optional input data path only required for the 'GEOTRACES' dataset

For instance: maris_to_nc ospar 191-OSPAR-2024.nc

`maris_db_to_nc`

The MARIS Master Database integrates two types of datasets:

Historical datasets retrieved from published scientific papers
Ongoing monitoring data from international programs like HELCOM, OSPAR, TEPCO, and GEOTRACES

This command-line utility converts MARIS datasets from their legacy format to NetCDF4, making them more accessible for modern data analysis workflows. Users can either convert the entire database or specify particular datasets by their reference IDs for selective conversion.

usage: maris_db_to_nc [-h] [--ref_ids REF_IDS] src dest

Convert MARIS legacy database to NetCDF4 format. If ref_ids is provided as comma-separated values, only encodes those subsets.

positional arguments:
  src                Path to MARIS database dump as `.txt` file
  dest               Output path for NetCDF file(s)

options:
  -h, --help         show this help message and exit
  --ref_ids REF_IDS  Optional comma-separated reference IDs (e.g., "123,456,789") (default: )

For instance:

maris_db_to_nc "~/pro/data/maris/2024-11-20 MARIS_QA_shapetype_id=1.txt" ~/pro/tmp/output
or maris_db_to_nc "~/pro/data/maris/2024-11-20 MARIS_QA_shapetype_id=1.txt" ~/pro/tmp/output --ref_ids="16,30" for a subset of the MARIS Master Database.

`maris_nc_to_csv`

This utility converts NetCDF files to CSV files that conform to the MARIS Standard format, originally designed for OpenRefine workflows.

Although MARISCO has now superseded OpenRefine in the data preparation pipeline, the MARIS master database continues to require CSV inputs in this legacy format. This command-line utility, built with the MARISCO library, handles the conversion process.

usage: maris_nc_to_csv [-h] src dest

Converts NetCDF files into CSV files that follow the MARIS Standard format.

positional arguments:
  src         Input path and filename for NetCDF file
  dest        Output path and filename (without extension) for CSV file

options:
  -h, --help  show this help message and exit

For instance: maris_nc_to_csv ~/pro/tmp/output/191-OSPAR-2024.nc ~/pro/tmp/output/191-OSPAR-2024

Note

When specifying the destination path (e.g., ~/pro/tmp/output/191-OSPAR-2024), the utility automatically appends the MARIS sample type to the filename. For example:

191-OSPAR-2024_BIOTA.csv for biological samples

While this specific example produces only a BIOTA file, the utility can generate multiple files (one per sample type) depending on the content of the source dataset. This reflects the NetCDF4 file structure, where each MARIS sample type is stored as a separate group within the file.

Development

The MARIS NetCDF template is generated from nbs/files/cdl/maris.cdl Common Data Language (CDL) file as defined by Unidata. To generate the MARIS NetCDF template nbs/files/nc/maris-template.nc, install the NetCDF-C utilities, once in Marisco home directory, run:

ncgen -4 -o nc/maris-template.nc cdl/maris.cdl