Stage 4: Write your first handler

Turn a small real dataset into a MARIS NetCDF file with the agent, from the first look at the data to a reviewed pull request

A handler is a notebook in nbs/handlers/ that converts one provider’s dataset to the MARIS format, and documents each curation decision next to the code that makes it. Writing a new handler gives the technical steps. This page gives the order of work, and explains how you share it with the agent and with the maintainer.

The maintainer chooses the dataset with you. A good first dataset is small, covers one sample type and a few nuclides, and is one that MARIS needs. The JOIS handler is the closest model: seawater only, with data downloaded from Zenodo.

Before you start

  1. Get the raw data and the provider’s documentation of its columns, units and codes. Read the documentation yourself before you start the agent.
  2. Open an issue for the handler. Describe the dataset, where the data comes from, and what you already know about its problems.
  3. Create a branch for the work, for example handler/<dataset>.
  4. Open a draft PR as soon as the branch has its first commit. The maintainer can then follow and comment on your work from the start.

Give the agent its context

Start claude in ~/aai-ws/marisco. Before any work, ask the agent to read these notebooks:

  • CRAFT.ipynb, the project overview for agents;
  • CRAFTs/handler-doc-style.ipynb, the documentation style for handlers;
  • nbs/how-to/writing-a-handler.ipynb;
  • nbs/handlers/jois.ipynb, or another handler the maintainer suggests.

Then describe the dataset and give the agent the path to the raw data. Ask it for a plan: which sample types the data covers, how the provider’s columns map to MARIS columns, and which steps the pipeline needs. Discuss the plan with the agent, and with the maintainer if anything is unclear. Do not let the agent write code until you agree with the plan.

Work one step at a time

Build the pipeline one step at a time, in the order of the notebook. Each step deals with one problem in the raw data, such as column names, a unit, a date format or a nomenclature. For each step:

  1. Ask the agent to show you the problem in the raw data, with a few example rows.
  2. Agree on what to do about it.
  3. Ask the agent to write the section: a paragraph that names the problem, the callback that fixes it, and a cell that runs the pipeline so far and checks the result.
  4. Read the section as a data provider would. The prose must say what was done and why.
  5. Run the notebook up to that section, and check the output.
  6. Commit the step.

Never ask the agent to write the whole handler at once. You cannot review a whole handler in one go, and a mistake in an early step spreads into every later one.

nbdev-test skips handler notebooks. They download or read provider data, and nbs/handlers/.notest excludes them. For a handler, nbdev-test prints No files were eligible for testing, which does not mean the tests passed. Run the handler notebook yourself in VS Code with Run All, or ask the agent to run all its cells and report the errors.

Curation decisions are yours

A curation decision changes what the data says: a unit conversion, a mapping from a provider’s name to a MARIS identifier, a detection limit code, or the rows the pipeline drops. The agent proposes these decisions. You make them, because you know the data and the domain. Check each one against the provider’s documentation, and against the curation rules in the how-to.

When you are not sure about a decision, record it in a “FEEDBACK TO MARIS DATA TEAM” callout next to the step, and do not guess. While any such callout is open, the handler’s status is Under review. When the data itself has a problem the provider should fix, such as a missing unit or an inconsistent code, record it in a “FEEDBACK TO DATA PROVIDER” callout. Handler documentation guide shows the callout format.

Check the output

When the pipeline is complete, encode the dataset and look at the result:

marisco-ingest <dataset> ~/output/<dataset>.nc
marisco-export ~/output/<dataset>.nc
  1. Open the CSV files in a spreadsheet. Check the number of rows, and the ranges of the values, dates, coordinates and depths.
  2. Choose a few measurements in the raw data, and find each one in the CSV. Check its value, unit, uncertainty, date and position by hand. This check catches mistakes that no test catches.
  3. Explore a MARIS NetCDF file shows how to read the NetCDF file itself, with its metadata and lookup tables.

Review with the maintainer

Mark the PR ready for review when the handler is complete. The maintainer reviews the notebook online, as a data provider and as a developer. Answer each comment with a change or an explanation. A first handler usually needs several rounds of review. That is the normal way to learn the curation conventions.

Stage 4 is complete when your handler is merged with the status Active or Under review, and you can walk a data provider through each curation decision in it.