Poll

Check data providers for new datasets, and for new versions of the datasets that handlers read

Each RECORDS entry of a handler that reads Zenodo data holds a record id in its zenodo field. A record is one version of a dataset. Zenodo never changes a published record, so the record id fixes the exact files the handler reads. Providers publish corrections and extensions as new records, under the same parent id. This module finds the records that handlers read and that have a newer version, and the datasets in a Zenodo community that no handler reads.

from fastcore.test import *

Records that handlers read

---

source

handler_records

def handler_records()->pandas.DataFrame: # One row per record: `handler`, `key` (the `RECORDS` key) and `record_id`

Zenodo records that the handlers’ RECORDS entries read

handler_records reads the record ids without calling Zenodo. fram_strait and jois read ten records:

recs = handler_records()
test_eq(recs.groupby('handler').size().to_dict(), {'fram_strait': 6, 'jois': 4})
recs

---

source

check_records

def check_records(
    recs:pandas.DataFrame, # Records, as `handler_records` returns them
)->pandas.DataFrame: # `recs` with the dataset's `parent_id`, its `latest_id`, and whether the record is `current`

Compare each record that a handler reads with the latest version of its dataset on Zenodo

check_records makes one request for each record. FS2024_i129 reads record 20761705, but its dataset has a newer version, record 23111574, which adds U-236. Moving fram_strait to the newer record is a curation decision for the handler:

checked = check_records(recs)
test_eq(checked.loc[~checked.current, 'key'].tolist(), ['FS2024_i129'])
checked

New datasets

---

source

new_datasets

def new_datasets(
    checked:pandas.DataFrame, # Records, as `check_records` returns them
    slug:str='titanica', # Zenodo community
)->pandas.DataFrame: # One row per dataset: `record_id` of its latest version, `parent_id`, `created` and `title`

Datasets in community slug that no handler reads

new_datasets compares datasets by parent id, not by record id. The community lists only the latest version of each dataset. The newer Fram Strait 2024 record therefore belongs to a dataset that fram_strait reads, through FS2024_i129, and is not a new dataset. The remaining records are datasets that no handler reads:

new = new_datasets(checked)
test_eq(23111574 in new.record_id.values, False)
new