Pipeline

Take IOM evaluation reports from PDF to SRF and GCM mappings, saving the state at each stage

A Report takes an evaluation through four stages:

  1. Download. download fetches the evaluation’s documents from the IOM repository.
  2. OCR. ocr converts the report PDF to Markdown pages, with fixed heading levels.
  3. Curation. A person selects the report’s core sections in the curator app (06_curator.ipynb).
  4. Mapping. The map_* methods score the core sections against the themes of the Strategic Results Framework (SRF) and the Global Compact for Migration (GCM).

run_pipeline runs the stages for one evaluation, and batch_run for many.

from cachy import enable_cachy
enable_cachy()

Report class

A Report wraps an Evaluation, with the state of each stage: pdf_path, md_path, curation_status, selected_headings and mappings. Creating a Report loads the saved state from results_path, when there is one.

source

Report

def Report(
    ev:iomeval.readers.Evaluation, # The evaluation
    pdf_url:str=None, # URL of the report PDF
    results_path:str='data/results', # Folder of the saved report JSON files
):

An evaluation report and its pipeline state

Exported source 
class Report:
    "An evaluation report and its pipeline state"
    def __init__(
        self,
        ev:Evaluation,                   # The evaluation
        pdf_url:str=None,                # URL of the report PDF
        results_path:str='data/results'  # Folder of the saved report JSON files
        ):
        store_attr()
        self.id = ev.id
        self.pdf_path,self.md_path = None,None
        self.mappings,self.selected_headings = {},[]
        self.curation_status = 'pending'
        self._load_existing()

source

Report.from_url

def from_url(
    url:str, # URL of the report PDF, one of the evaluation's documents
    evals:list, # Evaluations to search
    results_path:str='data/results', # Folder of the saved report JSON files
)->__main__.Report: # Report with `pdf_url` set to `url`

Create a report for the evaluation that has a document at url

source

Report.from_title

def from_title(
    title:str, # Evaluation title
    evals:list, # Evaluations to search
    results_path:str='data/results', # Folder of the saved report JSON files
)->__main__.Report: # Report with `pdf_url` from `eval_url`

Create a report for the evaluation titled title

source

Report.from_id

def from_id(
    id:str, # Evaluation ID
    evals:list, # Evaluations to search
    results_path:str='data/results', # Folder of the saved report JSON files
)->__main__.Report: # Report with `pdf_url` from `eval_url`

Create a report for the evaluation with ID id

Creating reports

Create a report from the URL of its PDF:

evals = load_evals('files/test/evaluations.json')

url = "https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/AAP%20Evaluation%20Report_final_.pdf"
report = Report.from_url(url, evals, results_path='files/test/results')
report

Report: Evaluation of IOM Accountability to Affected Populations

Year: 2025 | Organization: IOM
ID: 6c3c2cf3fa479112967612b0baddab72

Pipeline: ✓ Curated (1 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

Or from a title:

title = 'Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa'
report = Report.from_title(title, evals, results_path='files/test/results')
report

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

Or from its ID:

id = '49d2fba781b6a7c0d94577479636ee6f'
report = Report.from_id(id, evals, results_path='files/test/results')
report

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

Note

from_title and from_id find the report among the evaluation’s documents with eval_url. An evaluation’s documents can also include a brief, annexes and a management response. They raise ValueError when eval_url finds no report.

Persistence

save writes a report’s state to results_path/<id>.json. The file holds the evaluation, the report URL, the curation status, the selected headings, the mappings and a timestamp. ocr, each map_* method and run_pipeline save the report after they run. download does not. load_report reads the state back from a base folder, which holds the pdf/, md/ and results/ folders. It sets pdf_path and md_path to the report’s folders under pdf/ and md/, when they exist.

source

Report.save

def save(
    path:str=None, # Folder to write to; `None` uses `results_path`
)->__main__.Report: # The report, for chaining

Save the report’s state to <id>.json

source

load_report

def load_report(
    id:str, # Evaluation ID
    base_path:str='data', # Folder holding `pdf/`, `md/` and `results/`
)->__main__.Report: # The report, with its saved state

Load the report id saved under base_path

Resuming from a saved state

report = load_report('49d2fba781b6a7c0d94577479636ee6f', base_path='files/test')
report

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ OCR | ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

Pipeline methods

Run the methods in stage order. download comes first, then ocr, then the map_* methods once the report is curated.

Download

download saves every document of the evaluation in dst/<id>/ with download_eval, and sets pdf_path to that folder. It does nothing when pdf_path is already set, unless force is true.

source

Report.download

def download(
    dst:str='data/pdf', # Folder to create the report's folder in
    force:bool=False, # Download again when `pdf_path` is set
)->__main__.Report: # The report, for chaining

Download the evaluation’s documents to dst/<id>/

url = "https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/Abridged%20Evaluation%20Report_%20Final_Olta%20NDOJA.pdf"
report = Report.from_url(url, evals, results_path='files/test/results')
_ = report.download(dst='files/test/pdf')
Path('files/test/pdf/49d2fba781b6a7c0d94577479636ee6f').ls()
[Path('files/test/pdf/49d2fba781b6a7c0d94577479636ee6f/Abridged%20Evaluation%20Report_%20Final_Olta%20NDOJA.pdf'), Path('files/test/pdf/49d2fba781b6a7c0d94577479636ee6f/Final%20Evaluation%20Report%20Final_Olta%20NDOJA.pdf'), Path('files/test/pdf/49d2fba781b6a7c0d94577479636ee6f/HoA%20EU%20JI%20Final%20Eval%20-%20Management%20Response%20Matrix%20-%20Final.pdf'), Path('files/test/pdf/49d2fba781b6a7c0d94577479636ee6f/ISP_IOM_Case-Management-Return-Reintegr-JI-Review_final.pdf'), Path('files/test/pdf/49d2fba781b6a7c0d94577479636ee6f/Evaluation%20Learning%20Brief_Final_Olta%20NDOJA.pdf')]

OCR

ocr converts the report PDF to Markdown pages in dst/<id>/ with mistocr’s pdf_to_md. mistocr OCRs the PDF with Mistral and fixes the heading levels with an LLM. With add_img_desc, it also adds a description of each image. The PDF is the file named after pdf_url in pdf_path, or the first PDF there when pdf_url is not set. ocr does nothing when dst/<id>/ exists, unless force is true.

source

Report.ocr

async def ocr(
    dst:str='data/md', # Folder to create the report's Markdown folder in
    add_img_desc:bool=True, # Add a description of each image, written by an LLM
    force:bool=False, # OCR again when `dst/<id>/` exists
    progress:bool=False, # Log mistocr's progress
    fix_kwargs:dict=None, # Passed to mistocr's `fix_hdgs`
    desc_kwargs:dict=None, # Passed to mistocr's `add_img_descs`
)->__main__.Report: # The report, for chaining

OCR the report PDF into Markdown pages in dst/<id>/, and save the report

await report.ocr(dst='files/test/md', add_img_desc=False, force=False)

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ OCR | ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

report.md_path
Path('files/test/md/49d2fba781b6a7c0d94577479636ee6f')
report.md_path.ls()[:2]
[Path('files/test/md/49d2fba781b6a7c0d94577479636ee6f/page_21.md'), Path('files/test/md/49d2fba781b6a7c0d94577479636ee6f/page_15.md')]

Curate

A person reviews the OCR’d headings and selects the core sections in the curator app (06_curator.ipynb). The app then sets curation_status to 'sections_selected' and saves the headings in selected_headings.

source

Report.get_sections

def get_sections()->str: # Report title and the text of each selected section

Extract the sections under selected_headings from the OCR’d pages

Extracting sections

get_sections extracts the selected sections of a curated report. It raises ValueError when the report is not OCR’d, is not curated, or has no selected headings. For the curated test report:

report = load_report('49d2fba781b6a7c0d94577479636ee6f', base_path='files/test')
report.selected_headings
['## 1. Introduction ... page 4',
 '## 2. Background of the JI-HoA ... page 5',
 '## 3. Methodology ... page 8',
 '### 4.3. Effectiveness ... page 16',
 '## 5. Conclusions and Recommendations ... page 27']
print(report.get_sections()[:500])
# Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa ... page 1
## 1. Introduction ... page 4

In 2016, the EU and IOM launched the EU-IOM Joint Initiative for Migrant Protection and Reintegration, with as overall objective “To contribute to facilitating orderly, safe, regular and rights-based migration through the facilitation of dignified voluntary return and the implementation of development-focused and sustainable reintegration poli

Thematic mapping

Each map_* method maps the selected sections against one theme set, and saves the report. You can run each one on its own after curation. A method skips a theme set that is already in mappings, unless force is true. ensure_sp builds the report system prompt from the selected sections once, and the later mappings reuse it.

source

Report.ensure_sp

def ensure_sp()->None:

Build the report system prompt from the selected sections, unless the report already has one

source

map_single

async def map_single(
    sp, # Report system prompt from `mk_system_prompt`
    theme_type, # `'enbs'`, `'ccps'`, `'gcms'` or `'outs'`
    path:NoneType=None, # Folder of theme files; `None` uses the files shipped with the package
    model:str='claude-haiku-4-5', # LLM to use
    gcm_ids:NoneType=None, # GCM objective IDs whose linked outputs to map, for `'outs'`
    *,
    response_format:pydantic._internal._model_construction.ModelMetaclass=ThemeScores, # Pydantic model the reply must follow
    temperature:float=0, # Sampling temperature
    reasoning_effort:str=None, # `'low'`, `'medium'` or `'high'`; `None` for no extended thinking
)->list[dict]: # Score dicts from `parse_res`

Map the report in sp against one theme set, with its prompt and formatting

Map enablers

The SRF enablers are IOM’s organizational capabilities.

source

Report.map_enbs

async def map_enbs(
    force:bool=False, # Map again when the theme set is in `mappings`
    **kwargs
)->__main__.Report: # The report, for chaining

Map the selected sections against the SRF enablers, and save the report

Load the curated test report and map it against the enablers:

report = load_report('49d2fba781b6a7c0d94577479636ee6f', base_path='files/test')

await report.map_enbs(model='claude-haiku-4-5', force=False)

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ OCR | ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

sort_by_relevance(report.mappings['enbs'])[:2]
[{'theme_id': '4',
  'theme_title': 'Data and evidence',
  'relevance_score': 0.87,
  'reasoning': "This report would likely be essential for a synthesis on Enabler 4 (Data and Evidence). The evaluation explicitly examines IOM's data and evidence systems as a primary focus, with data availability and use appearing prominently in the program objectives, evaluation questions, and findings. Specific Objective 1 directly addresses 'Partner countries and relevant stakeholders developed or strengthened evidence-based return and reintegration procedures,' with substantial analysis of IOM's data production, data management systems, and evidence use. Key relevant sections include: the detailed findings on 'Data availability' (4.3.1.1) documenting that 'the JI exceeded the targets set for the number of field studies, surveys and other research conducted' and the 'increased availability of migration data...achieved mainly through the production and publication of migration data and research outputs by the Regional Data Hub'; the discussion of M&E systems (4.3.3.1) noting that 'a total of 36 planning, monitoring, learning, data collection and analysis tools were set up, implemented and/or strengthened'; and the analysis of evidence use in policymaking, with findings that '136 stakeholders reported that data produced has supported evidence-based policies, procedures, and programme design.' The report also discusses the development of the MiMOSA database, the Reintegration Sustainability Index (RSI) and RSS+ methodology, and capacity building on data collection and management. Recommendations 4 and 7 specifically address continuing support for the Regional Data Hub and building on impact evaluation methodologies. For a synthesis specialist, the detailed analysis of IOM's data systems, data production capacity, and evidence-based decision-making would be particularly valuable, especially sections 4.3.1 on data availability and capacity building, and the discussion of M&E systems implementation."},
 {'theme_id': '2',
  'theme_title': 'Partnership',
  'relevance_score': 0.79,
  'reasoning': "This report would likely contribute meaningful evidence to a synthesis on Enabler 2 (Partnership). The evaluation explicitly examines IOM's partnership model and effectiveness across the program, with substantive analysis of partnership development, coordination mechanisms, and sustainability of partnerships with national and local actors. Key relevant sections include: the discussion of the 'Regional MRC strategy with 40 partner organisations' as an important achievement; findings on the increase in 'number of stakeholders (state and non-state) involved in return and reintegration assistance...from 25 (baseline in 2017) to 180 by the end of the project in 2022'; and extensive analysis of partnerships with government entities, NGOs, and service providers. The recommendations explicitly address partnership strengthening, particularly Recommendation 1 on 'enhancing ownership and commitment of stakeholders' and Recommendation 3 on 'building partnerships with service providers who can function without (significant) funding channelled by IOM.' However, partnership is one of several operational themes covered; the report also examines program effectiveness, data systems, and reintegration outcomes with comparable depth. The content on partnership models, coordination challenges, and the analysis of how partnerships functioned across different contexts would add value to a synthesis, particularly the findings section discussing the increase in partner organizations and the recommendations on equitable partnership development."}]

Map cross-cutting priorities

source

Report.map_ccps

async def map_ccps(
    force:bool=False, # Map again when the theme set is in `mappings`
    **kwargs
)->__main__.Report: # The report, for chaining

Map the selected sections against the SRF cross-cutting priorities, and save the report

await report.map_ccps(model='claude-haiku-4-5', force=False)

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ OCR | ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

sort_by_relevance(report.mappings['ccps'])[:2]
[{'theme_id': '3',
  'theme_title': 'Protection-centred',
  'relevance_score': 0.79,
  'reasoning': "This report would likely contribute meaningful evidence to a synthesis on Cross-cutting Priority 3 (Protection-centred). The evaluation explicitly examines protection-related outcomes and approaches as core components of the program. The program's overarching objective directly references 'migrant protection' and 'rights-based migration,' and the evaluation assesses 'protection and voluntary assisted return' as one of five pillars of action. Multiple findings substantively address protection-centred commitments: Section 4.3.2 ('Safe, humane, dignified voluntary return processes') evaluates IOM's effectiveness in providing protection to stranded migrants, with findings showing 95% of assisted migrants satisfied with travel arrangements and 99.6% reporting safe, well-organized travel. The report documents extensive evidence of 'life-saving benefits for beneficiaries' and notes that returnees 'faced dire situations (including abuse, violence, and exploitation)' that IOM addressed. Section 4.3.3.1 discusses psychosocial support and vulnerability assessments, with evidence that 'continuous contact between IOM and returnees ensured that returnees' needs were identified and monitored from the moment of their identification.' The conclusions emphasize that 'the EU-IOM Joint Initiative has been of crucial importance to address the needs of migrants and returnees facing dire situations.' Recommendations 4 and 6 specifically address continuation of 'direct and specialized assistance in transit' and 'safe, humane, and orderly migration pathways,' directly supporting protection commitments. However, protection-centred approaches are one of several program quality themes evaluated alongside effectiveness, efficiency, and sustainability; the report does not systematically analyze PSEAH measures, child safeguarding, or meaningful participation of affected populations in decision-making as distinct protection-centred themes. For a synthesis on protection-centred programming, specialists should pay particular attention to Section 4.3.2 on AVR processes, the findings on vulnerability identification and psychosocial support, and Recommendations 4 and 6 on continuation of protection-focused activities."},
 {'theme_id': '2',
  'theme_title': 'Equality, Diversity & Inclusion',
  'relevance_score': 0.62,
  'reasoning': "This report contains some relevant content on equality and inclusion, though these are not central evaluation themes. The report explicitly mentions gender mainstreaming efforts: Section 4.3.2.1 notes that 'IOM Ethiopia organised awareness raising on sexual and gender-based violence to migrant returnees and host community members as part of the programme's gender mainstreaming effort within the reintegration process.' The integrated approach framework (Figure 1) addresses 'individual level' initiatives responding to 'specific needs and vulnerabilities of returnees,' which implicitly encompasses attention to diverse populations. The evaluation methodology section indicates that focus group discussions included beneficiaries, suggesting some attention to participation of affected populations. However, the report does not systematically analyze IOM's performance on gender equality, disability inclusion, age-appropriate programming, or meaningful participation of marginalized groups as distinct themes. Gender appears primarily as one aspect of vulnerability assessment and awareness-raising rather than as a cross-cutting institutional commitment. The report lacks dedicated findings on gender-responsive program design, representation of women and persons with disabilities in decision-making, or analysis of how the program addressed unequal outcomes based on sex, gender, age, or disability. A synthesis specialist focused on IOM's equality, diversity, and inclusion implementation would find this report has limited substantive content, though the brief mention of gender mainstreaming and the focus on vulnerable populations could provide supplementary context. Recommend prioritizing reports with more explicit gender and inclusion analysis."}]

Map GCM objectives

source

Report.map_gcms

async def map_gcms(
    force:bool=False, # Map again when the theme set is in `mappings`
    **kwargs
)->__main__.Report: # The report, for chaining

Map the selected sections against the GCM objectives, and save the report

await report.map_gcms(model='claude-haiku-4-5', force=False)

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ OCR | ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

sort_by_relevance(report.mappings['gcms'])[:2]
[{'theme_id': '21',
  'theme_title': 'Cooperate In Facilitating Safe And Dignified Return And Readmission, As Well As Sustainable Reintegration',
  'relevance_score': 0.92,
  'reasoning': "FUNDAMENTAL. This report would be essential evidence for a synthesis on GCM Objective 21. Safe, dignified return and sustainable reintegration are the core themes of the entire evaluation. The programme's overarching objective is 'To contribute to facilitating orderly, safe, regular and rights-based migration through the facilitation of dignified voluntary return and the implementation of development-focused and sustainable reintegration policies and processes' (page 4). The evaluation explicitly examines Specific Objective 2 ('Safe, humane, dignified voluntary return processes are enhanced along main migration routes,' pages 18-20) and Specific Objective 3 ('Returnees are sustainably integrated in host communities,' pages 20-23). Key findings show that the JI supported 9,025 migrants to return voluntarily (exceeding the target of 8,450), with 95% of assisted migrants satisfied with travel arrangements and 99.6% feeling travel was well-organized and safe (page 19). The evaluation provides extensive analysis of reintegration outcomes, including economic, social, and psychosocial dimensions, with findings that 93% of returnees reported sufficient levels of economic self-sufficiency, social stability, and psychosocial wellbeing (page 22). The report discusses the integrated approach to reintegration (individual support, community-based projects, structural-level interventions) and measures of reintegration sustainability using the Reintegration Sustainability Index (RSS+). Recommendations 4 and 5 specifically address continuation of AVR support and expansion of reintegration programming. For a synthesis specialist, the entire report would be valuable, with particular attention to: Specific Objective 2 findings (pages 18-20) on AVR effectiveness and safety; Specific Objective 3 findings (pages 20-23) on reintegration outcomes and sustainability; the discussion of the integrated approach (pages 23-24); and Recommendations 4-5 (pages 29-30) on continuation and expansion of return and reintegration support. This report provides direct evidence on implementation challenges (external factors, economic constraints, sustainability barriers) and effectiveness factors for return and reintegration programming."},
 {'theme_id': '1',
  'theme_title': 'Collect and utilize accurate and disaggregated data as a basis for evidence-based policies',
  'relevance_score': 0.88,
  'reasoning': "FUNDAMENTAL. This report would be essential evidence for a synthesis on GCM Objective 1. Data collection and utilization is a dominant theme throughout the evaluation. The report explicitly examines the Regional Data Hub's work on migration data production, capacity building, and harmonization of methodologies (Actions 1a, 1c, 1f). Key findings show the JI exceeded targets for field studies and surveys (20 vs. 19 target), with substantial analysis of data availability improvements and stakeholder capacity building on data use. The evaluation dedicates Section 4.3.1 entirely to 'Specific Objective 1: Partner countries and relevant stakeholders developed or strengthened evidence-based return and reintegration procedures,' with detailed findings on data availability, capacity of stakeholders, and use of data in policymaking. Notably, 136 stakeholders reported that data produced supported evidence-based policies (exceeding the target of 42). The report also discusses the establishment of 36 M&E tools and strengthening of 29 institutions for data collection. For a synthesis specialist, the sections on 'Data availability' (page 16), 'Capacity of stakeholders' (page 17), and findings on the Regional Data Hub's role would be particularly valuable. The report provides direct evidence on implementation challenges (staff turnover, competing priorities) and effectiveness factors for data-driven migration governance."}]

Map outputs

map_outs maps the SRF outputs linked to gcm_ids. By default, it uses the top GCM objective from map_gcms. When no GCM objective reaches the get_top_ids threshold, it maps nothing and leaves outs out of mappings.

source

Report.map_outs

async def map_outs(
    gcm_ids:NoneType=None, # GCM objective IDs whose linked outputs to map; `None` uses the top GCM objective
    force:bool=False, # Map again when the outputs are in `mappings`
    **kwargs
)->__main__.Report: # The report, for chaining

Map the selected sections against the SRF outputs linked to gcm_ids, and save the report

await report.map_outs(model='claude-haiku-4-5', force=False)

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ OCR | ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

sort_by_relevance(report.mappings['outs'])[:2]
[{'theme_id': '2b63',
  'theme_title': 'Returning migrants and returning, relocated and locally integrating displaced persons receive reintegration assistance in line with their needs and those of broader community members.',
  'relevance_score': 0.89,
  'reasoning': "This report would likely be essential for a synthesis on Output 2b63 (reintegration assistance for returning migrants and displaced persons). The evaluation explicitly examines IOM's reintegration programming across the Horn of Africa, with reintegration assistance appearing prominently in the program title, evaluation objectives, and throughout the report's structure. The report dedicates substantial analysis to individual and community-based reintegration across three dimensions (economic, social, psychosocial), directly addressing this Output's core deliverables. Key findings in Section 4.3.3 provide extensive evidence on reintegration assistance delivery: 15,161 beneficiaries received reintegration support (exceeding targets), with detailed analysis of satisfaction levels (55% average), economic support effectiveness, and community-based reintegration projects (54 projects supporting 76,348 beneficiaries). The report explicitly examines how assistance aligns with beneficiary needs through focus group discussions and monitoring surveys. Particularly valuable sections include: (1) Section 4.3.3.1 on achievement of outputs and results, detailing individual and community-based reintegration with specific findings on satisfaction drivers and dissatisfaction factors; (2) the integrated approach discussion (Section 4.3.4) explaining how economic, social, and psychosocial dimensions interconnect; (3) conclusions in Section 5.1 emphasizing the importance of individualized support and community-level approaches; and (4) Recommendation 2 on strengthening community-based reintegration efforts. The report provides direct evidence on reintegration assistance effectiveness, implementation challenges, and factors affecting beneficiary outcomes—all critical for understanding this Output's delivery."},
 {'theme_id': '3a43',
  'theme_title': 'Migrants of all genders, ages, abilities and other diversities are able to cover their immediate needs upon return to their countries of origin.',
  'relevance_score': 0.85,
  'reasoning': "This report would likely be essential for a synthesis on Output 3a43 (migrants' ability to cover immediate needs upon return). The evaluation provides extensive evidence on immediate assistance provided to returning migrants, which directly addresses this Output's deliverable. Key findings demonstrate: the JI provided reintegration assistance to 15,161 beneficiaries, with 95% of assisted migrants satisfied with travel arrangements and 99.6% reporting safe, well-organized travel; emergency support on arrival (cash, medical services, mental health and psychosocial support) was deemed 'highly relevant, important, and effective' (Section 5.1); and focus group discussions confirm returnees received crucial economic support, medical assistance, and psychosocial support upon return. The report explicitly examines the adequacy of immediate assistance: returnees noted economic support was 'crucial' as they returned 'with nothing,' helping them start businesses or search for employment. However, the report also identifies a critical gap—many returnees deemed the economic support insufficient to achieve sustainable reintegration, indicating that while immediate needs were addressed, the level of support was often inadequate for longer-term stability. Particularly valuable sections include: (1) Section 4.3.2.1 on assistance to stranded migrants, detailing AVR support and satisfaction metrics; (2) Section 4.3.3.1 on individual reintegration assistance, with findings on the types and adequacy of support provided; (3) Section 5.1 conclusions emphasizing the 'life-saving benefits' of immediate assistance; and (4) Recommendation 5 on extending support scope with focus on longer-term integration. The report provides direct evidence on immediate assistance delivery, beneficiary satisfaction, and gaps in adequacy—all critical for understanding this Output's effectiveness."}]

Map all themes

Report.map_all runs the four map_* methods in order.

source

Report.map_all

async def map_all(
    **kwargs
)->__main__.Report: # The report, for chaining

Run map_enbs, map_ccps, map_gcms and map_outs in order

Run the full pipeline

run_pipeline takes one evaluation, found by url, id or title, as far through the stages as it can. It first loads the report’s saved state from base_path, and it skips the stages already done. It returns a PipelineResult with one of three statuses:

  • 'awaiting_curation' when the report still needs curation. Curate it in the curator app, then run the same call again.
  • 'completed' after the four mappings.
  • 'failed' when a stage raises an error. run_pipeline logs the error, and the result holds its message in error and the stage in step.

Errors raised while creating the report, such as a missing report URL, are not caught. force runs stages again. Pass True for every stage, or a set of step names such as {'ocr', 'gcms'}. The step names are 'download', 'ocr', 'enbs', 'ccps', 'gcms' and 'outs'. By default, run_pipeline deletes the PDF folder after OCR.

source

should_force

def should_force(
    force, # `True` or `False` for every step, or a set of step names
    step, # Step name, such as `'ocr'`
)->bool: # Whether to run `step` again

Check whether force applies to step

source

PipelineResult

def PipelineResult(
    report, # The report, in its final state
    status, # `'completed'`, `'awaiting_curation'` or `'failed'`
    error:NoneType=None, # Error message, when `status` is `'failed'`
    step:NoneType=None, # Stage the run stopped at, when it did not complete
):

Outcome of run_pipeline for one report

source

get_current_step

def get_current_step(
    report, # Report to check
)->str: # `'ocr'`, `'curation'` or `'mapping'`

Infer the stage report is at from its state

source

run_pipeline

async def run_pipeline(
    evals:list, # Evaluations to search
    url:str=None, # URL of the report PDF
    id:str=None, # Evaluation ID
    title:str=None, # Evaluation title
    base_path:str='data', # Folder holding `pdf/`, `md/` and `results/`
    add_img_desc:bool=True, # Add image descriptions during OCR
    fix_kwargs:dict=None, # Passed to mistocr's `fix_hdgs`
    desc_kwargs:dict=None, # Passed to mistocr's `add_img_descs`
    force:bool | set=False, # `True` to run every step again, or a set of step names
    delete_pdf:bool=True, # Delete the PDF folder after OCR
    verbose:bool=False, # Log each step
    **kwargs
)->__main__.PipelineResult: # The report and how far the run got

Run the pipeline on one evaluation as far as it can go

evals = load_evals('files/test/evaluations.json')
id = '49d2fba781b6a7c0d94577479636ee6f'
result = await run_pipeline(evals, id=id, base_path='files/test', add_img_desc=False, 
                            force=False, verbose=True, model='claude-haiku-4-5')
__main__ - INFO - Mapping enablers...
__main__ - INFO - Mapping CCPs...
__main__ - INFO - Mapping GCM objectives...
__main__ - INFO - Mapping outputs...
__main__ - INFO - Pipeline complete!
result.status
'completed'
result.report

Report: Final Evaluation of the EU-IOM Joint Initiative for migrant protection and reintegration in the horn of Africa

Year: 2023 | Organization: IOM
ID: 49d2fba781b6a7c0d94577479636ee6f

Pipeline: ✓ OCR | ✓ Curated (5 headings) | ✓ Mappings (enbs, ccps, gcms, outs)
Curation: sections_selected

Report: View Evaluation Report

Batch

The IOM repository export, from evaluation-search-pdf, lists each evaluation’s documents with their type, such as brief, annex or final report. The types and URLs do not always match. A URL labelled “Evaluation Report” can point to an annex or a brief. IOM knows about this bug and is fixing it.

As a workaround, IOMRepoReader.to_json gets each evaluation’s report URL from IOM’s UNEG API, /json/api/evaluation-uneg. It adds that URL as the evaluation’s first document, with subtype 'Evaluation report (UNEG)', and eval_url prefers it. The UNEG API has less metadata than the export.

evals[0].docs
[{'subtype': 'Evaluation report',
  'url': 'https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/Annex%20VI%20Case%20Study%20-%20RDH%20East%2C%20Horn%20and%20Southern%20Africa.pdf',
  'desc': 'Evaluation Report'},
 {'subtype': 'Evaluation brief',
  'url': 'https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/Annex%20VII%20Case%20Study%20-%20RDH%20Asia-Pacific.pdf',
  'desc': 'Evaluation Brief'},
 {'subtype': 'Annexes',
  'url': 'https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/Annex%20VIII%20-%20Inception%20Report.pdf',
  'desc': 'Annex VI Case Study - RDH East'},
 {'subtype': 'Annexes',
  'url': 'https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/Evaluation%20Brief.pdf',
  'desc': 'Horn and Southern Africa'},
 {'subtype': 'Annexes',
  'url': 'https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/IOM%20MDS%20Evaluation%20Report%20-%20clean_0.pdf',
  'desc': 'Annex VII Case Study - RDH Asia-Pacific'},
 {'subtype': 'Special related reports/documents',
  'url': 'https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/Migration%20Data%20Evaluation%20infographics.pdf',
  'desc': 'Annex VIII - Inception Report'}]

BatchResult groups the evaluation IDs of a batch run by outcome.

source

BatchResult

def BatchResult(
    completed:NoneType=None, # IDs of the reports that completed
    awaiting_curation:NoneType=None, # IDs of the reports that wait for curation
    failed:NoneType=None, # One dict per failure, with `id`, `step` and `error`
    skipped:NoneType=None, # IDs of the reports skipped because they were already complete
):

Result of running pipeline on a batch of evaluations

source

filter_evals

def filter_evals(
    evals, # Evaluations to filter
    year:NoneType=None, # Keep the evaluations of this `Year`, such as `'2023'`
    ids:NoneType=None, # Keep the evaluations with these IDs; takes precedence over `year`
)->list: # The kept evaluations, or `evals` when neither filter is set

Keep the evaluations in ids, or else those of year

source

is_completed

def is_completed(
    report, # Report to check
)->bool: # Whether `mappings` has all four theme sets

Check whether report has been mapped against every theme set

batch_run runs run_pipeline on each evaluation in evals, after filter_evals. It skips the reports that have all four mappings, unless force is set. It counts an evaluation without a report URL as failed, at step 'url_lookup'. At the end, it saves the BatchResult to base_path/batch_runs/<timestamp>.json.

source

batch_run

async def batch_run(
    evals, # Evaluations to run
    base_path:str='../../data', # Folder holding `pdf/`, `md/` and `results/`
    year:str=None, # Keep the evaluations of this year, such as `'2023'`
    ids:list=None, # Keep the evaluations with these IDs; takes precedence over `year`
    force:bool | set=False, # `True` to run every step again, or a set of step names
    stop_after:int=None, # Stop after N reports; not implemented yet
    delete_pdf:bool=False, # Delete each report's PDF folder after OCR
    add_img_desc:bool=True, # Add image descriptions during OCR
    fix_kwargs:dict=None, # Passed to mistocr's `fix_hdgs`
    desc_kwargs:dict=None, # Passed to mistocr's `add_img_descs`
    **kwargs
)->__main__.BatchResult: # IDs of the evaluations, grouped by outcome

Run the pipeline on each evaluation, and save a summary of the run

test_ids = ['4341695461234eee3deb51ac68871109', '6c3c2cf3fa479112967612b0baddab72']
result = await batch_run(evals, base_path='files/test', ids=test_ids,
                         add_img_desc=False, verbose=False, model='claude-haiku-4-5')
result
__main__ - INFO - 43416954... awaiting_curation
__main__ - INFO - 6c3c2cf3... skipped (already completed)
<__main__.BatchResult>