Mapper

Score how relevant an evaluation report is to each SRF and GCM theme, with an LLM

The mapper scores a report against four theme sets: the SRF enablers, the SRF cross-cutting priorities, the GCM objectives and the SRF outputs. For each theme, the LLM returns a relevance score from 0 to 1 and its reasoning. The report is the same for every theme set, and prompt caching lets the calls after the first reuse it.

from cachy import enable_cachy
enable_cachy()

Response models

The LLM replies with a ThemeScores, which holds one ThemeScore per theme. structured_llm sends each model’s docstring to the LLM as part of the reply format.

source

ThemeScore

def ThemeScore(
    **data:Any
)->None:

Single theme’s relevance assessment with score, reasoning, and confidence

Exported source

class ThemeScore(BaseModel):
    "Single theme's relevance assessment with score, reasoning, and confidence"
    theme_id: str           # Theme ID, such as `'7'` or `'1a11'`
    theme_title: str        # Theme title
    relevance_score: float  # Relevance of the theme to the report, from 0 to 1
    reasoning: str          # The LLM's explanation of the score

source

ThemeScores

def ThemeScores(
    **data:Any
)->None:

Collection of theme relevance scores from a single mapping operation

Exported source

class ThemeScores(BaseModel):
    "Collection of theme relevance scores from a single mapping operation"
    scores: list[ThemeScore]  # All theme scores for a single mapping call

Parsing

parse_res turns a reply into a list of score dicts, the form the pipeline saves. sort_by_relevance orders the scores from most to least relevant. get_top_ids keeps the IDs of the themes that score at least min_score, 0.66 by default.

source

parse_res

def parse_res(
    res:__main__.ThemeScores, # Structured LLM reply from `map_themes`
)->list[dict]: # One dict per theme, with the `ThemeScore` fields

Convert the scores in an LLM reply to a list of dicts

source

sort_by_relevance

def sort_by_relevance(
    scores, # Score dicts from `parse_res`
)->list[dict]: # The scores, most relevant first

Sort scores by relevance_score, highest first

source

get_top_ids

def get_top_ids(
    scores, # Score dicts from `parse_res`
    min_score:float=0.66, # Lowest `relevance_score` to keep
)->list[str]: # Theme IDs, most relevant first

Get the IDs of the themes that score at least min_score

Mapping a report

map_themes asks the LLM to score one set of themes against a report. mk_system_prompt puts the report in the system prompt. map_themes marks that system prompt for prompt caching with cache_idxs=[0]. The later mappings of the same report read it from the cache, which costs less than sending it again.

source

mk_system_prompt

def mk_system_prompt(
    report:str, # Report text, usually the extracted core sections
)->str: # System prompt holding the report

Create the system prompt that every theme mapping of report shares

source

map_themes

async def map_themes(
    sp:str, # Report system prompt from `mk_system_prompt`
    themes:str, # Themes to score, formatted as Markdown
    prompt:str, # Mapping prompt, such as `load_prompt('gcms')`
    model:str='claude-haiku-4-5', # LLM to use
    response_format:pydantic._internal._model_construction.ModelMetaclass=ThemeScores, # Pydantic model the reply must follow
    temperature:float=0, # Sampling temperature
    reasoning_effort:str=None, # `'low'`, `'medium'` or `'high'`; `None` for no extended thinking
    **kwargs
)->__main__.ThemeScores: # The LLM's scores

Score themes against the report in sp, with the report cached

from iomeval.extract import extract_sections
from mistocr.core import read_pgs

md = read_pgs('files/test/AAP%20Evaluation%20Report_final_')
report = await extract_sections(md)

n_tokens(report)

To keep the examples cheap, the system prompt holds only the first 500 characters of the extracted sections:

sp = mk_system_prompt(report[:500])
print(sp[:200])

## Report to Analyze

## EXECUTIVE SUMMARY  ... page 6

This external evaluation assessed whether or not IOM is delivering on its commitments to accountability to affected populations (AAP), ${ }^{1}$

Map the report against the cross-cutting priorities, and show the most relevant ones:

ccps = load_ccps()
prompt = load_prompt('srf_ccps')
res = await map_themes(sp, fmt_ccps(ccps), prompt, reasoning_effort=None, temperature=0)

sort_by_relevance(parse_res(res))[:3]

[{'theme_id': '1',
  'theme_title': 'Integrity, Transparency and Accountability',
  'relevance_score': 0.82,
  'reasoning': "This report would likely contribute meaningful evidence to a synthesis on Cross-cutting Priority 1 (Integrity, Transparency and Accountability). The evaluation explicitly assesses IOM's delivery on Accountability to Affected Populations (AAP), which is a core institutional commitment within the transparency and accountability framework. The executive summary indicates that the 2023 MOPAN assessment found IOM has not consistently incorporated AAP across programmes and has not invested adequately in AAP relative to operational scale—directly addressing accountability mechanisms and organizational systems. The report appears to examine whether IOM has transparent, functional systems for accountability to the populations it serves, including feedback mechanisms and responsiveness to affected communities. For a synthesis specialist, the findings on AAP implementation gaps, the analysis of how systematically IOM incorporates accountability mechanisms across programmes, and recommendations for strengthening AAP would be particularly valuable. However, the excerpt provided does not reveal the full depth of analysis on broader integrity and transparency dimensions (e.g., ethical frameworks, financial transparency, anti-corruption measures), suggesting AAP is the primary accountability focus rather than a comprehensive institutional accountability assessment. The report would merit review for evidence on IOM's accountability systems and community-facing transparency practices."},
 {'theme_id': '3',
  'theme_title': 'Protection-centred',
  'relevance_score': 0.71,
  'reasoning': "This report would likely contribute relevant evidence to a synthesis on Cross-cutting Priority 3 (Protection-centred). Accountability to Affected Populations is closely aligned with protection-centred approaches, which emphasize placing migrants' human rights and well-being at the centre of operations and involving communities in decision-making. The evaluation's assessment of whether IOM has systematically incorporated AAP across programmes directly relates to IOM's commitment to involve individuals and communities in decision-making and to demonstrate responsiveness to their needs and concerns. The findings on AAP implementation gaps would likely reveal whether IOM is adequately centering affected populations' voices and protection concerns in programme design and delivery. However, the excerpt does not indicate whether the evaluation explicitly examines child safeguarding, PSEAH (protection from sexual exploitation and abuse), rights-based approaches, or vulnerability-focused programming—other core elements of protection-centred commitments. The report appears to focus specifically on the accountability/feedback dimension of protection-centred work rather than the full spectrum of protection commitments. A synthesis specialist would likely find the analysis of community participation and responsiveness mechanisms valuable, particularly in the findings section addressing AAP implementation, though they might prioritize reports with more comprehensive coverage of protection-centred programming elements."},
 {'theme_id': '2',
  'theme_title': 'Equality, Diversity & Inclusion',
  'relevance_score': 0.28,
  'reasoning': "No substantive discussion of equality, diversity, and inclusion as institutional commitments appears in the provided excerpt. The report's focus is on accountability to affected populations and organizational systems for AAP, which is distinct from EDI mainstreaming, gender equality, disability inclusion, or addressing barriers to equal participation and outcomes. While AAP frameworks may tangentially relate to inclusive participation, the excerpt does not indicate that the evaluation examines IOM's gender mainstreaming, disability inclusion, youth engagement, or efforts to address unequal outcomes based on protected characteristics. Recommend not reviewing this report for a synthesis on Cross-cutting Priority 2."}]

The GCM objectives are Markdown already. map_themes takes them as they are:

gcms = load_gcms()
prompt = load_prompt('gcms')
res = await map_themes(sp, gcms, prompt, reasoning_effort=None, temperature=0)

sort_by_relevance(parse_res(res))[:3]

[{'theme_id': '1',
  'theme_title': 'Collect and utilize accurate and disaggregated data as a basis for evidence-based policies',
  'relevance_score': 0.78,
  'reasoning': "This report would likely contribute relevant evidence to a synthesis on GCM Objective 1. The evaluation explicitly assesses IOM's accountability to affected populations (AAP), which requires systematic data collection and analysis on program effectiveness and beneficiary outcomes. The executive summary indicates the report reviews both internal and public documents related to AAP, suggesting substantive examination of how IOM collects and uses data to inform its accountability commitments. The evaluation's focus on whether IOM 'consistently incorporates AAP across all its programmes' implies analysis of data systems and evidence-based assessment mechanisms. However, data collection and utilization is not the primary focus—AAP implementation is. The report would likely contain relevant findings on data practices within AAP frameworks, particularly regarding how IOM monitors and measures accountability outcomes. Synthesis specialists should review sections examining IOM's data collection methodologies for AAP monitoring and any findings on gaps in evidence-based accountability practices."},
 {'theme_id': '15',
  'theme_title': 'Provide access to basic services for migrants',
  'relevance_score': 0.73,
  'reasoning': "This report would likely contribute relevant evidence to a synthesis on GCM Objective 15. Accountability to affected populations fundamentally requires ensuring that migrants have access to basic services and that service delivery is non-discriminatory and responsive to migrant needs. The evaluation's assessment of whether IOM 'consistently incorporates AAP across all its programmes' would likely examine how IOM ensures migrants can access health, education, and other basic services, and whether migrants have voice in service design. The finding that IOM 'did not invest adequately in AAP' suggests potential gaps in ensuring service access is truly migrant-centered and responsive. AAP frameworks in service delivery contexts address migrant participation in needs assessment, accessible information about services, non-discriminatory access, and complaint mechanisms—all relevant to Objective 15. Synthesis specialists should review findings on how IOM ensures migrants have equitable access to basic services, how service providers engage with migrant populations, and how AAP mechanisms facilitate service improvements. The report's analysis of AAP implementation in service delivery would add value to a synthesis on migrant access to basic services."},
 {'theme_id': '13',
  'theme_title': 'Use Migration Detention Only As A Measure Of Last Resort And Work Towards Alternatives',
  'relevance_score': 0.71,
  'reasoning': "This report would likely contribute relevant evidence to a synthesis on GCM Objective 13. Accountability to affected populations is particularly critical in detention contexts, where migrants' rights and dignity are at heightened risk. The evaluation's assessment of whether IOM 'consistently incorporates AAP across all its programmes' would likely include examination of AAP in detention settings or related to migrants in custody. AAP frameworks in detention contexts address migrant access to information about detention reasons, legal assistance, family communication, and complaint mechanisms—all core themes under Objective 13. The finding that IOM 'did not invest adequately in AAP' suggests potential gaps in ensuring detained migrants have voice and access to accountability mechanisms. Synthesis specialists should review findings on how IOM implements AAP in detention-related programs, ensures migrants in detention can report concerns, and facilitates access to remedies. The report's analysis of AAP gaps in detention contexts would add value to a synthesis on alternatives to detention and migrant rights in custody."}]

get_top_ids lists the GCM objectives that reach the threshold:

get_top_ids(parse_res(res))

['1', '15', '13']

Mapping against every framework

map_all maps a report against the SRF enablers, the cross-cutting priorities and the GCM objectives. It then maps the SRF outputs linked to the top GCM objective, which it looks up with get_srf_outs. When no GCM objective reaches the get_top_ids threshold, it maps no outputs and returns outs=None. The four calls share the cached report. Report.map_all in the pipeline runs the same steps and saves each result.

source

load_prompts

def load_prompts(
    path:pathlib.Path | str | None=None, # Folder of prompt files; `None` uses the prompts shipped with the package
)->fastcore.basics.AttrDict: # `enbs`, `ccps`, `gcms` and `outs`

Load the four mapping prompts

load_all_thms().keys()

dict_keys(['enbs', 'ccps', 'gcms', 'outs', 'gcm_lut'])

load_prompts().keys()

dict_keys(['enbs', 'ccps', 'gcms', 'outs'])

source

map_all

async def map_all(
    report:str, # Report text, usually the extracted core sections
    path:str='files/themes', # Folder of theme files, relative to the working directory
    prompt_path:str='files/prompts', # Folder of prompt files, relative to the working directory
    verbose:bool=True, # Log each step
    *, model:str='claude-haiku-4-5', # LLM to use
    response_format:pydantic._internal._model_construction.ModelMetaclass=ThemeScores, # Pydantic model the reply must follow
    temperature:float=0, # Sampling temperature
    reasoning_effort:str=None, # `'low'`, `'medium'` or `'high'`; `None` for no extended thinking
)->fastcore.basics.AttrDict: # Score dicts under `enbs`, `ccps`, `gcms` and `outs`; `outs` is `None` when no GCM objective reaches the threshold

Map report against the SRF enablers, cross-cutting priorities, GCM objectives and SRF outputs, in that order

Run every mapping on the same 500 characters:

res = await map_all(report[:500])

__main__ - INFO - Mapping SRF Enablers...
__main__ - INFO - Mapping Cross-cutting Priorities...
__main__ - INFO - Mapping GCM Objectives...
__main__ - INFO - Top GCM: 1 (from 3 candidates)
__main__ - INFO - Mapping 12 filtered SRF Outputs...

sort_by_relevance(res.enbs)[:2]

[{'theme_id': '6',
  'theme_title': 'Communication',
  'relevance_score': 0.74,
  'reasoning': "This report would likely contribute relevant evidence to a synthesis on Enabler 6 (Communication). An evaluation of accountability to affected populations (AAP) implementation necessarily examines IOM's communication with beneficiaries and affected communities—a core dimension of organizational communication capacity. The evaluation likely assesses how IOM communicates program information, solicits feedback, shares results, and ensures affected populations have access to information about their rights and program details. These practices directly relate to IOM's organizational capacity in external communication, beneficiary engagement, and transparency. The report probably contains findings on communication effectiveness, accessibility of information to diverse populations, and mechanisms for two-way dialogue with communities. However, communication systems are not the primary focus; AAP implementation is the dominant theme. The content on beneficiary communication, information accessibility, and feedback mechanisms would add value to a synthesis on communication capacities, particularly regarding IOM's communication with affected populations and marginalized groups. Specialists should review sections addressing how IOM communicates with beneficiaries, ensures information accessibility, and facilitates community voice in organizational processes."},
 {'theme_id': '2',
  'theme_title': 'Partnership',
  'relevance_score': 0.71,
  'reasoning': "This report would likely contribute relevant evidence to a synthesis on Enabler 2 (Partnership). The evaluation of accountability to affected populations (AAP) implementation appears to examine how IOM engages with and responds to beneficiary and community stakeholder feedback—a dimension of partnership and stakeholder engagement. AAP frameworks inherently involve partnerships with local actors, community representatives, and affected populations in program design and accountability mechanisms. The report likely contains findings on IOM's capacity to establish two-way communication channels, incorporate local voices in decision-making, and develop equitable engagement with communities—all relating to partnership quality and local actor involvement. However, partnership development and inter-organizational collaboration are not the primary focus; AAP is the dominant theme. The content on stakeholder engagement and community participation mechanisms would add value to a synthesis on partnership capacities, particularly regarding IOM's engagement with local and affected populations. Specialists should review sections addressing how IOM incorporates beneficiary feedback and community participation in program implementation."}]

sort_by_relevance(res.ccps)[:2]

[{'theme_id': '1',
  'theme_title': 'Integrity, Transparency and Accountability',
  'relevance_score': 0.82,
  'reasoning': "This report would likely contribute meaningful evidence to a synthesis on Cross-cutting Priority 1 (Integrity, Transparency and Accountability). The evaluation explicitly assesses IOM's delivery on Accountability to Affected Populations (AAP), which is a core institutional commitment within the integrity and accountability framework. The executive summary indicates that the 2023 MOPAN assessment found IOM has not consistently incorporated AAP across programmes and has not invested adequately in AAP relative to operational scale—directly addressing accountability and transparency commitments. The evaluation appears to examine both internal systems and public documentation related to AAP implementation. Key areas likely to be relevant for a synthesis include: the assessment of AAP integration across IOM programmes, analysis of organizational investment in accountability mechanisms, and findings on consistency of AAP implementation. However, while AAP is the primary focus, the report excerpt does not indicate whether broader integrity frameworks, ethical response systems, or transparency mechanisms beyond AAP are examined in depth. A synthesis specialist would benefit from reviewing the findings section on AAP implementation gaps and the recommendations for strengthening accountability systems, though the report's scope appears somewhat narrower than the full breadth of the Priority's institutional commitments (which also encompass organizational culture, ethical frameworks, and result visibility)."},
 {'theme_id': '3',
  'theme_title': 'Protection-centred',
  'relevance_score': 0.74,
  'reasoning': "This report would likely contribute relevant evidence to a synthesis on Cross-cutting Priority 3 (Protection-centred). Accountability to Affected Populations (AAP) is closely aligned with protection-centred approaches, particularly the commitment to involve individuals and communities in decision-making and to protect them from harm. The evaluation's assessment of AAP implementation directly relates to IOM's commitment to place migrants' human rights and well-being at the centre of operations and to involve communities in decision-making. The report appears to examine whether IOM has systematically incorporated mechanisms for community participation and accountability—core elements of protection-centred programming. Key findings likely to be valuable for a synthesis would include: the extent to which affected populations are meaningfully involved in programme design and decision-making, the functionality of feedback and complaint mechanisms, and barriers to effective community participation. However, the report excerpt does not explicitly indicate whether the evaluation addresses child safeguarding, PSEAH (sexual exploitation and abuse) prevention, or rights-based approaches more broadly—other important dimensions of the Protection-centred Priority. A synthesis specialist would benefit from reviewing the findings on AAP implementation and community participation mechanisms, though they should note that this report may address only one component of the broader protection-centred commitment."}]

get_top_ids(res.gcms)

['1', '15', '13']

sort_by_relevance(res.outs)[:1]

[{'theme_id': '1a31',
  'theme_title': 'Guidelines on data and information collection, sharing and management are in place that adhere to data protection standards, principles of confidentiality and a defined purpose, to protect the individuals and groups providing information from harm.',
  'relevance_score': 0.78,
  'reasoning': "This report would likely contribute meaningful evidence to a synthesis on Output 1a3.1 (data protection guidelines and standards). The evaluation explicitly assesses IOM's accountability to affected populations (AAP), which inherently involves responsible data collection, management, and protection practices. The executive summary indicates the evaluation reviewed both internal and public documents related to AAP, suggesting substantive examination of how IOM handles beneficiary information and data. Data protection and confidentiality are foundational to AAP frameworks, and the report's focus on whether IOM is 'consistently incorporating AAP across all its programmes' would likely include analysis of data safeguarding mechanisms. However, the excerpt provided does not explicitly foreground data protection guidelines as a primary evaluation focus—AAP is the dominant theme. A synthesis specialist would likely find value in the report's treatment of data protection within the broader AAP context, particularly in sections examining IOM's systems for protecting beneficiary information. Recommend reviewing findings and conclusions sections that address AAP implementation mechanisms and safeguards."}]