Maps evaluation reports to IOM’s SRF(Strategic Results Framework) and GCM (Global Compact for Migration UN General Assembly resolution) themes using LLM-based scoring. The module analyzes reports against four theme hierarchies (SRF Enablers, Cross-cutting Priorities, GCM Objectives, and SRF Outputs) using prompt caching for efficiency. Returns structured relevance scores with reasoning and confidence levels for each theme.
ThemeScore and ThemeScores are Pydantic models for structured LLM output. The LLM returns scores for each theme with reasoning and confidence levels.
Single theme’s relevance assessment with score, reasoning, and confidence
class ThemeScore(BaseModel):
"Single theme's relevance assessment with score, reasoning, and confidence"
theme_id: str # Unique identifier for the theme
theme_title: str # Human-readable theme name
relevance_score: float # 0-1 score indicating how relevant theme is to report
reasoning: str # LLM's explanation for the scoreCollection of theme relevance scores from a single mapping operation
Helper functions to extract and sort results from the LLM response.
Extract scores list from LLM completion response
Sort themes by relevance score descending
Get IDs of themes with relevance score >= min_score, sorted by relevance
The core mapping functions prepare the report for caching and call the LLM to score themes against the report content.
Create cached system message blocks from report text
def map_themes(
system_blocks:list, # Cached system blocks from mk_system_blocks
themes:str, # Formatted themes text to score against
prompt:str, # Mapping instruction prompt
model:str='claude-haiku-4-5', # Model to use for completion
response_format:ModelMetaclass=ThemeScores, # Pydantic model for structured output
max_tokens:int=8192, # Max tokens for completion
temperature:float=0, # Temperature for completion
reasoning_effort:str=None, # Reasoning effort for completion (low, medium, high)
kwargs:VAR_KEYWORD
)->ModelResponse: # LLM completion response with structured output
Map report against themes using cached system blocks
## Report to Analyze
## EXECUTIVE SUMMARY ... page 6
This external evaluation assessed whether or not IOM is delivering on its commitments to accountability to affected populations (AAP), ${ }^{1}$[{'theme_id': '1',
'theme_title': 'Integrity, Transparency and Accountability',
'relevance_score': 0.82,
'reasoning': "This report would likely contribute meaningful evidence to a synthesis on Cross-cutting Priority 1 (Integrity, Transparency and Accountability). The evaluation explicitly assesses IOM's delivery on Accountability to Affected Populations (AAP), which is a core institutional commitment within the transparency and accountability framework. The executive summary indicates that the 2023 MOPAN assessment found IOM has not consistently incorporated AAP across programmes and has not invested adequately in AAP relative to operational scale—directly addressing accountability mechanisms and organizational systems. The report appears to examine whether IOM has transparent, functional systems for accountability to the populations it serves, including feedback mechanisms and responsiveness to affected communities. For a synthesis specialist, the findings on AAP implementation gaps, the analysis of how systematically IOM incorporates accountability mechanisms across programmes, and recommendations for strengthening AAP would be particularly valuable. However, the excerpt provided does not reveal the full depth of analysis on broader integrity and transparency dimensions (e.g., ethical frameworks, financial transparency, anti-corruption measures), suggesting AAP is the primary accountability focus rather than a comprehensive institutional accountability assessment. The report would merit review for evidence on IOM's accountability systems and community-facing transparency practices."},
{'theme_id': '3',
'theme_title': 'Protection-centred',
'relevance_score': 0.71,
'reasoning': "This report would likely contribute relevant evidence to a synthesis on Cross-cutting Priority 3 (Protection-centred). Accountability to Affected Populations is closely aligned with protection-centred approaches, which emphasize placing migrants' human rights and well-being at the centre of operations and involving communities in decision-making. The evaluation's assessment of whether IOM has systematically incorporated AAP across programmes directly relates to IOM's commitment to involve individuals and communities in decision-making and to demonstrate responsiveness to their needs and concerns. The findings on AAP implementation gaps would likely reveal whether IOM is adequately centering affected populations' voices and protection concerns in programme design and delivery. However, the excerpt does not indicate whether the evaluation explicitly examines child safeguarding, PSEAH (protection from sexual exploitation and abuse), rights-based approaches, or vulnerability-focused programming—other core elements of protection-centred commitments. The report appears to focus specifically on the accountability/feedback dimension of protection-centred work rather than the full spectrum of protection commitments. A synthesis specialist would likely find the analysis of community participation and responsiveness mechanisms valuable, particularly in the findings section addressing AAP implementation, though they might prioritize reports with more comprehensive coverage of protection-centred programming elements."},
{'theme_id': '2',
'theme_title': 'Equality, Diversity & Inclusion',
'relevance_score': 0.28,
'reasoning': "No substantive discussion of equality, diversity, and inclusion as institutional commitments appears in the provided excerpt. The report's focus is on accountability to affected populations and organizational systems for AAP, which is distinct from EDI mainstreaming, gender equality, disability inclusion, or addressing barriers to equal participation and outcomes. While AAP frameworks may tangentially relate to inclusive participation, the excerpt does not indicate that the evaluation examines IOM's gender mainstreaming, disability inclusion, youth engagement, or efforts to address unequal outcomes based on protected characteristics. Recommend not reviewing this report for a synthesis on Cross-cutting Priority 2."}][{'theme_id': '1',
'theme_title': 'Collect and utilize accurate and disaggregated data as a basis for evidence-based policies',
'relevance_score': 0.78,
'reasoning': "This report would likely contribute relevant evidence to a synthesis on GCM Objective 1. The evaluation explicitly assesses IOM's accountability to affected populations (AAP), which requires systematic data collection and analysis on program effectiveness and beneficiary outcomes. The executive summary indicates the report reviews both internal and public documents related to AAP, suggesting substantive examination of how IOM collects and uses data to inform its accountability commitments. The evaluation's focus on whether IOM 'consistently incorporates AAP across all its programmes' implies analysis of data systems and evidence-based assessment mechanisms. However, data collection and utilization is not the primary focus—AAP implementation is. The report would likely contain relevant findings on data practices within AAP frameworks, particularly regarding how IOM monitors and measures accountability outcomes. Synthesis specialists should review sections examining IOM's data collection methodologies for AAP monitoring and any findings on gaps in evidence-based accountability practices."},
{'theme_id': '15',
'theme_title': 'Provide access to basic services for migrants',
'relevance_score': 0.73,
'reasoning': "This report would likely contribute relevant evidence to a synthesis on GCM Objective 15. Accountability to affected populations fundamentally requires ensuring that migrants have access to basic services and that service delivery is non-discriminatory and responsive to migrant needs. The evaluation's assessment of whether IOM 'consistently incorporates AAP across all its programmes' would likely examine how IOM ensures migrants can access health, education, and other basic services, and whether migrants have voice in service design. The finding that IOM 'did not invest adequately in AAP' suggests potential gaps in ensuring service access is truly migrant-centered and responsive. AAP frameworks in service delivery contexts address migrant participation in needs assessment, accessible information about services, non-discriminatory access, and complaint mechanisms—all relevant to Objective 15. Synthesis specialists should review findings on how IOM ensures migrants have equitable access to basic services, how service providers engage with migrant populations, and how AAP mechanisms facilitate service improvements. The report's analysis of AAP implementation in service delivery would add value to a synthesis on migrant access to basic services."},
{'theme_id': '13',
'theme_title': 'Use Migration Detention Only As A Measure Of Last Resort And Work Towards Alternatives',
'relevance_score': 0.71,
'reasoning': "This report would likely contribute relevant evidence to a synthesis on GCM Objective 13. Accountability to affected populations is particularly critical in detention contexts, where migrants' rights and dignity are at heightened risk. The evaluation's assessment of whether IOM 'consistently incorporates AAP across all its programmes' would likely include examination of AAP in detention settings or related to migrants in custody. AAP frameworks in detention contexts address migrant access to information about detention reasons, legal assistance, family communication, and complaint mechanisms—all core themes under Objective 13. The finding that IOM 'did not invest adequately in AAP' suggests potential gaps in ensuring detained migrants have voice and access to accountability mechanisms. Synthesis specialists should review findings on how IOM implements AAP in detention-related programs, ensures migrants in detention can report concerns, and facilitates access to remedies. The report's analysis of AAP gaps in detention contexts would add value to a synthesis on alternatives to detention and migrant rights in custody."}]The full pipeline maps a report against all theme hierarchies: SRF Enablers → Cross-cutting Priorities → GCM Objectives → SRF Outputs. The GCM lookup table filters which outputs to score based on top GCM objectives.
Load all mapping prompts
def map_all(
report:str, # Full report text to analyze
path:str='files/themes', # Directory containing theme JSON files
prompt_path:str='files/prompts', # Directory containing prompt files
verbose:bool=True, # Log progress messages
model:str='claude-haiku-4-5', # Model to use for completion
response_format:ModelMetaclass=ThemeScores, # Pydantic model for structured output
max_tokens:int=8192, # Max tokens for completion
temperature:float=0, # Temperature for completion
reasoning_effort:str=None, # Reasoning effort for completion (low, medium, high)
)->AttrDict: # Dict with enablers, ccp, gcm, outputs results
Map report against all theme classes: enablers → CCP → GCM → outputs
__main__ - INFO - Mapping SRF Enablers...
__main__ - INFO - Mapping Cross-cutting Priorities...
__main__ - INFO - Mapping GCM Objectives...
__main__ - INFO - Top GCM: 1 (from 3 candidates)
__main__ - INFO - Mapping 12 filtered SRF Outputs...[{'theme_id': '6',
'theme_title': 'Communication',
'relevance_score': 0.74,
'reasoning': "This report would likely contribute relevant evidence to a synthesis on Enabler 6 (Communication). An evaluation of accountability to affected populations (AAP) implementation necessarily examines IOM's communication with beneficiaries and affected communities—a core dimension of organizational communication capacity. The evaluation likely assesses how IOM communicates program information, solicits feedback, shares results, and ensures affected populations have access to information about their rights and program details. These practices directly relate to IOM's organizational capacity in external communication, beneficiary engagement, and transparency. The report probably contains findings on communication effectiveness, accessibility of information to diverse populations, and mechanisms for two-way dialogue with communities. However, communication systems are not the primary focus; AAP implementation is the dominant theme. The content on beneficiary communication, information accessibility, and feedback mechanisms would add value to a synthesis on communication capacities, particularly regarding IOM's communication with affected populations and marginalized groups. Specialists should review sections addressing how IOM communicates with beneficiaries, ensures information accessibility, and facilitates community voice in organizational processes."},
{'theme_id': '2',
'theme_title': 'Partnership',
'relevance_score': 0.71,
'reasoning': "This report would likely contribute relevant evidence to a synthesis on Enabler 2 (Partnership). The evaluation of accountability to affected populations (AAP) implementation appears to examine how IOM engages with and responds to beneficiary and community stakeholder feedback—a dimension of partnership and stakeholder engagement. AAP frameworks inherently involve partnerships with local actors, community representatives, and affected populations in program design and accountability mechanisms. The report likely contains findings on IOM's capacity to establish two-way communication channels, incorporate local voices in decision-making, and develop equitable engagement with communities—all relating to partnership quality and local actor involvement. However, partnership development and inter-organizational collaboration are not the primary focus; AAP is the dominant theme. The content on stakeholder engagement and community participation mechanisms would add value to a synthesis on partnership capacities, particularly regarding IOM's engagement with local and affected populations. Specialists should review sections addressing how IOM incorporates beneficiary feedback and community participation in program implementation."}][{'theme_id': '1',
'theme_title': 'Integrity, Transparency and Accountability',
'relevance_score': 0.82,
'reasoning': "This report would likely contribute meaningful evidence to a synthesis on Cross-cutting Priority 1 (Integrity, Transparency and Accountability). The evaluation explicitly assesses IOM's delivery on Accountability to Affected Populations (AAP), which is a core institutional commitment within the integrity and accountability framework. The executive summary indicates that the 2023 MOPAN assessment found IOM has not consistently incorporated AAP across programmes and has not invested adequately in AAP relative to operational scale—directly addressing accountability and transparency commitments. The evaluation appears to examine both internal systems and public documentation related to AAP implementation. Key areas likely to be relevant for a synthesis include: the assessment of AAP integration across IOM programmes, analysis of organizational investment in accountability mechanisms, and findings on consistency of AAP implementation. However, while AAP is the primary focus, the report excerpt does not indicate whether broader integrity frameworks, ethical response systems, or transparency mechanisms beyond AAP are examined in depth. A synthesis specialist would benefit from reviewing the findings section on AAP implementation gaps and the recommendations for strengthening accountability systems, though the report's scope appears somewhat narrower than the full breadth of the Priority's institutional commitments (which also encompass organizational culture, ethical frameworks, and result visibility)."},
{'theme_id': '3',
'theme_title': 'Protection-centred',
'relevance_score': 0.74,
'reasoning': "This report would likely contribute relevant evidence to a synthesis on Cross-cutting Priority 3 (Protection-centred). Accountability to Affected Populations (AAP) is closely aligned with protection-centred approaches, particularly the commitment to involve individuals and communities in decision-making and to protect them from harm. The evaluation's assessment of AAP implementation directly relates to IOM's commitment to place migrants' human rights and well-being at the centre of operations and to involve communities in decision-making. The report appears to examine whether IOM has systematically incorporated mechanisms for community participation and accountability—core elements of protection-centred programming. Key findings likely to be valuable for a synthesis would include: the extent to which affected populations are meaningfully involved in programme design and decision-making, the functionality of feedback and complaint mechanisms, and barriers to effective community participation. However, the report excerpt does not explicitly indicate whether the evaluation addresses child safeguarding, PSEAH (sexual exploitation and abuse) prevention, or rights-based approaches more broadly—other important dimensions of the Protection-centred Priority. A synthesis specialist would benefit from reviewing the findings on AAP implementation and community participation mechanisms, though they should note that this report may address only one component of the broader protection-centred commitment."}][{'theme_id': '1a31',
'theme_title': 'Guidelines on data and information collection, sharing and management are in place that adhere to data protection standards, principles of confidentiality and a defined purpose, to protect the individuals and groups providing information from harm.',
'relevance_score': 0.78,
'reasoning': "This report would likely contribute meaningful evidence to a synthesis on Output 1a3.1 (data protection guidelines and standards). The evaluation explicitly assesses IOM's accountability to affected populations (AAP), which inherently involves responsible data collection, management, and protection practices. The executive summary indicates the evaluation reviewed both internal and public documents related to AAP, suggesting substantive examination of how IOM handles beneficiary information and data. Data protection and confidentiality are foundational to AAP frameworks, and the report's focus on whether IOM is 'consistently incorporating AAP across all its programmes' would likely include analysis of data safeguarding mechanisms. However, the excerpt provided does not explicitly foreground data protection guidelines as a primary evaluation focus—AAP is the dominant theme. A synthesis specialist would likely find value in the report's treatment of data protection within the broader AAP context, particularly in sections examining IOM's systems for protecting beneficiary information. Recommend reviewing findings and conclusions sections that address AAP implementation mechanisms and safeguards."}]