/deep-research

Example: Policy Analysis (full mode)

User Request

Research comparing outcomes-based funding models in higher education across OECD countries

Phase 1: SCOPING

research_question_agent Output

Primary Research Question:

What are the design features, implementation strategies, and measured outcomes of performance-based funding (PBF) models in OECD higher education systems, and what contextual factors explain variation in effectiveness?

FINER Assessment:

Sub-questions:

1. What PBF design features (metrics, weights, mechanisms) are used across OECD countries?
2. What outcomes (intended and unintended) have been documented?
3. What contextual factors (governance, culture, market structure) mediate PBF effectiveness?

research_architect_agent Output

Paradigm: Pragmatist Method: Comparative policy analysis with systematic evidence review Framework: Modified Bardach's Eightfold Path + most-similar-systems design Data: OECD reports, national policy documents, academic literature Analysis: Cross-case comparison matrix + thematic synthesis

devils_advocate_agent — CHECKPOINT 1

Verdict: PASS

• Minor: "Effectiveness" needs operational definition — suggest specifying measurable dimensions
• Observation: Country selection will significantly affect conclusions — justify selection criteria

Phase 2: INVESTIGATION

bibliography_agent Output

22 academic articles + 8 OECD/government reports + 3 think tank publications = 33 sources

Key sources include:

• OECD (2019). Benchmarking Higher Education System Performance
• Hillman, N. W., Tandberg, D. A., & Fryar, A. H. (2015). Evaluating the impacts of "new" performance funding in higher education. Educational Evaluation and Policy Analysis, 37(4).
• de Boer, H., et al. (2015). Performance-based funding and performance agreements in fourteen higher education systems. CHEPS.

source_verification_agent Output

33 sources assessed: 28 Grade A-B, 4 Grade C, 1 Grade D (included with caveat)

Phase 3: ANALYSIS

synthesis_agent Output

Comparative Matrix (8 countries x 6 design features x 4 outcome dimensions)

Countries analyzed: US (Tennessee, Ohio), Australia, Denmark, Netherlands, Finland, UK (England), South Korea, Chile

Key Findings:

1. Design convergence: Most PBF models use graduation rates and retention as primary metrics, but weights and mechanisms vary dramatically. Tennessee allocates 85% of state funding based on outcomes (the highest proportion globally), while most European models tie less than 5% to performance. Despite this range, all models converge on a similar set of 4-6 core metrics: completion, retention, research output, and employment.

2. Modest positive effects on targeted metrics: Graduation rates show small positive effects (2-5 percentage points) in most contexts where PBF has been implemented for more than 3 years. Tennessee's model, the most studied, shows a 3.2 percentage point increase in 6-year graduation rates after controlling for student demographics (Hillman et al., 2015). However, several studies note difficulty in attributing causality given concurrent policy changes.

3. Significant unintended consequences: Grade inflation, cream-skimming, narrowing of institutional mission documented in multiple countries. In Australia, the demand-driven system (2012-2017) led to rapid enrollment growth but declining completion rates at some institutions. In the US, evidence from Indiana suggests institutions may have shifted enrollment toward better-prepared students rather than improving outcomes for existing students (Umbricht et al., 2017).

4. Context is king: Governance structure (centralized vs. decentralized) is the strongest moderator of PBF effectiveness. Centralized systems (Denmark, Finland, Netherlands) show more consistent implementation and fewer gaming behaviors than decentralized systems (US states, Australia). The mechanism appears to be: centralized systems can enforce compliance and adjust metrics more rapidly, while decentralized systems create more opportunities for strategic behavior.

5. Equity concerns: Institutions serving disadvantaged students consistently report greater difficulty meeting PBF targets. This is documented across all 8 countries studied, though the magnitude varies. Tennessee partially addresses this through "premium" weights for Pell-eligible students and adult learners, while most European models lack explicit equity adjustments.

Major Contradiction: US state-level studies show mixed results (Tennessee positive, Ohio neutral, Indiana negative), while European models show more consistent modest effects — likely explained by funding proportion (US: 5-25% of budget; Europe: often < 5%).

devils_advocate_agent — CHECKPOINT 2

Verdict: PASS

• Major (downgraded from Critical): Selection of 8 countries needs stronger justification — why not Germany, Japan, Canada?
• Minor: "Context is king" is an assertion — specify which contextual variables and their mechanisms

Phase 4: COMPOSITION

report_compiler_agent Output

Title: Performance-Based Funding in OECD Higher Education: A Comparative Analysis of Design, Outcomes, and Contextual Mediators

Word Count: 6,800 words

Structure:

1. Introduction (600 words): PBF as global policy trend; rationale for comparative analysis
2. Literature Review (1,200 words): Evolution of PBF models from 1970s to present; theoretical foundations (principal-agent theory, new public management, institutional theory)
3. Methodology (800 words): Comparative policy analysis framework; country selection criteria (most-similar-systems with variation on key variables); data sources and analytical approach
4. Findings:
   • 4.1 Design Feature Comparison (1,200 words): Cross-case matrix of 8 countries across 6 design dimensions (metrics, weights, mechanism type, funding proportion, adjustment frequency, equity provisions)
   • 4.2 Outcome Evidence (800 words): Intended effects on graduation, retention, research output
   • 4.3 Unintended Consequences (600 words): Gaming, cream-skimming, mission drift, administrative burden
   • 4.4 Contextual Mediators (600 words): Governance, market structure, institutional culture, data infrastructure
5. Discussion (600 words): Implications for policy design; transferability considerations
6. Recommendations (400 words): 5 evidence-based design principles
   • R1: Keep performance-linked proportion below 10% initially
   • R2: Include explicit equity adjustments from the start
   • R3: Use improvement metrics alongside absolute metrics
   • R4: Build in 3-year review cycles with stakeholder input
   • R5: Invest in institutional data capacity before implementation
7. Limitations & Future Research (300 words)
8. References (33 sources in APA 7.0)

Phase 5: REVIEW

editor_in_chief_agent Output

Verdict: Accept with Minor Revision (Score: 4.1/5.0)

• Strength: Well-structured comparative framework; balanced treatment of evidence
• Minor: Table 2 (design features) needs a notes column to explain country-specific nuances
• Minor: Recommendation #5 (data capacity) needs supporting evidence — currently assertion-based

ethics_review_agent Output

Verdict: CLEARED

• AI disclosure: Present and accurate
• References spot-checked: 7/33 (21%) verified — all confirmed
• Dual-use risk: Low (public policy analysis)
• Fair representation: Adequate — noted absence of non-OECD perspectives as limitation
• Potential bias: Report slightly favors European models over US models; flagged for author awareness

devils_advocate_agent — CHECKPOINT 3

Verdict: PASS

• Observation: Conclusion recommendation #3 ("use improvement metrics alongside absolute metrics") is the strongest recommendation but has the weakest evidence base — only 2 countries have implemented this
• "So what?" test: Passed — clear implications for policymakers designing or reforming PBF systems
• Counterfactual check: What if PBF effects are entirely explained by the Hawthorne effect (attention to metrics, not funding mechanism)? This alternative explanation is not adequately addressed

Phase 6: REVISION

report_compiler_agent (Revision 1)

Changes:

1. Table 2 expanded with notes column explaining country-specific nuances (Minor, Editor)
2. Recommendation #5 supported with evidence from Finnish data infrastructure investment (Minor, Editor)
3. Discussion section expanded to address Hawthorne effect alternative explanation (Observation, Devil's Advocate)
4. Country selection justification strengthened in Methodology — added explicit criteria and acknowledged Germany, Japan, Canada as excluded cases with rationale (Major, Devil's Advocate from Checkpoint 2)

Final Word Count: 6,800 words Revision Loops Used: 1 of 2

Final Output Summary

• Full APA 7.0 report: 6,800 words
• 33 cited sources (Levels I-VII)
• 8-country comparative matrix
• 5 key findings
• 5 evidence-based policy recommendations
• Ethics cleared
• 1 revision loop completed