/scientific-problem-selection

SKILL 3: Optimization Function Selection

Overview

This skill helps scientists articulate HOW their project should be evaluated and define what success means. While Skill 2 focused on likelihood of success (the X-axis), this skill focuses on impact if successful (the Y-axis). The key insight: value is in the eye of a belief system—the value creation framework must be explicitly stated and led with.

Core Principle

"Pick the right optimization function."

Different types of projects should be evaluated by different metrics. A common source of conflict between trainees and PIs, or authors and referees, is a misunderstanding about which category a project falls under. The root cause is often failure to articulate evaluation criteria clearly.

The Fundamental Truth

The default state of:

1. Every new discovery is irrelevance
2. Every new technology is non-use
3. Every company is death

Scientists must actively work against these defaults by choosing the right metrics and scoring well on at least one axis.

The Skill Workflow

Phase 1: Project Categorization (5 minutes)

First, Claude should determine what type of project the user is pursuing:

Question 1: What is the primary goal? A. Understand how biology works (fundamental knowledge) B. Enable new experiments or capabilities (tool/technology) C. Solve a practical problem (invention/application) D. Something else (please describe)

Question 2: What would "success" look like in 3-5 years?

• 1-2 sentences describing the ideal outcome

Question 3: Who cares if this succeeds?

• Academic researchers in the subfield?
• Broader scientific community across fields?
• Clinicians or practitioners?
• Industry partners or companies?
• General public or specific communities?
• All of the above?

Based on the answers, Claude should help identify the right optimization function.

Phase 2: Understanding the Three Main Frameworks

Framework 1: Basic Science

Axes: How much did we learn? × How general/fundamental is the object of study?

Philosophy: A high score on EITHER axis yields substantial impact. You don't need both.

Examples:

• High Generality, Medium Learning: Ribosome stalling complex

  • Updates understanding of translation (fundamental process)
  • Scores well because translation is universal

• Medium Generality, High Learning: Oxytricha germ-line nucleus

  • Genomic acrobatics may not be common to other organisms
  • BUT elegant mapping scores highly on how much we learned
  • May yield tools for genome editing (bonus)

• High on Both Axes (Landmark): RNA interference, biomolecular condensates

  • These are rare—don't expect every project to be here
  • But aim to score well on at least one axis

Key Questions:

• How many systems/organisms does this apply to?
• Does it update understanding of a fundamental process?
• Will textbooks need to be rewritten?
• What new questions does this open?

Framework 2: Technology Development

Axes: How widely will it be used? × How critical is it for the application?

Philosophy: Again, high score on EITHER axis is sufficient.

Examples:

• Widely Used, Not Critical: BLAST

  • Used in countless projects
  • Rarely THE critical tool, but enormous cumulative impact

• Not Widely Used, Highly Critical: Cryo-electron tomography

  • Too complicated for broad adoption
  • But generates stunning data that's impossible to get otherwise
  • When you need it, nothing else works

• High on Both Axes (Game-Changing):

  • GFP, CRISPR, AlphaFold (the famous ones)
  • But also: lentiviral delivery, cell sorting, massively parallel sequencing
  • Technologies we cannot imagine living without

Key Questions:

• How many labs would adopt this?
• For what fraction of experiments is this THE enabling technology?
• What becomes possible that wasn't before?
• How hard is it to implement?

Critical Rule: A tool that won't be widely used AND isn't critical for an application probably isn't worth building.

Framework 3: Typical Invention/Application

Axes: How much good? × For how many people?

Philosophy: Useful for translational work, frugal science, global health.

Examples:

• Foldscope: Paper microscope accessible to millions of students globally
• Neglected tropical disease intervention: Quality-adjusted life years per $100
• Medical device: Number of patients who can access treatment

Key Questions:

• What problem does this solve?
• How many people have this problem?
• How much better is their life if you solve it?
• What's the cost per person helped?

Phase 3: Selecting and Articulating Your Framework

Based on your Phase 1 responses, let me help you choose:

If you selected A (fundamental knowledge): → Basic Science Framework If you selected B (enable experiments): → Technology Development Framework
If you selected C (solve practical problem): → Invention Framework

Now, let's be explicit:

1. State Your Framework: "This project should be evaluated as [basic science/technology development/invention]."

2. Define Your Axes:

   • X-axis measures: [specific metric]
   • Y-axis measures: [specific metric]

3. Make Your Case:

   • X-axis score (Low/Medium/High): [Your assessment + reasoning]
   • Y-axis score (Low/Medium/High): [Your assessment + reasoning]

4. Threshold Check:

   • Do you score at least MEDIUM-HIGH on one axis?
   • If both are LOW-MEDIUM, you have a problem

Phase 4: Alternative or Custom Metrics

Sometimes standard frameworks don't fit. Examples where custom metrics work:

Alternative Metric Examples:

• Frugal Science: How many children in low/middle-income countries gain access to microscopy?
• Neglected Disease: Quality-adjusted life years saved per $100 invested
• Sustainability: Tons of CO₂ equivalent prevented × cost-effectiveness
• Equity: Reduction in disparity metric × number of people affected

When to propose alternative metrics:

• Your work addresses a specific underserved need
• Standard metrics miss your core value proposition
• You're working in an emerging area without established norms
• Your work crosses traditional boundaries

How to propose alternative metrics:

1. Explain why standard metrics are insufficient
2. Define your proposed metric clearly
3. Provide a value creation index (two axes)
4. Show how your project scores on these axes

Phase 5: Comparative Assessment

Even if absolute impact is hard to estimate, comparative assessment is valuable:

Exercise: Compare 3 Related Projects

For your project and two alternatives (either from literature or hypothetical):

Comparative Questions:

• Which would be most impactful if they all work?
• Which has the best risk-adjusted impact?
• Are you pursuing the best option?
• If not, why? (Sometimes there are good reasons: resources, expertise, timing)

Phase 6: Avoiding Metric Mismatch

Common Mismatches:

Mismatch 1: Basic Science vs. Technology Evaluation

Scenario: You're doing fundamental biology, but reviewers ask "How widely will this be used?"

Problem: They're evaluating basic science with technology metrics

Solution: Explicitly frame as basic science. Lead with: "This updates our understanding of [fundamental process], which is conserved across [many systems]."

Mismatch 2: Technology vs. Basic Science Evaluation

Scenario: You're building a tool, but reviewers ask "How much did we learn about biology?"

Problem: They're evaluating technology with basic science metrics

Solution: Explicitly frame as technology development. Lead with: "This enables experiments that are currently impossible, which [X] labs need for [Y] applications."

Mismatch 3: Within-Category Confusion

Scenario: Your basic science is specific but deep, but reviewers want broad generality

Problem: They think both axes are required, rather than either/or

Solution: Explicitly acknowledge: "While this may not be universal, the depth of mechanistic insight scores highly on the learning axis."

Mismatch 4: Time Horizon Mismatch

Scenario: You're working on long-term fundamental research, but reviewers want immediate impact

Problem: Different value systems about when impact should materialize

Solution: Articulate your time horizon explicitly and provide historical examples of similar timelines

Phase 7: Value System Discussion

This is where Claude explicitly discusses the user's belief system about what matters:

Questions for Reflection:

1. What drives the user?

   • Discovery and understanding?
   • Enabling others?
   • Solving problems?
   • Building things?

2. What would make the user proud?

   • Paper in Cell/Nature/Science?
   • Tool used by hundreds of labs?
   • Treatment reaching patients?
   • Opening a new field?

3. How does the user want to be remembered?

   • "Discovered X"
   • "Built Y that enabled Z"
   • "Solved problem W"
   • "Trained students who went on to..."

4. Whose approval matters?

   • Specific senior scientists in the field?
   • Broader community across fields?
   • Practitioners who use tools?
   • People whose lives are improved?

There are no wrong answers—but alignment matters:

• The project should match the user's value system
• The evaluation framework should match the project type
• Communication should lead with the framework

Phase 8: Literature Benchmarking

Claude should use PubMed to benchmark impact in the user's area:

Searches should include:

1. Impact Exemplars: Papers the user considers high-impact in the field

   • What framework did they use (implicitly or explicitly)?
   • How did they score on the axes?
   • What made them successful?

2. Analogous Projects: Similar approaches or systems

   • How were they evaluated?
   • What impact did they achieve?
   • What can be learned from their framing?

3. Field Expectations: What's typical for the area?

   • Are basic science papers common?
   • Is technology development valued?
   • What level of impact is "good enough"?

Questions to ask the user:

• What papers should be analyzed as benchmarks?
• What search terms capture the field's impact exemplars?
• Are there specific journals or authors whose framing to emulate?

Phase 9: Communication Strategy

Once the framework is selected, here's how to lead with it:

In Talks:

Opening Frame (within first 2 slides):

• "The goal of this work is to understand [fundamental process X] in [general system Y]" → Basic science
• "We're developing a technology that will enable [critical experiment X] for [community Y]" → Technology
• "This invention addresses [problem X] affecting [N] people" → Application

In Papers:

Abstract Structure:

• State your framework implicitly through word choice
• Basic science: "reveals," "demonstrates," "shows that"
• Technology: "enables," "provides," "makes it possible to"
• Application: "solves," "addresses," "improves"

In Grants:

Broader Impact Section:

• Explicitly name your evaluation framework
• Provide the two-axis assessment
• Score yourself on each axis with evidence

With Your PI/Committee:

Alignment Conversation:

• "I want to make sure we're aligned on how this should be evaluated"
• "I see this as [framework], scoring [X] on [axis 1] and [Y] on [axis 2]"
• "Do you agree, or do you see it differently?"
• "This matters because..." [explain downstream implications]

Output Deliverable

Claude should produce a 2-page Impact Assessment Document:

Page 1: Framework and Scoring

Project Categorization:

• Type: Basic Science / Technology Development / Invention / Custom
• Rationale: [Why this categorization fits]

Optimization Function:

• X-Axis: [Metric name and definition]
• Y-Axis: [Metric name and definition]
• Custom Rationale (if applicable): [Why standard metrics don't fit]

Self-Assessment:

X-Axis Score: [Low/Medium/High]

• Evidence: [Specific reasons for this score]
• Examples: [Comparable projects or benchmarks]
• PubMed Support: [Key papers that inform assessment]

Y-Axis Score: [Low/Medium/High]

• Evidence: [Specific reasons for this score]
• Examples: [Comparable projects or benchmarks]
• PubMed Support: [Key papers that inform assessment]

Overall Assessment:

• Score on at least one axis: ☑ Yes / ☐ No
• Strong justification: ☑ Yes / ☐ No
• Aligned with your values: ☑ Yes / ☐ No

Visual Framework:

         [Your Project Type]
              
Y-Axis    |           ★ Your Project
[Metric]  |         /
          |       /
          |     /
          |   /
          |_________________
              X-Axis [Metric]
              
★ = Your project
Reference projects plotted for context

Page 2: Communication and Alignment

Value System Alignment:

• What Drives You: [Discovery/Enabling/Problem-solving/Building]
• Success Definition: [What would make this worthwhile]
• Approval Sources: [Whose opinion matters and why]
• Framework Fit: [How project aligns with values]

Potential Mismatches to Avoid:

1. [Specific mismatch type]

   • Scenario: [When this might happen]
   • Prevention: [How to frame to avoid it]

2. [Another mismatch]

   • Scenario: [When this might happen]
   • Prevention: [How to frame to avoid it]

Communication Strategy:

For Talks:

• Opening frame: [Exact language to use in first 2 slides]
• Key phrases: [Vocabulary that signals your framework]

For Papers:

• Abstract structure: [Framework-appropriate language]
• Impact statement: [How to articulate in discussion]

For Grants:

• Broader impact: [How to score yourself explicitly]
• Justification: [Evidence for scores]

For Mentors:

• Alignment question: [Exact question to ask]
• Your perspective: [How you see it]
• Discussion points: [What matters for alignment]

Comparative Analysis:

Action Items:

1. [Specific step to strengthen X-axis score or argument]
2. [Specific step to strengthen Y-axis score or argument]
3. [Communication alignment with key stakeholders]

Practical Examples

Example 1: Ribosome Stalling (Basic Science)

• Framework: Basic science
• X-Axis (Generality): HIGH—translation is universal
• Y-Axis (Learning): MEDIUM—mechanism of one quality control system
• Assessment: High on generality alone = substantial impact
• Communication: "Updates our understanding of translation quality control"

Example 2: BLAST (Technology)

• Framework: Technology development
• X-Axis (Widely Used): VERY HIGH—used by virtually all molecular biologists
• Y-Axis (Critical): LOW-MEDIUM—helpful but rarely essential
• Assessment: Extreme breadth of use = enormous cumulative impact
• Communication: "Enables rapid sequence comparison across all biological databases"

Example 3: Cryo-EM Tomography (Technology)

• Framework: Technology development
• X-Axis (Widely Used): LOW—complex, expensive, specialized
• Y-Axis (Critical): VERY HIGH—generates impossible-to-get-otherwise data
• Assessment: Extreme criticality for niche = high impact
• Communication: "Enables 3D visualization of molecular machines in native cellular context"

Example 4: Foldscope (Invention)

• Framework: Invention (custom metric)
• X-Axis (Good): MEDIUM—functional microscopy
• Y-Axis (People): VERY HIGH—millions of students globally
• Assessment: Massive reach × modest utility = transformative for education
• Communication: "Democratizes microscopy for global education"

Key Principles to Remember

1. Value Is in the Eye of a Belief System: Make yours explicit.

2. Lead with Your Metric: Don't assume others share your framework.

3. Either Axis Suffices: You don't need both—just score well on one.

4. Articulate Early: Discuss with mentors before you're 2 years in.

5. Avoid Default State: Work actively against irrelevance/non-use.

6. Compare, Don't Absolute: Even rough comparison beats ignoring impact.

7. Align Communication: Your words should signal your framework.

8. Match Project to Values: Life is too short for misaligned work.

Warning Signs

Warning signs include:

• Inability to articulate which framework applies
• Scoring LOW on both axes
• Project type and evaluation framework don't match
• User and PI have different frameworks but haven't discussed it
• Using basic science metrics for a tool or vice versa
• Never explicitly discussing impact assessment

Good shape indicators:

• Clear statement of optimization function
• MEDIUM-HIGH score on at least one axis
• Framework matches project type
• Alignment with key stakeholders
• Communication signals framework clearly
• Benchmarking against comparable work

Getting Started

Claude should begin Phase 1 by asking:

1. What is the primary goal? (A/B/C/D)
2. What would success look like in 3-5 years?
3. Who cares if this succeeds?

Together, Claude and the user will select the right optimization function and position the work for maximum impact.

Remember: Impact assessment isn't about ego—it's about ensuring work matters in the way the scientist wants it to matter. Explicit framing prevents years of misalignment.