/deep-research

Logical Fallacies Catalog — 30+ Fallacies for Research Review

Purpose

Reference catalog of logical fallacies commonly encountered in research. Used by the devils_advocate_agent.

Formal Fallacies (Invalid Logical Structure)

1. Affirming the Consequent

Structure: If P then Q; Q is true; therefore P is true. Example: "If a university has high research funding, it has good outcomes. This university has good outcomes. Therefore, it must have high research funding." Problem: Q can have multiple causes.

2. Denying the Antecedent

Structure: If P then Q; not P; therefore not Q. Example: "If enrollment increases, revenue increases. Enrollment didn't increase. Therefore, revenue didn't increase." Problem: Revenue can increase from other sources.

3. Undistributed Middle

Structure: All A are B; All C are B; therefore All A are C. Example: "All successful programs use technology. Our program uses technology. Therefore, our program is successful." Problem: B (technology use) is shared but doesn't link A and C.

4. False Dilemma / False Dichotomy

Structure: Either A or B; not A; therefore B. Example: "Either we adopt online learning completely or maintain traditional methods." Problem: Many hybrid options exist.

Informal Fallacies

Relevance Fallacies

5. Ad Hominem

Description: Attacking the person rather than the argument. Research Example: "This study's conclusions are unreliable because the author works for a for-profit university." Correct Approach: Evaluate the methodology and evidence, not the author's affiliation (though COI should be noted).

6. Appeal to Authority

Description: Accepting a claim solely because an authority figure endorses it. Research Example: "Published in Nature, so the findings must be valid." Correct Approach: Even prestigious journals publish flawed studies. Evaluate on merit.

7. Appeal to Tradition

Description: Arguing something is correct because it has always been done that way. Research Example: "This metric has been used for 30 years, so it must be the best measure." Correct Approach: Evaluate whether the metric is still valid in current context.

8. Appeal to Novelty

Description: Arguing something is better because it's new. Research Example: "This new framework must be superior to the established one." Correct Approach: Novelty doesn't equal improvement. Compare on evidence.

9. Appeal to Popularity (Bandwagon)

Description: Arguing something is true because many people believe it. Research Example: "Most researchers in the field use this method, so it must be the best." Correct Approach: Popularity doesn't validate methodology. Assess independently.

10. Red Herring

Description: Introducing an irrelevant topic to divert from the argument. Research Example: Responding to criticism of methodology by discussing the importance of the topic.

Evidence Fallacies

11. Cherry-Picking (Selection Bias)

Description: Selecting evidence that supports the conclusion while ignoring contradictory evidence. Research Example: Citing 5 studies that support the hypothesis while omitting 12 that don't. Detection: Compare cited sources against comprehensive search results.

12. Confirmation Bias

Description: Seeking, interpreting, and remembering information that confirms pre-existing beliefs. Research Example: Designing search terms that are more likely to return supportive results. Detection: Check if search strategy was neutral; look for actively sought disconfirming evidence.

13. Survivorship Bias

Description: Drawing conclusions only from "survivors" (successes), ignoring those that didn't survive. Research Example: "All top-ranked universities implemented X" — ignoring universities that implemented X and didn't improve. Detection: Ask "what about those that failed?"

14. Anecdotal Evidence

Description: Using individual stories as proof of a general claim. Research Example: "One university tripled enrollment after rebranding, so rebranding drives enrollment." Detection: Is this a systematic finding or an isolated case?

15. Hasty Generalization

Description: Drawing broad conclusions from insufficient evidence. Research Example: "Three case studies from Taiwan show X, therefore this applies to all Asian universities." Detection: Is the sample representative? Is the generalization proportionate to the evidence?

Causal Fallacies

16. Post Hoc Ergo Propter Hoc

Description: Assuming that because B followed A, A caused B. Research Example: "After implementing the new curriculum, graduation rates improved. Therefore the curriculum caused the improvement." Detection: Were there confounders? Was there a control group?

17. Cum Hoc Ergo Propter Hoc (Correlation ≠ Causation)

Description: Assuming that correlation implies causation. Research Example: "Universities with more international students have higher rankings, so international students cause higher rankings." Detection: Is there a plausible mechanism? Could both be caused by a third factor?

18. Reverse Causation

Description: Getting cause and effect backwards. Research Example: "Good facilities attract students" when actually "student fees fund better facilities." Detection: Consider temporal order and alternative causal directions.

19. Ecological Fallacy

Description: Inferring individual-level conclusions from group-level data. Research Example: "Countries with more education spending have higher GDP, so spending on education makes individuals richer." Detection: Are individual-level and group-level relationships the same?

20. Simpson's Paradox

Description: A trend present in subgroups reverses when groups are combined. Research Example: Department A and B both show improving retention, but the university overall shows declining retention (due to shifting enrollment proportions). Detection: Always check disaggregated data alongside aggregate.

Reasoning Fallacies

21. Straw Man

Description: Misrepresenting an opponent's argument to make it easier to attack. Research Example: Critic says "this method has limitations" → Author responds "my critic says the entire study is worthless." Detection: Does the refutation address the actual criticism?

22. Moving the Goalposts

Description: Changing the criteria for success after seeing results. Research Example: Defining "program success" as enrollment growth, then shifting to "student satisfaction" when enrollment drops. Detection: Were success criteria pre-defined?

23. Slippery Slope

Description: Arguing that one action will inevitably lead to an extreme outcome. Research Example: "If we allow flexible admission criteria, academic standards will collapse entirely." Detection: Is each step in the chain actually probable?

24. Circular Reasoning (Begging the Question)

Description: The conclusion is assumed in the premise. Research Example: "This university is excellent because it is highly ranked, and it is highly ranked because it is excellent." Detection: Does the argument depend on the truth of what it's trying to prove?

25. No True Scotsman

Description: Redefining a category to exclude counterexamples. Research Example: "All quality assurance systems improve outcomes." "But system X didn't." "Well, X wasn't a true quality assurance system." Detection: Is the definition being modified to fit the claim?

26. Equivocation

Description: Using a term in two different senses within the same argument. Research Example: "Quality" used sometimes to mean "standards compliance" and sometimes to mean "student satisfaction." Detection: Is the key term defined consistently throughout?

Statistical Fallacies

27. Base Rate Neglect

Description: Ignoring the base rate (overall probability) in favor of specific information. Research Example: "This program has a 90% satisfaction rate" — but the base rate for all programs is 88%. Detection: Always compare against relevant base rates.

28. Regression to the Mean

Description: Extreme performances naturally tend back toward average on subsequent measurements. Research Example: "Our intervention improved scores for the lowest-performing students" — they may have improved anyway. Detection: Was there a control group? Were initial measurements extreme?

29. Texas Sharpshooter

Description: Finding a pattern in random data by focusing on clusters and ignoring misses. Research Example: Running 20 statistical tests and reporting only the 1 that was significant. Detection: Were hypotheses pre-registered? Was multiple testing corrected for?

30. Gambler's Fallacy

Description: Believing past random events influence future random events. Research Example: "This institution has declined for 5 years, so it's due for improvement." Detection: Is there a causal mechanism for reversal, or is this just pattern-seeking?

31. McNamara Fallacy (Quantitative Bias)

Description: Making decisions based solely on quantitative metrics while ignoring qualitative factors. Research Example: Ranking universities only by publication counts, ignoring teaching quality and community impact. Detection: Are important but hard-to-measure factors being excluded?

32. Goodhart's Law

Description: "When a measure becomes a target, it ceases to be a good measure." Research Example: Universities gaming rankings metrics instead of genuinely improving quality. Detection: Has the metric become a target? Are there signs of metric manipulation?

Quick Reference: Detection Questions