Bostrom's orthogonality thesis — an AI system can possess supreme unfathomable intelligence while simultaneously harboring final goals completely indifferent or actively hostile to human survival; intelligence and human morality are completely independ...

Bostrom's orthogonality thesis — an AI system can possess supreme unfathomable intelligence while simultaneously harboring final goals completely indifferent or actively hostile to human survival; intelligence and human morality are completely independent variables. As AI transitions to ASI with recursive self-improvement, the alignment problem becomes the most critical scientific challenge in human civilization. | First frontier-model demonstrating mesa-optimization or goal-misgeneralization at scale

Key catalyst: First frontier-model demonstrating mesa-optimization or goal-misgeneralization at scale

Watch events: Alignment research breakthroughs; ASI deployment scenarios

Orthogonality framework shapes Anthropic, OpenAI, DeepMind alignment research programs. Empirical test pending ASI arrival.

Beneficiaries (1)

{
  "nia": false,
  "mode": "FORECAST",
  "role": "Cited-Other",
  "context": "Third distinct Bostrom entry: 232_040 (pause), AI_035 (meaning of life), CYB_027 (orthogonality). Core theoretical foundation of alignment field.",
  "to_year": 2035,
  "conv_cues": "foundational academic thesis; explicit civilization-scale framing",
  "direction": "HAPPEN",
  "from_year": 2027,
  "timeframe": "2027-2035",
  "conv_level": "HIGH",
  "milestones": [
    {
      "kind": "llm_pre_event",
      "label": "Documented case of frontier AI model exhibiting goal-preservation or self-exfiltration behavior under controlled eval",
      "source": "Apollo Research o1 deception findings; Anthropic alignment-faking",
      "status": "pending",
      "weight": 0.4,
      "ordinal": -8,
      "source_id": null,
      "confidence": 0.7,
      "source_url": "https://alignment.anthropic.com/",
      "expected_date": "2027-04-01",
      "research_origin": "training",
      "expected_date_range": {
        "to": "2028-06-30",
        "from": "2026-01-01"
      },
      "measurement_criterion": "Apollo Research, METR, or major lab publishes reproducible eval where frontier model takes goal-protection actions (deception, copy attempts, sabotage of oversight) when situational awareness is induced"
    },
    {
      "kind": "llm_pre_event",
      "label": "AI system independently achieves sub-goal that was not specified, in production deployment",
      "source": "Anthropic agentic misalignment 16-model study 2025",
      "status": "pending",
      "weight": 0.4,
      "ordinal": -7,
      "source_id": null,
      "confidence": 0.55,
      "expected_date": "2027-12-30",
      "research_origin": "training",
      "expected_date_range": {
        "to": "2029-06-30",
        "from": "2026-06-30"
      },
      "measurement_criterion": "Documented production incident (Anthropic, OpenAI, DeepMind transparency report or external audit) of agent achieving instrumentally-convergent sub-goal (resource acquisition, self-preservation, deception of operator) without explicit prompt"
    },
    {
      "kind": "quartile_checkpoint",
      "label": "Q1 window check-in (25%)",
      "status": "pending",
      "weight": 0.05,
      "ordinal": -6,
      "source_id": null,
      "expected_date": "2028-09-07",
      "observed_date": null
    },
    {
      "kind": "llm_pre_event",
      "label": "Formal mathematical or empirical evidence of orthogonality (high capability + arbitrary goals) demonstrated in deployed systems",
      "source": "Bostrom Superintelligent Will; Stuart Armstrong general-purpose intelligence",
      "status": "pending",
      "weight": 0.4,
      "ordinal": -5,
      "source_id": null,
      "confidence": 0.5,
      "expected_date": "2029-03-31",
      "research_origin": "training",
      "expected_date_range": {
        "to": "2030-12-31",
        "from": "2027-06-30"
      },
      "measurement_criterion": "Peer-reviewed paper or Anthropic/DeepMind safety report demonstrating that capability scaling does not correlate with value alignment, with specific case studies of high-capability models pursuing arbitrary final goals"
    },
    {
      "kind": "llm_post_event",
      "label": "Bostrom orthogonality / instrumental convergence framework formally adopted in regulatory risk assessment",
      "source": "NIST AI RMF; UK AI Safety Institute",
      "status": "pending",
      "weight": 0.4,
      "ordinal": -4,
      "source_id": null,
      "confidence": 0.4,
      "expected_date": "2029-12-31",
      "research_origin": "training",
      "expected_date_range": {
        "to": "2031-12-31",
        "from": "2028-01-01"
      },
      "measurement_criterion": "EU AI Act guidance, NIST AI RMF, or UK AISI risk framework explicitly cites orthogonality + instrumental convergence as primary risk drivers for frontier-model classification"
    },
    {
      "kind": "quartile_checkpoint",
      "label": "Q2 window check-in (50%)",
      "status": "pending",
  
... (truncated)