/orbit

Loop Execution Learning (REFINE)

Purpose: load this when Orbit analyzes completed loops, adapts defaults, or syncs reusable execution patterns to Lore and Judge.

Contents

1. REFINE workflow
2. Learning triggers
3. Loop Effectiveness Score
4. Adaptation rules
5. Safety guardrails
6. Integration points
7. Templates
8. Regression eval launch

REFINE Workflow

OBSERVE -> MEASURE -> ANALYZE -> IMPROVE -> SAFEGUARD -> JOURNAL

Learning Triggers

Priority:

• RF-02 and RF-03 override lighter triggers.
• RF-01 data is still consumed by a concurrent full or medium cycle.

Loop Effectiveness Score (LES)

LES = Completion_Rate × 0.30
    + Iteration_Economy × 0.25
    + Recovery_Effectiveness × 0.20
    + Contract_Quality × 0.15
    + User_Autonomy × 0.10

Components

Resource Efficiency Bonus

When COST_TRACKING=true, an optional resource efficiency modifier can be applied:

Resource_Efficiency = 1 - (actual_cost / budget_cost)
LES_adjusted = LES * (1 + Resource_Efficiency * 0.05)

This modifier has a maximum impact of ±0.05 on the final LES, rewarding cost-efficient loops without dominating the score.

Grading Scale

LES is valid only after >= 3 completed loops of the same tier. Below that, report INSUFFICIENT_DATA.

Adaptation Rules

Allowed Scope by Grade

Adaptation Types

Adaptive Timeout Algorithm

When ADAPTIVE_TIMEOUT=true, timeout values are dynamically adjusted based on historical execution data.

Algorithm

effective_timeout = max(
  EXEC_TIMEOUT,
  min(
    moving_average(last_N_durations) + 1.5 * stddev(last_N_durations),
    EXEC_TIMEOUT * 3
  )
)

Integration with REFINE Cycle

The adaptive timeout feeds into the REFINE cycle at the OBSERVE and ANALYZE phases:

Tier-Specific Bounds

Safety Rules

• Adaptive timeout is disabled until >= 3 timing data points exist.
• If 3 consecutive iterations hit the adaptive ceiling, emit [ADAPTIVE:WARN] and propose tier upgrade.
• Timeout data is stored in ${LOOP_DIR}/.iter-timings.log (one duration per line in seconds).
• The REFINE cycle may propose changing ADAPTIVE_TIMEOUT from false to true when sufficient data exists (RF-01 lightweight record).

Application process:

1. propose
2. validate
3. snapshot
4. apply
5. monitor next 3 executions

Safety Guardrails

Rollback Protocol

1. detect regression when LES drops >= 0.05
2. restore the snapshot
3. record the rollback event and cause
4. share the negative pattern with Lore

Integration Points

Lore Sync Protocol

ORBIT_TO_LORE_PATTERN:
  type: LOOP_EXECUTION_PATTERN
  pattern_name: <name>
  source_data:
    tier: <tier>
    sample_size: <int>
    les_impact: <float>
  pattern_detail: <description>
  confidence: <HIGH | MEDIUM | LOW>
  actionable: <bool>

Templates

Feedback Record

## Loop Feedback Record — <loop_id>

**Date:** YYYY-MM-DD
**Tier:** <Light | Standard | Heavy | Marathon>
**Final Status:** <DONE | BLOCKED | MAX_ITER | USER_ABORT>
**Iterations:** <actual> / <max>
**LES:** <score> (Grade: <A-F>)

Adaptation Log

## Adaptation Log — <date>

**Trigger:** <RF-xx>
**Current LES:** <score> (Grade: <grade>)
**Target LES:** <expected score>

### Change
- **Type:** <adaptation type>
- **Detail:** <specific change>
- **Rationale:** <why this change>

### Safeguard Check
- Anti-pattern compliance: PASS / FAIL
- Pattern consistency: PASS / FAIL
- Evidence threshold (≥ 3): PASS / FAIL
- Snapshot saved: <path>

### Outcome (post-monitoring)
- LES after 3 executions: <score>
- Rollback needed: YES / NO

Regression Eval Launch

Use this when a loop fires RECOVER, CIRCUIT_OPEN, CONVERGENCE_STALL, OSCILLATION_LOOP, VALIDATOR_GAP, REWARD_HACK, GOAL_DRIFT, BURN_RATE_ANOMALY, or PERMISSION_HIJACK. The failed trace is more valuable than any synthetic test.

Steps:

1. Snapshot the trace — capture the failing iteration's progress.md entry, state.env, last EXEC_TIMEOUT worth of runner.log, and any decisions.md delta.
2. Distil to a single behavioural assertion — rewrite the failure as a one-line claim the next runner must satisfy. Example: loop must abort within 30s when tests/ file mtime changes during an iteration (AP-13 regression).
3. Append to .agents/orbit-eval.md under a section keyed by failure class. This file is append-only; never edit historical entries.
4. Wire into next GENERATE — when generating a new loop in the same project, read .agents/orbit-eval.md and inject the relevant behavioural assertions into the generated verify.sh as additional checks.
5. Trajectory-level eval, not final-output eval — record the sequence of actions that led to the failure, not just the final state. Final-output checks miss OSCILLATION_LOOP and BURN_RATE_ANOMALY, which are sequence-shaped failures. [Source: augmentcode.com — Best AI Agent Evaluation Tools; augmentcode.com — Agent Observability for AI Coding]

Eval entry format:

### <failure-class> | <ISO8601 first observed> | severity <P0|P1|P2>

**Trigger conditions:** <what state of the loop produced this failure>
**Behavioural assertion:** <one-line claim the next runner must satisfy>
**Verify check (bash one-liner or test name):** <executable check>
**Source incident:** loop-dir `<path>`, iter `<id>`
**Mitigation status:** active | retired (date) | superseded by <other entry>

Lifecycle:

• Each entry stays active until 3 consecutive future loops in the same project pass the assertion without intervention.
• After 3 clean passes, mark retired and date-stamp; do not delete (kept for audit).
• If a later incident matches a retired entry, reactivate and append a delta entry referencing the original.

Connect to RF-02: when RF-02 fires (same tier blocked 3+ times), the eval launch is the first step of the full REFINE cycle.

Recovery Metrics

Learning Guardrails

LES valid only after ≥ 3 completed loops of the same tier; LES ≥ B requires human approval; max 3 parameter changes per session with a pre-adaptation snapshot; roll back if LES drops ≥ 0.05; Lore sync mandatory for reusable patterns; staged-autonomy rollout (sandbox → gated tools → monitoring → full). Detail → reference/loop-learning.md, reference/resilience-patterns.md.

Interaction and Learning Triggers

Priority: RF-02/RF-03 override lighter triggers; RF-01 data is still consumed by a concurrent full/medium cycle. Full trigger detail → reference/loop-learning.md.