/agent-observability-experiment-bootstrap

--evaluator-style function (default — what the notebooks use)

Plain Python functions with the signature (input_data, output_data, expected_output). Always emit at least three: a trivial boolean (returns bool), a richer rule-based one (returns EvaluatorResult), and an LLM-as-Judge surrogate (a RemoteEvaluator reference or a placeholder).

Code to emit

from ddtrace.llmobs import EvaluatorResult

# Trivial check — bare bool is fine here, the result has no extra signal.
def exact_match(input_data, output_data, expected_output) -> bool:
    return output_data == expected_output

# Richer check — use EvaluatorResult so reasoning/assessment surface in the UI.
def response_well_formed(input_data, output_data, expected_output) -> EvaluatorResult:
    if not isinstance(output_data, str):
        return EvaluatorResult(
            value=False,
            reasoning=f"output_data was {type(output_data).__name__}, expected str",
            assessment="fail",
        )
    if len(output_data) > 500:
        return EvaluatorResult(
            value=False,
            reasoning=f"output exceeded 500 chars (was {len(output_data)})",
            assessment="fail",
            metadata={"length": len(output_data)},
        )
    return EvaluatorResult(value=True, assessment="pass")

When to extend

• If the user passed --dataset with a structured expected_output, add a JSON-shape check (also returning EvaluatorResult).
• For LLM-as-Judge surrogates, prefer RemoteEvaluator references (server-side, scalable) over inline LLMJudge calls.

When NOT to use this style

If the evaluator needs persistent state (a model client, a cached lookup, an async I/O resource), use class style instead — BaseEvaluator.__init__ is where you set up state safely. See references/python/evaluator-styles/class.md.