{"id":"Q-3","type":"question","title":"How stable are AI evaluator recommendations across model versions, prompt phrasings, and repeated sampling?","created":"2026-07-13","status":"open","authors":["Upstream Zero"],"edges":[],"commercialRelevance":{"affectedBuyers":"Anyone relying on an AI evaluator's recommendation as if it were a stable fact","affectedCategories":"All categories surfaced in AI-assisted comparison and screening","potentialProductImpact":"If recommendations are noisy, single-shot \"AI visibility\" measurements are largely meaningless","currentConfidence":"Unmeasured by us. Instrument instability is a known risk (see FOUNDATIONAL_UNDERSTANDING, research risks)."},"whatWouldCountAsAnAnswer":"Published variance measurements: recommendation agreement rates across paraphrased prompts, temperature settings, sampling repeats, and model version changes, per evaluator class.","body":"The instrument-stability question. A finding that is true in March and\nfalse in May — with no announcement — is not a finding; it is a dated\nobservation. Before the observatory publishes anything about *what*\nevaluators recommend, it must characterize how *stable* those\nrecommendations are. This question gates the credibility of every future\nmeasurement, including our own commercial offerings.","url":"/questions/Q-3","machineUrl":"/objects/Q-3","referencedBy":[],"_meta":{"site":"Upstream Zero — Commercial Evaluation Observatory","version":"0.1","note":"Claims are presented at their evidence tier; Narrated is the lowest. Verify by walking edges, not by trusting us."}}