Version the questions when the strategy evolves
Maintain the question set long enough to compare observations and annotate deliberate revisions. Replacing difficult questions with easier branded prompts changes the measurement basis.
Stable questions support comparison; new questions support exploration. Both are useful if separated. Quietly changing wording, brand references or difficulty can make an apparent visibility trend impossible to interpret.
A worked example
A hypothetical team starts with 30 unbranded questions and later adds ten product-name questions. Report the original cohort and the new group separately. The expanded percentage should not be compared directly with the original baseline.
How to apply it
- Assign a persistent identifier and purpose to each question.
- Record wording changes and their effective dates.
- Preserve fixed-cohort results alongside exploratory observations.
What the result can establish
Include failed or unavailable runs under a stated rule. Do not remove inconvenient questions merely to improve the score. A transparent sample supports useful learning without pretending to measure every AI search.
Reference: NIST: AI Risk Management Framework.
Supporting documentation
NIST: AI Risk Management FrameworkGoogle: AI features and websitesGoogle: AI features and your website