Why is GEO testing hard?
Three properties break classical SEO testing instincts. First, non-determinism: the same prompt to the same engine yields different answers across sessions — one citation is an anecdote, not a result. Second, opacity: engines do not announce algorithm changes on a schedule you can align to. Third, shared causality: a visibility change may come from your intervention, a competitor’s move, or an engine update. A testing framework exists to make decisions defensible despite all three.
What are the components of the framework?
- Fixed question set. 50-100 prompts per engine, covering your fan-out tree, frozen for the test window. Changing questions mid-test invalidates everything.
- Repeated sampling. Each prompt run multiple times per cycle; Citation Rate is a proportion across runs, not a single check. See how we measure citation rate.
- Control group. A matched set of pages or topics you do not touch. If test and control both jump, an engine change — not your work — is the likely cause.
- Four-dimension scoring. Visibility, Citation Rate, Semantic Coverage and Brand Perception, each 0-100, per the open standard in measurement.
- Evidence archive. Screenshots and timestamps for every run, so results are auditable.
How do you run an intervention test?
Follow the 90-day POC shape. Week 1: baseline — full question set, both test and control groups, plus a crawler-log snapshot. Weeks 2-10: apply exactly one class of intervention to the test group — say, answer-block rewrites, or structured-data rollout, or source-graph work. Week 12: re-measure with the identical question set and compare score deltas between test and control. Attribute cautiously: a meaningful lift is one that beats the control delta and exceeds run-to-run noise.
What sample sizes and windows work?
Practical guidance: score changes under five points are usually noise; treat them as “no signal”. Run cycles monthly — weekly cycles mostly measure randomness. For per-page tests, group pages by pattern (all comparison pages, all pricing pages) rather than testing singletons, so each cell carries enough prompts to detect movement. And keep the question set stable for at least a quarter before refreshing it, so trends remain comparable.
Common pitfalls
The classic error is testing once and declaring victory: a single post-intervention check that catches a lucky sample. Second, changing everything at once — technical fixes, rewrites and PR in the same window — which makes attribution impossible; one intervention class per cycle. Third, dropping questions that perform badly mid-test, which flatters the scores and destroys the evidence. Fourth, ignoring engine mix: a lift in Perplexity with flat ChatGPT may simply reflect Bing index lag — check the crawler layer before concluding. Finally, keep humans in the loop on Brand Perception scoring; rubric-based judgment beats vibes, as detailed in measuring AI visibility.
Finally, socialize the framework before the results. Walk stakeholders through the rubric, the noise thresholds and the control-group logic at kickoff, not after the first flat month. Tests that surprise people get re-litigated; tests whose rules were agreed in advance get accepted. The framework’s real product is not the score — it is an organization that believes the score enough to act on it.