$ cat Research Studies & Results
A reproducible archive-search evaluation with frozen data and measured results, plus four unexecuted external search and AI study protocols.
Known-tool retrieval and text alignment
We evaluated the archive’s preserved v9 index against a frozen corrected v9.1 index using the same search algorithm and all 45 tool-card IDs shared by both editions. This measures our local search software on actual archive data. It does not measure Google rankings, AI citations or the efficacy of a manipulation technique.
| Predeclared metric | Preserved v9 | Corrected snapshot |
|---|---|---|
| Target record’s own text contains every query token | 2 / 45 (4.4%) | 45 / 45 (100%) |
| Exact target is the first result | 42 / 45 (93.3%) | 45 / 45 (100%) |
| Exact target appears within five results | 42 / 45 (93.3%) | 45 / 45 (100%) |
| Mean reciprocal rank | 0.933333 | 1.000000 |
Interpretation: matching titles hid misattached body text in the old index. The corrected snapshot returns the right target with aligned text for every matched record. Design: queries use current tool names; two removed venue records and three newly separated product cards are excluded because no stable paired ID exists. The protocol and input hashes were frozen before evaluation, after the author already knew the defect. The author also made the corrections; this is not independent or blind validation.
Limits: content and indexing code both changed, so these differences do not isolate a causal code-fix effect. Name queries are easier than natural research questions. Token presence is a narrow test of text alignment. The 45 records are the complete matched set, not a random sample of readers or queries; no general-population confidence interval is claimed. Later site edits do not alter the retained snapshots.
Download study, data and code · Protocol and SHA-256 hashes · All paired observations · Method and rerun instructions
node run-study.cjs
# Checks frozen input hashes; writes run-output/results.jsonThese four protocols describe questions that the archive has not answered empirically. They contain no measured ranking effects, citation effects or original field-study findings. The earlier one-target link comparison, duplicate-page citation comparison, same-domain content experiment and “CTR alone” interpretation were not adequate causal designs. The revisions below state the unit of analysis, controls, required records and remaining limits before any study is attempted.
Sample sizes, stopping rules and analysis plans still need to be fixed using a justified design before collection. No expected numerical result is supplied. The six offline demonstrations have actual local outputs, but their invented inputs provide no evidence about the following public-web hypotheses. A protocol is not an experiment result.
Every future study needs a dated protocol, access authorization, sampling frame, treatment assignment or selection rules, raw records, exclusion log and analysis code. Use the blank observation record as a starting structure. Preserve failed retrievals, null results and contradictory observations. Register the primary outcome and stopping rule before looking at treatment results.
Study 1 · What can observations say about linking-page traffic?
- Question
- Among naturally occurring editorial links, is a linking page's measured audience associated with a target's later search visibility after accounting for recorded differences?
- Hypothesis
- An association may exist; its direction and size are unmeasured here. Traffic, topical relevance, publication quality and indexing can co-vary.
- Unit and sample
- Use independent target-page/source-link observations from authorized records, with dates and an explicit sampling frame. Exclude units where both compared sources feed the same target outcome, or model their combined exposure without assigning individual credit. Cluster uncertainty by domain and target where appropriate.
- Comparison
- Match or stratify sources by topic, publication period, indexation and observable audience differences. Include target histories and contemporaneous comparison targets. Record additional links, content edits and update windows. Prefer pre-existing editorial events; the design does not call for purchased links, manufactured traffic or link networks.
- Measurement
- Fix target/query groups, source audience measure, baseline window, follow-up window and visibility outcome in advance. Preserve timestamps and the underlying data rather than only an aggregate rank screenshot. Do not use an unopposed invented query as evidence of competitive ranking effects.
- Analysis and limits
- Estimate a conditional association with uncertainty and sensitivity to matching choices. Unmeasured quality, selection, crawl timing and site-level interference remain. This observational design cannot identify Google's internal link tiers or establish that traffic caused link value.
- Result
- Not run. No dataset or estimated effect exists. Lab 1 demonstrates why one target receiving two links cannot separate their effects.
Study 2 · Does presentation relate to AI-search citation?
- Question
- For a defined product, query population and time period, does answer-first presentation change the probability that an eligible source is cited?
- Unit and assignment
- Use distinct, useful topic/query units with one canonical version each. Match units by topic and baseline retrieval, then randomize presentation within blocks. Two near-duplicate pages competing for the same query on one domain are not independent treatment units. Assign at domain level when site-wide effects could contaminate a page-level comparison.
- Controls
- Hold facts, author/source attribution, publication eligibility and substantive information constant. Record canonical/indexation status. Avoid updating both content quality and presentation in the same treatment. Keep a stable comparison group and predeclare how unavailable or ineligible features enter the denominator.
- Sampling
- Define the query set and repeated observation schedule before collection. Record the exact product (for example a search feature rather than a base chatbot), model/version when exposed, timestamp, locale, device, session state and full answer. Separate products in the analysis; repeated responses for one query are clustered observations.
- Outcome
- Primary: fraction of scheduled eligible runs containing the assigned source URL. Count one source at most once per run; retain uncited runs. Secondary measures such as time to first citation must be declared separately. Report confidence intervals and all exclusions.
- Limits
- Discovery delays, canonicalization, output variability and product changes can confound results. Randomization improves the design only if assignment, compliance and independence assumptions are defensible. Results would concern the evaluated query population and product version, not “GEO” generally.
- Result
- Not run. The citation-scoring exercise verifies counting on six fictional records, not this hypothesis.
Study 3 · What changes after an editorial quality review?
- Question
- How do authorized existing pages' reader-quality assessments and visibility change after a documented editorial review?
- Scope
- Audit an existing corpus instead of publishing a new batch of low-value pages. Identify its original production method only where documented; do not infer AI authorship from a detector score.
- Unit and comparison
- Use site/topic clusters rather than assuming fifty pages on one domain are fifty independent sites. Compare reviewed clusters with comparable, unchanged clusters when possible. Record exposure to shared templates, navigation, links and domain-wide changes.
- Treatment description
- Log each intervention separately: factual corrections, sources, original evidence, clarity, accessibility and relevance. These form an editorial package; its effect cannot be attributed solely to an “AI versus human” distinction.
- Outcomes
- Predefine blind quality-review criteria and a search-performance outcome with a baseline and follow-up period. Retain removals, redirects and deindexed pages with explicit outcome codes. Record core/spam update timing as context, not as a classifier label for individual pages.
- Analysis and limits
- Compare changes with uncertainty clustered at the appropriate level; test whether baseline trends are comparable before using difference in differences. Selection into editing, topic demand, updates and domain spillovers may remain. A visibility change does not reveal whether a particular Google policy classifier fired.
- Result
- Not run. No survival percentage or causal effect is reported. Policy documentation is evidence about the rule; it is not an experimental measurement.
Study 4 · What is the total effect of a title/snippet revision?
- Question
- For a defined page/query population, does an accurate title and description revision change search clicks and impressions relative to an unchanged comparison group?
- Unit and assignment
- Match pages by topic, baseline impressions, position and query mix. Randomize revisions where traffic and independent units support a planned analysis. Prevent treated/control pages from competing for the same target query; cluster at the site or topic level where necessary.
- Measurements
- Primary outcome: predeclared click or impression change over a fixed interval. Secondary: CTR and position, segmented by stable query/device/country groups where the data permits. Preserve daily exports, deployment time, actual displayed titles when observed, and omitted/anonymized-query limitations.
- Controls
- Freeze unrelated page edits when feasible and log unavoidable changes, seasonality, search features and update periods. Inspect baseline trends and use an unchanged comparison group. Do not buy clicks or manufacture impressions.
- Interpretation
- The revision can change topical relevance, the displayed result and click behavior simultaneously. Position also affects CTR. Even a randomized title experiment estimates the total revision effect; it does not isolate “CTR alone,” prove mediation through clicks, or establish a NavBoost mechanism.
- Result
- Not run. No page-one/page-five response pattern, ranking lift or effect size has been measured.
A future report must include the actual data, assignment/selection procedure, analysis code, uncertainty, null results and limitations. Rename a proposal “completed” only when those artifacts exist and the reported conclusion matches what the design can identify.