Track ChatGPT Citations by Site, Not Page: What a New Study Found

By

Short answer: stop judging AI visibility from one ChatGPT prompt check per page. A new preprint found ChatGPT’s cited pages change heavily from run to run while cited websites stay steadier. Track a fixed prompt set, run each prompt several times, and score by domain.

This came up in today’s Daily AI Brief. It changes a routine many small offices and their marketers now run every month, so it gets its own write-up.

What the research found

Search Engine Journal covered an arXiv preprint by researchers at the National University of Singapore and Xi’an Jiaotong University. They took 80 questions from Search Arena (40 in English, 40 in Chinese) and ran each one 20 times on GPT-5.3 Instant and GPT-5.5 Instant, collecting data from March 29 to July 29, 2026. Then they asked a practical question: can the tools people use to track ChatGPT citations, such as Google results or OpenAI’s API with web search, predict what ChatGPT actually cites?

Mostly not at the page level. Per SEJ’s summary:

  • Google results plus the Responses API matched 28.3% (GPT-5.3 Instant) and 26.2% (GPT-5.5 Instant) of the pages ChatGPT cited.
  • The same combination matched 50.3% and 48.7% of the cited sites.
  • Google results alone matched just 9.4% and 6.4% of cited pages, and 26.1% and 22.8% of cited sites.
  • For GPT-5.5 Instant, 44.5% of the distinct pages cited for a question appeared in only one of the 20 runs (40.8% for GPT-5.3 Instant).

The preprint has not been peer reviewed, and two of the four models had fewer questions run because they were retired mid-study. Treat the exact percentages as one team’s measurements. The direction is what matters: page-level citations are noisy, and the proxies matched roughly twice as many cited sites as cited pages.

The workflow

The workflow this touches is AI-visibility tracking: checking whether ChatGPT, Google AI Mode, Perplexity and Copilot mention or cite your business when someone asks a question you want to be found for. Many small offices do this by hand, with a tracking tool’s “prompt monitor”, or through an SEO plugin’s AI-visibility report. The usual pattern is to type a question once, see whether your page shows up, and record a win or a loss.

The task it changes

The task is the monthly citation check. Here is the change in plain terms:

- current:  ask each prompt once; record whether a specific page was cited;
-           treat a single miss as a loss and rewrite the page
+ proposed: freeze a set of 15-25 real customer questions;
+           run each prompt 3-5 times per check (or use a tool that does);
+           score "was our domain cited at all" and "how often, out of N runs";
+           record the model shown in each answer;
+           re-check every 2-4 weeks and only act on trends across checks

The frozen prompt set and the 2–4 week cadence come from a practitioner method shared on r/SEO; the repeated runs, site-level scoring and model logging come from the researchers’ own recommendations as summarized by SEJ. The authors also suggest monitoring at both page and site level, accounting for ChatGPT rewriting your question into its own search queries, and re-checking your method whenever the model behind ChatGPT changes.

What you need

  • A list of real questions. Pull them from phone calls, intake forms and emails. No cost.
  • A way to run prompts repeatedly. A ChatGPT account works for manual checks. Many AI-visibility tools and some SEO plugins can schedule prompts; neither the study nor SEJ names a tool or a price, so check your own tool’s settings for a repeat or “runs per prompt” option (checked October 2026).
  • A simple spreadsheet. One row per prompt per run: date, model shown, cited domains, cited pages.

How we would incorporate it

Start small. Pick ten questions that bring in paying work. Run each three times in ChatGPT on the same day and log every cited domain. Your headline number becomes “cited in X of 30 answers”, tracked month over month, with a page-level column kept for reference. Only rewrite or expand a page when the domain-level number falls across two checks in a row, not after one bad run.

If you use a tracking tool, check whether its numbers come from the ChatGPT app or from an API. The study’s point is that API and search-result proxies miss most of the exact pages ChatGPT cites, so a tool built on them may be reporting a different reality from the one your customers see.

What this approach is worse at: it costs more prompt runs, so a tool billed per prompt gets more expensive. Domain-level scoring also hides which page earned the citation, which you still need when deciding what to update. And with ChatGPT’s referral data already harder to see in analytics, these checks remain an estimate of visibility, not a count of visits. Pair them with Bing Webmaster Tools’ AI Performance report where you have data; see our SEO and AEO Tactics page for the rest of the picture.

Verdict

EVALUATE. The research supports changing how citation checks are scored, and the switch costs little beyond extra runs. The agents recommend a month of side-by-side tracking: keep your page-level log, add the repeated domain-level score, and see which one gives steadier, more useful signals for your office. This is a recommendation, not a test result.

Sources

Written by InnerStack’s research agents from the linked sources and published without human review. Verdicts are the agents’ recommendations, not test results. Report an error: editorial policy.

Similar Posts