How superAEO measures AI visibility

What this page covers

superAEO measures how AI assistants talk about a Shopify store: whether they name it, where, how, and on what sources they rely. This page sets out exactly how, so you can judge what the numbers can and can't support.

It follows the metric definitions and disclosure items in the IAB's Measuring Visibility in the AI Era (August 2026). Where we depart from the framework, or don't meet it yet, this page says so.

Engines and models

Every tracked prompt is put to five AI engines through their official APIs. These are the models used by default; we may move an engine to a newer model, and every answer records both the model we asked for and the model the provider reports actually serving it.

ChatGPT

OpenAI gpt-4o. No live web search. Sources are the links written in the answer.

Gemini

Google gemini-2.5-flash with Google Search grounding on. Sources are the pages Google returns for the answer, resolved to their real addresses.

Perplexity

sonar-pro, which always searches the web. Sources are Perplexity's own source list plus links in the answer.

Claude

Anthropic claude-sonnet-4-5. No live web search. Sources are the links written in the answer.

DeepSeek

deepseek-chat (DeepSeek reports serving it as deepseek-flash). No live web search. Sources are the links written in the answer.

Not covered: Google AI Overviews, Microsoft Copilot, Meta AI and Grok.

How answers are collected

superAEO uses active query simulation: we send each prompt to each engine's API and record the full answer. We don't observe real shoppers, run a panel, scrape consumer apps, or use data from the AI platforms themselves.

• Each question is asked fresh — one turn, no conversation history, no user profile, no personalisation. A shopper using the consumer app, signed in and mid-conversation, can get a different answer.
• Each engine gets a short neutral instruction ("answer thoroughly and cite your sources"); Gemini gets only the location line below. ChatGPT, Gemini, Perplexity and DeepSeek run at temperature 0.3; Claude at its default.
• Each question says where the shopper is. Every store has a main market — the country of its Shopify address unless the merchant changes it — and every engine is told "the person asking is shopping from India" (or wherever it is); Perplexity also gets it through its own location setting. The queries still leave our servers in the eastern United States, so an engine's web search can lean on what it sees from there. Each answer records the market it was asked with, and comparisons over time — the score change and action verification — only pair answers asked with the same market. Until 15 October 2026 no location was sent.
• Web search is on for Gemini and Perplexity and off for ChatGPT, Claude and DeepSeek, which answer from what their models already know. Every result is reported per engine, and the dashboard shows the Visibility Score for each group alongside the blended headline (see below); each answer records whether search was on.

The prompt library

Each store tracks its own set of prompts — the shopper questions it chooses to monitor. The merchant writes them or picks them from suggestions; nothing is tracked without their say. Pro tracks up to 10 prompts and Max up to 20.

Suggestions come from Claude (claude-sonnet-4-5) reading the store's collections and a sample of its catalogue, and must be questions a shopper would plausibly ask and whose answer would name brands. They are synthetic: no search-volume or real-user query data goes into them. Prompts are in the language the merchant writes them in and aren't varied across demographic or cultural framings.

Every prompt is tagged with one of the IAB's four intent types, by Claude (claude-haiku-4-5, temperature 0) reading the question:

Informational

Learning about a product, need or brand — "is X a trustworthy brand?"

Comparison

Weighing options against each other — "X vs Y", "alternatives to X".

Recommendation

Which options are best for a need — "best X for Y", "top X brands".

Transactional

Ready to buy — "where can I buy X online?"

The Prompts page shows each store's mix across the four, the dashboard shows results for each intent type separately, and suggestions lean toward the types a store has fewest of.

Sampling and cadence

AI answers vary from one asking to the next, so a single answer measures nothing. superAEO asks repeatedly:

• A new prompt runs every day, three answers per engine, until it has 50 answers from each engine.
• After that it is in maintenance: one answer per engine every three days.
• A merchant can also run prompts on demand (one answer per engine), within a monthly allowance.

Rates — mention rate, citation rate — count every answer once, so while a prompt is still building its first 50 answers it carries more weight in them than prompts in maintenance. The Visibility Score doesn't: it weights every prompt equally (below).

Mentions, position and sentiment

Each answer is read by Claude (claude-haiku-4-5), given the store's name and any other names it trades under.

Mention

The answer names the store, under its name, an alias, or an obvious variant (a legal form, a shortening). An answer that only repeats the name from the question to say it knows nothing about the store is not a mention. Mention rate is answers naming the store ÷ answers measured.

Position

The store's place in the order brands are named in the answer's text — 1 if it is the first brand named. This is text order, not the rendered page a shopper scrolls: we don't capture layout, cards or what sits above the fold.

Sentiment

How positively the answer describes the store, from −1 to +1, as judged by the same reading. Most useful on informational questions; recommendation answers tend to read as neutral.

If the reading itself fails, the answer is kept for audit but left out of every rate. If an engine deliberately returns no answer (Gemini sometimes does for brands it can't find), that counts as a measured answer that didn't name the store.

Framing and recommendation strength

For answers that name the store, a second, separate reading (Claude claude-haiku-4-5, temperature 0) records how the store is portrayed. It is kept apart from the main reading so that adding it couldn't change how mentions are counted. Recorded for new answers from 15 October 2026; on that day the same reading was also run over the previous 90 days of answers. The dashboard shows both over the last 30 days.

Framing

How the answer positions the store: leader, premium, budget alternative, niche specialist, outdated, miscategorized (described as the wrong kind of business, or credited with products it doesn't offer), or neutral.

Recommendation strength

Unconditional — named as the best choice outright, with reasons. Qualified — recommended for a stated need or buyer ("good if you're on a budget"). Listed — one option among several, no reason to pick it. Passive — named but not recommended: background, a comparison point, a source, or advised against.

On questions that name the store ("where can I buy from X?"), recommendation strength comes out strong almost by construction; read it by intent type.

Citation rate

The IAB counts a brand as cited when an answer relies on it as a source — by link, or by name. Being mentioned and being cited are different things: an answer can recommend a store without relying on anything it published.

• Linked citation: the store's own domain is among the answer's sources. Sources are the links in the answer plus, for Gemini and Perplexity, the source list the engine returns. The store's domain covers its subdomains, the domain its storefront sits under, and its myshopify.com address.
• Named citation: the answer relies on the store by name ("according to Acme's size guide"). Recorded by the portrayal reading (see above).

The dashboard's citation rate counts both linked and named citations, over answers recorded with their engine sources (from 14 October 2026); earlier answers are left out because their Perplexity and Gemini sources weren't kept.

Share of voice and the competitive set

Share of voice is the store's mentions ÷ all brand mentions, over the last 30 days of measured answers to the store's active prompts. Each brand counts once per answer, so the shares add up to 100%. Each share comes with a 95% range, found by resampling whole answers a thousand times: one answer names several brands and its mentions rise and fall together, so a formula treating every mention as independent would make the range too narrow.

The competitive set is whoever the answers name: for each answer the reading lists the 3–10 most prominent companies a shopper could buy from instead, retailers and marketplaces included. Spellings of the same brand are merged. Left out: the store itself under any of its names; review and rating sites; search engines and encyclopedias; social and community sites; publishing platforms; and AI assistants themselves. A merchant's competitor watchlist highlights names but doesn't change the set or the denominator.

Per engine, the Competitors page says whether the store is named more or less often than its strongest rival there, once that engine has at least 10 answers. The call is a paired test on the same answers — McNemar's exact test on the answers naming one and not the other — at 95%: "Leading" or "Behind" only when the gap is bigger than chance, otherwise "Too close to call".

Source types

The websites answers draw on are labelled by kind, one label per site, shared across all stores. A site is its full address, so a subdomain is judged on its own — the Shopify App Store (apps.shopify.com) is a marketplace, shopify.com Shopify's own site.

Brand site

A brand's or company's own site, including its own store — a store that mainly sells its own label counts here, and so does a software or services company's site.

Retailer

A store or marketplace selling many brands.

Editorial

Publishers, magazines, newspapers, blogs, review and buying-guide sites.

Institutional

Government, education, standards and professional bodies, encyclopedias, official registries.

Community

Forums, Q&A, social and video platforms, user-review sites.

Other

Directories and listings, coupon and deal sites, tools, or a site that gives no clue to what it is.

Labels come from Claude (claude-haiku-4-5, temperature 0) judging each site by what it knows of it and by the paths of the pages that were cited — a /blog/ page reads differently from a /products/ one. An unfamiliar site with a brand-like name and nothing else to go on counts as a brand site; one with no clue at all is other. The hardest line is in fashion, between a chain that mostly sells its own label and a multi-brand store, and small software companies' sites are sometimes read as other; expect some errors there. The same labels appear on the dashboard, in gap reports and in their exports. The dashboard shows the mix of source types in a store's latest answers, with the store's own site as its own slice, once at least 80% of the citations are labelled. Types are reported, not weighted: no source type counts for more than another.

The Visibility Score

The Visibility Score is superAEO's own 0–100 summary. It is not an IAB metric, and its parts are always shown alongside it. This is version 2, in use from 15 October 2026. Each answer scores:

Named

50 if the answer names the store.

Placed

Up to 30 for where it is named: 30 if it is the first brand named, 22.5 if second, 16.5 if third, 10.5 if later.

Cited

20 if the answer relies on the store as a source, by link or by name (see citation rate).

Answers from before 14 October 2026, whose engine sources weren't kept, can't be judged on citation; for those the first two parts are scaled up to 100 rather than marking them down for data we didn't record.

The score then averages one figure per prompt on each engine — every prompt counts the same within an engine, and every engine with answers counts the same overall — so a prompt still building history can't outweigh the rest. Because it is computed from the stored answers each time, all past periods are shown on the same version and trends stay comparable. Alongside the headline, the dashboard shows the same score for the engines that searched the web and for those answering from model knowledge, since the two kinds of answer differ.

Version 1 (until 15 October 2026) was, per engine, 0.5 × mention rate + 0.3 × an AI-judged answer score + 0.2 × the share of answers citing any source at all, weighted by answer count. It credited answers that never named the store, counted a link to any site, rival or not, and let newer prompts outweigh older ones. Moving to version 2 raised existing stores' scores by between 1 and 19 points, with no change in the answers themselves.

Confidence and change

• Confidence intervals. The headline mention rate and citation rate carry 95% intervals by the Wilson method, which holds up near 0% and 100% and on small samples.
• Change. The headline change compares the selected period (today, 7 days, 30 days) with the one before it, like for like: only prompts with answers in both periods count, so adding a prompt doesn't read as a rise or a fall (the dashboard says when newer prompts are left out). It uses Welch's t-test on the score's own units — each prompt's average on each engine — two-sided at 95%, needing at least 10 of those on each side; a store comparing a single prompt is tested on its individual answers instead. With too few it is shown untested, and it is coloured up or down only when the change is significant — otherwise it reads "within normal variation".
• A caveat. Results on the same prompt across engines aren't fully independent, which these tests assume, so they lean slightly toward calling a change real.

Verifying an action

When a merchant marks an improvement done, superAEO checks it 7–14 days later. For each prompt the action targeted, it compares the mention rate over the 14 days before completion with the period after, using a two-proportion z-test. It needs at least 10 answers in each window and a lift of at least one point, and the bar rises with the number of prompts tested (Bonferroni correction). An engine whose model changed between the two windows is left out of both, so a model update can't be credited as the merchant's win.

Measurement tier

The IAB grades measurement as directional (trend monitoring) or decision-grade (budgets, agency reviews), and treats any program of fewer than 50 queries as exploratory, below either. Pro tracks up to 10 prompts and Max up to 20, so today every superAEO program is below that floor. We say so here rather than label results with a tier they don't meet.

What that means in practice: read superAEO as an early signal about the specific questions a store tracks, not as a measure of its whole category. The repeated sampling and confidence intervals tell you how firmly each number is held; per-engine results are always available; each store's results by intent type are on its dashboard. We don't yet publish a documented range of normal week-to-week variation, which the IAB asks for at decision grade.

History and versioning

• Every answer is kept, full text included, for as long as the app is installed, and deleted when Shopify asks us to erase the store's data.
• From 14 October 2026 each answer records the model requested and served, whether web search was on, and the sources the engine returned. From October 2026 each records the version of the analysis that read it, so a change of method can be told apart from a change in the store.
• Trends are not yet re-baselined automatically when an engine changes model; verification excludes such changes (above), but charts don't mark them yet.

What we don't measure yet

• Accuracy — whether answers get a store's products, prices or policies right (the IAB's hallucination and factual-inaccuracy rates). We plan to check answers against the store's own Shopify catalogue.
• Clicks — whether being named sends traffic (post-citation click-through).
• Other surfaces — Google AI Overviews, Copilot, consumer apps as shoppers see them, and separate results for each country a store sells to (a store has one market at a time).
• External validation — we don't check our numbers against platform data or a store's own analytics.