--- title: "How AI Search Engines Decide Which Sources to Cite" date: 2026-10-13 author: "Matt Travers" featured_image: "https://leadtoconversion.com/wp-content/uploads/how-ai-search-engines-choose-sources.webp" categories: - name: "SEO" url: "/blog/category/seo.md" --- # How AI Search Engines Decide Which Sources to Cite *By [Matthew Travers](https://leadtoconversion.com/about/matthew-travers/), President, Lead to Conversion.* ## Direct Answer > AI search engines choose sources by running a live web search, often as several related searches, and citing the pages whose passages answer part of the question. A page can be cited only if that engine’s search crawler can reach it and the page is allowed to show a snippet. ## TL;DR - AI answers cite pages found in a live search, not pages the model remembers from training. - Each engine uses a named search crawler: Googlebot, OAI-SearchBot and PerplexityBot. Block it, and that engine can’t cite you. - Training crawlers such as GPTBot and Google-Extended are separate switches, so you can opt out of training and keep your citations. - Query fan out means you can be cited for related questions you rank for, beyond the one a patient typed. - Google says optimizing for AI search is still SEO. Its AI features need no llms.txt file, chunking or special schema. - Check the Search generative AI control in Search Console, live for all sites since August 31, 2026. - Pages that answer one piece of a question clearly, with first-hand detail, are the ones that get picked. **One liner:** Get your pages in front of each engine’s search crawler, then give it answers only your practice can give. ## Key Numbers FactValueSourceSearch generative AI control rollout to all sitesAugust 31, 2026Search Console Help [\[6\]](#source-6)Default setting for the controlIncludeSearch Console Help [\[6\]](#source-6)Time for an exclusion to take effect1 to 2 days after the control goes live, some content longerSearch Console Help [\[6\]](#source-6)Time for robots.txt changes to reach ChatGPT searchAbout 24 hoursOpenAI [\[3\]](#source-3)Time for robots.txt changes to reach PerplexityUp to 24 hoursPerplexity [\[4\]](#source-4)Extra technical requirements for AI Overviews and AI ModeNone beyond SearchGoogle, updated December 10, 2025 [\[5\]](#source-5)Special schema.org markup needed for Google’s AI featuresNoneGoogle, updated July 10, 2026 [\[1\]](#source-1) A prospective patient opens ChatGPT and types a simple question: which concierge doctors near me are taking new patients? A few seconds later, the answer names three practices, each with a small link beside it. Your practice isn’t one of them, even after years of work on your website. So how do AI search engines choose sources? They run a search, read what comes back, and cite the pages they used. The details of that search decide who gets named. Below, you’ll see how retrieval and selection work in Google’s AI features, ChatGPT and Perplexity. You’ll also see what the platforms themselves say matters, and the steps a practice can take to become a source AI can cite. ## How Do AI Search Engines Choose Sources? ***In short:** they search, they read, and they cite what they used.* AI search engines choose sources by running a web search, often as several related searches at once, then reading the pages that come back. They keep the passages that answer part of the question, write a response from them, and link to the pages those passages came from [\[1\]](#source-1)[\[2\]](#source-2). Here’s how that plays out: 1. **Retrieve.** The engine turns your question into one or more searches. Google calls this query fan out, a set of related searches that pull in more results than the original question would [\[1\]](#source-1). ChatGPT does something similar, rewriting a question into targeted queries it sends to its search partners [\[2\]](#source-2). 2. **Select.** The model reviews the specific information on the pages it retrieved and keeps what helps answer the question [\[1\]](#source-1). 3. **Cite.** The response links to the pages that support what it says. Those links are the AI citations you see beside an answer [\[1\]](#source-1). [![Diagram of how AI search engines choose sources in three stages: retrieve pages, select passages, and cite the source pages](https://leadtoconversion.com/wp-content/uploads/ai-search-retrieve-select-cite.webp "1")](https://leadtoconversion.com/wp-content/uploads/ai-search-retrieve-select-cite.webp "Retrieve, Select, Cite: How AI Answers Are Built") Think of a research assistant who can only quote from books already on the library shelf. However well they write, the shelf decides what they can quote. For AI search, the shelf is a search index, and a search crawler decides what makes it onto the shelf. ## AI Citations Come From a Live Search, Not the Model’s Memory ***In short:** a page gets cited only if that engine’s search crawler can reach it.* The links in an AI answer come from a live search of an index that a named search crawler keeps stocked. Training runs on a separate crawler with a separate switch [\[3\]](#source-3)[\[4\]](#source-4). So being in a model’s training data doesn’t make a page citable. And staying out of training doesn’t have to cost you citations. OpenAI is direct about this. To be eligible for ChatGPT search, a site must allow OAI-SearchBot and let OpenAI’s published crawler IP addresses through its host or CDN [\[2\]](#source-2). Sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers,” though they can still appear as navigational links [\[3\]](#source-3). GPTBot, the crawler tied to model training, is a separate setting [\[3\]](#source-3). Perplexity says PerplexityBot surfaces and links sites in its answers and isn’t used to crawl content for AI foundation models [\[4\]](#source-4). Google’s AI Overviews and AI Mode are built on its core Search ranking systems and Search index [\[1\]](#source-1). Robots.txt rules for Googlebot are how site owners manage that access [\[5\]](#source-5). EngineCrawler that decides if you can be citedCrawler tied to trainingWhat the platform says about accessGoogle AI Overviews and AI ModeGooglebotGoogle-Extended, for training and grounding in some of Google’s other systemsA page must be indexed and eligible for a snippet to be a supporting link [\[5\]](#source-5)ChatGPT searchOAI-SearchBotGPTBotSites opted out of OAI-SearchBot are left out of search answers [\[3\]](#source-3)PerplexityPerplexityBotPerplexityBot is not used for model trainingAllow PerplexityBot to appear in search results [\[4\]](#source-4) Picture a practice that asks its web vendor to block “AI bots” to keep patient pages out of model training. A broad rule catches OAI-SearchBot and PerplexityBot along with GPTBot. The practice drops out of ChatGPT and Perplexity answers. A narrower rule, blocking GPTBot and allowing the search crawlers, keeps both the training opt-out and the citations [\[3\]](#source-3)[\[4\]](#source-4). [![Comparison of a broad robots.txt rule blocking GPTBot, OAI-SearchBot and PerplexityBot versus blocking only GPTBot to keep AI citations](https://leadtoconversion.com/wp-content/uploads/ai-search-crawlers-vs-training-crawlers.webp "2")](https://leadtoconversion.com/wp-content/uploads/ai-search-crawlers-vs-training-crawlers.webp "Search Crawlers vs Training Crawlers in robots.txt") A few more details matter. ChatGPT searches the web on its own when a question would benefit from current information, and responses that use web search may include citations [\[2\]](#source-2). ChatGPT-User and Perplexity-User visit pages when a person asks a question, so robots.txt rules may not apply to them [\[3\]](#source-3)[\[4\]](#source-4). OpenAI says ChatGPT-User isn’t used to decide what appears in search; OAI-SearchBot is [\[3\]](#source-3). And robots.txt changes take up to about 24 hours to show in ChatGPT and Perplexity search [\[3\]](#source-3)[\[4\]](#source-4). So start with access: can each engine’s search crawler reach your pages? In our [audit of 100 concierge practice websites](https://leadtoconversion.com/blog/concierge-practice-marketing/), none of the robots.txt files we could read blocked OAI-SearchBot or PerplexityBot. ## LLM Retrieval: How AI Search Engines Find Candidate Pages ***In short:** the engine searches first, often several times, and only pages those searches return can be cited.* ### The search index makes the first cut Google calls its method retrieval augmented generation, or RAG, also known as grounding. Its AI features rely on Google’s core Search ranking systems to pull relevant, up to date pages from the Search index [\[1\]](#source-1). The answer is built from what those pages say, so the index decides what the model has to work with. Google says grounding exists to make AI answers more accurate and more current [\[1\]](#source-1), so pages you keep up to date have a better shot. ChatGPT follows the same principle with different plumbing. OpenAI says ChatGPT search sometimes partners with other search providers, and its help page points to Microsoft’s privacy statement among them [\[2\]](#source-2). A strong Google ranking helps with Google’s AI features, but it isn’t the only index that matters. Answers vary even inside Google. AI Overviews and AI Mode may use different models and techniques, so the links they show differ [\[5\]](#source-5). Expect each engine to cite a different mix of pages. ### Query fan out turns one question into many searches Google’s AI Overviews and AI Mode may use query fan out [\[1\]](#source-1). The model runs several related searches at once to gather more results than the original question would. ChatGPT does something close. It rewrites a question into targeted queries, reviews the first results, and may send more specific follow-up queries [\[2\]](#source-2). For a “near me” question, it may add the person’s general location to the search [\[2\]](#source-2). Here’s how one question, “is a concierge doctor worth it?”, might fan out. The searches below are an illustration, not output from any engine. [![Query fan out example: the question "is a concierge doctor worth it" expanding into three related searches and the pages each can cite](https://leadtoconversion.com/wp-content/uploads/ai-query-fan-out-example.webp "3")](https://leadtoconversion.com/wp-content/uploads/ai-query-fan-out-example.webp "Query Fan Out Example for a Concierge Medicine Question") Related search the engine might runPage that could earn the citationconcierge medicine membership costA membership or pricing pageconcierge doctor vs primary care physicianA plain comparison articleconcierge doctors taking new patients near AkronA location page with current availability That’s how a practice gets cited for a question it never ranked for. The citation comes from a related search where one of its pages did show up. Tracking a single keyword undercounts where you can appear. Don’t respond by building a page for every variation. Google says creating separate content for each fan out query mainly to manipulate rankings or AI answers violates its scaled content abuse policy [\[1\]](#source-1). A few strong pages that answer the real sub-questions serve patients better and hold up longer. ## Why AI Cites Some Sites and Not Others ***In short:** of the pages retrieved, the cited ones answer part of the question clearly, add something new, and allow their text to be shown.* Retrieval gets a page into the pool. Selection decides whether it’s quoted. Google says its systems review the specific information on retrieved pages [\[1\]](#source-1). They link to the pages that support the response. OpenAI says ChatGPT ranks results on several factors meant to surface relevant, reliable information [\[2\]](#source-2). Placement isn’t guaranteed. ### The page answers one piece of the question Each link in an AI answer supports part of what the answer says [\[1\]](#source-1). A page that answers its piece plainly, under a clear heading, is easy to match to that part. Google’s advice is to organize content in paragraphs and sections with headings that help readers follow it [\[1\]](#source-1). ### It says something other pages don’t Google puts the most weight here. It says unique, useful content will likely do more for your presence in AI search over time than any other tactic it lists [\[1\]](#source-1). It warns against commodity content based on common knowledge and favors first-hand, expert takes [\[1\]](#source-1). Here’s the difference for a practice, modeled on Google’s own example: Commodity pageNon-commodity page“5 Benefits of Concierge Medicine”A physician on what members use most in their first year, and what the membership doesn’t coverRestates what other sites already sayAdds detail only the practice hasNo clear authorWritten by a named clinician with credentials listed For a practice, first-hand usually means a clinician writing from what they see with patients. That gives an engine, and a patient, a reason to trust the passage. ### Trust and snippet settings decide what can be shown Some pages rule themselves out. Google says nosnippet, data-nosnippet, max-snippet and noindex limit what it shows from a page in Search, AI features included [\[5\]](#source-5). If these were added to stop copying, they also limit what AI answers can use. Shortcuts don’t help either. Google says chasing inauthentic mentions isn’t as helpful as it might seem [\[1\]](#source-1). Its AI features depend on both its quality ranking systems and its spam systems [\[1\]](#source-1). ## What Google Says You Do Not Need ***In short:** Google says its AI features run on the same SEO basics, so special files, chunking and AI-only markup don’t help in its results.* Terms like AEO and GEO come up a lot. Google’s view is that optimizing for generative AI search “is optimizing for the search experience, and thus still SEO” [\[1\]](#source-1). Its July 2026 guide lists several tactics site owners can skip for Google Search [\[1\]](#source-1). Common adviceWhat Google saysWhat to do insteadAdd an llms.txt or other AI text fileGoogle Search doesn’t use them. Keeping one for other services neither helps nor harms you in Google [\[1\]](#source-1)Make sure crawlers can reach your real pagesBreak content into small chunksThere’s no requirement to split content into tiny pieces for AI [\[1\]](#source-1)Write pages at the length your patients needRewrite pages for every long-tail variationYou don’t need to write a special way for AI, since its systems understand synonyms and meaning [\[1\]](#source-1)Answer the questions patients actually askAdd special schema for AIStructured data isn’t required, and there’s no special markup for AI features [\[1\]](#source-1)Keep standard schema accurate and matched to the page Schema still earns its place. Google recommends it as part of an overall SEO strategy because it helps pages qualify for rich results [\[1\]](#source-1). It also asks that structured data match the visible text on the page [\[5\]](#source-5). For a practice, that means markup for the providers, services and locations the page itself describes. For the rich results side, see [how structured data improves healthcare SEO](https://leadtoconversion.com/blog/how-does-structured-data-improve-healthcare-seo/). A simple test before paying for an AI-only tactic: if Google says it doesn’t use it, put that budget into crawler access and better pages. ## How to Get Cited by AI: Six Steps for Healthcare Practices ***In short:** open the door for search crawlers, confirm Google’s AI setting, then give each engine pages worth quoting.* These steps follow the order the engines work in, with access first and content second. [![Six steps for healthcare practices to get cited by AI, grouped into crawler access steps and content steps](https://leadtoconversion.com/wp-content/uploads/how-to-get-cited-by-ai-six-steps.webp "4")](https://leadtoconversion.com/wp-content/uploads/how-to-get-cited-by-ai-six-steps.webp "Six Steps for Healthcare Practices to Get Cited by AI") ### 1. Let the search crawlers in Check that robots.txt, your host and any CDN allow Googlebot, OAI-SearchBot and PerplexityBot [\[3\]](#source-3)[\[4\]](#source-4)[\[5\]](#source-5). OpenAI also asks sites to let its published search crawler IP addresses through [\[2\]](#source-2). To stay out of model training, block the training crawler by name instead of blocking every AI bot [\[3\]](#source-3): ``` User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / ``` ### 2. Check Google’s Search generative AI control Since August 31, 2026, every site has a Search generative AI control in Search Console, under Settings [\[6\]](#source-6). “Include” is the default. “Exclude” removes your links from AI Overviews and AI Mode and stops your content from grounding those answers [\[6\]](#source-6). The setting doesn’t affect training, which Google-Extended handles [\[6\]](#source-6). Properties can inherit the setting from a parent property, so groups with several locations should check each one [\[6\]](#source-6). ### 3. Make key pages indexable and quotable A page must be indexed and eligible for a snippet to be a supporting link in Google’s AI features [\[5\]](#source-5). Look for stray noindex or nosnippet tags on service pages. Keep important details in text rather than images, and link to key pages from elsewhere on your site [\[5\]](#source-5). ### 4. Cover the related questions patients ask Map the questions around your core services: cost, comparisons, availability and location. Build one strong page per real question and skip near-duplicates for every phrasing [\[1\]](#source-1). That’s the heart of [generative engine optimization for healthcare](https://leadtoconversion.com/blog/what-is-generative-engine-optimization/). ### 5. Publish answers only your practice can give Have clinicians write or review pages on what they see with patients, such as what a membership covers or how a first visit runs. Google says unique, first-hand content will likely matter most over time [\[1\]](#source-1). Good [healthcare content marketing](https://leadtoconversion.com/healthcare/content-marketing/) starts there. ### 6. Keep your Google Business Profile current Google says Business Profiles can help your services show up in AI responses as well as regular results [\[1\]](#source-1). Keep hours, services and contact details up to date [\[5\]](#source-5). The same care sits at the center of [local SEO for healthcare](https://leadtoconversion.com/healthcare/seo/). ## How to Check Your AI Search Visibility ***In short:** use Google’s own report for its AI features, then look at real answers in each engine.* Google counts traffic from AI Overviews and AI Mode in Search Console [\[5\]](#source-5). Its Generative AI performance report shows how people find your content through those features [\[1\]](#source-1). For ChatGPT and Perplexity, you check the answers themselves. CheckWhereWhat it tells youGenerative AI performance reportGoogle Search ConsoleHow your pages perform in Google’s AI features [\[1\]](#source-1)Ask the related questions patients askChatGPT, Perplexity and GoogleWhich pages get cited, yours or someone else’sCrawler visitsServer or hosting logsWhether OAI-SearchBot and PerplexityBot reach your pages [\[3\]](#source-3)[\[4\]](#source-4)Free AI visibility check[Lead to Conversion’s AI Visibility Tool](https://leadtoconversion.com/ai-visibility-tool/)How AI engines describe your practice, with a list of fixes Keep one caution in mind. Google says no third-party tool has access to its internal ranking or AI systems [\[1\]](#source-1). Use tools that show you real AI answers, and be wary of any that promise a score from inside Google. To see where your practice stands today, [run a free AI visibility check](https://leadtoconversion.com/ai-visibility-tool/). It shows how ChatGPT, Perplexity and Google AI describe your practice and gives you a prioritized list of fixes. ## Frequently Asked Questions About AI Citations ***In short:** fast answers on access, files, schema and timing.* ### [How do I rank in AI Overviews?](#432fe4d27875bfdf4) Google says there are no extra requirements to appear in AI Overviews or AI Mode. A page must be indexed and eligible to show with a snippet, and the usual SEO basics apply [\[5\]](#source-5). Specific, first-hand pages for the questions patients ask give you the best chance [\[1\]](#source-1). ### [How do I show up in ChatGPT?](#7a72a2f2fda851f1e) Allow OAI-SearchBot in robots.txt and make sure your host or CDN lets OpenAI’s published search crawler IP addresses through [\[2\]](#source-2). From there, ChatGPT ranks results on several factors, and OpenAI says placement isn’t guaranteed [\[2\]](#source-2). ### [If I block GPTBot, will ChatGPT stop citing me?](#61a42cc74bfa4c7d0) No. GPTBot covers content that may be used for model training, and OAI-SearchBot covers ChatGPT search. OpenAI says each setting works independently, so you can block one and allow the other [\[3\]](#source-3). ### [Do I need an llms.txt file?](#c2bd5db6ff67bb0fe) Not for Google. Google Search doesn’t use llms.txt or other AI text files, and keeping one neither helps nor harms your visibility there [\[1\]](#source-1). You can keep one for other services if you like. ### [Does schema help my practice get cited by AI?](#2a99b26097c9664e2) Google says structured data isn’t required for its AI features and there’s no special markup to add [\[1\]](#source-1). It’s still worth using for rich results, as long as it matches what the page says [\[1\]](#source-1)[\[5\]](#source-5). ### [Why does ChatGPT cite different sources than Google?](#df1ac067deb57d726) Each engine searches a different index and ranks results on its own factors [\[2\]](#source-2)[\[5\]](#source-5). Even Google’s AI Overviews and AI Mode may use different models and techniques, so the links they show vary [\[5\]](#source-5). Track each engine on its own. ### [How long do changes take to show up?](#06fefe187ceb80409) OpenAI and Perplexity say robots.txt changes take up to about 24 hours to reflect in search [\[3\]](#source-3)[\[4\]](#source-4). Google says a change to the Search generative AI control generally takes effect within a few days, though some content can take longer [\[6\]](#source-6). Want help putting these steps to work on your practice’s site? You can [book a consultation](https://leadtoconversion.com/consultation/) with our [AI digital marketing](https://leadtoconversion.com/ai-digital-marketing-agency/) team, and we’ll start with what AI engines see today. ## Definition Bank TermPlain-English definitionAI citationA linked source shown beside an AI answer, pointing to a page that supports part of what the answer saysRetrieval augmented generation (RAG)A method where the AI searches an index first, then writes its answer from the pages it foundGroundingGoogle’s other name for RAG: tying an AI answer to real, retrieved web pagesQuery fan outThe AI running several related searches at once to gather more results than one question wouldSearch indexThe stored copy of the web a search engine searches, built by its crawlerSearch crawlerThe bot that visits pages so an engine can show and cite them in search answersTraining crawlerA separate bot that collects content that may be used to train AI modelsOAI-SearchBotOpenAI’s search crawler; sites that block it are left out of ChatGPT search answersGPTBotOpenAI’s training crawler; blocking it does not affect ChatGPT searchPerplexityBotPerplexity’s crawler for surfacing and linking sites; Perplexity says it is not used for model trainingGoogle-ExtendedGoogle’s control for limiting AI training and grounding in some of its other systemsSearch generative AI controlA Search Console setting that includes or excludes a site from AI Overviews, AI Mode and AI features in DiscoverSnippet controlsTags such as nosnippet, data-nosnippet and max-snippet that limit what Google can show from a pageNon-commodity contentContent with first-hand or expert detail that goes beyond common knowledgellms.txtA text file some sites add for AI tools; Google Search does not use it ## Entity Cards ### Google AI Overviews and AI Mode PropertyValueRetrieval methodRAG built on Google’s core Search ranking systems and Search index [\[1\]](#source-1)Uses query fan outYes, may run related searches across subtopics [\[1\]](#source-1)[\[5\]](#source-5)Crawler that decides eligibilityGooglebot [\[5\]](#source-5)EligibilityIndexed and eligible to show with a snippet [\[5\]](#source-5)Site-level switchSearch generative AI control in Search Console [\[6\]](#source-6)ReportingGenerative AI performance report in Search Console [\[1\]](#source-1) ### ChatGPT search PropertyValueRetrieval methodRewrites questions into targeted queries sent to search partners [\[2\]](#source-2)Search crawlerOAI-SearchBot [\[3\]](#source-3)Training crawlerGPTBot, a separate setting [\[3\]](#source-3)User-triggered fetcherChatGPT-User, not used to decide search inclusion [\[3\]](#source-3)EligibilityAllow OAI-SearchBot and OpenAI’s published IP addresses [\[2\]](#source-2)PlacementNot guaranteed [\[2\]](#source-2) ### Perplexity PropertyValueSearch crawlerPerplexityBot [\[4\]](#source-4)Used for model trainingNo, per Perplexity [\[4\]](#source-4)User-triggered fetcherPerplexity-User, which generally ignores robots.txt [\[4\]](#source-4)Time for robots.txt changesUp to 24 hours [\[4\]](#source-4) ### Search generative AI control PropertyValueWhereSearch Console, Settings, Search generative AI [\[6\]](#source-6)Rolled out to all sitesAugust 31, 2026 [\[6\]](#source-6)DefaultInclude [\[6\]](#source-6)Effect of ExcludeNo links and no grounding in AI Overviews, AI Mode and Discover AI features [\[6\]](#source-6)Affects trainingNo; Google-Extended covers that [\[6\]](#source-6) ## Sources 1. Google Search Central, Optimizing your website for generative AI features on Google Search, updated July 10, 2026: 2. OpenAI Help Center, Searching the web with ChatGPT: 3. OpenAI, Overview of OpenAI Crawlers: 4. Perplexity, Perplexity Crawlers: 5. Google Search Central, AI features and your website, updated December 10, 2025: 6. Google Search Console Help, Search generative AI control, rolled out to all sites August 31, 2026: