Short answer: AI assistants recommend companies they can find clear, verifiable, quotable information about, on sites their crawlers can reach. There's no trick and no secret file. You need three things: good, answer-first content, unambiguous company information, and the right bots let in. Everything else is fine-tuning.
More and more decision-makers don't search Google for a supplier; they ask ChatGPT, Claude or Perplexity: "Who can help an SME implement an ERP in Central Europe?" If your company doesn't come up there, a competitor does. We write this partly from our own practice: our website was designed first and foremost for visibility in AI assistants.
How much does it matter today?
The numbers are still small, but growing fast. SE Ranking's study of more than 100,000 websites found that traffic from AI platforms made up 0.02% of all traffic in 2024, 0.24% in 2025 and 0.32% in the first four months of 2026. Within that, ChatGPT accounted for 74.78% of AI referrals, Gemini 11.56% and Perplexity 7.23%.
Google is changing too. Pew Research Center's 2025 study found that when an AI summary appeared, users clicked a traditional result on 8% of visits, against 15% without one. They clicked a link inside the summary on just 1% of visits.
Low volume, but high intent: someone arriving because an AI recommended you often comes with a concrete need. And in many languages and niches, good, factual business content is still scarce, which means a clear page has a better chance of being cited.
What the research says
The term "generative engine optimisation" (GEO) was introduced in a 2023 paper later presented at KDD 2024. The researchers found that certain content techniques can increase a source's visibility in generative engine responses by up to 40%, with effects varying by domain. The point: it's not tricks that work, but better, more quotable content.
First: let the right bots in
If a crawler can't reach your site, the best content won't help. The big providers use separate bots for search and for model training, and you can allow or block them separately in your robots.txt.
OpenAI (ChatGPT):
- OAI-SearchBot: for ChatGPT search. Block it, and your site won't appear in ChatGPT search answers.
- GPTBot: for model training. You can block this separately if you don't want your content used for training.
- ChatGPT-User: visits made on a user's request, to which, according to OpenAI, robots.txt may not apply.
Anthropic (Claude):
- Claude-SearchBot: to improve search result quality. Blocking it may reduce your visibility in Claude's search answers.
- ClaudeBot: for model training. Blocking it signals that your site's future content should be excluded from training data.
- Claude-User: access on a user's request. Blocking it may reduce visibility in user-initiated searches.
Anthropic says its bots respect robots.txt directives.
Perplexity:
- PerplexityBot: for Perplexity's results. Allow it if you want to appear.
- Perplexity-User: visits on a user's request, which according to Perplexity generally don't follow robots.txt.
Perplexity says changes take effect within 24 hours.
Google: AI Overviews and AI Mode build on the regular Google index. The Google-Extended token controls whether your content may be used to improve Gemini models.
The decision is simple: if you want to appear in AI search, allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot). Whether to allow the training bots (GPTBot, ClaudeBot, Google-Extended) is a separate, business decision rather than a technical one.
Second: the content
Google's official guidance says there are no additional requirements to appear in its AI features: a page must be indexed and eligible to be shown as a snippet. No new AI files and no special structured data are needed. So good SEO fundamentals are the foundation of AI visibility too.
In practice:
- Answer first. The first paragraph of every page and article should be a self-contained, quotable answer: who you are, what you do, for whom, with what result. AI assistants typically pick this up.
- Specific, verifiable claims. Numbers, dates, sources. Generic marketing copy doesn't get quoted, because there's no fact in it.
- Your own experience. Content that adds new information is cited most readily: case studies, measurements, your own data.
- Question-and-answer structure. FAQ sections match how people ask AI assistants.
- Freshness. A visible date, and an "updated" date when the content changes.
Third: unambiguous company information
An AI only recommends you if it's sure who you are. Your basic details must be the same everywhere: name, activity, service area, contact. Mark it up with structured data (schema.org Organization) on your site, and present each service on its own page with a clear title and description. Keep your company registry entry, business profiles and professional networks consistent with the site: AI assistants combine several sources, and a mismatch between them is a reason to doubt.
llms.txt: useful, but no magic
llms.txt is a community proposal from 2024: a Markdown file at the root of a website that briefly summarises its content for AI agents. It isn't a standard, and Google's guidance says it's not needed for Google's AI features. We use it anyway because it's cheap and may help agents that look for it. But it's not what makes an AI recommend you.
What we do on our own site
Our website was designed first and foremost for AI visibility. What anyone can copy:
robots.txtexplicitly allows the search bots;- the first paragraph of every page is a self-contained, quotable answer;
- every page and article has a Markdown version that AI agents read easily;
- there's an
llms.txtand a full-textllms-full.txt; - structured data marks up the company, the services and the FAQ sections;
- in Google Analytics, traffic from AI sources shows up in its own channel group.
It's too early to talk about measured results, but the method is the same one we recommend to clients. None of it required special tools: it's mostly careful writing, a few configuration lines and structured data that a modern website platform can generate automatically.
A quick self-check: what does an AI know about you now?
Before you change anything, look at the starting point. Ask ChatGPT, Claude and Perplexity these questions and note the answers:
- "What does [your company] do?" Is the answer accurate, and where does it come from?
- "Who would you recommend for [your service] in [your region]?" Are you there, and which competitors are?
- "What's the difference between [your company] and [competitor]?" What does it know about you, and what's missing?
If an AI describes you wrongly or incompletely, it's rarely the AI's fault. Usually your website has no single clear sentence saying who you are and what you do, or other sources say something different.
Five common mistakes
Blocking every bot. Many companies block all AI bots over a security or copyright concern and shut out the search bots in the process. If you want to appear, treat search and training bots separately.
Marketing copy only. "Leading, innovative, customer-centric" sentences contain no fact an AI could quote. No specifics, no citations.
Information hidden in images or PDFs. If your service descriptions live in an image or a downloadable brochure, crawlers struggle to read them. Put important information on the page as text.
Contradictory details. If your website, company registry and social profiles show different names, addresses or activities, the AI becomes unsure and recommends someone else.
Treating it as a one-off project. AI visibility isn't a setting; it's a process. Your services, prices and market change, and if your site shows the state of two years ago, the AI will repeat it. Outdated information is worse than missing information, because the customer arrives with it. Review your main pages and FAQ quarterly, and add short, precise answers to new questions customers ask; those are exactly the sentences an AI likes to quote.
How to measure it
Measuring AI visibility is still in its early days, but a few methods already work:
- Analytics: Google Search Console shows traffic from AI features under the "Web" search type. In Google Analytics, a separate channel group shows how many visitors come from ChatGPT, Claude, Perplexity and other AI sources.
- Manual checks: every month, ask four or five major AI assistants the same ten questions a customer would ask, and note whether you appear and which page they cite.
- Citations: watch which of your content gets cited, and make more of it.
- Enquiries: ask new leads how they heard about you. "An AI recommended you" often doesn't show in analytics at all, because the visitor later types your address directly.
What to do now
- Check your
robots.txt: are the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed? - Rewrite the first paragraph of your main pages into a self-contained, quotable answer.
- Add specifics to your content: numbers, cases, sources.
- Make company information consistent everywhere, and mark it up with structured data.
- Set up measurement: an AI channel group in analytics, plus a monthly ten-question check.
- Publish expert content regularly on the real questions your customers ask, in every language your customers use.
If you'd like us to prepare your website for AI search, from crawlers to content and measurement, have a look at our SEO and digital marketing service.
FAQ
What is GEO?
Generative engine optimisation (GEO) aims to get AI assistants and AI search (ChatGPT, Claude, Perplexity, Google AI Overviews) to cite your content and recommend your company. It builds on good SEO, plus quotable, factual content.
Do we need an llms.txt file?
It's not required, and Google says it's not needed for its AI features. It's cheap and may help some AI agents, but it's not what makes an AI recommend you.
Which bots should we allow?
The search bots: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity) and Googlebot. Whether to allow the training bots (GPTBot, ClaudeBot, Google-Extended) is a separate decision.
How much traffic do AI search engines bring?
Still little, but growing fast: one large study found it was 0.32% of all traffic in early 2026, against 0.02% in 2024. The value can be high, though, because visitors arriving on an AI's recommendation often come with a concrete need.
How long until an AI starts recommending us?
It varies by provider and topic. Allowing bots takes effect within days; content changes take weeks to months. Regular measurement shows what works.
Sources
- Aggarwal et al. – GEO: Generative Engine Optimization (arXiv 2311.09735)
- Google Search Central – AI features and your website
- OpenAI – Overview of OpenAI Crawlers
- Anthropic – Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity – Perplexity Crawlers
- Pew Research Center – Google users are less likely to click on links when an AI summary appears
- SE Ranking – AI traffic research study
- llms.txt – a proposal for LLM-friendly website summaries
