For most of my 15 years in digital marketing, the scoreboard was simple: rank on page one and the clicks follow. That deal is breaking. In a randomised field study by researchers at Carnegie Mellon University and the Indian School of Business, organic clicks fell 38% on queries where Google showed an AI Overview, and zero-click searches rose from 54% to 72%.

Your page can still rank first and lose the click to an AI-written answer that cites someone else. The practice of fixing that has a name: Generative Engine Optimization (GEO) — and it comes from an actual research paper by researchers from Princeton, Georgia Tech, the Allen Institute for AI, and IIT Delhi.

This is the checklist I now use on any website before I spend a single rupee on content. It is engineering first, copywriting second.


First, Cut Through the Noise

A lot of “AI SEO” advice being sold right now is pure guesswork. Start with what the platforms actually say.

Google published its official guide to generative AI features in Search on 15 May 2026. Its position is blunt: to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show a snippet, and there are no additional technical requirements. The guide even names llms.txt and artificial content “chunking” among the things site owners can safely ignore for Google Search.

However, that does not settle it for ChatGPT, Claude, or Perplexity, which run their own independent crawlers and behave very differently from Googlebot. That difference is where most websites are quietly failing.


Layer 1: Make Sure AI Crawlers Can Get In

Most site owners don’t realise that OpenAI runs separate bots for separate jobs:

Each one obeys its own directive block in robots.txt, so blocking GPTBot does not block OAI-SearchBot.

If you want to be cited but would rather not feed model training, the optimal configuration looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

robots.txt is only half the story. Since July 2025, Cloudflare has blocked known AI crawlers by default for new domains, and from 15 September 2026 it extended default blocking of training and agent crawlers to ad-supported pages on new sites and existing free-plan customers. A huge number of small Indian business sites sit behind Cloudflare’s free plan. If nobody on the team made an active choice, the CDN made it for you.

Verify, don’t assume. Your server logs will reveal the truth:

grep -Ei "oai-searchbot|chatgpt-user|gptbot|claudebot|perplexitybot" access.log \
  | awk '{print $9}' | sort | uniq -c

If you see 403s, or no AI bot hits at all, fix that infrastructure leak before you touch a single heading.


Layer 2: Serve Real HTML, Not an Empty Shell

This is the most expensive mistake in modern website builds. Google renders JavaScript. The AI crawlers mostly don’t.

Vercel and MERJ analysed AI crawler traffic across Vercel’s network: in one month GPTBot made 569 million requests and ClaudeBot 370 million. GPTBot downloaded JavaScript files in only 11.5% of its requests and ClaudeBot in 23.84%, and neither executed any of it.

So a React or Vue single-page app that loads pricing, course details, or product specs after the page loads can rank perfectly on Google and still look completely blank to ChatGPT. The two-minute terminal test:

curl -s -A "OAI-SearchBot" https://yoursite.com/pricing | grep -i "₹"

If your key facts aren’t in that raw HTML response, AI search cannot quote them. The fix is server-side rendering or static generation (Next.js, Nuxt, Astro), or prerendering for legacy apps. A WordPress site with a sensible theme is usually fine; a heavily scripted client-side builder often isn’t.


Layer 3: Write Pages a Model Can Lift From

Once crawlers can read you, the content has to be worth quoting. The Princeton-led GEO paper tested nine content changes across 10,000 queries. Adding statistics, quotations, and cited sources produced the biggest gains — up to roughly 40% higher visibility in generated answers. Keyword stuffing did nothing.

In practice, that means:


Layer 4: Build the Entity, Not Just the Links

This is where the old SEO playbook changes most. Ahrefs studied 75,000 brands and found that branded web mentions had a correlation of 0.664 with visibility in AI Overviews, against just 0.218 for raw backlinks. Branded anchors (0.527) and branded search volume (0.392) also beat raw link counts.

Language models learn what a brand is from how the broader web talks about it. Consistent mentions across industry publications, directories, podcasts, community forums, and partner sites build that neural representation. One caution: Google’s guide explicitly warns against inauthentic mention campaigns. Earn the mentions; don’t manufacture spam.


Layer 5: Measure What Matters Now

Rankings alone no longer reflect commercial impact. Track three metrics instead:

  1. AI referrals in GA4: Segment sessions originating from chatgpt.com, perplexity.ai, gemini.google.com, and copilot.microsoft.com.
  2. Citation share: Maintain a fixed set of 20 to 30 core prompts your buyers ask, run them monthly across ChatGPT, Gemini, and Perplexity, and log who gets cited.
  3. Crawler health: A weekly log check for AI bot HTTP status codes catches a CDN or firewall mistake before it costs you a month of organic reach.

It is well worth the effort. When Seer Interactive analysed results pages carrying AI Overviews, brands cited inside the Overview earned 35% more organic clicks and 91% more paid clicks than non-cited competitors on the same page.


So, What About llms.txt?

It is cheap to implement, so ship it if you like, but keep expectations realistic. Google has confirmed it doesn’t use the file, and studies of crawler logs show most AI bots skip it and read your raw HTML directly. It is not a magical ranking switch. Spend your time on Layers 1 and 2 — that is where real visibility is lost.


The Owner’s View

As someone who runs a business, I look at GEO the way I look at any infrastructure investment: where is the leak, and what does it cost to plug? For most websites, the leak isn’t a missing AI gimmick. It is a firewall blocking the right bots, a JavaScript build that hides critical facts, and content that never states a clear answer.

None of that requires a new tool. It requires marketers who can read a robots.txt file, run a curl command, and talk to developers in their language. That combination of technical literacy and marketing judgement is what separates the brands that will be cited from the brands that will be summarised away.


About the Author

Shrey Ranjan Sinha is Co-Founder of US Soft Lucknow, a premier AI-integrated digital marketing institute in Lucknow. He brings over 15 years of corporate digital marketing experience, has built scalable web properties ranging from single-page web apps to full-scale e-commerce architectures, and holds an executive Certificate in Digital Marketing and Strategy from IIM Indore. At US Soft, he leads technical curricula across SEO, Google Ads, Meta Ads, and Web Architecture at the Advanced Digital Marketing Programs.

Campus Address: US Soft Lucknow, 2nd & 3rd Floor, 633/020, Ayodhya Road, Kamta, Lucknow, Uttar Pradesh 226028
Phone / WhatsApp: +91 98892 71727 | Email: ussoftlucknow@gmail.com
Official Website: https://ussoftlucknow.in