Is my website AI-ready? 8 things crawlers and agents need

Your website is AI-ready when AI crawlers are allowed in, your facts are written in the page itself, and a machine can use the page without guessing. Most of that is ordinary SEO work. Below are eight things to check, which ones Kashif checks, and a free Chrome tool that covers the agent side.

Google says you don't need new machine-readable files or special markup to appear in AI Overviews or AI Mode. In its own guide, optimizing for AI search is "still SEO". So read this list as the basics machines depend on. None of it is a trick.

1. Let the AI crawlers you want in through robots.txt

robots.txt is the file at yourdomain.com/robots.txt that tells crawlers where they may go. OpenAI runs separate crawlers for separate jobs, and you can treat each one differently:

• OAI-SearchBot fetches pages for ChatGPT search. OpenAI says sites that opt out of it "will not be shown in ChatGPT search answers".
• GPTBot collects pages for training OpenAI's models. Blocking it is a separate choice from search.
• ChatGPT-User visits a page when a person asks ChatGPT to open it. OpenAI notes that robots.txt rules may not apply to these visits.

So a site can say no to training and still allow search. Look for an old rule that blocks every bot from the whole site, and check that it is not shutting out the crawlers you want.

2. Check your Cloudflare or hosting setting

robots.txt is a request. A firewall can turn a crawler away before it ever reads that file. Cloudflare has its own AI-crawler setting: block, block only on pages with ads, allow, or disallow training only. Its docs, last updated on 2 November 2026, say new domains allow search, agents and training by default. A domain set up earlier may have a different setting, so open your dashboard and look.

3. Put your words in the HTML

Not every crawler runs JavaScript. If your hours, services or prices appear only after a script runs, a crawler that reads the page as the server sends it finds a gap where they should be. You can test this yourself: open the page, choose "View page source" and search for a sentence you care about. If it is not there, it depends on a script.

4. Write menus and prices as text

A photo of a menu works for a person standing at the counter. A crawler that reads text gets nothing from it. Put dishes, services, prices and hours on the page as text, so they can be read and quoted. Keep the photo next to it if you like it.

5. Make contact details easy to find

When someone asks an assistant how to reach a business, the answer can only use details it can read. Put your phone number, address and email as text on the homepage and the contact page. A tap-to-call link and a WhatsApp link help people on phones as well.

6. Add structured data where it fits

Structured data is code that labels your facts: this is the business name, this is the address, these are the opening hours. Google says it isn't required for its AI search features, and that it is still useful because it makes pages eligible for rich results. Mark up facts that are true and visible on the page. Don't add ratings or reviews that visitors cannot see.

7. Make pages an agent can use

An AI agent may do more than read. It may try to fill a form or press a button for someone. web.dev, Google's site for web developers, lists best practices for agent-friendly sites. Four of them most small sites can act on now:

• Use real buttons, links and headings (semantic HTML) instead of boxes made clickable by a script.
• Link every form field to its label, so "Phone number" is attached to the phone box.
• Keep the layout stable, so buttons stay where an agent expects to find them.
• Avoid ghost overlays: transparent layers on top of the page that can hide the buttons underneath from an agent.

The others are clear cursor cues, a minimum size so an agent can see the things it needs to click, and WebMCP tools.

8. Know what WebMCP is, and don't rush it

WebMCP lets a site describe actions an agent can call directly, such as booking a table, instead of the agent clicking through the page. It is currently in a Chrome origin trial, which means it is still being tested. If your site runs on a common platform, waiting for the platform to add it is a reasonable choice.

What Kashif checks, and what Lighthouse checks

Kashif reads up to 8 pages of your site as your server sends them, without running scripts. Text that only appears after a script runs may be missed, which is the same gap item 3 describes.

• Readable by AI: whether robots.txt lets AI crawlers in, facts in a form assistants can quote, enough text to answer a customer's question, your business name on the homepage, whether a restaurant menu is text or a picture, and whether you have an llms.txt file.
• Getting found: among other things, business details in a form search engines can read.
• Easy to reach: a WhatsApp link, a tap-to-call number, a contact form, an email link, and how easy they are to find.
• Ready for AI agents: image descriptions, named form fields, whether the layout holds still, and whether you publish a file of actions an agent can call. Not having that file does not count against you.

Kashif does not ask ChatGPT or any other assistant about your business, and it does not test whether an agent can finish a booking. A strong score means your facts can be read. It does not mean an assistant will name you.

For the agent checks, Chrome's Lighthouse is a free alternative. Its Agentic Browsing category looks for llms.txt, WebMCP tools, names and labels on page elements, and layout shift. It is experimental, needs Chrome 150 or later, and gives a pass ratio instead of a score out of 100.

llms.txt has its own post: llms.txt in 2027: does Google use it? To see these checks on your own site, run the free scan, or read how we score first.

Sources

• Google: Optimizing your website for generative AI features on Google Search (seen 11 November 2026)
• Google Search Central: AI features (AI Overviews and AI Mode) (seen 11 November 2026)
• OpenAI: crawlers and user agents (seen 11 November 2026)
• Cloudflare: AI crawler settings (page updated 2 November 2026, seen 11 November 2026)
• web.dev: Build agent-friendly websites (seen 11 November 2026)
• Chrome: Lighthouse Agentic Browsing scoring (seen 11 November 2026)