I Tested Markdown for AI Search & the Crawlers Never Asked for it

I Tried to Test Markdown for AI Search. The Crawlers Never Asked for It.
A controlled GEO case study across 3 sites, 170 posts, 245 prompts and 6 AI engines. In 22 days and 16,298 logged bot requests, not one AI crawler asked for a .md file, llms.txt barely got touched, and AI citations changed more on their own than any change I could have made.
Illustration built from real rows in the study's bot log. Paths shortened.
Every GEO checklist says to publish markdown and an llms.txt file so AI engines can read you. I set up a controlled test on three of my own sites to prove it. In 22 days and 16,298 bot requests, not one AI crawler asked for markdown.
Background: my markdown for AI search and llms.txt best practices guide covers how to set these up. This case study is the follow-up: what actually happened when I measured it.
What's in this case study
Why I ran this test
Markdown for AI is everywhere in 2026. Plugins generate a .md copy of every post in one click, and llms.txt files have become a standard line item in GEO audits. The pitch is simple: language models prefer clean text, so give it to them and they will read you, understand you and cite you more.
I could not find anyone who had tested that with a control group. So I set one up on my own sites. The hypothesis: if AI crawlers prefer markdown, pages with a .md version should get fetched more by AI bots and cited more in AI answers than matched pages without one.
The setup

I used three sites with very different audiences: vuss.io (culture and entertainment), infographi.ac (infographics) and optimizationtheory.com (SEO). Eligible posts were split at random into two matched groups, balanced on Search Console impressions, word count and crawl depth. Any page with more than 25% of its site's impressions was held out so one big page could not swing the result.
For each post I wrote search-style prompts and froze the panel at 245 prompts before any data came in. Citations were collected from ChatGPT, Claude, Gemini, Perplexity, Google AI Mode and Google AI Overviews through the DataForSEO API. A small WordPress snippet logged every bot request with its user agent, IP and Accept header, so I could see exactly what each crawler asked for. This case study reports what the crawlers requested over 22 days and how AI citations behaved across two identical runs, two weeks apart.
Finding 1: No AI crawler asked for markdown

Across 16,298 bot requests, no crawler requested a .md URL or sent an Accept: text/markdown header. Every major AI crawler asked for plain HTML.
Bar widths on a log scale so the small numbers stay visible. Source: MD GEO bot logger on vuss.io, infographi.ac and optimizationtheory.com.
The only agent that asked for markdown was Claude Code's web fetcher, an agentic coding tool rather than a search crawler. It sent 7 requests, and some were likely my own sessions. Every one of them asked for markdown the same way:
Accept: text/markdown, text/html, */*
That detail matters. When an AI agent wants markdown, it asks for it at the normal URL through the Accept header. It does not go looking for a separate .md file.
Finding 2: llms.txt barely gets touched
All three sites publish an llms.txt file. In 22 days the logs showed 16 requests for it, all on optimizationtheory.com. vuss.io and infographi.ac got none. Meanwhile crawlers pulled sitemaps more than a thousand times.
Requests from all bots, all 3 sites, 22 days. All three sites publish an llms.txt.
16 requests total, all on optimizationtheory.com. The three biggest AI crawlers never asked.
The crawlers that did fetch llms.txt were mostly Amazonbot and Googlebot. GPTBot, ClaudeBot and PerplexityBot, the three crawlers people usually have in mind when they publish an llms.txt, never requested it.
Finding 3: Who actually crawls your site
AI and LLM crawlers sent nearly half of all bot requests. The single busiest bot was not an AI crawler at all: Huawei's PetalBot made about 4,700 requests, more than half of them on infographi.ac. Live fetches, the requests a chatbot makes when someone asks it a question right now, were about 1% of bot traffic.
All 3 sites, Oct 24 to Nov 14. Live AI fetches are ChatGPT-User, Claude-User and Perplexity-User.
AI and LLM crawler requests by hour of day, all 3 sites, 22 days. Bar length is to scale.
Full days Oct 25 to Nov 13. Oct 30 is low because the logger missed most of that day.
Each site attracts a different crawler mix. ClaudeBot and GPTBot concentrate on optimizationtheory.com, Bytespider and bingbot prefer vuss.io, and PetalBot hammers infographi.ac.
Darker cells mean more requests (square-root scale so smaller bots stay readable). Oct 24 to Nov 14.
Finding 4: AI citations churn enough to fake a result

This is the finding I did not expect. I ran the same 245 prompts on Perplexity, Google AI Mode and Google AI Overviews twice, on Oct 29 and Nov 13. Nothing changed on any of the three sites in between. 24 answers cited one of my sites the first time, 15 the second time, and only 8 citations showed up both times.
Cited answers on Perplexity, Google AI Mode and Google AI Overviews. Circle areas are to scale.
Google AI Overviews went from 4 citations to none. 731 paired calls.
Because the posts were already split into two matched groups, I could see what a before and after comparison looks like when nothing changes at all:
Cited answers by random group. Nothing changed on these pages between the two runs, so any difference here is noise.
If I had published .md files between these two runs, this exact data would have read as "markdown cut AI citations by two thirds."One group fell from 15 to 5 cited answers. The other rose from 9 to 10. Not a single page changed.
Most GEO case studies compare one snapshot before a change to one snapshot after. This is why that is not enough. You need a control group, repeated runs, or both.
Finding 5: Pages AI bots crawl are the pages that get cited
Of the 170 posts in the test, 33 were cited at least once. When I grouped posts by how many AI crawler visits they got in 22 days, the citation rate climbed with every step.
Share of posts cited at least once in either run, grouped by AI crawler visits. Correlation, not proof of cause.
16 of the 17 posts Perplexity cited had been visited by PerplexityBot, compared with about half of the posts it did not cite. The 6 posts no AI bot ever visited were never cited. This is a correlation: bots may also visit pages more because those pages already appear in answers. Either way, a page AI crawlers never reach is a page they cannot cite.
Finding 6: Engines disagree, and cost very different amounts to track
23 of the 33 cited prompts were cited by only one of the six engines. Visibility in one AI tool tells you very little about the others.
33 prompts cited on Oct 29, out of 6 engines. 23 were cited by just one.
Oct 29 run, cited by at least one of 6 engines.
Tracking costs vary just as much. Perplexity found 14 citations for about $1.51 per run. Claude found 10 for about $12.81. Gemini cited by far the most sources per answer, which gives a mid-sized site more chances to show up.
Oct 29 run via DataForSEO. Bubble size shows how many sources each engine cites per answer.
More sources per answer means more chances for a mid-sized site to appear.
Two quirks if you track AI referrals: ChatGPT adds ?utm_source=openai to every link it cites, which makes those clicks easy to segment in GA4 dashboards. Gemini hides cited pages behind Google redirect URLs, so URL matching misses it unless you match on the citation title.
Finding 7: Some "AI bots" are scanners in disguise

44 requests probed for secret files like /proc/self/environ, credentials.json and secrets.env. 19 of them used an AI bot's name, including ChatGPT-User, Claude-User and PerplexityBot. Others posed as Meta and Cohere crawlers from cloud server IPs.
Each square is one request for files like /proc/self/environ, credentials.json or secrets.env. Live AI fetchers: ChatGPT-User, Claude-User, Perplexity-User.
If your AI traffic report trusts the user agent, part of what it counts as AI is attackers. Verify bots against the IP ranges each company publishes.
I built a free plugin for this. AI Crawler Companion logs AI bot visits in WordPress and checks each one against OpenAI, Perplexity, Google, Bing and Apple's published IP ranges.
What to do with this
- ✓Get crawled firstCheck robots.txt, your CDN and your firewall for PerplexityBot, OAI-SearchBot, Claude-SearchBot and Googlebot. Every cited post in this test had AI bot visits.
- ✓Keep .md and llms.txt expectations lowThey are cheap to add, but nothing here suggests crawlers use them yet. Do not sell them to clients as a citation lever.
- ✓Serve markdown at the same URLIf you offer markdown, return it when a client asks for
text/markdown. That is how the one agent that wants it asks. - ✓Answer narrow, named questionsThe citations that held up across both runs were specific: a named person, a named work, a named tool.
- ✓Never trust one before and one afterIdentical pages swung by two thirds. Use a control group and repeat your runs.
- ✓Track Perplexity and Google firstThey cost cents per citation. Add ChatGPT, Claude and Gemini when the budget allows.
Limitations, method and FAQ
- 1This study measured crawler requests and citation stability. It did not publish .md versions of the test pages, so it does not measure their direct effect on citations.
- 2Two citation runs are enough to show the noise, not to measure it precisely.
- 3The bot logger runs inside WordPress, so requests answered from the CDN cache were not logged. All counts are a floor.
- 4Three small to mid-sized sites. Large publishers may see different crawler behavior.
Method. Random split of 170 eligible posts into two matched groups (87 and 83) with seeded assignment and balance checks. Prompt panel of 245 prompts frozen Oct 24, 2026. Citation runs on Oct 29 (6 engines, 1,468 calls) and Nov 13 (Perplexity, Google AI Mode and Google AI Overviews, 733 calls) through DataForSEO. Bot logs from Oct 24 to Nov 14, 2026, with QA test requests removed.
Do AI crawlers read markdown (.md) files?
In this 22-day test, no. None of the major AI crawlers requested a .md URL or asked for markdown in their Accept header. The only agent that asked for markdown was Claude Code's web fetcher.
Does llms.txt help with AI search?
There was no sign of it here. Three sites got 16 llms.txt requests in 22 days, mostly from Amazonbot and Googlebot. GPTBot, ClaudeBot and PerplexityBot never requested it.
How stable are AI citations?
Not very. Running the same 245 prompts two weeks apart with no site changes, only 8 of 24 citations repeated, and Google AI Overviews went from 4 citations to none.
What gets a page cited by AI engines?
In this data, being crawled. Citation rates rose from 0% for posts with no AI bot visits to 32% for posts with 20 or more. Narrow questions about named people, works and tools also held their citations best.
Which AI engine is cheapest to track?
Perplexity, at about $0.11 per citation found through DataForSEO, followed by Google AI Mode and Google AI Overviews at under $0.20.