I Tested Markdown for AI Search & the Crawlers Never Asked for it

Three robot crawlers walking past a glowing markdown file toward an HTML page
GEOOriginal dataAI crawlersCase study

I Tried to Test Markdown for AI Search. The Crawlers Never Asked for It.

A controlled GEO case study across 3 sites, 170 posts, 245 prompts and 6 AI engines. In 22 days and 16,298 logged bot requests, not one AI crawler asked for a .md file, llms.txt barely got touched, and AI citations changed more on their own than any change I could have made.

By Jourdan Rombough, Optimization TheorySources: first-party bot logs (3 sites), DataForSEO citation runs

Jump to the findings

mdgeo-bot-log.csv · 16,298 rows · Oct 24 to Nov 14, 2026GET/peter-pans-shadow-the-psychological-depth/GPTBot/1.2text/htmlGET/sitemap.xmlClaudeBot/1.0text/html,application/xhtml+xmlGET/the-ten-theories-of-magic-visual-guide/PerplexityBot/1.0text/htmlGET/feed/GPTBot/1.2application/rss+xmlGET/interactive-pagerank-tool/OAI-SearchBot/1.0text/htmlGET/ga4-seo-dashboards/Googlebot/2.1text/htmlGET/case-studies-ga4-bot-traffic-ai-visitors/Claude-User (claude-code)text/markdown, text/htmlGET/high-resolution-garbage-pail-kids-images/Amazonbot/0.1text/htmlCrawler requests asking for markdown:0

Illustration built from real rows in the study's bot log. Paths shortened.

Every GEO checklist says to publish markdown and an llms.txt file so AI engines can read you. I set up a controlled test on three of my own sites to prove it. In 22 days and 16,298 bot requests, not one AI crawler asked for markdown.

Background: my markdown for AI search and llms.txt best practices guide covers how to set these up. This case study is the follow-up: what actually happened when I measured it.

16,298bot requests logged across 3 sites
0crawler requests that asked for markdown
16llms.txt requests in 22 days
1 in 3AI citations repeated when I reran the same prompts
What's in this case study
  1. Why I ran this test
  2. The setup
  3. No crawler asked for markdown
  4. llms.txt barely gets touched
  5. Who actually crawls your site
  6. AI citations churn on their own
  7. Crawled pages get cited
  8. Engines disagree and cost differs
  9. Some AI bots are scanners
  10. What to do with this
  11. Limitations, method and FAQ

Why I ran this test

Markdown for AI is everywhere in 2026. Plugins generate a .md copy of every post in one click, and llms.txt files have become a standard line item in GEO audits. The pitch is simple: language models prefer clean text, so give it to them and they will read you, understand you and cite you more.

I could not find anyone who had tested that with a control group. So I set one up on my own sites. The hypothesis: if AI crawlers prefer markdown, pages with a .md version should get fetched more by AI bots and cited more in AI answers than matched pages without one.

The setup

Three websites split into two balanced groups of pages, with prompts sent to six AI engines

I used three sites with very different audiences: vuss.io (culture and entertainment), infographi.ac (infographics) and optimizationtheory.com (SEO). Eligible posts were split at random into two matched groups, balanced on Search Console impressions, word count and crawl depth. Any page with more than 25% of its site's impressions was held out so one big page could not swing the result.

3 sitesvuss.io, infographi.ac,optimizationtheory.com170 postsrandom splittwo matched groups245 promptsfrozen Oct 24one post per prompt6 AI enginesvia DataForSEO2 runs, 2,201 callsBot logger on all 3 sites: every bot request with user agent, IP and Accept header · 16,298 rows in 22 days

For each post I wrote search-style prompts and froze the panel at 245 prompts before any data came in. Citations were collected from ChatGPT, Claude, Gemini, Perplexity, Google AI Mode and Google AI Overviews through the DataForSEO API. A small WordPress snippet logged every bot request with its user agent, IP and Accept header, so I could see exactly what each crawler asked for. This case study reports what the crawlers requested over 22 days and how AI citations behaved across two identical runs, two weeks apart.

Finding 1: No AI crawler asked for markdown

Robot crawlers lining up to enter an HTML doorway while a markdown file and llms.txt sit unvisited under a cobweb

Across 16,298 bot requests, no crawler requested a .md URL or sent an Accept: text/markdown header. Every major AI crawler asked for plain HTML.

From 16,298 bot requests to zero markdown requests
16,298bot requests logged16requests for llms.txt7requests that asked for markdown (all Claude Code)0crawler requests that asked for markdown

Bar widths on a log scale so the small numbers stay visible. Source: MD GEO bot logger on vuss.io, infographi.ac and optimizationtheory.com.

The only agent that asked for markdown was Claude Code's web fetcher, an agentic coding tool rather than a search crawler. It sent 7 requests, and some were likely my own sessions. Every one of them asked for markdown the same way:

Accept: text/markdown, text/html, */*

That detail matters. When an AI agent wants markdown, it asks for it at the normal URL through the Accept header. It does not go looking for a separate .md file.

Finding 2: llms.txt barely gets touched

All three sites publish an llms.txt file. In 22 days the logs showed 16 requests for it, all on optimizationtheory.com. vuss.io and infographi.ac got none. Meanwhile crawlers pulled sitemaps more than a thousand times.

What crawlers fetched besides your pages
sitemaps1,011RSS feeds645robots.txt455llms.txt16.md files0

Requests from all bots, all 3 sites, 22 days. All three sites publish an llms.txt.

Who asked for llms.txt
Amazonbot6Googlebot4OAI-SearchBot2Claude-SearchBot2bingbot1Applebot1GPTBot0ClaudeBot0PerplexityBot0

16 requests total, all on optimizationtheory.com. The three biggest AI crawlers never asked.

The crawlers that did fetch llms.txt were mostly Amazonbot and Googlebot. GPTBot, ClaudeBot and PerplexityBot, the three crawlers people usually have in mind when they publish an llms.txt, never requested it.

Finding 3: Who actually crawls your site

AI and LLM crawlers sent nearly half of all bot requests. The single busiest bot was not an AI crawler at all: Huawei's PetalBot made about 4,700 requests, more than half of them on infographi.ac. Live fetches, the requests a chatbot makes when someone asks it a question right now, were about 1% of bot traffic.

Who sent the 16,298 requests
16,298bot requestsAI and LLM crawlers7,788 · 47.8%Other bots (mostly PetalBot)5,130 · 31.5%Search engines3,196 · 19.6%Live AI fetches184 · 1.1%

All 3 sites, Oct 24 to Nov 14. Live AI fetches are ChatGPT-User, Claude-User and Perplexity-User.

When AI crawlers show up
00:0006:0012:0018:00UTChour16:00 UTC847 AI crawler requests,2.6x the hourly averageBusiest hourOther hours

AI and LLM crawler requests by hour of day, all 3 sites, 22 days. Bar length is to scale.

Daily bot requests, all 3 sites
All botsAI and LLM crawlers

06001,2001,800Nov 3: 1,619 (GPTBot burst, 550 requests on optimizationtheory.com)logger gapOct 25Oct 28Oct 31Nov 3Nov 6Nov 9Nov 12

Full days Oct 25 to Nov 13. Oct 30 is low because the logger missed most of that day.

Each site attracts a different crawler mix. ClaudeBot and GPTBot concentrate on optimizationtheory.com, Bytespider and bingbot prefer vuss.io, and PetalBot hammers infographi.ac.

Bot requests by site
vuss.io
infographi.ac
optimization­theory.com
PetalBot
1,069
2,712
964
Amazonbot
668
784
898
Googlebot
201
313
812
PerplexityBot
163
403
630
meta-externalagent
62
282
757
Bytespider
627
24
394
Applebot
430
300
293
bingbot
549
17
281
ClaudeBot
33
157
515
GPTBot
34
45
593
Claude-SearchBot
276
15
78
OAI-SearchBot
70
129
107
ChatGPT-User
40
53
43

Darker cells mean more requests (square-root scale so smaller bots stay readable). Oct 24 to Nov 14.

Finding 4: AI citations churn enough to fake a result

Two identical AI answer cards dated Oct 29 and Nov 13, with most citation sources swapped out and one staying in place

This is the finding I did not expect. I ran the same 245 prompts on Perplexity, Google AI Mode and Google AI Overviews twice, on Oct 29 and Nov 13. Nothing changed on any of the three sites in between. 24 answers cited one of my sites the first time, 15 the second time, and only 8 citations showed up both times.

Same prompts, same pages, two weeks apart
16Oct 29 only8Middle: cited both times7Nov 13

Cited answers on Perplexity, Google AI Mode and Google AI Overviews. Circle areas are to scale.

Citations by engine, no site changes
Oct 29Nov 13Perplexity 1410Google AI Mode 65Google AI Overviews 40

Google AI Overviews went from 4 citations to none. 731 paired calls.

Because the posts were already split into two matched groups, I could see what a before and after comparison looks like when nothing changes at all:

The fake effect: identical pages, opposite moves
Oct 29Nov 13

051015155Group A-67%910Group B+11%

Cited answers by random group. Nothing changed on these pages between the two runs, so any difference here is noise.

If I had published .md files between these two runs, this exact data would have read as "markdown cut AI citations by two thirds."One group fell from 15 to 5 cited answers. The other rose from 9 to 10. Not a single page changed.

Most GEO case studies compare one snapshot before a change to one snapshot after. This is why that is not enough. You need a control group, repeated runs, or both.

Finding 5: Pages AI bots crawl are the pages that get cited

Of the 170 posts in the test, 33 were cited at least once. When I grouped posts by how many AI crawler visits they got in 22 days, the citation rate climbed with every step.

The more AI bots visit a post, the more often it gets cited
0%10%20%30%0%00 of 6 posts12%1 to 48 of 67 posts22%5 to 98 of 36 posts26%10 to 1911 of 42 posts32%20+6 of 19 postsAI bot visits per post in 22 days

Share of posts cited at least once in either run, grouped by AI crawler visits. Correlation, not proof of cause.

16 of the 17 posts Perplexity cited had been visited by PerplexityBot, compared with about half of the posts it did not cite. The 6 posts no AI bot ever visited were never cited. This is a correlation: bots may also visit pages more because those pages already appear in answers. Either way, a page AI crawlers never reach is a page they cannot cite.

Finding 6: Engines disagree, and cost very different amounts to track

23 of the 33 cited prompts were cited by only one of the six engines. Visibility in one AI tool tells you very little about the others.

How many engines cited the same prompt
081624231 engine42 engines33 engines14 engines25 engines

33 prompts cited on Oct 29, out of 6 engines. 23 were cited by just one.

Share of prompts that cited each site
13.5%vuss.io20 of 148 prompts cited12.1%infographi.ac7 of 58 prompts cited15.4%optimizationtheory.com6 of 39 prompts cited

Oct 29 run, cited by at least one of 6 engines.

Tracking costs vary just as much. Perplexity found 14 citations for about $1.51 per run. Claude found 10 for about $12.81. Gemini cited by far the most sources per answer, which gives a mid-sized site more chances to show up.

What each engine costs to track, and what it finds
0481216$0$2$4$6$8$10$12$14Cost per full run of 245 promptsCited answersPerplexityGoogle AI ModeGoogle AI OverviewsGeminiClaudeChatGPT

Oct 29 run via DataForSEO. Bubble size shows how many sources each engine cites per answer.

Sources cited per answer
Gemini31.1Claude16.0Perplexity10.0ChatGPT5.8Google AI Overviews3.6Google AI Mode2.1

More sources per answer means more chances for a mid-sized site to appear.

Two quirks if you track AI referrals: ChatGPT adds ?utm_source=openai to every link it cites, which makes those clicks easy to segment in GA4 dashboards. Gemini hides cited pages behind Google redirect URLs, so URL matching misses it unless you match on the citation title.

Finding 7: Some "AI bots" are scanners in disguise

A disguised robot wearing an AI BOT badge blocked by a verification shield while a verified crawler reaches the vault

44 requests probed for secret files like /proc/self/environ, credentials.json and secrets.env. 19 of them used an AI bot's name, including ChatGPT-User, Claude-User and PerplexityBot. Others posed as Meta and Cohere crawlers from cloud server IPs.

44 requests hunting for secret files
Posing as live AI fetchers · 12Posing as AI crawlers · 7Other user agents · 25

Each square is one request for files like /proc/self/environ, credentials.json or secrets.env. Live AI fetchers: ChatGPT-User, Claude-User, Perplexity-User.

If your AI traffic report trusts the user agent, part of what it counts as AI is attackers. Verify bots against the IP ranges each company publishes.

I built a free plugin for this. AI Crawler Companion logs AI bot visits in WordPress and checks each one against OpenAI, Perplexity, Google, Bing and Apple's published IP ranges.

Get it on GitHub

What to do with this

  • ✓Get crawled firstCheck robots.txt, your CDN and your firewall for PerplexityBot, OAI-SearchBot, Claude-SearchBot and Googlebot. Every cited post in this test had AI bot visits.
  • ✓Keep .md and llms.txt expectations lowThey are cheap to add, but nothing here suggests crawlers use them yet. Do not sell them to clients as a citation lever.
  • ✓Serve markdown at the same URLIf you offer markdown, return it when a client asks for text/markdown. That is how the one agent that wants it asks.
  • ✓Answer narrow, named questionsThe citations that held up across both runs were specific: a named person, a named work, a named tool.
  • ✓Never trust one before and one afterIdentical pages swung by two thirds. Use a control group and repeat your runs.
  • ✓Track Perplexity and Google firstThey cost cents per citation. Add ChatGPT, Claude and Gemini when the budget allows.

Limitations, method and FAQ

  • 1This study measured crawler requests and citation stability. It did not publish .md versions of the test pages, so it does not measure their direct effect on citations.
  • 2Two citation runs are enough to show the noise, not to measure it precisely.
  • 3The bot logger runs inside WordPress, so requests answered from the CDN cache were not logged. All counts are a floor.
  • 4Three small to mid-sized sites. Large publishers may see different crawler behavior.

Method. Random split of 170 eligible posts into two matched groups (87 and 83) with seeded assignment and balance checks. Prompt panel of 245 prompts frozen Oct 24, 2026. Citation runs on Oct 29 (6 engines, 1,468 calls) and Nov 13 (Perplexity, Google AI Mode and Google AI Overviews, 733 calls) through DataForSEO. Bot logs from Oct 24 to Nov 14, 2026, with QA test requests removed.

Do AI crawlers read markdown (.md) files?

In this 22-day test, no. None of the major AI crawlers requested a .md URL or asked for markdown in their Accept header. The only agent that asked for markdown was Claude Code's web fetcher.

Does llms.txt help with AI search?

There was no sign of it here. Three sites got 16 llms.txt requests in 22 days, mostly from Amazonbot and Googlebot. GPTBot, ClaudeBot and PerplexityBot never requested it.

How stable are AI citations?

Not very. Running the same 245 prompts two weeks apart with no site changes, only 8 of 24 citations repeated, and Google AI Overviews went from 4 citations to none.

What gets a page cited by AI engines?

In this data, being crawled. Citation rates rose from 0% for posts with no AI bot visits to 32% for posts with 20 or more. Narrow questions about named people, works and tools also held their citations best.

Which AI engine is cheapest to track?

Perplexity, at about $0.11 per citation found through DataForSEO, followed by Google AI Mode and Google AI Overviews at under $0.20.