Crawl Budget Analysis Template

Technical SEO Toolkit, template 2 of 6

Crawl Budget Analysis Template

Group verified Googlebot requests by URL pattern, measure crawl waste and pick the right fix for each pattern.

Which URL patterns soak up crawl requests without returning value, how much crawl activity each wastes, and what to do about it.

  • Excel workbook (.xlsx), opens in Google Sheets
  • PDF field guide, 9 to 11 pages
  • Blank and worked CSV templates
  • DataForSEO import script

Free, no email required. Version 3.0, updated November 2026.

Crawl Budget Analysis Template.xlsx
I13fx=IF(H13="","",IF(H13<>"Ready","n.a.",B13-C13))

Start HereDashboardWorkspaceDelivery QAField Guide

The formula above is the real one from cell I13. Scroll it sideways to read the whole rule.

Connected to the DataForSEO API

This template is designed around one DataForSEO endpoint. Pull the data, convert it with one command, and the workbook does the rest.

Setup steps

DataForSEO On-Page APIA crawl of your site: URLs, status codes, canonicals, inlinks and sitemap membershipon_page/task_post

Save the JSON responseThe full API response, kept as your source of truth.response.json

Run the import scriptOffline. No API calls, no credentials.dataforseo_import.py import --template 14

Paste at A13Dashboard, checks and action list update instantly.Crawl Budget Analysis Template.xlsx

Column map for this template:Filled by the APIAudit trail (automatic)You add

URL patternVerified bot requestsRequests to valuable URLsUnique requested URLsAverage internal inlinksNext actionSource reference
1 of 7 input columns come straight from DataForSEO (the audit trail; this one is a human sign-off workflow)

What the crawl budget analysis template decides

Crawl budget is a real constraint on large sites: ecommerce catalogs with faceted navigation, marketplaces, publishers with deep archives. When Googlebot spends hundreds of thousands of requests on sort orders, filter combinations and session IDs, new and updated pages wait longer to be crawled.

This crawl budget analysis template works from verified Googlebot log data grouped by URL pattern. For each pattern you enter total verified requests and how many hit valuable URLs, plus unique URLs and internal links. It calculates requests to review, the share of the pattern that's waste, and requests per URL, so you can match each pattern to the right fix.

Who it's for

  • Technical SEOs on large ecommerce or publisher sites
  • Developers deciding how to handle faceted navigation and parameters
  • Teams investigating slow indexing of new pages
  • Anyone with server logs and a suspicion something's wrong

Where it fits in your workflow

The crawl budget analysis template answers one question. These three templates answer the questions around it:

  • Technical SEO Prioritization Template: which technical issues to fix first, ranked by an explicit estimate of recoverable clicks and monthly profit per hour of effort.
  • Indexability Audit Template: which pages that should be indexed have a technical conflict, which are technically fine but still not indexed, and which were never meant to be indexed.
  • XML Sitemap Audit Template: which URLs should be removed from your XML sitemap (or fixed), which eligible URLs are missing from it, and which are fine.

What's inside the download

One ZIP with the workbook, a 7-column input template (blank and filled), the PDF field guide and an offline script that converts saved DataForSEO responses into the input CSV. The workbook has five tabs.

Workspace. The working table. 7 input columns, 4 calculated columns, room for 250 rows. Dropdowns, Yes/No fields and range checks catch bad inputs as you paste.
Start Here. What it decides, a four-step setup and the colour key.
Dashboard. KPI cards, charts, a ranked Top 10 and an automatic action list.
Delivery QA. Pass, fail or exclude each validation step with evidence, owner and date.
Field Guide. Every field explained, plus each formula in plain English.

How the log file analysis SEO calculation works

Requests to review = verified requests − valuable requests. Review share = requests to review ÷ verified requests. Requests per URL = verified requests ÷ unique URLs.

The key formula, exactly as it appears in the workbook

=IF(H13="","",IF(H13<>"Ready","n.a.",B13-C13))
Result column Plain-English formula What it tells you
Requests to review Verified bot requests-Requests to valuable URLs Requests outside the approved valuable subset. Not all are waste.
Review share IF(Verified bot requests=0,"n.a.",Requests to review ÷ Verified bot requests) Share of verified requests outside the valuable subset.
Requests / URL Verified bot requests ÷ Unique requested URLs Request frequency for this observed window.

How to use the crawl budget analysis template

  1. Verify the botFilter logs to verified Googlebot using reverse DNS. User agents alone are easily spoofed.
  2. Group by patternBucket URLs into patterns: parameters, folders, templates. Count requests and unique URLs per pattern for a fixed window.
  3. Mark valuable requestsValuable means the request hit an indexable, canonical URL you want crawled. Join with your crawl data.
  4. Choose an action per patternHigh review share with many internal links usually means fixing links; high requests per URL means a trap.

The worked example in the Workspace tab. Clear the lime input cells, paste your CSV at A13 (values only) and every result recalculates.

Fill the crawl budget analysis template with DataForSEO

Every template in this library is built to run on DataForSEO data. This one uses DataForSEO On-Page API: a crawl of your site: URLs, status codes, canonicals, inlinks and sitemap membership.

DataForSEO supplies the URL or query list and the audit trail; you add the measurements listed below.

  1. Request the dataSend this request with your DataForSEO API login (HTTP Basic auth).
    POST https://api.dataforseo.com/v3/on_page/task_post [ { "target": "yourdomain.com", "max_crawl_pages": 1000 } ]

    Post the crawl task, wait for it to finish, then request on_page/pages with {"id": "<task id>", "limit": 1000} and save that response.

  2. Save the full JSON responseSave the complete response, not just the task ID, next to the template folder.
  3. Convert it with the included scriptIt runs offline, makes no API calls and never sees your credentials.
    python3 Automation/dataforseo_import.py import --template 14 --input response.json --output my-inputs.csv
  4. Add what the API can't knowURL pattern, Verified bot requests, Requests to valuable URLs, Unique requested URLs, Average internal inlinks, Next action
  5. Paste at A13Clear the demo rows, paste the CSV values at Workspace cell A13 and the dashboard updates.
Fills automatically from DataForSEO You add
Source reference (audit trail) URL pattern, Verified bot requests, Requests to valuable URLs, Unique requested URLs, Average internal inlinks, Next action

Source reference fills automatically with the task ID, endpoint, location, language, device and snapshot time, so every row is traceable. Before a big pull, estimate the cost with the DataForSEO pricing calculator. The same steps, plus an example request file, are in the download's Automation folder.

Worked example: Northbeam Outdoor

Every template ships filled with data from Northbeam Outdoor, a fictional outdoor-gear retailer that appears across all 40 workbooks, so you can see what useful output looks like before you add your own. This workbook has 12 example rows.

12Records
0Rows with input issues
665,800Requests to review
1,145,900Verified requests in scope

Filter combinations and sort parameters account for most of the example's waste. Product URLs are 95% valuable and should be left alone.

Top 10 by requests to review

# URL pattern Requests to review
1 /*?color=*&size=* 231,200
2 /*?sort=* 177,900
3 /search?q=* 96,000
4 /*?sessionid=* 41,000
5 /*?page=* 33,000

Input fields

Keep the columns in this order when you prepare a CSV. A missing observation stays blank; use 0 only for a measured zero or a verified No. Fields marked DataForSEO can be filled automatically by the included script.

CSV field Column Meaning and rule Source
pattern URL pattern Mutually exclusive URL pattern or template group. Your records
verified_requests Verified bot requests Count from server logs after verifying the bot identity. Your records
valuable_requests Requests to valuable URLs Requests inside the same group to approved canonical indexable URLs. Your records
unique_urls Unique requested URLs Distinct URLs requested by the verified bot. Your records
inlinks Average internal inlinks Crawl-derived average inlinks for this same group. Your records
action Next action Document a proposed navigation, canonical, status or URL-generation change. Your records
source_ref Source reference Task ID, export filename and observation scope. Retain the untouched source response separately. Optional. DataForSEO: source_ref

Reading the result honestly

Redirect checks, robots fetching and discovery can be legitimate. The metric finds work to inspect.

Limits. On-Page is a site crawl, not Googlebot logs. Server request data must come from your verified logs.

Log file analysis SEO mistakes to avoid

Using unverified user agentsFake Googlebots are common. Verify before analyzing.
Blocking parameters that carry canonical signalsIf a parameter URL is canonicalized, blocking it in robots.txt stops Google from seeing the canonical. Choose the fix deliberately.
Treating crawl budget as a problem on small sitesSites with a few thousand URLs rarely have crawl budget issues. Check indexation first.

Tips from practice

  • Cut internal links to wasteful patterns before blocking them; links create the demand.
  • Track the same patterns monthly to confirm fixes reduce waste.
  • Pair with the indexability audit to check what's valuable.

Frequently asked questions

What is crawl budget?

The number of URLs Googlebot can and wants to crawl on your site in a given time. It matters mostly for very large or frequently changing sites.

How do I analyze crawl budget with log files?

Filter server logs to verified Googlebot, group requests by URL pattern, and compare requests on valuable URLs with requests on parameters, duplicates and dead ends.

What causes crawl waste?

Faceted navigation, sort and filter parameters, session IDs, internal search pages, infinite calendars and expired content that keeps getting linked.

Should I block parameters in robots.txt?

Sometimes. It's effective for pure waste, but it hides canonicals and noindex from Google. For duplicates with signals, prefer removing links and using canonicals.

Part of the Technical SEO Toolkit

Get all 6 Technical SEO templates in one ZIP, or the complete set of 40 SEO and GEO templates with the 401-page field handbook.

Download the Technical SEO Toolkit

See the complete SEO & GEO toolkit

Jourdan Rombough

Built by Jourdan Rombough, technical SEO and GEO consultant with 15+ years in search. About Jourdan or get an SEO plan.

More on digital marketing in our blog

See all posts