Crawl Budget Analysis Template (Optimization Theory, v3.0, November 2026)
====================================================================

Group verified Googlebot requests by URL pattern, measure crawl waste and pick the right fix for each pattern.

What it decides
  Which URL patterns soak up crawl requests without returning value, how much crawl activity each wastes, and what to do about it.

Files
  Crawl Budget Analysis Template.xlsx ............ the workbook (Start Here, Dashboard, Workspace, Delivery QA, Field Guide)
  Crawl Budget Analysis Template - Field Guide.pdf  the illustrated guide
  Input-Template-blank.csv ..... column headers in paste order
  Input-Template-example.csv ... the worked example (Northbeam Outdoor, fictional data)
  HOW-TO-USE-WITH-DATAFORSEO.txt step-by-step: request, save, convert, complete, paste
  Automation/ .................. import script, example request, field mapping (python3 Automation/dataforseo_import.py --help)

Built to run on DataForSEO
  Source: DataForSEO On-Page API
  Fills automatically: the audit trail (Source reference)
  Full steps: HOW-TO-USE-WITH-DATAFORSEO.txt (also in the Automation folder), with example-request.json

Quick start
  1. Verify the bot: Filter logs to verified Googlebot using reverse DNS. User agents alone are easily spoofed.
  2. Group by pattern: Bucket URLs into patterns: parameters, folders, templates. Count requests and unique URLs per pattern for a fixed window.
  3. Mark valuable requests: Valuable means the request hit an indexable, canonical URL you want crawled. Join with your crawl data.
  4. Choose an action per pattern: High review share with many internal links usually means fixing links; high requests per URL means a trap.

Works in Microsoft Excel, Google Sheets (upload, then open with Sheets) and LibreOffice Calc.
Paste values only at Workspace cell A13. Lime cells are inputs; blue cells are formulas.

Full tutorial: https://optimizationtheory.com/download-view/crawl-budget-analysis-template/
Part of the Technical SEO Toolkit: https://optimizationtheory.com/download-view/technical-seo-toolkit-6-audit-and-validation-workbooks/
All 40 templates: https://optimizationtheory.com/download-view/free-seo-geo-toolkit-40-spreadsheets-and-field-guides/

Free to use in your own and client work. Please don't resell the templates themselves.
Built by Jourdan Rombough, optimizationtheory.com
