How to fill the Crawl Budget Analysis Template with DataForSEO
============================================================

Data source: DataForSEO On-Page API (a crawl of your site: URLs, status codes, canonicals, inlinks and sitemap membership).
DataForSEO supplies the URL or query list and the audit trail; you add the measurements listed below.

1. Request the data
   POST https://api.dataforseo.com/v3/on_page/task_post
   Body:
   [
     {
       "target": "yourdomain.com",
       "max_crawl_pages": 1000
     }
   ]
   Post the crawl task, wait for it to finish, then request on_page/pages with {"id": "<task id>", "limit": 1000} and save that response.
   Auth: HTTP Basic with your DataForSEO API login and password. Keep credentials out of shared files.

2. Save the complete JSON response (not just the task ID) next to this folder, e.g. response.json.

3. Convert it to the input CSV (offline, Python 3.10+, no API call, no credentials):
   python3 Automation/dataforseo_import.py import --template 14 --input response.json --output my-inputs.csv

4. Complete the columns DataForSEO can't supply:
   - URL pattern
   - Verified bot requests
   - Requests to valuable URLs
   - Unique requested URLs
   - Average internal inlinks
   - Next action

5. Open the workbook, clear the demo rows in Workspace (lime input cells only), and paste the CSV values at cell A13.

Filled automatically: the audit trail only (Source reference)
Source reference fills automatically with the task ID, endpoint, location, language, device and snapshot time, so every row is traceable.

Costs: estimate them first with the DataForSEO Cost Calculator (template 38). Live endpoints cost more than task_post/task_get.
Docs: https://docs.dataforseo.com/v3/