# Example policy for public pages
# Keep search; decline selected collection

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Separate controls, not extra default rules:
# Google-Extended covers future Gemini training and selected grounding in Gemini Apps / Vertex AI.
# It is a product token, not a separate fetcher, and does not control Google Search inclusion/ranking.
# Google Search AI inclusion has a separate Search Console control.
# robots.txt is not authentication or reliable index removal; Google must fetch a noindex rule to read it.
# Choose your purpose and verify the current provider/CDN settings before changing production rules.
