# Koru — robots.txt # # Default: search engines and AI crawlers are welcome on every public surface. # The dynamic, authenticated app lives behind /app, /dashboard, /websites, … # and isn't intended for indexing — those paths are explicitly disallowed # below regardless of the user-agent. # # AI crawlers are listed by name so the intent is unambiguous (some servers # default-deny anything that isn't a recognised search engine; we want LLM # discovery to work). # --- Search engines (default) ------------------------------------------------- User-agent: * Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # --- AI / LLM crawlers -------------------------------------------------------- # OpenAI — used to discover content for ChatGPT search + browsing. User-agent: GPTBot Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # OpenAI's user-triggered ChatGPT browsing. User-agent: ChatGPT-User Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # OpenAI training corpus (not the same as GPTBot — separate user-agent). User-agent: OAI-SearchBot Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # Anthropic Claude. User-agent: ClaudeBot Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ User-agent: anthropic-ai Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # Google's "AI features" crawler — controls inclusion in Gemini, AI Overviews, # Vertex AI Search. Note this is DIFFERENT from Googlebot (regular search) — # both should be allowed for full coverage. User-agent: Google-Extended Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # Perplexity AI. User-agent: PerplexityBot Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # Apple Intelligence (Apple Spotlight / Siri summarisation, on-device LLM). User-agent: Applebot-Extended Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # Common Crawl — open web index used by many LLM training pipelines. User-agent: CCBot Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # DuckDuckGo / Brave AI Answer. User-agent: DuckAssistBot Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # Meta AI training crawler. User-agent: meta-externalagent Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ User-agent: FacebookBot Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # Bytespider (TikTok / ByteDance LLMs). User-agent: Bytespider Allow: / Disallow: /app/ Disallow: /dashboard Disallow: /websites Disallow: /portfolio Disallow: /settings Disallow: /billing Disallow: /login Disallow: /register Disallow: /oauth/ Disallow: /api/ # --- Pointers ---------------------------------------------------------------- # Markdown-summarised site map for AI agents that can't usefully render JS. # Discoverable three ways so crawlers find it whichever path they take: # # 1. This file: see the LLMs-Index line below (a Koru-defined hint). # 2. The HTML # in every page's . # 3. The matching HTTP "Link: ; rel=alternate; type=text/markdown" # response header (set by Cloud CDN on every frontend response). # # Sitemap — the public-content index for classical crawlers. AI # crawlers also read this; standard XML format documented at # https://www.sitemaps.org/protocol.html. Lists only public # (un-authenticated) routes; auth-walled paths (/dashboard, # /websites, etc.) are explicitly Disallow'd above and absent from # the sitemap. Sitemap: https://koru-seo.com/sitemap.xml # LLMs-Index is a non-standard hint some AI crawlers honour (and others # safely ignore), pointing at the markdown summary directly. LLMs-Index: https://koru-seo.com/llms.txt