# robots.txt — https://www.arivaanai.com # # Lives in frontend/public/, so the Next.js static export copies it to # frontend/out/robots.txt, which server.js already serves at the site root via # express.static(FRONTEND_DIR). No backend route was added for this. # # Policy: the public marketing pages are open to everything, including AI # training and retrieval crawlers. Being retrievable is the entire point of # generative engine optimisation — a blocked crawler cannot cite you. Signed-in # surfaces, payment flows and the API are closed to everyone. # ─── Default: all crawlers ─── User-agent: * Allow: /$ Allow: /tools Allow: /pricing Allow: /contact Allow: /privacy Allow: /terms Allow: /refund-policy Allow: /legal/ # /about and /models retired 2026-08-08 — both answer 410. Deliberately not # Disallowed: a blocked URL cannot be crawled, so the crawler never sees the 410 # and the page lingers in the index. Let them through to read the tombstone. # Authenticated, transactional or machine-only surfaces. Nothing here has search # value, and crawling it just burns budget on redirects to the login page. Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= # App launch URLs (/chat, /studio, …). Each is a redirect: into the app for a # signed-in visitor, into the login modal for everyone else — including a # crawler, so there is nothing to index and no reason to spend budget here. Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # ─── Answer engine / AI assistant crawlers ─── # Named explicitly rather than relying on the wildcard above: several of these # only honour a directive addressed to them by name, and an explicit Allow is # the clearest signal that citation is welcome. # # Each group repeats the full Disallow list on purpose. robots.txt has NO # inheritance (RFC 9309 §2.2.1): a crawler obeys only the most specific group # that matches its name and ignores `User-agent: *` entirely. These groups # previously carried four Disallow rules while the wildcard carried thirteen, # which left /login, /signup, /auth/, /device, /uploads/ and both payment # callbacks open to every named crawler — including bingbot. Repetition is # verbose but it is the only correct form. Keep this list identical to the # wildcard group above. # OpenAI — ChatGPT browsing, search index, and model training User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # Anthropic — Claude search and browsing User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # Google — Gemini grounding and AI Overviews User-agent: Google-Extended Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # Perplexity User-agent: PerplexityBot User-agent: Perplexity-User Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # Microsoft — Bing index, which also powers ChatGPT search and Copilot User-agent: bingbot Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # Apple Intelligence User-agent: Applebot-Extended Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # Meta AI User-agent: meta-externalagent Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # Common Crawl — the corpus most open models are trained from User-agent: CCBot Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social # Mistral, You.com, Amazon, ByteDance User-agent: MistralAI-User User-agent: YouBot User-agent: Amazonbot User-agent: Bytespider Allow: / Disallow: /api/ Disallow: /account Disallow: /admin Disallow: /checkout Disallow: /auth/ Disallow: /login Disallow: /signup Disallow: /device Disallow: /uploads/ Disallow: /payment-success Disallow: /payment-cancel Disallow: /*?auth= Disallow: /*?redirect= Disallow: /chat Disallow: /assistant Disallow: /studio Disallow: /fitmind Disallow: /flashcards Disallow: /trip Disallow: /social Sitemap: https://www.arivaanai.com/sitemap.xml