# ============================================================================== # Robots.txt for AIDO India (https://aidoworld.com) # Standard Web Crawlers & AI-Friendly Search & Training Bots # ============================================================================== # Global Crawler Permissions User-agent: * Allow: / Allow: /uploads/ Allow: /assets/ Allow: /category/ Allow: /product/ Allow: /brochures/ Allow: /installation-manuals/ Allow: /product-flyers/ Allow: /image-gallery/ Allow: /videos/ Allow: /pdf-view/ # Disallow Private & Admin Endpoints Disallow: /admin Disallow: /admin/ Disallow: /superadmin/ Disallow: /dashboard Disallow: /dashboard/ Disallow: /login Disallow: /register Disallow: /logout Disallow: /authenticate Disallow: /registerUser Disallow: /users/ Disallow: /settings/ Disallow: /notifications/ Disallow: /categories/ Disallow: /products/ Disallow: /seo-manager/ Disallow: /banners/ Disallow: /galleries/ Disallow: /video-categories/ Disallow: /brochure-categories/ Disallow: /installation-manual-categories/ Disallow: /product-flyer-categories/ Disallow: /manage-videos/ Disallow: /manage-brochures/ Disallow: /manage-installation-manuals/ Disallow: /manage-product-flyers/ Disallow: /manage-distributors/ Disallow: /manage-brochure-downloads/ Disallow: /warranty/ # ============================================================================== # AI Search, LLM Agents & Content Discovery Bots (Explicitly Allowed) # ============================================================================== # OpenAI (ChatGPT, GPTBot, SearchGPT) User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Google AI & Extended Crawlers User-agent: Google-Extended User-agent: Googlebot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Anthropic Claude & Research Bots User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Perplexity AI Search User-agent: PerplexityBot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Apple Intelligence User-agent: Applebot User-agent: Applebot-Extended Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Microsoft / Bing Copilot User-agent: Bingbot User-agent: Copilot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Meta / Facebook AI User-agent: Meta-ExternalAgent User-agent: FacebookBot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Amazon AI User-agent: Amazonbot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Cohere AI User-agent: cohere-ai Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # Common Crawl (Open AI datasets) User-agent: CCBot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /dashboard # ============================================================================== # Sitemap # ============================================================================== Sitemap: https://aidoworld.com/sitemap.xml