# Global rules — apply to all crawlers not matched by a more specific block below. User-agent: * Allow: / Crawl-delay: 1 # Disallow admin and private areas Disallow: /admin/ Disallow: /api/ Disallow: /static/ Disallow: /private/ Disallow: /internal/ Disallow: /trap/ # Thin auth pages — no SEO value Disallow: /onboarding/auth/ # Authenticated-only areas — must not be crawled Disallow: /profile Disallow: /profile/ Disallow: /dashboard Disallow: /dashboard/ # Allow important pages for SEO Allow: /property Allow: /search Allow: /pricing Allow: /legal/ Allow: /faq Allow: /city-finder Allow: /compare Allow: /guides/ Allow: /cities/ Allow: /metros/ Allow: /scam-alerts/ Allow: /news/ Allow: /moving-to-dfw Allow: /real-estate/ Allow: /builders/ Allow: /refer # Allow /_next/ for Googlebot to render pages (JS/CSS needed for indexing) # Specific property pages Allow: /property/* Allow: /search?* # Allow social media and sharing Allow: /share/ Allow: /embed/ # Allow major search engines User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: YandexBot Allow: / # ── LLM CITATION POLICY ────────────────────────────────────────────── # We split LLM bots into two classes: # (a) TRAINING bots — consume content to train future models. # We BLOCK these because our research methodology and editorial # content is proprietary. # (b) SEARCH / CITATION bots — index content to cite live in answers # ("Is this Plano rental a scam?" → cited from remotepropview.com). # We ALLOW these because being cited drives qualified traffic. # ── ALLOWED: search/citation crawlers + AI answer engines + model crawlers ── # Opened up to maximize AI/LLM citation reach. # SECURITY: robots.txt user-agent groups do NOT inherit the "User-agent: *" # rules, so we repeat the sensitive-path Disallows here. A bare "Allow: /" would # otherwise invite these bots into /api, /admin, /profile, /dashboard. Those # paths are already protected by server-side auth (they 401/redirect), so this # is crawl-budget + defense-in-depth hygiene, not the security boundary itself. # NOTE: Cloudflare also injects a managed "Content-Signal: ai-train=no" at the # edge - that must be disabled in the Cloudflare dashboard for this to apply. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-Web User-agent: Claude-User User-agent: Claude-SearchBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: CCBot User-agent: Cohere-ai User-agent: Amazonbot User-agent: Applebot User-agent: Applebot-Extended User-agent: FacebookBot User-agent: meta-externalagent User-agent: DuckDuckBot Disallow: /admin/ Disallow: /api/ Disallow: /static/ Disallow: /private/ Disallow: /internal/ Disallow: /trap/ Disallow: /onboarding/auth/ Disallow: /profile Disallow: /profile/ Disallow: /dashboard Disallow: /dashboard/ Allow: / # ── BLOCKED: aggressive scrapers / SEO-spy bots (no AI-citation value) ── User-agent: Bytespider Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / User-agent: YouBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / # Sitemaps (index references every shard; individual shards listed too for # crawlers that don't follow a sitemap index) # AI answer files (full-content answers for LLM citation): # https://www.remotepropview.com/llms.txt # https://www.remotepropview.com/llms-full.txt Sitemap: https://www.remotepropview.com/sitemap-index.xml Sitemap: https://www.remotepropview.com/sitemap.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/michigan.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/ohio.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/illinois.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/new-york.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/kansas.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/arkansas.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/california.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/florida.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/georgia.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/colorado.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/washington.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/tennessee.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/arizona.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/north-carolina.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/pennsylvania.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/texas.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/virginia.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/new-jersey.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/massachusetts.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/maryland.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/indiana.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/missouri.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/wisconsin.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/minnesota.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/oregon.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/nevada.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/alabama.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/alaska.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/connecticut.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/delaware.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/hawaii.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/idaho.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/iowa.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/kentucky.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/louisiana.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/maine.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/mississippi.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/montana.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/nebraska.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/new-hampshire.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/new-mexico.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/north-dakota.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/oklahoma.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/rhode-island.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/south-carolina.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/south-dakota.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/utah.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/vermont.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/west-virginia.xml Sitemap: https://www.remotepropview.com/real-estate/sitemap/wyoming.xml Sitemap: https://www.remotepropview.com/builders/sitemap.xml