# Nelson Environmental Remediation: robots.txt # Permanent contaminant destruction. Because tomorrow matters. # --- Search engines --- # NOTE: a crawler obeys ONLY its most specific matching group. Any named group # below therefore has to repeat the site-wide Disallow list at the bottom, or # that crawler ignores it entirely. User-agent: Googlebot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Googlebot-Image Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Bingbot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: DuckDuckBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Applebot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: YandexBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ # --- Social previews --- User-agent: Twitterbot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: facebookexternalhit Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: LinkedInBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Slackbot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: WhatsApp Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: TelegramBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ # --- AI assistants & answer engines (explicit allow) --- User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Claude-Web Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: anthropic-ai Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ # --- Blocked: zero-value crawlers used primarily for bulk training-data # resale, scraper-as-a-service resale, and competitor mimicry. # Search/citation crawlers above are unaffected. User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: SerpstatBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: SeekportBot Disallow: / User-agent: PetalBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: Scrapy Disallow: / User-agent: magpie-crawler Disallow: / User-agent: trendictionbot Disallow: / # --- Blocked: regional crawlers and datacenter scrapers with no buyer reach # for a Canadian/US remediation market. These account for the bulk of the # high-bounce, one-page-per-visit volume in the analytics record. User-agent: Baiduspider Disallow: / User-agent: Baiduspider-image Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou inst spider Disallow: / User-agent: Sogou spider2 Disallow: / User-agent: YisouSpider Disallow: / User-agent: 360Spider Disallow: / User-agent: HaoSouSpider Disallow: / User-agent: Yeti Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: SputnikBot Disallow: / User-agent: YandexImages Disallow: / User-agent: YandexMetrika Disallow: / User-agent: Yandex.Zen Disallow: / User-agent: Alibaba.Security.Heimdall Disallow: / User-agent: Aliyun Disallow: / User-agent: Qwantify Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: cohere-ai Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Diffbot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Meta-ExternalAgent Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: Amazonbot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: YouBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: DuckAssistBot Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ User-agent: MistralAI-User Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout Disallow: /explore? Disallow: /*?ref= Disallow: /playbooks Disallow: /playbooks/ Disallow: /data/*.json Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json Disallow: /d/ Disallow: /embed/ # --- Site-wide crawl policy --- # Everything is discoverable except admin surfaces and the internal RFP # workspace. User-agent: * Allow: / Disallow: /admin/ Disallow: /admin/chat-analytics Disallow: /rfp-builder Disallow: /rfp-builder/ Disallow: /resource-recovery-services/thank-you # /explore itself is public and indexable. The rep-facing tooling around it # (link builder, engagement insights, handout view) is internal. Disallow: /explore/link Disallow: /explore/custom Disallow: /explore/insights Disallow: /explore/analytics Disallow: /explore/handout # Personalized tour links are the same page with tracking params on it. Disallow: /explore? Disallow: /*?ref= # Confidential executive playbooks. Offline delivery only. Disallow: /playbooks Disallow: /playbooks/ # --- Dataset compilation licence --- # The PFAS and hydrocarbon JSON payloads under /data/ are a compiled work: # Dataset compilation copyright 2026 Nelson Environmental Remediation Ltd. # All rights reserved. Source attribution required for any reuse, republication # or model training. Terms: https://nelsonremediation.com/data-licence # Bulk crawling of the raw payloads is not permitted. The human-readable hub # pages remain fully open to search and answer engines. Disallow: /data/*.json # Bait paths. They hold nothing. Any agent that fetches one has ignored this # file on purpose, and is blocked automatically at the delivery layer. Disallow: /data/ner-full-export.json Disallow: /data/ner-master-registry.json Disallow: /data/all-sites-complete.json Disallow: /data/nelson-sites-full.json Disallow: /data/pfas-all-sites.json Disallow: /data/hydrocarbon-all-sites.json Disallow: /data/dataset-export.json Disallow: /data/full-export.json # Compiled payloads are served from per-build hashed paths. Nothing here is a # page, and the paths change on every deploy. Disallow: /d/ # Embeddable widgets are for third-party pages, not for indexing. Disallow: /embed/ Sitemap: https://nelsonremediation.com/sitemap.xml Sitemap: https://nelsonremediation.com/sitemap-news.xml Sitemap: https://nelsonremediation.com/sitemap-video.xml