# robots.txt — flowrunner.ai (marketing site) # Managed by: Midnight Coders, Inc. # Last updated: 2026-09-08 # ───────────────────────────────────────────────────────────── # Policy: this is a MARKETING property. Its job is to get FlowRunner # discovered, learned, and recommended. Every legitimate AI crawler is # welcome, whichever of the three jobs it does: # # training — builds the model's memory (GPTBot, ClaudeBot, # meta-externalagent, Google-Extended, Applebot-Extended, # Amazonbot, CCBot, cohere-ai, Bytespider) # retrieval — builds the answer-engine index that produces citations # (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, # DuckAssistBot, Googlebot, Bingbot) # user fetch — reads a page live on behalf of a person asking an assistant # (ChatGPT-User, Claude-User, Perplexity-User, # meta-externalfetcher) # # Blocking a retrieval or user-fetch agent removes FlowRunner from AI # answers. Blocking a training agent only affects future model memory. We # want both, so all of them share the open group below. Listing them by name # in ONE group matters: a named group replaces the `*` group for that agent, # so an agent split into its own group would lose the Content-Signal line. # # Content Signals (https://contentsignals.org): # search=yes — index and return hyperlinks/excerpts in search # ai-input=yes — use in real-time AI answers (Perplexity, ChatGPT search, AI Overviews) # ai-train=yes — permitted to use content to train/fine-tune AI models # ai-train=yes deliberately reverses the 2026-05-28 opt-out: for marketing # content, being ingested into model memory is the goal, not a risk. # Proprietary/product surfaces (app.flowrunner.ai) are governed separately # and are NOT covered by this file. # # Bulk content-resellers, scrapers, and plagiarism-fingerprinting bots stay # out: they crawl our pages but give no model-recognition benefit. # # Machine-readable entry points for agents: # https://flowrunner.ai/llms.txt short orientation index # https://flowrunner.ai/llms-full.txt narrative pages inlined, catalog pages linked # Every page has a Markdown twin at the same path plus ".md" # (homepage: /index.md). Requesting a page with "Accept: text/markdown" # returns the same Markdown document. # ───────────────────────────────────────────────────────────── User-agent: * User-agent: Googlebot User-agent: Bingbot User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: meta-externalfetcher User-agent: Amazonbot User-agent: DuckAssistBot User-agent: CCBot User-agent: cohere-ai User-agent: Bytespider Content-Signal: search=yes,ai-input=yes,ai-train=yes Allow: / Disallow: /cdn-cgi/ Sitemap: https://flowrunner.ai/sitemap-index.xml # Blog feed. Not a robots.txt directive any crawler is required to honour, but # several feed discovery tools and AI crawlers look here before parsing HTML. # Feed: https://flowrunner.ai/rss.xml # ───────────────────────────────────────────────────────────── # BULK SCRAPERS / RESELLERS / FINGERPRINTING — blocked # These harvest content for resale or plagiarism checks, not to build the # assistants we want to be recommended by. No recognition benefit, so out. # ───────────────────────────────────────────────────────────── # Diffbot — structured-data extraction for resale User-agent: Diffbot Disallow: / # Omgili / Webz.io — bulk content harvesting for resale User-agent: omgili User-agent: omgilibot Disallow: / # DataForSeo — bulk SEO scraping User-agent: DataForSeoBot Disallow: / # Turnitin / iThenticate — content fingerprinting User-agent: TurnitinBot Disallow: /