# Thrifle robots.txt # # Policy in one line: if you fetch for a person, you are welcome; if you # harvest the databases in bulk, you need a licence. # # Licensing terms and rate card: https://thrifle.com/data-licensing # Licensing contact: data@thrifle.com # # Prefer calling us as a tool? Thrifle is also an MCP server (read-only, # no auth, 24 tools over the same databases; fair-use limits apply): # MCP endpoint: https://api.thrifle.com/api/mcp # MCP docs: https://thrifle.com/mcp # # This file is kept deliberately in sync with two enforcement layers. If you # change a group here, change the matching layer too, or the site will say one # thing and do another: # 1. Cloudflare AI Crawl Control — per-crawler Block, returns 403 at the edge # 2. Origin toll booth — /etc/nginx/conf.d/tollbooth-map.conf, # returns 402 + licensing terms # # ─── DEAL PAGES: A 90-DAY LIFESPAN, ENFORCED WITH 410 ───────────────── # Deal listings are perishable, so we retire them on a schedule rather than # leaving dead offers on the web. A deal page is live for 90 days from the # date it was posted (or from its last revive); on day 91 the URL returns # 410 Gone — permanently, at every spelling of that URL. Offers flagged # evergreen (ongoing memberships and the like) are exempt and stay live. # # We do NOT Disallow expired deals here, and that is deliberate: a URL you # are forbidden to fetch can never be recrawled, so it lingers in an index # indefinitely. Crawl it once, take the 410, and drop it. That is exactly # the outcome we want, and it is why the status is 410 rather than a 404 # or a redirect — the page is gone on purpose and it is not coming back. # # The underlying price records are retained in our own database, where they # feed price history and research. They are simply no longer published as # pages. Every deal page we intend to be indexed is listed in # https://thrifle.com/sitemap.xml — nothing outside it is meant for your index. # NOTE ON GROUPS: robots.txt has no inheritance. A crawler named in its own # group ignores the "*" group entirely, so the private-area rules below are # repeated in every group on purpose. Do not "de-duplicate" them. # ─── Answer engines: fetch on behalf of a live person ───────────────── # These send us human readers. Never block, never toll. They are exempt at # both the Cloudflare layer and the origin toll booth. # 2026-10-16: Meta's crawlers (meta-externalagent, meta-webindexer, FacebookBot) # joined this group — Haider's call, reversing the 07-26 toll. User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: Meta-ExternalFetcher User-agent: meta-externalagent User-agent: Meta-ExternalAgent User-agent: meta-webindexer User-agent: FacebookBot User-agent: facebookexternalhit Disallow: /profile/ Disallow: /dashboard/ Disallow: /login Disallow: /signin Disallow: /signup Disallow: /employee/ Disallow: /super-profile/ Disallow: /deal-alerts/ Disallow: /my-alerts/ Allow: / # ─── Search engines ─────────────────────────────────────────────────── User-agent: Googlebot User-agent: Googlebot-Image User-agent: Bingbot User-agent: Applebot User-agent: DuckDuckBot Disallow: /profile/ Disallow: /dashboard/ Disallow: /login Disallow: /signin Disallow: /signup Disallow: /employee/ Disallow: /super-profile/ Disallow: /deal-alerts/ Disallow: /my-alerts/ # ─── Compare pages: only the curated matchups are crawlable ─────────── # 2026-08-19. /return-policy/compare/{a}-vs-{b} is generated by the rival strip # on every merchant page — 8,521 reachable URLs today, and the set grows every # time the return-policy DB changes (rivals are top-4 BY GRADE, so a regrade # mints new pairs). Everything outside the curated list already carried # noindex, but noindex still costs a fetch: 76,065 origin hits in 15 days, # 13% of ALL site traffic, and GSC shows ZERO impressions across the entire # subtree over 90 days. So the crawl bought nothing and starved /blog. # Allow lines precede the Disallow on purpose — longest-match parsers (Google, # Bing) ignore order, but first-match parsers need the exceptions first. # GENERATED BLOCK — after editing utils/comparePairs.js run: # node scripts/verify-compare-robots.js --write # CI runs the same script in verify mode, so the two cannot drift. # BEGIN compare-allow (generated) Allow: /return-policy/compare/adidas-vs-nike Allow: /return-policy/compare/aldi-vs-traderjoes Allow: /return-policy/compare/amazon-vs-best-buy Allow: /return-policy/compare/amazon-vs-costco Allow: /return-policy/compare/amazon-vs-target Allow: /return-policy/compare/amazon-vs-walmart Allow: /return-policy/compare/apple-vs-best-buy Allow: /return-policy/compare/apple-vs-samsung Allow: /return-policy/compare/backcountry-vs-rei Allow: /return-policy/compare/best-buy-vs-walmart Allow: /return-policy/compare/bjs-wholesale-club-vs-costco Allow: /return-policy/compare/bjs-wholesale-club-vs-sams-club Allow: /return-policy/compare/chewy-vs-petsmart Allow: /return-policy/compare/costco-vs-sams-club Allow: /return-policy/compare/costco-vs-target Allow: /return-policy/compare/costco-vs-walmart Allow: /return-policy/compare/cvs-vs-walgreens Allow: /return-policy/compare/gap-vs-old-navy Allow: /return-policy/compare/h-and-m-vs-zara Allow: /return-policy/compare/ikea-vs-wayfair Allow: /return-policy/compare/jcpenney-vs-kohls Allow: /return-policy/compare/kohls-vs-macys Allow: /return-policy/compare/kroger-vs-walmart Allow: /return-policy/compare/lowes-vs-the-home-depot Allow: /return-policy/compare/macys-vs-nordstrom Allow: /return-policy/compare/marshalls-vs-tjmaxx Allow: /return-policy/compare/ross-vs-tjmaxx Allow: /return-policy/compare/sephora-vs-ulta-beauty Allow: /return-policy/compare/target-vs-walmart Allow: /return-policy/compare/traderjoes-vs-wholefoods Disallow: /return-policy/compare/ # END compare-allow Allow: / # ─── Training-control tokens (no fetching) ──────────────────────────── # These two are robots.txt CONTROL TOKENS, not crawlers — nothing ever fetches # with these user-agents, so they cannot be tolled at the origin. Listing them # is how we express training consent for Gemini and Apple Intelligence. # We keep them allowed: being known by name is worth more than the crawl cost. # # 2026-07-31: GPTBot and Google-CloudVertexBot moved OUT of this group. They are # real fetching crawlers and are now tolled — see the next group. User-agent: Google-Extended User-agent: Applebot-Extended Disallow: /profile/ Disallow: /dashboard/ Disallow: /login Disallow: /signin Disallow: /signup Disallow: /employee/ Disallow: /super-profile/ Disallow: /deal-alerts/ Disallow: /my-alerts/ Allow: / # ─── Tolled at the origin, but still welcome to come and see the price ─ # 2026-07-31. These crawl to build training corpora or to resell our data. # They get HTTP 402 + the rate card at the origin instead of free content. # # They are deliberately NOT Disallowed. A Disallow makes a compliant crawler # stop fetching entirely, so it never sees the 402 and never learns the price — # and the 402 IS the sales pitch. A 402 costs us 554 bytes / 74 ms (nginx only, # never touches Next.js or Mongo) versus ~137 KB for a real page, so leaving # the door open is close to free. User-agent: GPTBot User-agent: Google-CloudVertexBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: DotBot User-agent: DataForSeoBot User-agent: YouBot User-agent: Scrapy Disallow: /profile/ Disallow: /dashboard/ Disallow: /login Disallow: /signin Disallow: /signup Disallow: /employee/ Disallow: /super-profile/ Disallow: /deal-alerts/ Disallow: /my-alerts/ Allow: / # ─── Licensed access required ───────────────────────────────────────── # Bulk extractors. All of these are tolled at the origin (402 + terms). The # five Allow lines below are the same paths the toll booth keeps free, so a # crawler that hits the wall can always read why and what it costs. # # Enforcement as of 2026-07-31 — the Cloudflare per-crawler BLOCK was retired. # It returned a mute 403, which taught nobody the price; the origin toll booth # now answers all of these with a 402 and the rate card instead. The Cloudflare # WAF rule "AI Crawl Control - Block AI bots by User Agent" still exists but is # disabled, so it can be flipped back in one click if a bot misbehaves. # # ClaudeBot, anthropic-ai, Amazonbot, # Bytespider, CCBot, PetalBot, TikTokSpider, Novellum, Timpibot, # Diffbot, omgili, omgilibot, cohere-ai, ImagesiftBot # ....... origin toll booth, HTTP 402 # # Every one of the nine crawlers the retired Cloudflare rule used to block now # reaches the origin and gets the rate card instead of a silent 403. # # 🔑 A name list cannot catch an unnamed crawler. The Tencent/Alibaba-Singapore # fleet found 2026-07-31 sends plain rotating Chrome user-agents and is # invisible to every rule in this file. Blocking by name is also a treadmill. # (Meta's crawlers meta-externalagent, meta-webindexer and FacebookBot were # removed from this group 2026-10-16 and are welcome — see the first group.) User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: ImagesiftBot User-agent: Amazonbot User-agent: Bytespider User-agent: CCBot User-agent: PetalBot User-agent: TikTokSpider User-agent: Novellum User-agent: Timpibot User-agent: Diffbot User-agent: omgili User-agent: omgilibot User-agent: cohere-ai Allow: /robots.txt Allow: /sitemap Allow: /data-licensing Disallow: / # ─── Blocked outright (no toll, no pitch) ───────────────────────────── # 2026-08-17. Unlike the licensed-access group above, SleepBot is NOT tolled # at the origin (yet) — this Disallow is the enforcement, and it is known to # obey robots.txt. SleepBot is an anonymous GCP-based scraper with a dead # cover domain; it swept 928 price-predict products once each. No one to sell # a licence to. If the query log shows it still fetching a week after this # ships, escalate it to the origin toll booth # (/etc/nginx/conf.d/tollbooth-map.conf, HTTP 402). User-agent: SleepBot Disallow: / # ─── Treated like any other crawler ─────────────────────────────────── # 2026-08-23 (per Haider). The price-predict Disallow added here on 2026-08-17 # is REMOVED. Its stated reason was "our Semrush subscription is expired, so # that crawl buys us nothing" — that premise no longer holds: the Site Audit is # in active use, and the block was hiding /price-predict from our own audit. # Cost of letting it back in is ~592 fetches/day at peak on the heaviest pages # we serve; that is the price of auditing them. # # This group now matches the "*" group exactly, and is kept only because # robots.txt has NO INHERITANCE — deleting it would drop SemrushBot into "*", # which is the same ruleset, but the compare-block verifier # (scripts/verify-compare-robots.js) asserts EXPECTED_GROUPS = 3 and this is # one of the three. The single token covers all SemrushBot-* variants (they # match on "SemrushBot"), including SemrushBot-SA used by Site Audit. User-agent: SemrushBot Disallow: /profile/ Disallow: /dashboard/ Disallow: /login Disallow: /signin Disallow: /employee/ Disallow: /super-profile/ Disallow: /deal-alerts/ Disallow: /my-alerts/ # ─── Compare pages: only the curated matchups are crawlable ─────────── # 2026-08-19. /return-policy/compare/{a}-vs-{b} is generated by the rival strip # on every merchant page — 8,521 reachable URLs today, and the set grows every # time the return-policy DB changes (rivals are top-4 BY GRADE, so a regrade # mints new pairs). Everything outside the curated list already carried # noindex, but noindex still costs a fetch: 76,065 origin hits in 15 days, # 13% of ALL site traffic, and GSC shows ZERO impressions across the entire # subtree over 90 days. So the crawl bought nothing and starved /blog. # Allow lines precede the Disallow on purpose — longest-match parsers (Google, # Bing) ignore order, but first-match parsers need the exceptions first. # GENERATED BLOCK — after editing utils/comparePairs.js run: # node scripts/verify-compare-robots.js --write # CI runs the same script in verify mode, so the two cannot drift. # BEGIN compare-allow (generated) Allow: /return-policy/compare/adidas-vs-nike Allow: /return-policy/compare/aldi-vs-traderjoes Allow: /return-policy/compare/amazon-vs-best-buy Allow: /return-policy/compare/amazon-vs-costco Allow: /return-policy/compare/amazon-vs-target Allow: /return-policy/compare/amazon-vs-walmart Allow: /return-policy/compare/apple-vs-best-buy Allow: /return-policy/compare/apple-vs-samsung Allow: /return-policy/compare/backcountry-vs-rei Allow: /return-policy/compare/best-buy-vs-walmart Allow: /return-policy/compare/bjs-wholesale-club-vs-costco Allow: /return-policy/compare/bjs-wholesale-club-vs-sams-club Allow: /return-policy/compare/chewy-vs-petsmart Allow: /return-policy/compare/costco-vs-sams-club Allow: /return-policy/compare/costco-vs-target Allow: /return-policy/compare/costco-vs-walmart Allow: /return-policy/compare/cvs-vs-walgreens Allow: /return-policy/compare/gap-vs-old-navy Allow: /return-policy/compare/h-and-m-vs-zara Allow: /return-policy/compare/ikea-vs-wayfair Allow: /return-policy/compare/jcpenney-vs-kohls Allow: /return-policy/compare/kohls-vs-macys Allow: /return-policy/compare/kroger-vs-walmart Allow: /return-policy/compare/lowes-vs-the-home-depot Allow: /return-policy/compare/macys-vs-nordstrom Allow: /return-policy/compare/marshalls-vs-tjmaxx Allow: /return-policy/compare/ross-vs-tjmaxx Allow: /return-policy/compare/sephora-vs-ulta-beauty Allow: /return-policy/compare/target-vs-walmart Allow: /return-policy/compare/traderjoes-vs-wholefoods Disallow: /return-policy/compare/ # END compare-allow Allow: / # ─── Default (all other crawlers) ───────────────────────────────────── # Specific rules precede the blanket "Allow: /" so first-match parsers cannot # swallow them. No blank line inside a group either: a blank line ends a group, # which previously orphaned every private-area rule in this file. User-agent: * Disallow: /profile/ Disallow: /dashboard/ Disallow: /login Disallow: /signin Disallow: /signup Disallow: /employee/ Disallow: /super-profile/ Disallow: /deal-alerts/ Disallow: /my-alerts/ # ─── Compare pages: only the curated matchups are crawlable ─────────── # 2026-08-19. /return-policy/compare/{a}-vs-{b} is generated by the rival strip # on every merchant page — 8,521 reachable URLs today, and the set grows every # time the return-policy DB changes (rivals are top-4 BY GRADE, so a regrade # mints new pairs). Everything outside the curated list already carried # noindex, but noindex still costs a fetch: 76,065 origin hits in 15 days, # 13% of ALL site traffic, and GSC shows ZERO impressions across the entire # subtree over 90 days. So the crawl bought nothing and starved /blog. # Allow lines precede the Disallow on purpose — longest-match parsers (Google, # Bing) ignore order, but first-match parsers need the exceptions first. # GENERATED BLOCK — after editing utils/comparePairs.js run: # node scripts/verify-compare-robots.js --write # CI runs the same script in verify mode, so the two cannot drift. # BEGIN compare-allow (generated) Allow: /return-policy/compare/adidas-vs-nike Allow: /return-policy/compare/aldi-vs-traderjoes Allow: /return-policy/compare/amazon-vs-best-buy Allow: /return-policy/compare/amazon-vs-costco Allow: /return-policy/compare/amazon-vs-target Allow: /return-policy/compare/amazon-vs-walmart Allow: /return-policy/compare/apple-vs-best-buy Allow: /return-policy/compare/apple-vs-samsung Allow: /return-policy/compare/backcountry-vs-rei Allow: /return-policy/compare/best-buy-vs-walmart Allow: /return-policy/compare/bjs-wholesale-club-vs-costco Allow: /return-policy/compare/bjs-wholesale-club-vs-sams-club Allow: /return-policy/compare/chewy-vs-petsmart Allow: /return-policy/compare/costco-vs-sams-club Allow: /return-policy/compare/costco-vs-target Allow: /return-policy/compare/costco-vs-walmart Allow: /return-policy/compare/cvs-vs-walgreens Allow: /return-policy/compare/gap-vs-old-navy Allow: /return-policy/compare/h-and-m-vs-zara Allow: /return-policy/compare/ikea-vs-wayfair Allow: /return-policy/compare/jcpenney-vs-kohls Allow: /return-policy/compare/kohls-vs-macys Allow: /return-policy/compare/kroger-vs-walmart Allow: /return-policy/compare/lowes-vs-the-home-depot Allow: /return-policy/compare/macys-vs-nordstrom Allow: /return-policy/compare/marshalls-vs-tjmaxx Allow: /return-policy/compare/ross-vs-tjmaxx Allow: /return-policy/compare/sephora-vs-ulta-beauty Allow: /return-policy/compare/target-vs-walmart Allow: /return-policy/compare/traderjoes-vs-wholefoods Disallow: /return-policy/compare/ # END compare-allow Allow: / # ─── Discoverability ────────────────────────────────────────────────── Sitemap: https://thrifle.com/sitemap.xml # Machine-readable resources: # https://thrifle.com/feed.xml (RSS — latest 50 active deals) # https://thrifle.com/data-licensing (bulk access terms and rate card)