TailorAIBot

TailorAIBot builds the product index behind tailor.ai, a fashion search that people use through AI assistants. Shoppers are sent to the retailer's own product page to buy.

What it fetches

• Public catalogue pages only: product pages, category listings and product sitemaps. Never accounts, carts, checkout, search or anything behind a login.
• One modest image (about 600px) per product, used to make the product findable by look.
• No personal data.

How it behaves

• It identifies itself as TailorAIBot/1.0 with a link to this page.
• It reads and obeys robots.txt (for TailorAIBot and *), includingCrawl-delay. If robots.txt can't be read, it doesn't crawl.
• It sends one request at a time per site, with a pause of at least 1.5 seconds between requests.
• It caches every page and re-visits a product at most once a week or two.
• A 403, a 429 or a challenge page makes it stop. It does not rotate identities or proxies, and never attempts a CAPTCHA.

Controlling it

To stop it entirely, add this to your robots.txt:

User-agent: TailorAIBot
Disallow: /

To slow it down, set a Crawl-delay (in seconds) for TailorAIBot. Changes take effect within 24 hours.