JetSiftBot

JetSiftBot is the crawler of JetSift, which shows restaurants, cafés and bars how AI recommends them.

What JetSiftBot fetches

Two kinds of page, and nothing else:

• A page that AI cited in one of its answers. JetSiftBot fetches that page alone, to see whether it lists the venue the report is about.
• The venue's own website, as the venue gave it to us: its home page and up to three pages it links to, such as the menu or the opening hours. There JetSiftBot looks for opening hours, prices and the menu as text, and for schema.org markup.

On that home page only, JetSiftBot also checks whether the site lets in the crawlers of OpenAI, Anthropic and Perplexity, which AI assistants read the web with: one request for each, whose user agent names that crawler and JetSiftBot together, so your logs show who asked.

It fetches each page once, when a venue's monthly report is made, and follows no other links. It keeps only the result, never the page itself.

robots.txt

JetSiftBot obeys robots.txt, on every site and after every redirect. Where your robots.txt has rules for JetSiftBot, it follows those. Otherwise it follows the rules for all crawlers (User-agent: *). If your robots.txt cannot be read, JetSiftBot fetches nothing from your site.

How to block JetSiftBot

Add these two lines to your robots.txt:

User-agent: JetSiftBot
Disallow: /

JetSiftBot reads your robots.txt again for every report, so the block holds from the next one.

How to recognise JetSiftBot

Every request it makes carries this user agent, but for the three on a venue's own home page, which name the crawler being checked as well:

Mozilla/5.0 (compatible; JetSiftBot/1.0; +https://jetsift.com/bot)

Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://jetsift.com/bot) JetSiftBot/1.0
Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://jetsift.com/bot) JetSiftBot/1.0
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://jetsift.com/bot) JetSiftBot/1.0

Questions

Write to hello@jetsift.com.