AIAssistantBot, the AnswerWeave crawler
If you found AIAssistantBot in your server logs, this page is for you. It is the crawler behind AnswerWeave: it reads a website so that a chat assistant on that website can answer from what the site actually says. It reads a site because the site’s owner told us to, and it obeys the same robots.txt rules search engines do.
Only sites its owner asked us to read
AIAssistantBot does not roam the web. A customer of AnswerWeave adds their own website, and the crawler reads that site — the same origin, never a link out to somewhere else — so an assistant on it can answer visitors from the site's own pages.
One request at a time, at a set pace
Requests to a site go one at a time. When the site’s owner starts a sync themselves the pause between requests is about a quarter of a second; automatic, scheduled re-syncs are slower still — no more than one request a second. A Crawl-delay in your robots.txt raises that pause, up to 30 seconds.
It backs off when you ask it to
If your server answers 429 or 5xx the crawler slows down, and if it sends a Retry-After it waits that long for the whole host before asking again. After a run of failures it stops contacting the host for several minutes and records on the customer’s sync report that some pages were skipped.
A person reads the mailbox
Tell us if it is doing something you do not want — a path it should stay out of, a rate that is too high, a site that never asked for it. Write to crawler@answerweave.ai with the hostname and roughly when you saw it.
How to recognise it
Its requests carry this User-Agent:
AIAssistantBot/1.0They are plain GET requests, and on a re-sync the crawler asks whether a page has changed (If-None-Match / If-Modified-Since) so an unchanged page costs your server a short “304 Not Modified” instead of the whole page. The crawler does not log in, does not submit forms and does not follow links to other sites. There is one exception to the label: when a page comes back as an empty shell that only a browser can fill in, the crawler reads that one page a second time with a headless browser, straight after the labelled request for the same address. That browser request carries the browser’s own User-Agent, not this one.
How it respects robots.txt
The crawler reads /robots.txt before it reads your pages, and checks every page against it at the moment it fetches that page — for every kind of source, including a list of addresses the site’s owner typed in.
- A group addressed to
AIAssistantBotis followed; without one, theUser-agent: *group is. The longest matchingAlloworDisallowrule wins, and*and$work as they do for search engines. Crawl-delayis honoured as a minimum pause between requests, up to 30 seconds.- A
robots.txtthat answers 404 (or another 4xx) means “no restrictions”. One that cannot be read at all — a 5xx, a timeout, a certificate error, a 429 — is not treated as permission: the crawler does not read the site until it can see the file. - A page the rules disallow is not fetched, and is reported to the customer as blocked by robots.txt, so they know why it is missing or out of date. What was already indexed from it is kept, never treated as removed.
- Sitemaps you declare in
robots.txtare read as a list of pages to visit.
How to keep it out
Add this to robots.txt and it will not read anything on that host:
User-agent: AIAssistantBot
Disallow: /To keep it out of one section only, disallow that path instead. If the site belongs to you and you did not add it to AnswerWeave yourself, write to us at the address below and we will find out who did.
Contact
crawler@answerweave.ai. Include the hostname and a rough time so we can find the requests in our own records.