KashifBot, our crawler

Who runs it

KashifBot belongs to Kashif (kashifaudit.com), a service operated by Chatwiser Inc. If anything about it worries you, write to hello@kashifaudit.com.

How it introduces itself

Every request it makes carries this user-agent:

KashifBot/0.1 (+https://kashifaudit.com/bot)

The name to use in robots.txt is KashifBot.

What it reads

• robots.txt first, then the sitemap, llms.txt and /.well-known/webmcp if they exist.
• Then the site’s public pages: up to 8 pages in a full audit, and up to 3 pages in a quick check or a competitor check.
• It waits 0.5 seconds between requests, and 1 second between requests in a competitor check.
• It gives up on a page after 15 seconds and on a site file after 6 seconds.
• If a customer gave us a social profile link, it makes one request to that profile page to see whether it is public. It does not read the social platform’s robots.txt.
• If a server leaves out its intermediate certificate, KashifBot downloads that certificate from the address the site’s own certificate names (usually the certificate provider), the way a browser does. It never does this when a customer names your site as a competitor.
• Once per check it also opens one secure connection to your site to read the dates on its security certificate. It requests no page on that connection.

What it does not do

It sends plain GET requests. It does not log in, fill in or submit forms, or make payments.

When it visits

• When someone asks Kashif to check a site: the free scan on kashifaudit.com, or a customer checking their own site, on the schedule their plan sets.
• When a customer names your site as a competitor, Kashif reads up to 3 pages, and no more than once every 24 hours per site, however many customers named it.

It follows robots.txt

For your own site, KashifBot reads robots.txt before anything else and follows it. Rules under User-agent: * apply to it as well as rules that name KashifBot. If robots.txt cannot be read (a server error or a timeout), KashifBot treats the site as having no rules. To keep it out of your whole site, add this:

User-agent: KashifBot
Disallow: /

What is not KashifBot

Every audit, the free scan included, asks Google PageSpeed Insights to measure speed. A full audit also opens the home page and one contact page in a headless browser to take screenshots. Neither uses the KashifBot name, so robots.txt rules for KashifBot or for * do not cover them.

Kashif also looks up your domain's public DNS records and its registry entry. These lookups go to DNS servers and the domain registry, not to your site.

Ask us to stop

Email hello@kashifaudit.com with your site’s address. Our team adds it to our competitor opt-out list. A site on the list cannot be added as a competitor, and a competitor check already waiting for it is not run. The list does not stop a check someone runs on your site itself, such as the free scan. To keep KashifBot off your pages for every kind of check, use the robots.txt rule above.