The Levies crawler

levies.fyi is the home of Levies, a small crawler run by StayLevy.

StayLevy tells short-term rental hosts which local taxes and permits apply to them. Each rule it shows cites an official source: a state tax page, a city or county ordinance, a rate table or a filing form. Levies keeps a copy of those pages so we notice when a rule changes and can correct what we show.

What it fetches, and how often

• Only the pages our published data cites, about 2,200 in all. It does not follow links or crawl a site.
• Once a week, on Mondays.
• One request at a time per site, with a pause between them.
• It obeys robots.txt.

How to recognize it

Its User-Agent contains Levies/1.0 and links here. Its requests are signed with Web Bot Auth (HTTP Message Signatures), so a site can check that a request really comes from us; the public key is at /.well-known/http-message-signatures-directory.

Some pages are read in a standard Chrome browser: those that only render with JavaScript, and those that refuse a plain request. The browser sends the same signature and names us in its User-Agent too. It does not disguise itself or solve CAPTCHAs.

Opting out

To stop it, disallow it in your robots.txt:

User-agent: Levies
Disallow: /

If it causes you any trouble, or you would rather tell us about a change yourself, write to hello@staylevy.com.