Wuizard

[ Crawler ]

WuizardBot, our crawler

When someone starts a design from a public website in Wuizard, WuizardBot reads one of that site’s public pages to learn its colours and type. Here’s how to recognise it, what it keeps, and how to keep it away.

How to recognise it

Every request it makes carries this user agent:

Mozilla/5.0 (compatible; WuizardBot/1.0; +https://wuizard.com/crawler)

Its robots.txt token is WuizardBot. It never sends cookies, passwords or any other credential, and never fills in or submits a form.

What it reads

  • Your robots.txt, before anything else on your site. It follows it for every request after that: pages and stylesheets.
  • One page and up to 8 of its stylesheets, when someone reads a site’s colours and type to start a sheet.
  • Only the site it was asked about. It doesn’t follow links to other sites. The one exception is a stylesheet a page loads from another address, and that address’s own robots.txt applies to it.
  • Never anything behind a sign-in. With no cookies or passwords, it only sees what any visitor sees.

What it keeps

Design tokens, re-made in Wuizard’s own terms. Nothing from your site is reproduced.

Kept

  • Design tokens: colour values, the names of type faces, corner radii, spacing and motion timing.
  • The site’s address, shown only to the person who asked and to our staff. For a single page, its title too.

Never kept

  • Any other text from the site: headings, copy, prices, quotes or names.
  • Images, logos, icons, videos or screenshots, or their addresses.
  • HTML, CSS, JavaScript or fonts.
  • Anything behind a sign-in.

A page’s HTML and CSS are held in memory only while it’s being read, then dropped. They’re never written to a database, a file or a log.

robots.txt

To keep WuizardBot out of your whole site, add this to your robots.txt:

User-agent: WuizardBot
Disallow: /
  • It follows the group for WuizardBot, or the * group when there isn’t one. The longest matching rule wins, and Allow wins a tie.
  • Crawl-delay is honoured as the least time between its requests to your site.
  • If your robots.txt can’t be read, or our request for it is refused, it doesn’t read your site at all.
  • Pages marked noai or noimageai, or that reserve text and data mining (tdm-reservation: 1), aren’t used.

How fast it goes

  • At least 1 second between two requests to your site, across all of Wuizard, and longer when your Crawl-delay asks.
  • For a single page and its stylesheets, nothing is asked for after 25 seconds.
  • If your site asks it to slow down, or refuses a page, it stops.

Questions

Email support@wuizard.com. The form on this page is the quickest way to opt out.