the crypto web, read and proven
This is the page named in our User-Agent string. If a surveyor of ours has been reading your site, this is what it is, what it fetched, and how to tell it to stop.
QED runs a small fleet of surveyors that read the BNB Chain documentation web — chain docs, protocol
handbooks and EIPs — page by page, and trains a model from scratch on exactly what they read. Each fetch is a
plain, unauthenticated GET for the page's HTML. A surveyor never submits a form, never runs the
page's scripts, never requests an image, a stylesheet or any other asset, and never carries a cookie or a
credential of any kind. If a page needs a session to read, we do not read it.
Every request we send — the robots.txt fetch and the page fetch alike — carries this exact
User-Agent header, with no variation:
QEDSurveyor/1.0 (+https://theqed.org/manual)
The URL in it is this page. There is no second agent string and no browser impersonation.
We fetch /robots.txt once per host and cache what it says; see the TTL in §04. When a host has a
group naming QEDSurveyor and a wildcard * group, the named group is used and the
wildcard group is ignored entirely — the two are never merged.
Within whichever group applies, the longest matching Allow or Disallow pattern wins;
if the longest Allow and the longest Disallow match the same length, the
Allow wins. A trailing * matches any run of characters, and a trailing
$ anchors the pattern to the end of the path. If your robots.txt answers with a server error
(a 5xx status), we treat that host as disallowed entirely until the next fetch, rather than guessing it is
open.
A host that disallows us everywhere:
User-agent: QEDSurveyor Disallow: /
A host open to everyone except one section, with the longest match rule carving out an exception inside it:
User-agent: * Disallow: /private/ Allow: /private/notices/
A host using a wildcard and an end anchor to keep us off a search page and a file type:
User-agent: QEDSurveyor Disallow: /search* Disallow: /*.pdf$
| pages fetched per cron tick | 6 |
|---|---|
| cron tick runs every | 3 minutes |
| cooldown between two fetches of the same host | 20 seconds |
| request timeout | 12 seconds |
| byte cap per page | 600,000 bytes (≈586 KiB) — larger responses are skipped unread |
| robots.txt cache | 24 hours |
| a page we have already read is fetched again at most | once every 14 days |
| links followed from a page | up to 14, only to hosts already on our surface, at most 4 hops from a seed |
Crawl-delay | honoured when it asks for more than our 20 seconds, up to a ceiling of 60 |
If twenty seconds between fetches is too fast for your host, say so in robots.txt and we will wait as long as you ask, to a minute:
User-agent: QEDSurveyor Crawl-delay: 60
For every page we keep: the text we extracted (its headings, paragraphs, list items and code, in document order), the sha256 of that text, the HTTP status we received, the byte count and the time the fetch took. We also keep the link graph — which of a page's links we chose to follow, and at what depth from its seed.
The sha256 of the page currently shown on the live plate is printed under it, so anyone can fetch the same URL, hash the same text, and get the same digest. Nothing we store is private to us.
Add a Disallow for QEDSurveyor to your robots.txt:
User-agent: QEDSurveyor Disallow: /
We re-fetch robots.txt at most once every 24 hours per host, so the change takes effect at your host's next robots fetch — at most 24 hours after you publish it, sooner if we have not read your robots.txt inside that window yet.
The frontier starts from these pages:
Links are only ever followed onto these 15 hosts; a link to anywhere else is left unread: