FlightBriefArchive crawler

FlightBriefArchive keeps dated copies of public pages about drones: regulators' lists and guidance, procurement notices, programme pages, and companies' product and news pages. The copies are FlightBrief's research record. They let a reference table or an article show what a page said on a given date, after the page has changed.

How it behaves

It identifies itself as FlightBriefArchive in the User-Agent header and gives[email protected] in the From header, the field HTTP sets aside for a crawler's contact address. It makes at most one request to a site every ten seconds, one at a time. It visits public pages only: it never logs in and never gets past a paywall, a CAPTCHA or a block. It does not build records about individuals.

It reads robots.txt and keeps a copy with each visit, but a Disallow line does not stop it: the pages it keeps are public records of what organisations said and when. If you want it to stay away, write to us instead.

Asking it to stay away

Email [email protected] with the site, or the part of it, you want left alone. Once the request is recorded, the crawler stops visiting those pages and does not add them again. Copies already made are kept privately as a dated record.

What happens to the copies

They are kept privately. FlightBrief publishes what it finds, with short quotes and links, not copies of other sites' pages. Each day's record is fingerprinted, signed and timestamped; the fingerprints and proofs are on the daily seals page.

Contact

Questions, or a request to leave a site or page alone:[email protected].