Every Iowan Hydra crawler

Business-site crawler requests identify as EveryIowan-Hydra/2.0 (+https://everyiowan.com/bot; contact: https://everyiowan.com/contact)

What it does

Hydra discovers Iowa businesses, identifies their owned public websites, crawls permitted pages, and collects source evidence for accurate Every Iowan business listings. It reconciles names, locations, contact details, hours, services, and other useful facts while preserving where each fact came from.

Hydra also uses known feeds, APIs, calendars, and public data sources for information such as events, schools, sports, weather, and news.

Scope

  • Our listings center on Iowa businesses and organizations.
  • An official or evidentiary source may be hosted outside Iowa when it describes an Iowa subject.
  • Hydra reviews robots.txt for the exact public pages it plans to crawl and blocks disallowed targets.
  • Requests are rate limited per host, bounded in size and time, and restricted from private network addresses.
  • Other public-data requests may identify as EveryIowan-Hydra/2.0 (+https://everyiowan.com/bot; contact: https://everyiowan.com/contact).

Verifying our requests

Hydra is moving to signed requests using Web Bot Auth (HTTP Message Signatures). Signed requests carry a Signature-Agent header pointing to our public key directory at /.well-known/http-message-signatures-directory, so site owners and their security providers can confirm a request really came from us.

AI training

Hydra is not an AI training crawler, and Every Iowan does not use collected content to train a foundation model. Automated extraction or review may use an approved model to turn source material into structured, source-linked facts; model output is not treated as source evidence by itself.

Licensing

Every Iowan publishes structured facts, limited excerpts, attribution, and links according to the source's policy and the applicable rights. Hydra retains source evidence so published information can be checked and corrected.

Opt out

Site operators may disallow EveryIowan-Hydra in robots.txt. For a correction, removal request, or crawler concern, email [email protected] and include the affected URL.

Contact

[email protected]

Related: How Every Iowan works . DMCA notice