About KalevalaBot
What our crawler fetches, how often, what it keeps, and how to tell it to stop.
What it is
KalevalaBot is the small set of programs we run to find work: it reads public procurement notices, public company registers and a short list of web pages we have chosen, and tells us which ones are worth a look. It identifies itself on every request as
KalevalaBot/0.1 (+https://kalevala.dev/docs/bot; toimeksianto@kalevala.dev)Where it reads
- Official data interfaces first: the EU's TED notice search, Finland's Hilma, the Finnish Patent and Registration Office's open data, and news summaries. These are published for exactly this kind of reuse.
- A small list of ordinary web pages (mostly other software companies' public pages), fetched at most once a week.
It never logs in anywhere, never fills in forms, never solves CAPTCHAs, and never uses proxies to hide where it comes from.
How it behaves on your site
- It reads
robots.txtfirst and follows it, including rules written for*. A server error onrobots.txtcounts as "do not fetch". - At most one request every five seconds per site, and it backs off when a site answers 429 or 503.
- It asks whether a page changed before downloading it again (ETag and Last-Modified).
- It honours
tdm-reservationheaders and other machine-readable opt-outs.
What it keeps
Facts about organisations and opportunities: a buyer's name, a notice's title, deadline and value, a company's registered line of business, a short summary we write with the help of a language model. Raw copies of what was fetched are deleted after 90 days; the facts after 24 months. Nothing about people is collected: no names, no emails, no profiles.
To make it stop
Add a robots.txt rule for KalevalaBot, or write to toimeksianto@kalevala.dev with your domain. Opt-outs are honoured within a day and kept permanently.