Contact sales

About KalevalaBot

What our crawler fetches, how often, what it keeps, and how to tell it to stop.

What it is

KalevalaBot is the small set of programs we run to find work: it reads public procurement notices, public company registers and a short list of web pages we have chosen, and tells us which ones are worth a look. It identifies itself on every request as

KalevalaBot/0.1 (+https://kalevala.dev/docs/bot; toimeksianto@kalevala.dev)

Where it reads

  • Official data interfaces first: the EU's TED notice search and Finland's Hilma for public tenders. These are published for exactly this kind of reuse.
  • The public job-board interfaces (Greenhouse, Lever, Ashby, Recruitee) of a short list of companies, read once a day: role titles and counts only.
  • The RSS feeds of a few business and startup news sites, every six hours: headlines and the feed's own summary, nothing behind a paywall.
  • A small list of ordinary web pages, mostly other software companies' own pages, fetched once a week to notice what changed.
  • The homepage of organisations we already know about, once a month, to note what the site is built on.

It never logs in anywhere, never fills in forms, never solves CAPTCHAs, and never uses proxies to hide where it comes from.

How it behaves on your site

  • It reads robots.txt first and follows it, including rules written for *. A server error on robots.txt counts as "do not fetch".
  • At most one request every five seconds per site, and it backs off when a site answers 429 or 503.
  • It asks whether a page changed before downloading it again (ETag and Last-Modified).
  • It honours tdm-reservation headers and other machine-readable opt-outs.

What it keeps

Facts about organisations and opportunities: a buyer's name, a notice's title, deadline and value, the technical roles a company has open, what a site is built on, a short summary we write with the help of a language model. Raw copies of what was fetched are deleted after 90 days; the facts after 24 months. Nothing about people is collected: no names, no emails, no profiles.

To make it stop

Add a robots.txt rule for KalevalaBot, or write to toimeksianto@kalevala.dev with your domain. Opt-outs are honoured within a day and kept permanently.