DATUM

DatumBot

User-Agent: DatumBot/1.0 (+https://<domain>/bot)

Datum is a dashboard for AI news, models and measurements. This page describes exactly what our crawler fetches, because the URL in its User-Agent points here.

What DatumBot does

  • ·Polls the RSS and Atom feeds in our source inventory on a fifteen-minute cycle.
  • ·Polls one sitemap — anthropic.com/sitemap.xml, filtered to /news/ — because that publisher has no feed.
  • ·Fetches article pages only from sources whose class and licence permit it: labs and research organisations publishing their own announcements. Press articles are never body-fetched; we take the feed's own snippet and link out.

What DatumBot does not do

  • ·It does not fetch anything robots.txt disallows. A robots.txt we cannot retrieve is treated as a disallow, not as an absence of one — only an explicit 404 reads as allow-all.
  • ·It does not use archive mirrors, cache proxies, or any route around a paywall.
  • ·It does not crawl for training data. Nothing DatumBot fetches trains a model.
  • ·It does not republish your text. Publisher body text is stored transiently to classify and summarise it, and is deleted seven days after fetch.
  • ·It does not reuse your headlines. Every headline and summary on Datum is our own. Your headline appears in exactly one place: the source list on a story page, under your name, as the link to your article.

Rate

  • ·One request per second per host as a floor, and slower whenever robots.txt sets a longer Crawl-delay — we take the larger of the two.
  • ·Feeds are polled in preference to HTML, and requests are serialised per host, so two of our sources on one domain still share a single budget.

Opting out

  • ·robots.txt — disallow DatumBot, or disallow *. We re-read it on every run.
  • ·Email — ask us to remove your source and we will. We do not require you to justify it and we do not negotiate. Rows already held from your site are deleted rather than left to age out.

If we are getting something wrong, report it. A crawler that misbehaves and is never told is a crawler that keeps misbehaving; we would rather fix it than be blocked.