TrustVendorBot.
We crawl publicly accessible vendor pages — trust centres, subprocessor lists, security pages, DPAs, changelogs, and filings. We sell to compliance teams, not to advertisers, and we follow the policy below exactly.
User agent
TrustVendorBot/1.0 (+https://trustvendor.co/bot) The rules
Seven promises.
Identifiable user agent
Every request carries TrustVendorBot/1.0 (+https://trustvendor.co/bot) so you can identify and block us if you choose.
robots.txt respected
We cache robots.txt per eTLD+1 for 24 hours and log every check. Disallowed paths are skipped and recorded with robots_ok = false so we can prove compliance.
No authentication, no bypass
We never log in, never bypass a paywall, never solve a CAPTCHA, never use residential proxies to evade blocks. If a page requires a login, we do not fetch it.
Polite rate limits
Default: 1 request per 2 seconds per host. We back off on 429 and 503 with exponential jitter and do not resume until the host recovers.
APIs & RSS preferred
Structured feeds reduce load on your origin and give us cleaner data. We prefer them over HTML scraping wherever both exist.
Opt-out within 5 business days
Email [email protected] with your domain. Opted-out vendors still get a profile built from public registry data (NVD, EDGAR, CT), clearly labelled as such.
No verbatim republishing
We store snapshots for evidence integrity; public vendor pages show claims and short cited spans with attribution and a link to the source. Full snapshots are visible only to tenants monitoring that vendor.
Opt out, ask a question
Reach the crawler team.
We respond within 2 business days. Opt-outs honoured within 5.
[email protected]