Amazonbot ignores robots.txt and falls for honeypot trap, user reports
A Hacker News user reports that Amazonbot aggressively scraped their personal website despite robots.txt directives and was caught accessing a hidden honeypot with fake HTTP endpoints.
Briefing
- Hacker News: A Hacker News user reports that Amazonbot aggressively scraped their personal website despite robots.txt directives and was caught accessing a hidden honeypot with fake HTTP endpoints.
Why it matters
This incident highlights ongoing tensions between web publishers and large AI crawlers that disregard standard opt-out mechanisms like robots.txt. The use of a honeypot to expose non-compliant scraping demonstrates a growing arms race between site owners and bot operators, with implications for trust, bandwidth costs, and the effectiveness of existing blocking tools.
Sources
- Reported · Hacker News · 2026-08-01 — Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt
Watch next
Will Amazon update Amazonbot's behavior to respect robots.txt after this public exposure?