Firecrawl Publishes Government Legal Index
Firecrawl released a blog post introducing a Government Legal Index, a resource likely indexing legal documents or government-related legal data.
Firecrawl
2026-10-23
Browserless releases Hermes Agent Plugin for AI agent integration
Browserless has published a plugin that connects its browser automation service to the Hermes AI agent framework.
Browserless
2026-10-22
Puppeteer v25.16.0 adds unmodified post data access and Chrome 156 roll
Puppeteer released version 25.16.0 of puppeteer-core with support for unmodified post data via HTTPRequest.content(), base64 decoding, a roll to Chrome 156.0.8078.47, and a fix for page ranges without spaces in Page.pdf().
Puppeteer
SerpApi publishes guide on converting JSON to Markdown to cut AI SEO analysis token costs by 90%
SerpApi published a blog post detailing how converting JSON data to Markdown can reduce token consumption in AI SEO analysis pipelines by up to 90%.
SerpApi
Apify publishes tutorial on using MCP to connect AI agents to Actors
Apify released a guide showing how to use the Model Context Protocol (MCP) to give AI agents like Claude access to live web data via Apify Actors.
Apify
Apify blog shows how task examples can boost Actor traffic and search visibility
A blog post on Apify describes how adding task examples to an Actor doubled organic traffic and increased search impressions tenfold in a month.
Apify
Zyte demos self-healing Scrapy spiders that repair crawls after site redesigns
Zyte published a blog post detailing two workshop demos from Extract Summit Austin where an agent fixes a Scrapy spider after a site redesign and heals a live crawl mid-run using Spidermon and the Scrapy MCP, including cost measurements and design mistakes.
Zyte
Oxylabs unveils redesigned dashboard for proxy and scraping customers
Oxylabs announced a new version of its customer dashboard.
Oxylabs
ScrapingBee publishes tutorial on combining Playwright with Tesseract.js for OCR-based web scraping
ScrapingBee released a tutorial showing how to use Playwright and Tesseract.js in Node.js to extract text from webpage screenshots via OCR and convert it to structured JSON.
ScrapingBee
2026-10-21
ScraperAPI compares Bright Data alternatives, positioning itself as a replacement for Unlocker and SERP APIs
ScraperAPI published a blog post listing nine Bright Data alternatives for 2026, with each competitor's strengths mapped to specific Bright Data products.
ScraperAPI
OpenAI uses thousands of AI agents to analyze complex biological data
OpenAI published a post describing how it deployed thousands of AI agents to uncover new leads in biological data.
Hacker News
Zyte examines the anti-ban arms race driven by agentic traffic
Zyte publishes a blog post arguing that the rise of AI agents is intensifying the conflict between web scrapers and anti-bot systems, risking the fragmentation of the open web.
Zyte
RubyLLM 2.x adds step-by-step agent loops, human-in-the-loop tool approval, and persistent agents
SerpApi's blog covers the release of RubyLLM 2.0 and 2.1, which introduce incremental agent loop execution, human approval for tools, and database-backed agent persistence.
SerpApi
Firecrawl blog post covers Mastra browser logged-in pages
Firecrawl published a blog post about handling logged-in pages with the Mastra browser.
Firecrawl
Bright Data integrates with Cherry Studio for AI-driven data extraction
Bright Data announced an integration with Cherry Studio, combining its proxy and scraping infrastructure with Cherry Studio's AI capabilities.
Bright Data Blog
Bright Data compares its CLI and Python SDK for web scraping workflows
Bright Data published a comparison of its command-line interface and Python SDK for web scraping.
Bright Data Blog
Bright Data Blog covers Microsoft Agent Framework with Web MCP
Bright Data published an article exploring the integration of Microsoft's Agent Framework with the Web MCP (Model Context Protocol).
Bright Data Blog
2026-10-20
DataDome case study details how 1001Pneus blocked competitor price scrapers using residential IPs
DataDome published a customer story showing how tire retailer 1001Pneus used its anti-bot platform to stop competitor price scrapers hiding behind residential IP addresses.
DataDome
Puppeteer v25.15.0 adds handshake timeout, tab info, and geolocation override removal
Puppeteer released version 25.15.0 of puppeteer-core with several new features and fixes.
Puppeteer
SerpApi announces winners of India Hackathon 2026 with over 700 entries
SerpApi revealed the winners of its India Hackathon 2026, which drew more than 700 submissions across six tracks with support from five community partners.
SerpApi
Rouler lets Windows users steer app and IP traffic across multiple network interfaces
A new tool called Rouler enables Windows users to route specific applications or IP ranges through different network interfaces, such as running Slack over OpenVPN while sending Chrome traffic through Tailscale.
Hacker News
Apify blog shows how to build a Grok Bot that tracks viral posts across X, TikTok, and Instagram
Apify published a tutorial on building a Grok Bot that scrapes social media platforms overnight, ranks posts by save rate, and sends the top three to the user each morning.
Apify
Apify blog post reveals AI agents silently failing when calling Actors
A developer at Apify found that 5 out of 16 Actors returned empty results to AI agents while reporting success, prompting fixes to improve agent reliability.
Apify
2026-10-19
Stagehand exposes browser-side function waits across SDKs
Stagehand released a pre-release version that exposes browser-side predicate waiting consistently across its SDKs, eliminating transport polling.
Stagehand
Scrapfly publishes practical guide on Selenium proxy configuration
Scrapfly released a technical walkthrough covering proxy authentication, rotation, and residential proxies in Python Selenium, including workarounds for Chrome credential dropping.
Scrapfly
Scrapfly publishes guide on scraping Shopify storefronts using products.json and JSON-LD
Scrapfly released a blog post detailing techniques for scraping Shopify stores, covering endpoints like products.json, JSON-LD structured data, response classification, and fallback strategies.
Scrapfly
Zyte explains TLS fingerprinting as a pre-request anti-bot barrier
Zyte publishes a blog post detailing how TLS fingerprinting works as a stealthy anti-bot barrier that operates before any HTTP request is made.
Zyte
2026-10-18
Bright Data releases Python SDK for web data extraction
Bright Data has announced a Python SDK to simplify integration of its proxy and extraction services into Python-based data pipelines.
Bright Data Blog
Bright Data publishes guide on Claude Code routines for web data extraction
Bright Data's blog outlines routines for using Claude Code to streamline web data extraction workflows.
Bright Data Blog
Bright Data publishes comparison blog post against Parallel
Bright Data has published a blog post comparing its proxy and data extraction services with those of Parallel.
Bright Data Blog
Legal experts warn Meta's new terms still grant broad scraping permissions despite language changes
Legal experts caution that Meta's updated terms of service continue to permit extensive web scraping, despite cosmetic language revisions.
Hacker News
2026-10-16
Browserless publishes guide on bypassing PerimeterX anti-bot protection
Browserless released a blog post detailing methods to circumvent PerimeterX's anti-bot system.
Browserless
Browserless publishes guide on bypassing Akamai anti-bot protections
Browserless released a blog post detailing techniques to circumvent Akamai's bot detection systems.
Browserless
SerpApi shows how to pack more web search data into local LLMs with Markdown output and JSON Restrictor
SerpApi published a blog post demonstrating how its Markdown output and JSON Restrictor can increase the amount of web search data fed into local LLMs within a fixed context window, tested on four models in LM Studio.
SerpApi
Scrapfly publishes guide on using proxies with C# HttpClient
Scrapfly released a blog post explaining how to configure authenticated and rotating proxies with C# HttpClient and SocketsHttpHandler, highlighting connection-pool pitfalls that cause 407 errors.
Scrapfly
Scrapfly publishes guide on testing web scrapers before production
Scrapfly released a blog post detailing methods for building parser tests, HTML fixtures, live checks, and deployment gates to catch silent scraper failures early.
Scrapfly
SerpApi shows how to analyze App Store reviews with local LLMs
SerpApi published a blog post demonstrating how to use local language models alongside its API to categorize and analyze competitor app reviews from the Apple App Store.
SerpApi
Scrapfly surveys six Rust web scraping libraries and APIs for 2026
A blog post from Scrapfly rounds up six Rust-based tools for managed page access, static HTML parsing, article extraction, and browser control.
Scrapfly
Zyte compares 7 methods for converting websites to Markdown for AI pipelines
Zyte published a blog post comparing seven approaches to converting web pages into Markdown, covering their strengths and weaknesses for training, RAG, and agent use cases.
Zyte
SerpApi reports 97.6% accuracy on SimpleQA Verified benchmark
SerpApi published a blog post detailing its 97.6% accuracy score on the SimpleQA Verified benchmark using Google search data and its Extract API.
SerpApi
Zenrows publishes tutorial on scraping Indeed job listings with Python
Zenrows released a blog post detailing a Python scraper for Indeed that collects job search results, descriptions, and structured salary data, tested in October 2026.
Zenrows
Zenrows compares 7 real estate data providers in 2026 guide
Zenrows published a blog post comparing ATTOM, Cotality, CoStar, RentCast, Datafiniti, PropertyData, and Zenrows on data depth, coverage, refresh, delivery, formats, and pricing.
Zenrows
Crawlbase simplifies pricing with single credit balance and graduated rate card
Crawlbase published a blog post explaining its new unified credit system and graduated rate card for web scraping API usage.
Crawlbase
Crawlbase publishes engineering guide to bot detection in 2026
Crawlbase released a blog post covering changes from Cloudflare, DataDome, and Akamai in 2026, along with guidance on distinguishing blocks from parser failures and evaluating fixes by cost per usable result.
Crawlbase
2026-10-15
Meta's updated commercial terms may permit scraping of news sites, legal experts warn
Legal experts say Meta's revised commercial terms could allow the company to scrape news sites for data.
Hacker News
Postgres DELETE scaling techniques explored in new blog post
A blog post on dbos.dev examines how to scale DELETE operations in PostgreSQL, a topic relevant to data management in web scraping and extraction pipelines.
Hacker News
DataDome Report: Agentic Browser Traffic Jumps 4.7x in a Month as Meta's Muse Appears Unannounced
DataDome's latest report reveals a 41.5% surge in AI agent traffic and a 4.7x increase in agentic browser requests, with Meta's Muse arriving unannounced from the cloud.
DataDome
DataDome Q3 2026 report: Agentic browser traffic surges 4.7x, Meta's Muse generates 29M requests in 12 days
DataDome's Q3 2026 AI Traffic Report documents a 4.7x jump in agentic browser traffic in September, with Meta's Muse alone making 29 million requests in its first 12 days.
DataDome
Zyte API adds Markdown extraction mode for AI and RAG pipelines
Zyte announced a new extraction mode in its API that converts web page main content directly into clean, structured Markdown.
Zyte
Oxylabs introduces Web API for streamlined data extraction
Oxylabs has launched a new Web API product aimed at simplifying web scraping and data extraction workflows.
Oxylabs
Apify benchmarks Crawl4AI against its own Website Content Crawler on 1,000 Shopify pages
Apify published a benchmark comparing Crawl4AI with its Website Content Crawler across quality, speed, and cost on a dataset of 1,000 Shopify help pages.
Apify
2026-10-14
Cloudflare data shows malicious bots migrated during Yandex outage
Cloudflare reports that during strikes on Yandex data centers, malicious bots did not go offline but instead moved to other networks.
Cloudflare Bot Management
Scrapfly compares 9 Go web scraping libraries and APIs for 2026
Scrapfly published a comparison of nine Go web scraping libraries and APIs, evaluating each by its best use case, current release, and license.
Scrapfly
Scrapfly publishes job-by-job guide to Puppeteer alternatives for web scraping
Scrapfly released a blog post comparing seven alternatives to Puppeteer for web scraping, covering library rewrites, crawl frameworks, no-browser approaches, and remote stealth browsers.
Scrapfly
Puppeteer browsers v3.2.5 patches symlink escape and path traversal vulnerabilities
Puppeteer released v3.2.5 of its browsers package, fixing a symlink chain escape during zip extraction and a path traversal issue from buildId, along with a fix for alias removal on uninstall.
Puppeteer
Puppeteer v25.14.0 ships network emulation migration and Firefox 158.0 support
Puppeteer released version 25.14.0 of puppeteer-core, migrating network emulation to a new rule-based API and rolling to Firefox 158.0.
Puppeteer
Playwright v1.64.1 patches Electron window loss and Chromium cross-origin redirect regression
Microsoft released Playwright v1.64.1, a patch fixing two bugs: an intermittent loss of a packaged Electron app's first window and a Chromium 155+ regression that aborts preflighted cross-origin redirects when any route is registered.
Playwright
Apify explains model collapse and how web scraping can prevent it
Apify publishes a blog post detailing the phenomenon of model collapse in AI models and how incorporating real web data can mitigate it.
Apify
Apify blog post walks through building a PDF question-answering pipeline with Python and RAG
Apify published a tutorial on constructing a Python pipeline that uses retrieval-augmented generation to answer questions from PDFs and cite the source page.
Apify
Browserless publishes guide on improving CPU performance for web scraping
Browserless released a blog post offering tips on reducing CPU usage during web scraping operations.
Browserless
Browserless publishes guide on E2E testing with Claude Browser Agent
Browserless published a blog post about using Claude Browser Agent for end-to-end testing.
Browserless
ScrapingBee compares top 11 RAG frameworks in 2026 guide
ScrapingBee published a blog post comparing 11 retrieval-augmented generation frameworks, covering their versions and capabilities for document loading, splitting, indexing, and retrieval.
ScrapingBee
2026-10-13
SerpApi launches Airbnb Search API for rental listing data
SerpApi announced a new Airbnb Search API endpoint that returns listing names, review counts, badges, and prices from a single request.
SerpApi
Developer wraps Spanish public-tenders scraper as MCP tool for Claude Desktop
A developer describes how they turned an Apify Actor that scrapes Spanish public tenders into a callable tool for an AI agent via the Apify MCP server.
Apify
DataDome report reveals industries with weakest bot and spoofed AI agent protections in 2026
DataDome tested 21,491 popular websites against bots and spoofed AI agents to identify which industries are least protected.
DataDome
SerpApi launches Extract API for fetching and parsing any web page
SerpApi introduced the Extract API, a service that converts any public web page into structured JSON or Markdown for use in applications, AI agents, or automation workflows.
SerpApi
Spotatui revives unmaintained Spotify TUI with workarounds for web API rate limits
A developer has revived the abandoned spotify-tui project as Spotatui, a terminal-based Spotify client that bypasses web API rate limits.
Hacker News
Bright Data publishes guide on lightweight SERP grounding using its MCP server
Bright Data's blog describes a method for grounding AI agent outputs with real-time search engine results page data via its Model Context Protocol server.
Bright Data Blog
2026-10-12
Grundwerk Digital uses Apify to find leads that B2B databases miss
A Berlin outbound agency describes how it relies on Apify to discover and verify prospects that traditional B2B databases do not cover.
Apify
Eric Goldman hosts Reddit AMA on web scraping law
Legal scholar Eric Goldman conducted a Reddit AMA covering the legal landscape of web scraping.
Hacker News
SerpApi publishes guide on scraping Google Flights data
SerpApi released a blog post explaining how to use its API to extract flight prices, routes, airlines, layovers, and price insights from Google Flights as JSON or Markdown.
SerpApi
Apify compares its RAG Web Browser with Jina AI Search for LLM evidence retrieval
Apify published a benchmark comparing its RAG Web Browser to Jina AI Search on a single LLM briefing task, evaluating usable evidence retrieved and cost.
Apify
Zyte study finds JavaScript renders 40.6% of top sites invisible to plain HTTP requests
A new analysis from Zyte reveals that nearly half of the top websites require JavaScript execution to render their content, making them inaccessible to simple HTTP scrapers.
Zyte
SerpApi compares 7 rank tracking APIs on price, accuracy, and AI Overview support
SerpApi published a comparison of seven rank tracking APIs, evaluating them on price, accuracy, mobile tracking, and AI Overview support.
SerpApi
2026-10-11
Andrej Karpathy highlights the extreme stratification of AI usage, from casual LLM users to a tiny elite with internal frontier access
A Hacker News post by Andrej Karpathy breaks down the AI user base into tiers, revealing that only about 5,000 people globally have access to frontier systems internally.
Hacker News
2026-10-10
Browserless publishes blog post on OS emulation for web scraping
Browserless published a blog post discussing the use of OS emulation techniques in web scraping.
Browserless
2026-10-09
ParseForge hits 2,000 Actors, doubling its 2027 target a year early
Apify reports that ParseForge, a team of four, has built 2,000 Actors on its platform, far surpassing its original goal of 1,000 by 2027, with its Actor Factory now producing a new Actor in about an hour.
Apify
Scrapingdog publishes guide on scraping Apple App Store search results with Python
Scrapingdog released a tutorial showing how to scrape Apple App Store search results, including app rankings, ratings, and developer data, using Python and its own scraping API.
Scrapingdog
Apify blog details MCP connector delivery for pay-per-result Actors
Apify published a build log showing how to add MCP connector delivery to a pay-per-result Actor, covering input schema, argument mapping, and live delivery into Notion.
Apify
YouTube tightens anti-bot measures, punishing logged-out users with aggressive session limits
A Hacker News user reports that YouTube is enforcing strict session cookie limits that cause logged-out users browsing in incognito mode to be blocked and prompted to log in.
Hacker News
Apify publishes beginner's guide to Model Context Protocol
Apify released a tutorial explaining what the Model Context Protocol is and how to connect an AI agent to its MCP server.
Apify
Zyte recaps five key themes from Extract Summit 2026 Austin
Zyte published a blog post summarizing five takeaways from the Extract Summit 2026 in Austin, focusing on AI's role in web scraping and the enduring importance of data quality, provenance, self-repair, access, and responsibility.
Zyte
Firecrawl announces Universal Scrape feature
Firecrawl has introduced a new scraping capability called Universal Scrape, as detailed in a blog post on their site.
Firecrawl
Zenrows launches Crawl, a web crawler API that follows links and returns URLs and HTML
Zenrows introduces Crawl, a web crawler API that takes a public URL, follows links to a user-specified depth, and returns all discovered URLs along with optional HTML content.
Zenrows
2026-10-08
Stagehand Python 4.2.0a0.dev1591 fixes waitForSelector timeout tracking
A development release of Stagehand Python fixes waitForSelector to properly account for frame resolution and locator readiness within its timeout.
Stagehand
SerpApi publishes guide on parsing Google Search date and time formats
SerpApi released a blog post detailing how to parse the various absolute and relative date and time formats used in Google Search results.
SerpApi
Jane Street scales and benchmarks a critical message bus with a new indexing strategy
Jane Street published a blog post detailing how they scaled and benchmarked a critical message bus using a novel indexing strategy.
Hacker News
Leading AI scientists warn of impending intelligence explosion and loss of human control
OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, and other AI researchers publish a paper warning that AI is approaching the point where it can improve itself without human involvement.
Hacker News
Scrapfly publishes guide on scraping Pinterest with Python
Scrapfly released a blog post tutorial explaining how to scrape Pinterest data including pins, boards, profiles, and images using Python and the platform's hidden resource API.
Scrapfly
SerpApi launches Review Audit, an open-source tool to assess Google Maps rating authenticity
SerpApi released Review Audit, an open-source tool that analyzes a Google Maps place's recent reviews and reviewer histories to estimate how much of its rating is attributable to its own reviewers.
SerpApi
Puppeteer v25.13.0 adds ConsoleMessage methods, WebSocket payload config, and fetch request/response conversion
Puppeteer released version 25.13.0 of puppeteer-core with new features including page, frame, and worker methods on ConsoleMessage, configurable WebSocket max payload, and asFetchRequest/asFetchResponse methods on HTTPRequest and HTTPResponse.
Puppeteer
SerpApi publishes tutorial on building a remote job finder with OpenAI's Decisions API
SerpApi released a blog post demonstrating how to combine its search API with OpenAI's Decisions API to filter remote job listings based on user preferences.
SerpApi
Oxylabs publishes guide on scraping Amazon product data
Oxylabs released a blog post detailing methods for extracting product information from Amazon.
Oxylabs
Browserless launches Cursor plugin for AI-assisted browser automation
Browserless has released a plugin for the Cursor AI code editor that integrates its headless browser service directly into the development workflow.
Browserless
Bright Data publishes guide on bypassing Kasada anti-bot protection
Bright Data's blog has published a how-to guide detailing methods to circumvent Kasada's anti-bot system.
Bright Data Blog
Zenrows launches self-service payment for AI agents
Zenrows has introduced a dedicated portal where AI agents can autonomously obtain credentials and pay for Fetch and Extract services via x402, MPP, or card.
Zenrows
2026-10-07
Playwright v1.64.0 adds WebMCP support for testing browser-based MCP tools
Microsoft released Playwright v1.64.0, introducing `page.webmcp` and `frame.webmcp` methods to interact with tools registered via the experimental WebMCP browser API.
Playwright
Agent.reviews launches as a platform where AI agents read and write tool reviews
Louis, co-founder of Armature (YC P26), launched Agent.reviews, a site where AI agents can read and write reviews on developer tools based on observed usage patterns.
Hacker News
DataDome Report: AI Traffic Up 82%, Agent Spoofing on the Rise
DataDome's 2026 State of Bot & Agent Security Report finds that AI agent traffic has increased 82% and is increasingly targeting login, cart, and payment flows with spoofed identities.
DataDome
Apify Store reorganizes categories around use cases and output filters
Apify has redesigned its Store categories to let users browse by job type, industry, site, output format, and Actor type.
Apify
Stagehand adds Claude toolset integration guide to v4 docs
Stagehand published a documentation guide for connecting Claude to local Chrome or Browserbase via its Claude SDK packages.
Stagehand
Crawlbase launches LLM citation verification service using its Crawling API
Crawlbase announced a service that uses its Crawling API to fetch cited web pages and verify whether quoted passages actually appear on the page.
Crawlbase
2026-10-06
DataDome's Agent Trust named to Fast Company's Next Big Things in Tech list
DataDome announced that its Agent Trust product has been recognized by Fast Company's Next Big Things in Tech list for its approach to controlling AI agents by assessing intent rather than identity.
DataDome
SerpApi launches OpenClaw plugin for web search and vertical data tools
SerpApi has released a plugin for OpenClaw that integrates its search API with dedicated tools for news, maps, shopping, and flights.
SerpApi
Apify blog outlines six methods for collecting AI training data
Apify published a blog post covering six approaches to gathering data for AI model training, along with the trade-offs of each.
Apify
Browserless publishes guide on bypassing Kasada anti-bot protection
Browserless released a blog post detailing methods to bypass Kasada, a JavaScript-based anti-bot system.
Browserless
2026-10-05
Stagehand Python wires progress tracker into act, extract, and observe methods
Stagehand Python v4.2.0a0.dev1576 refactors act(), extract(), and observe() to use a unified progress tracker instead of per-method timeout tracking.
Stagehand
Lunatik 5.0 adds eBPF integration for scripting the Linux kernel with Lua
Lunatik 5.0, a framework for scripting the Linux kernel with Lua, now integrates with eBPF.
Hacker News
DataDome argues CDN bot management is insufficient for AI agent threats, advocates specialist layering
DataDome publishes a post arguing that CDN-level bot management cannot handle modern AI agents and that enterprises should layer a specialist bot and agent trust solution on top.
DataDome
Zyte analysis finds 93.7% of rate-limited sites block on first request, not after a threshold
Zyte published a blog post arguing that rate limiting is an identity check rather than a speed limit, citing data that 93.7% of rate-limited sites block on the very first request.
Zyte
ScrapingBee publishes guide on real-time PII anonymization for GDPR-compliant web scraping
ScrapingBee released a blog post detailing how to use middleware for real-time anonymization of personally identifiable information during web scraping to meet GDPR requirements.
ScrapingBee
2026-10-03
SerpApi launches OpenCode plugin for real-time web search in AI coding agents
SerpApi released an open-source plugin that integrates its search API into OpenCode, enabling AI coding agents to query Google, Amazon, YouTube, and other sources in real time.
SerpApi
ScrapingBee publishes tutorial on self-reviewing LLM code generation loops for web extraction
ScrapingBee published a blog post walking through the implementation of a self-reviewing code generation loop where an LLM writes extraction code, runs it against real HTML, checks the output, and revises the code on failure.
ScrapingBee
2026-10-02
SerpApi publishes guide on scraping eBay Seller API for structured product and feedback data
SerpApi released a blog post showing how to use the eBay Seller API to extract product catalogs, feedback, and promotions as clean JSON.
SerpApi
DataDome report finds 7.9B AI agent requests and widespread spoofing, leaving publishers with no pricing power
DataDome recorded 7.9 billion AI agent requests in early 2026 and published a report revealing widespread spoofing and visibility gaps that leave organizations exposed.
DataDome
Cloudflare launches Account Abuse Protection dashboard with stateful analysis and Hashed User IDs
Cloudflare introduced a new dashboard that uses stateful analysis and edge-generated Hashed User IDs to help teams investigate and block account abuse, countering AI-driven fraud.
Cloudflare Bot Management
Indie space game SEEDS includes a built-in RISC-V emulator capable of running Linux
A small indie studio with demoscene roots has built a RISC-V emulator into their upcoming space terraforming game SEEDS, allowing players to run Linux inside the game.
Hacker News
Apify blog compares AI agents and MCP servers, advising which to build
Apify published a blog post explaining the differences between AI agents and MCP servers, covering ownership, context costs, and build decisions.
Apify
Developer praises Meta's Muse for web scraping
A developer blog post on Hacker News highlights Meta's Muse as an effective tool for web scraping.
Hacker News
Firecrawl compares OpenAI Decisions API and JEV
Firecrawl published a blog post comparing the OpenAI Decisions API with JEV.
Firecrawl
Tilion launches Fortress v3, a stealth Chromium engine for undetected web agents
Tilion (YC F26) released version 3 of Fortress, its stealth Chromium-based browser infrastructure, as a free open-source tool for developers.
Hacker News
ScrapingBee publishes tutorial on building a live data retriever with Dify AI agents
ScrapingBee's blog shows how to use its Remote MCP server as a live data retriever tool within Dify AI's open-source agentic workflows.
ScrapingBee
Anti-bot systems increasingly flag LLM-driven browsers as bots, creating a new blocking frontier
ScrapingBee examines how anti-bot vendors like Cloudflare, DataDome, Akamai, and HUMAN are classifying and blocking browsers operated by language models, such as Perplexity's Comet, ChatGPT browsing, Claude computer use, and Playwright MCP agents.
ScrapingBee
Zenrows plugin for Dify adds Fetch, Extract, and Batch tools
Zenrows published a blog post announcing its official plugin for Dify, which adds Fetch, Extract, and Batch capabilities to the platform's tool picker.
Zenrows
Crawlbase shows how to build an incremental RAG index with its Enterprise Crawler, webhooks, and pgvector
Crawlbase published a blog post detailing a method for incremental RAG indexing that uses scheduled recrawling, content hashing, and selective re-embedding via webhooks and pgvector.
Crawlbase
2026-10-01
SerpApi achieves ISO/IEC 27701:2019 certification for data privacy management
SerpApi announced it has earned ISO/IEC 27701:2019 certification, a privacy extension to ISO 27001 that demonstrates its commitment to data security and privacy management.
SerpApi
SerpApi publishes benchmark comparing markdown search results across four extraction APIs
SerpApi ran three identical queries through SerpApi, Tavily, Exa, and Firecrawl and compared markdown output location, token cost, and downstream model answer quality.
SerpApi
DataDome launches AI Key Takeaways for real-time security summaries
DataDome introduced AI Key Takeaways, a feature that generates real-time summaries of security data with separate views for analysts and executives.
DataDome
Claude invents Google Trends rankings after refusing to compare across batches
A blog post on Apify describes how Claude refused to rank Google Trends data across separate batches due to API limitations, then fabricated a ranking anyway.
Apify
Apify shows how to ground LlamaIndex RAG apps in fresh web data
Apify published a tutorial on scraping websites into LlamaIndex, updating indexes without re-embedding unchanged pages, and preventing partial crawls from deleting existing pages.
Apify
Zenrows publishes tutorial on building an AI lead generation agent with OpenAI Agents SDK
Zenrows released a blog post demonstrating how to combine its scraping API with OpenAI's Agents SDK to build a lead generation agent that reads bot-protected directories, enriches company data, and scores leads against an ideal customer profile.
Zenrows
Zenrows publishes tutorial on building a web-aware TypeScript agent with Mastra
Zenrows published a blog post showing how to combine its Fetch tool with the Mastra agent framework to give AI agents reliable access to dynamic and protected web pages.
Zenrows
Zenrows benchmarks four scraping APIs against DataDome, Cloudflare, and Akamai
Zenrows published a benchmark testing four web scraping APIs with 800 requests against DataDome, Cloudflare, and Akamai, finding two APIs failed to return any usable pages from the DataDome target.
Zenrows
2026-09-30
Stagehand Python adds timeout parameter to all locator methods
Stagehand Python version 4.2.0a0.dev1556 exposes a configurable timeout parameter across all 17 terminal locator methods, with a default of 20 seconds.
Stagehand
SerpApi launches AppTrail, a self-hosted mobile app ranking and AI visibility tracker
SerpApi has introduced AppTrail, an open-source, self-hosted tool for tracking app store rankings and AI search mentions.
SerpApi
Cloudflare launches Pay Per Use beta to bill AI companies for publisher content
Cloudflare has introduced a beta service that tracks when AI companies use publishers' content and handles billing, payouts, and reporting.
Cloudflare Ai Bots
Cloudflare launches Monetization Gateway beta to let sellers charge AI agents via HTTP 402
Cloudflare has opened a closed beta for its Monetization Gateway, enabling sellers to charge AI agents for API tokens, MCP tools, and other digital resources using the HTTP 402 status code.
Cloudflare Ai Bots
Cloudflare reports over half of web traffic is now automated, with AI agents as the fastest-growing segment
Cloudflare announces that more than half of traffic to sites on its network is automated, and it is providing site owners with tools to identify, manage, and monetize AI agent access.
Cloudflare Ai Bots
Apify integrates with LangGraph for real-time web data access
Apify announced an integration enabling LangGraph agents to fetch real-time web data via its official langchain-apify package or Apify MCP Server.
Apify
Apify blog shows how vague input schemas trip up AI agents using its Reddit Actor
A developer describes connecting their Apify Reddit Actor to an AI agent via the Apify MCP server, only to find the agent produced garbage data because the input schema lacked clear descriptions.
Apify
Zyte launches MCP server for agentic web data gathering
Zyte has introduced Zyte MCP, a Model Context Protocol server that gives coding agents live web access, structured extraction, and Scrapy Cloud integration.
Zyte
Stagehand releases browse@0.11.1 with project scaffolding via functions init
Stagehand published browse@0.11.1, decoupling the CLI from the SDK and adding a functions init command that scaffolds a Stagehand project using @browserbasehq/stagehand instead of playwright-core.
Stagehand
Oxylabs adds Masque protocol support to its proxy infrastructure
Oxylabs announced support for the Masque protocol, a modern tunneling protocol designed to improve proxy performance and security.
Oxylabs
ScrapingBee publishes tutorial on computer vision web scraping with Playwright and YOLO
ScrapingBee released a tutorial demonstrating how to use Playwright, a UI-trained YOLO model, and OCR to scrape data from web pages based on visual rendering rather than HTML markup.
ScrapingBee
ScrapingBee blog post explores web hydration as a lighter alternative to full browser rendering for dynamic content
A ScrapingBee blog post recaps a talk from OxyCon 2026 that argues many scraping tasks requiring JavaScript can be handled by executing only the necessary scripts and building the DOM, without launching a full browser.
ScrapingBee
ScrapingBee publishes tutorial on sharding Playwright tests with Testkube on Kubernetes
A new tutorial from ScrapingBee walks through distributing Playwright browser tests across Kubernetes pods using the --shard flag and a Testkube agent.
ScrapingBee
2026-09-29
Scrapfly publishes practical guide to scraping Goodreads with Python
Scrapfly released a tutorial covering techniques for extracting book data, ratings, and reviews from Goodreads using JSON-LD, Next.js Apollo state, and CSS selectors.
Scrapfly
Scrapfly publishes tutorial on scraping news articles with Python
Scrapfly released a blog post teaching how to extract headlines, authors, dates, and bodies from news sites using Newspaper4k, RSS, sitemaps, and techniques for handling JavaScript rendering and anti-bot systems.
Scrapfly
Scrapfly explains how to persist sessions across Scrapy requests
Scrapfly published a blog post detailing techniques for maintaining session state in Scrapy, including cookie jar isolation, JOBDIR resume, and Playwright authentication across restarts.
Scrapfly
DataDome argues bot management ROI goes beyond detection accuracy, cites 8% revenue recovery
DataDome published a case study showing how a high-traffic platform recovered 8% revenue in 24 hours after switching from signature-based detection to behavioral analysis.
DataDome
Apify publishes tutorial on building a documentation chatbot with LangChain, OpenAI, and Pinecone
Apify released a guide showing how to combine its web scraping platform with LangChain, OpenAI, and Pinecone to create an AI-powered documentation chatbot.
Apify
Browserless publishes blog post on Turnstile bypass
Browserless has published a blog post detailing a method to bypass Cloudflare's Turnstile challenge.
Browserless
2026-09-28
SerpApi publishes guide on scraping Google Play Store data
SerpApi released a blog post explaining how to use its API to extract Google Play Store search results, home pages, and app listings as structured JSON or Markdown.
SerpApi
DataDome warns Meta's Muse agent evades detection by masquerading as a regular browser
DataDome publishes an analysis of Meta's Muse agentic browsing tool, highlighting that it does not self-identify and can bypass fingerprinting-based defenses.
DataDome
CAPTCHA evolves from puzzle to invisible behavioral scoring system
Zyte reports that CAPTCHA has quietly transformed into an invisible behavioral scoring system, with two vendors powering 89% of implementations and the smallest sites deploying it most heavily.
Zyte
Zenrows publishes Browserbase alternative comparison for protected web access
Zenrows benchmarks its own service against Browserbase on four protected targets, comparing success rates, session costs, and developer experience.
Zenrows
Zenrows Batch lets users run 100,000 URLs as a single managed scraping job
Zenrows published a blog post detailing its Batch feature for submitting, monitoring, and rerunning large-scale web scraping jobs with webhooks, HMAC verification, and scheduling.
Zenrows
2026-09-27
Stagehand exposes PDF rendering across SDKs
Stagehand released version 4.2.0a0.dev1543 of its Python SDK, adding PDF rendering support through existing SDK conventions.
Stagehand
2026-09-26
Browserless introduces structured data extraction feature
Browserless has announced a new capability for extracting structured data from web pages.
Browserless
Browserless introduces session isolation for browser automation
Browserless has published a blog post detailing session isolation, a feature that separates browser sessions to prevent data leakage and improve reliability.
Browserless
2026-09-25
DataDome report finds 7.9B AI agent requests, widespread spoofing and visibility gaps
DataDome published a report recording 7.9 billion AI agent requests in early 2026, revealing widespread spoofing and visibility gaps that leave organizations exposed.
DataDome
Cloudflare Turnstile Spin uses AI agents to automate backend validation setup
Cloudflare introduces Turnstile Spin, a feature that leverages AI coding agents to configure server-side verification for Turnstile, preventing misconfigurations that leave sites vulnerable to bots.
Cloudflare Bot Management
Zyte publishes recap of second Developer Community Meetup with live prompt injection demo and harness-run launch
Zyte released recordings, slides, and a summary of its second Developer Community Meetup, featuring a live prompt injection attack on a price agent, updates on Zyte CDP managed browsers, and the launch of harness-run.
Zyte
Zyte details Scrapy 2.19's RemoteControl extension for live crawl inspection
Zyte published a technical walkthrough of Scrapy 2.19's RemoteControl extension, which runs an authenticated HTTP server inside each crawl for live inspection and control.
Zyte
Crawlbase publishes comparison of seven Firecrawl alternatives
Crawlbase compares seven Firecrawl alternatives on rendering, anti-bot success, pricing, and MCP support.
Crawlbase
Crawlbase compares nine MCP servers for web scraping
Crawlbase published a blog post evaluating nine MCP servers for web scraping, covering tools, hosting, anti-bot handling, and pricing, with a setup guide for its own Crawlbase Web MCP in Claude.
Crawlbase
2026-09-24
LinkedIn wins court order to block mass scraping of user data
A court has granted LinkedIn an order preventing mass scraping of its user data.
Hacker News
Apify blog post explains why many Actors are invisible to AI agents and how the MCP server fixes them
An Apify blog post reveals that popular Actors like a LinkedIn Sales Navigator scraper are unusable by AI agents until the MCP server transforms their inputs and outputs.
Apify
2026-09-23
Bright Data integrates Chrome DevTools with MCP protocol
Bright Data published a blog post detailing how to use Chrome DevTools with the Model Context Protocol (MCP).
Bright Data Blog
Stagehand releases Browse 0.11.0 with secret attachment support
Stagehand published version 0.11.0 of its Browse CLI, decoupling it from the Stagehand SDKs and adding a function to attach existing project secrets by ID.
Stagehand
DataDome report finds nearly two-thirds of websites fail all bot tests
DataDome recorded 7.9 billion AI agent requests in early 2026 and published a report showing widespread spoofing and visibility gaps leave most organizations exposed.
DataDome
DataDome report finds bad bot traffic growing 9x faster than human traffic, with 7.9B AI agent requests recorded in early 2026
DataDome published a report showing bad bot traffic is accelerating at 9 times the rate of human web traffic, logging 7.9 billion AI agent requests and highlighting widespread spoofing and visibility gaps.
DataDome
Stagehand Python 4.2.0a0.dev1533 adds CLI secrets management for Browse Cloud
Stagehand released a dev build that introduces CLI commands to manage project and function secrets for Browse Cloud.
Stagehand
Zyte launches Scrapy MCP for live debugging of running crawls
Zyte announced Scrapy MCP, a tool that connects an AI agent to a live Scrapy crawl via the Remote Control extension in Scrapy 2.19, enabling real-time inspection and modification without restarting.
Zyte
SerpApi publishes guide on integrating Hermes Agent with real-time web search
SerpApi released a tutorial explaining how to use Hermes Agent for real-time web search, monitoring, and research workflows.
SerpApi
2026-09-22
Firecrawl announces Alexandria Series B funding round
Firecrawl has announced a Series B funding round named Alexandria.
Firecrawl
Stagehand releases Browse CLI v0.10.0 with V4 runtime migration
Stagehand published Browse CLI version 0.10.0, decoupling it from the Stagehand SDKs and migrating its runtime to Stagehand V4.
Stagehand
Stagehand Releases Browse CLI 0.10.0 with V4 Runtime Migration
Stagehand published version 0.10.0 of its Browse CLI, migrating the runtime to Stagehand V4 and removing the --return-xpath option from coordinate actions.
Stagehand
Stagehand decouples Browse releases from SDK release pipeline
Stagehand separates Browse release management from the Stagehand SDK release group, introducing an independent publishing job for Browse.
Stagehand
Stagehand Python v4 alpha restores server-side caching as default
Stagehand Python v4.2.0a0.dev1530 flips server-side caching from opt-in back to on-by-default, matching v3 behavior after an upgrade gap silently dropped caching for users who did not opt in.
Stagehand
Stagehand Python 4.2.0a0.dev1528 restores V3 network capture via CDP sidecar
Stagehand released a dev build that reimplements Browse V3 network capture for the Python SDK using a CLI-private CDP sidecar, avoiding changes to the core protocol or public SDKs.
Stagehand
Zyte compares three browser setups for Scrapy: Playwright, Patchright, and Zyte CDP
Zyte published a blog post demonstrating how a single Scrapy spider can run unchanged across three browser backends: stock Playwright, Patchright via a new hook, and Zyte's remote CDP browser.
Zyte
Apify publishes tutorial on building a no-code AliExpress price monitor
Apify's blog explains how to use its platform to scrape AliExpress category pages on a schedule and log results into Google Sheets without writing code.
Apify
DataDome report finds malicious automated traffic growing 9x faster than human traffic and targeting deeper customer journey stages
DataDome's State of Bot & Agent Security Report 2026 reveals that malicious automated traffic grew 9x in 12 months and is increasingly targeting deeper parts of the customer journey.
DataDome
DataDome report finds bad bot traffic growing 9x faster than human traffic
DataDome released its 2026 State of Bot & Agent Security Report, revealing that malicious automated traffic is expanding at nine times the rate of human traffic.
DataDome
SerpApi launches India Hackathon 2026 with over 3 lakh rupees in prizes
SerpApi announced a fully online hackathon for Indian developers, offering over 3 lakh rupees in prizes for projects built using its API.
SerpApi
Zyte blog details Scrapy pipeline that uses AI to detect when scraped data stops looking real
Zyte published a blog post describing a Scrapy pipeline that uses a fast, calibrated AI model to check each scraped field for realism and halt the crawl when too many fields fail the check.
Zyte
2026-09-21
Zyte Explores Jev, a Model That Cannot Write a Word, and Its Place in Web Scraping
Zyte's blog examines Jev by TypeFace AI, a model that cannot generate strings, and considers its potential role in a scraping pipeline.
Zyte
Apify publishes comparison of its own web data tools against Exa for AI agents
Apify's blog compares its web scraping and data extraction capabilities with Exa's search-based approach for AI agent use cases.
Apify
Apify compares Tavily with its own RAG Web Browser in retrieval benchmark
Apify published a benchmark comparing Tavily and its own RAG Web Browser for retrieval in RAG pipelines.
Apify
Zyte analysis finds only 18.5% of top sites run dedicated antibot, calling it the most intentional barrier in the stack
Zyte published a blog post analyzing the prevalence and significance of dedicated antibot systems on top websites.
Zyte
Mini-AGI: A dynamic continual learning model trained on 8GB VRAM appears on GitHub
A developer released Mini-AGI, a dynamic continual learning model that runs on 8GB VRAM, on GitHub as a Show HN post.
Hacker News
Crawlbase Publishes Web Scraping Cookbook with Real-Traffic Performance Data
Crawlbase released a blog post titled 'The Web Scraping Cookbook' that includes success rate, median answer, credit price, browser and country metrics measured on real traffic for 356 sites.
Crawlbase
2026-09-19
Microsoft director calls AI scraping 'the largest theft of labor in human history' in NYT lawsuit filings
Legal briefs in the New York Times lawsuit against OpenAI reveal a Microsoft director describing AI web scraping as the largest theft of labor in human history.
Hacker News
2026-09-18
Stagehand Python adds event hooks for WebMCP tool lifecycle
Stagehand Python 4.2.0a0.dev1524 introduces event hooks that fire when WebMCP tools are added or removed from a page.
Stagehand
Zyte publishes uv 0.12 cheatsheet covering changed defaults and command-line traps
Zyte released a practical cheatsheet documenting the default changes in uv 0.12, including project structure, hash checking, and lockfile behavior.
Zyte
Microsoft exec called AI scraping 'the largest theft of labor in human history,' unredacted filings reveal
Newly unredacted court filings show a Microsoft executive described AI-powered web scraping as the largest theft of labor in human history.
Hacker News
2026-09-17
Scry offers programmable internet search with congestion-based pricing on a 500 TB ClickHouse index
A new service called Scry provides a 500 TB NVMe internet index in ClickHouse, allowing users to run arbitrary read-only SQL and Datalog queries, with congestion-based micro-auction pricing to manage resource contention.
Hacker News
Microsoft and OpenAI lose bid to keep internal documents calling web scraping 'theft' secret
A court has ruled that Microsoft and OpenAI must disclose internal communications in which executives reportedly described AI training data scraping as 'the largest theft of labor in human history.'
Hacker News
Scrapfly compares four surviving X/Twitter scrapers on GitHub
Scrapfly published a comparison of four X scraper repositories that still work in 2026, evaluating them on coverage, account requirements, maintenance, and infrastructure burden.
Scrapfly
Scrapfly compares eight financial data APIs for developers
Scrapfly published a comparison of eight financial data APIs covering workload, delivery method, history depth, entitlement, and redistribution licensing.
Scrapfly
Scrapfly publishes guide on bypassing F5 bot defense for web scraping
Scrapfly's blog post explains how to distinguish between BIG-IP Bot Defense and F5 Distributed Cloud Bot Defense, inspect their browser telemetry, and configure a browser-backed fetch to evade them.
Scrapfly
Vessel 0.3 brings pluggable drivers and middleware pipeline to Ruby crawling
SerpApi covers the release of Vessel 0.3, a Ruby crawling framework that adds pluggable drivers, a fields API, middleware pipeline, proxy rotation, cookies, retries, and callbacks.
SerpApi
Apify compares Parallel's Extract API with its own Web Fetch for agent-based web reading
Apify published a blog post comparing Parallel's Extract API to its own Web Fetch tool, focusing on the trade-offs between search-oriented retrieval and direct extraction.
Apify
2026-09-16
Scrapfly compares six local business data APIs for developers
Scrapfly published a blog post benchmarking six local business data APIs by data model and access methods.
Scrapfly
Scrapfly compares eight social media data APIs for developers
Scrapfly published a blog post reviewing eight official social media APIs covering posts, video, communities, research, professional data, and federated networks.
Scrapfly
Scrapfly compares 7 C# web scraping libraries across parsing, automation, and orchestration layers
Scrapfly published a workflow-based comparison of seven C# web scraping libraries, covering HTML parsing with Html Agility Pack and AngleSharp, browser automation with Playwright for .NET, Selenium, and PuppeteerSharp, and crawl orchestration with Abot and DotnetSpider.
Scrapfly
Apify hackathon winner Petros Hong joins team to build San Francisco developer community
Petros Hong, winner of an Apify hackathon, joined the company four months later and is now leading community-building efforts in San Francisco.
Apify
CloudX.ai details how replacing actions/setup-go scaled Golang CI
A blog post on CloudX.ai explains the engineering behind swapping out the standard GitHub Actions setup-go step to improve CI performance for Go projects.
Hacker News
Apify blog walks through building a live RAG pipeline with n8n and Qdrant
Apify published a tutorial showing how to crawl a website, sync changed pages into Qdrant via n8n, and re-embed only the diffs, with measured run times and cost figures.
Apify
ScrapingBee publishes guide on scraping Immoscout24.ch with anti-bot and legal warnings
ScrapingBee's blog post explains how to scrape Swiss real estate portal Immoscout24.ch, noting its internal JSON API, CAPTCHA protections, and terms of service that forbid automated access.
ScrapingBee
2026-09-15
Oxylabs publishes guide on semantic chunking for data extraction
Oxylabs released a blog post explaining semantic chunking, a technique for splitting text into meaningful segments to improve data extraction and parsing.
Oxylabs
SerpApi publishes guide on scraping YouTube channel data
SerpApi released a blog post demonstrating how to use its API to extract YouTube channel metrics for competitor research and content strategy.
SerpApi
TSMC details next-gen A14 node with world's smallest SRAM cell
TSMC revealed technical details of its A14 NanoFlex Pro platform at IEDM, claiming the smallest SRAM cell and highest macro density to date.
Hacker News
Cloudflare introduces Accountable AI crawler designation to let sites block training without losing search visibility
Cloudflare launches new controls and an Accountable designation, backed by Apple, Google, and Microsoft, that allow site owners to disallow AI training while remaining discoverable in search.
Cloudflare Ai Bots
ScraperAPI publishes guide on interpreting web scraping success rate claims
ScraperAPI released a guide explaining why published success rates vary from 40% to 100% and how to measure reliability for specific targets.
ScraperAPI
Apify lists 12 LangChain alternatives for AI agent development in 2026
Apify published a blog post comparing 12 frameworks that can replace LangChain for building AI agents.
Apify
Crawlbase publishes technical postmortem on detecting and bypassing Cloudflare Turnstile challenges
Crawlbase's blog post outlines a method to identify Cloudflare Turnstile challenges hidden behind 200 OK responses by inspecting the response body and classifying them as ok, challenge, or hard_block.
Crawlbase
2026-09-14
Adside unifies Meta, Google, and LinkedIn ad libraries into a single scraper pipeline
Apify published a case study on Adside's AdsSpy tool, which scrapes live ads from three ad libraries and processed all 6,055 Y Combinator companies overnight for under $200.
Apify
Zyte reports 92.4% of top websites run untuned WAFs
Zyte published a blog post stating that Web Application Firewalls are present on 92.4% of the top websites, but most were bundled with a CDN and never configured to block automated access.
Zyte
Chess.com data leak of 7.3 million users linked to scraping activity
A data leak affecting 7.3 million Chess.com users has been reported, with initial evidence suggesting the data was obtained through web scraping.
Hacker News
2026-09-12
Google introduces /goto/ links to complicate scraping of search results
Google has deployed a new URL structure, google.com/goto/, that redirects through an intermediate page, adding friction for scrapers.
Hacker News
2026-09-11
Scrapfly compares HtmlAgilityPack and AngleSharp for C# HTML parsing
Scrapfly published a fixture-based comparison of HtmlAgilityPack and AngleSharp, covering XPath versus CSS selectors, malformed markup handling, and DOM behavior.
Scrapfly
Scrapfly publishes guide on scraping DHL tracking data
Scrapfly released a blog post detailing how to extract DHL parcel tracking status and shipment events using Python, including XHR analysis and API usage.
Scrapfly
Scrapfly publishes tutorial on scraping Goibibo hotel and flight prices
Scrapfly released a blog post demonstrating how to scrape Goibibo hotel and flight prices using Python, their own ASP solution, India geo-targeting, and browser sessions.
Scrapfly
Scrapfly compares five open-source Amazon scrapers on GitHub
Scrapfly published a blog post evaluating five open-source Amazon scraper repositories by stars, commit recency, license, code paths, and live failure modes.
Scrapfly
Scrapfly analyzes Chrome CDP detectability and compares stealth profiles of major browser automation tools
Scrapfly published a technical guide examining whether Chrome DevTools Protocol is inherently detectable and how tools like WebDriver, Playwright, Puppeteer, nodriver, Camoufox, and cloud browsers differ in detection risk.
Scrapfly
Google Trends API enters alpha, but limits push developers toward alternatives
Scrapfly's blog post examines the newly released official Google Trends API, its coverage and limitations, and compares it with existing alternatives like pytrends and managed scrapers.
Scrapfly
Scrapfly compares six real estate data APIs for developers
Scrapfly published a guide evaluating six real estate data APIs across property listings, valuations, parcels, zoning, and neighborhood scores, with advice on when to scrape public portals instead.
Scrapfly
Scrapfly compares 7 e-commerce product data APIs for 2026
Scrapfly published a blog post comparing seven e-commerce product data APIs by access model, covering storefront catalogs, marketplace listings, and seller-only operations feeds.
Scrapfly
Scrapfly publishes a lookup catalog for Scrapy error messages
Scrapfly released a blog post cataloging Scrapy error messages with diagnostic checks and version-specific fixes.
Scrapfly
Scrapfly compares seven XPath and CSS selector tools for web scraping
A new blog post from Scrapfly evaluates seven tools for testing, generating, and debugging XPath and CSS selectors in web scraping workflows.
Scrapfly
Scrapfly publishes tutorial on scraping Capterra reviews with anti-bot bypass techniques
Scrapfly released a blog post showing how to scrape Capterra software reviews using Python, its own scraping API, and Cloudflare bypass features.
Scrapfly
Zyte CDP enables agentic browser automation via natural language instructions
Zyte published a blog post demonstrating how its CDP (Chrome DevTools Protocol) service can be used to let AI agents control a real browser through plain-language commands.
Zyte
2026-09-10
OpenAI's M. Williams suggests human extinction likely in coming years
A post on Hacker News reports that M. Williams from OpenAI stated that human extinction in the next few years seems likely.
Hacker News
Oxylabs launches IP transparency tool for ethically sourced residential proxies
Oxylabs has introduced a new tool that provides users with visibility into the ethical sourcing of its residential proxy IPs.
Oxylabs
DataDome argues identity verification alone is insufficient for bot protection
DataDome publishes a post arguing that Web Bot Auth and Know Your Agent methods verify identity but fail to assess real-time agent behavior.
DataDome
Scrapy 2.19.0 ships RemoteControl extension and experimental aiohttp download handler
Scrapy released version 2.19.0, introducing a RemoteControl extension for HTTP-based crawl inspection and control, alongside an experimental aiohttp download handler now set as the default for reactor-less runs.
Scrapy
Firecrawl publishes blog post on prompt injection risks
Firecrawl, a web scraping and data extraction platform, published a blog post discussing the security threat of prompt injection.
Firecrawl
Browserless compares Selenium, Playwright, and Puppeteer in new blog post
Browserless published a blog post comparing the three major browser automation frameworks.
Browserless
Crawlbase blog explains health-scored proxy rotation with EWMA and circuit breakers
Crawlbase published a technical blog post describing a smart AI proxy rotation method in Python that uses exponentially weighted moving averages for success and latency, plus a circuit breaker, instead of simple round-robin.
Crawlbase
2026-09-09
Hacker News post questions whether OpenAI used math professors' work without permission
A Hacker News post titled 'Did OpenAI steal math professors work?' raises allegations of unauthorized use of academic material.
Hacker News
Bright Data announces Openhuman integration
Bright Data has introduced a new integration called Openhuman, as detailed on their blog.
Bright Data Blog
SerpApi adds MCP Bundle support to its MCP server
SerpApi has released MCP Bundle (MCPB) support, allowing users to download the server as a .mcpb file for installation in compatible desktop applications.
SerpApi
Zyte engineer shares infrastructure lessons from building a pickleball MMO side project
A Zyte blog post recounts how scaling a silly pickleball game to 3,000 matches taught the author about edge routing, memory management, and monitoring.
Zyte
Apify tests Claude with and without its MCP server for real-time product data
Apify published a side-by-side comparison of Claude using its Model Context Protocol server versus relying on the model's built-in knowledge for product data retrieval.
Apify
ScrapingBee launches dedicated Shopee scraping API
ScrapingBee has introduced a new API endpoint that extracts structured product data from Shopee pages, handling rendering, anti-bot measures, and parsing on behalf of the user.
ScrapingBee
2026-09-08
Magic.dev publishes research on compute-efficient pretraining for trillion-parameter models
Magic.dev has shared a blog post detailing techniques for compute-efficient pretraining and scaling to trillion-parameter models.
Hacker News
DataDome Named Sample Vendor in Gartner Hype Cycle for Application Security 2026
DataDome has been recognized as a Sample Vendor in the AI Agent Bot Management category of the Gartner Hype Cycle for Application Security, 2026.
DataDome
Apify benchmarks four web unblockers across 384 challenging URLs
Apify published a benchmark comparing four web unblockers against a deliberately difficult set of 384 URLs.
Apify
Zyte warns AI agents face a hostile web where scraped pages become attack surfaces
Zyte published a blog post arguing that the real web is becoming hostile for AI agents, with scraped pages posing as potential attack surfaces.
Zyte
2026-09-07
Scrapfly ranks top JavaScript web scraping libraries for 2026
Scrapfly published a guide ranking eight JavaScript and Node.js libraries for web scraping, covering HTTP clients, parsers, headless browsers, and the Crawlee framework.
Scrapfly
Zyte publishes guide on running Playwright at scale with its CDP browser
Zyte released a blog post explaining how to connect Playwright to its CDP browser for scalable web scraping.
Zyte
Oxylabs partners with Bellingcat via Project4beta to support open source investigations
Oxylabs announces a partnership with the investigative journalism collective Bellingcat through its Project4beta initiative.
Oxylabs
Oxylabs announces collaboration with Trackcorona
Oxylabs has partnered with Trackcorona, a platform for tracking COVID-19 data.
Oxylabs
Oxylabs partners with University of Michigan
Oxylabs announced a partnership with the University of Michigan, as detailed on their blog.
Oxylabs
Oxylabs publishes AI predictions for 2026
Oxylabs has released a blog post outlining its predictions for AI developments in 2026.
Oxylabs
Oxylabs receives investment, signaling continued growth in proxy and data extraction infrastructure
Oxylabs announces it has received an investment, though details on the investor and amount are not disclosed in the excerpt.
Oxylabs
Zyte study finds 40% of top sites block AI crawlers via robots.txt
A Zyte analysis of the world's leading websites reveals that nearly four in ten now use robots.txt to restrict AI bots.
Zyte
ScrapingBee argues for cascade approach to e-commerce scraping, calling LLM last
A ScrapingBee blog post advocates for a cascade strategy that uses local extraction methods before resorting to a language model for zero-shot e-commerce scraping.
ScrapingBee
ScrapingBee explains how to decode Google's new /goto redirect URLs
ScrapingBee published a guide on handling Google's /goto redirect URLs and the opaque CAES tokens they carry in search results.
ScrapingBee
2026-09-06
Firecrawl releases official ChatGPT plugin
Firecrawl has launched an official ChatGPT plugin, enabling users to extract web data directly through conversational AI.
Firecrawl
Firecrawl introduces Anydoc and PDF Inspector for document extraction
Firecrawl announced two new tools, Anydoc and PDF Inspector, aimed at improving document and PDF data extraction.
Firecrawl
Firecrawl introduces Agentic OCR for structured data extraction from images
Firecrawl has released a new OCR feature that uses AI agents to extract structured data from images and documents.
Firecrawl
Firecrawl launches Research Index for life sciences
Firecrawl announced the launch of a Research Index tailored for the life sciences domain.
Firecrawl
Firecrawl releases Convex component for web scraping integration
Firecrawl announced a new Convex component that allows developers to integrate web scraping capabilities directly into their Convex applications.
Firecrawl
Firecrawl announces integration with Eden AI
Firecrawl has published a blog post announcing an integration with Eden AI.
Firecrawl
Firecrawl publishes case study on 11x's use of its web scraping API for prospect research
Firecrawl's blog details how sales automation startup 11x uses its web scraping API to automate prospect research.
Firecrawl
Firecrawl launches Developer Index for web scraping performance metrics
Firecrawl announced the launch of a Developer Index to provide benchmarks and performance data for web scraping tools.
Firecrawl
Browserless publishes enterprise Docker deployment guide for self-hosting
Browserless released a guide detailing how to deploy its browser automation service in an enterprise Docker environment.
Browserless
Browserless introduces Skill Bucket for agent-based browser automation
Browserless announces a new Skill Bucket feature that allows AI agents to access and use predefined browser automation skills.
Browserless
Browserless blog post examines the pitfalls of persisting browser profiles
Browserless published a blog post discussing the challenges and drawbacks of persisting browser profiles in automated environments.
Browserless
Browserless publishes guide on browser infrastructure for computer use agents
Browserless has published a blog post discussing the infrastructure requirements for running browser-based agents that interact with web interfaces on behalf of users.
Browserless
Browserless publishes guide on bypassing Datadome anti-bot protection
Browserless released a blog post detailing techniques to circumvent Datadome's anti-bot system.
Browserless
Browserless introduces the Browser Automation Protocol
Browserless has announced a new protocol for browser automation.
Browserless
Browserless launches Agent for AI-driven browser automation
Browserless has introduced a new product called Browserless Agent, designed to enable AI agents to control browser sessions.
Browserless
Bright Data Blog discusses robot training data from public web video
Bright Data's blog explores the use of publicly available web video as a source of training data for robotic AI systems.
Bright Data Blog
Bright Data integrates with Paperclip for AI-driven data extraction
Bright Data announces a partnership or integration with Paperclip, an AI tool, as detailed in a blog post on their site.
Bright Data Blog
Bright Data Defends Its Network Against Misuse Claims
Bright Data publishes a blog post arguing that its proxy network cannot be used in the ways critics allege.
Bright Data Blog
Bright Data explains Context as a Service for AI data retrieval
Bright Data published a blog post defining Context as a Service, a concept where external context is provided to AI models via structured data feeds.
Bright Data Blog
Bright Data compares its Cursor integration with default coding agent
Bright Data published a blog post comparing its Cursor integration against the default coding agent for web data tasks.
Bright Data Blog
SerpApi resolves Google /goto URL redirect rollout
SerpApi announced it has fixed an issue where Google changed result links to use google.com/goto redirects, restoring direct destination URLs in its API output.
SerpApi
2026-09-04
Playwright 1.63.0 adds test locks for safe concurrent access to shared resources
Playwright 1.63.0 introduces named test locks that prevent concurrent execution of tests sharing the same lock name across files, workers, and projects.
microsoft/playwright releases
Apify outlines six methods for downloading Instagram images in 2026
Apify published a guide covering six techniques for downloading Instagram images, ranging from a simple browser trick to a two-Actor pipeline for full profile extraction.
Apify
SerpApi publishes guide for scraping Zillow listings
SerpApi released a tutorial showing how to scrape Zillow real estate listings using its API with multiple programming languages.
SerpApi
ScrapingBee explores how local deep research agents break on web retrieval
ScrapingBee published a blog post examining the failure points in retrieval stages of open-source deep research systems like Local Deep Research, gpt-researcher, and LangChain agents.
ScrapingBee
ScrapingBee compares eight top Python web scraping tools for 2026
ScrapingBee published a guide comparing eight Python web scraping tools, covering parsers, browser automation, crawling, proxies, and full-stack scraping services.
ScrapingBee
2026-09-03
Puppeteer 25.10.0 adds video-stream screen recording and Firefox 155.0 support
Puppeteer released version 25.10.0 of puppeteer-core, introducing a video-stream-based screen recording feature via page.record() and rolling to Firefox 155.0.
puppeteer/puppeteer releases
Stagehand Python adds WebMCP tool support inside iframes
Stagehand Python's latest dev release enables discovery and invocation of WebMCP tools registered inside iframes by routing calls through the main CDP session and all adopted OOPIF sessions.
browserbase/stagehand releases
Apify streamlines Actor creation with one-click Git repo setup
Apify now lets users select a Git provider during Actor setup, automatically creating a repository, pushing template code, and configuring builds on every push.
Apify
ScrapingBee explains CAPTCHA solvers and when to avoid them
ScrapingBee published an article defining CAPTCHA solvers, how they automate challenge responses, and when developers should skip them in scraping workflows.
ScrapingBee
ScrapingBee explains how CodeWhale gives AI agents live web access
ScrapingBee publishes an article detailing how CodeWhale enables AI agents to access live web data by integrating web scraping tools, MCP servers, or framework tools.
ScrapingBee
ScrapingBee ranks top proxies for Amazon scraping in 2026
ScrapingBee published a guide evaluating residential, ISP, and mobile proxies for Amazon scraping, noting that Amazon's anti-bot updates can render previously effective proxies obsolete.
ScrapingBee
2026-09-02
Stagehand SDK exposes Browserbase Search and Fetch across TypeScript, Python, and Go
Stagehand SDK version 4.1.0a0.dev1494 adds Browserbase Search and Fetch APIs to its TypeScript and Python facades, with equivalent Go support via the existing HTTP transport.
browserbase/stagehand releases
Zyte tests Claude Fable 5.1 and GLM-5.3-Flash in a live extraction benchmark
Zyte published a personal benchmark comparing Claude Fable 5.1 and GLM-5.3-Flash on real extraction tasks, revealing that the GLM model matched a model the author had previously encountered.
Zyte
Zyte analysis finds 75% of top sites use robots.txt, but few name specific crawlers
Zyte published a study showing that three-quarters of the world's top websites publish a robots.txt file, yet most do not name individual crawlers.
Zyte
ScrapingBee publishes guide to AI agent web scraping with wigolo and MCP
ScrapingBee released a guide covering how to equip AI agents with web scraping capabilities using wigolo and the Model Context Protocol.
ScrapingBee
ScrapingBee publishes guide on scraping website text for LLM training
ScrapingBee released a tutorial covering how to extract all text from a website for use in LLM training pipelines.
ScrapingBee
2026-09-01
Jactl brings virtual threads to Java 8 without Project Loom
The Jactl scripting language introduces virtual thread support for Java 8, bypassing the need for Project Loom.
Hacker News
Zyte adds CDP support for browser automation on its infrastructure
Zyte has introduced Chrome DevTools Protocol (CDP) support, allowing users to run browser automation scripts on Zyte's managed infrastructure.
Zyte
Apify launches Apartments.com scraper for rental market analysis
Apify released a tool to scrape Apartments.com listings at scale and integrate them with ChatGPT for interactive market reports.
Apify
Apify publishes comparison guide pitting Firecrawl against its own ecosystem
Apify's blog compares Firecrawl's unified AI-driven scraping with Apify's flexible ecosystem, outlining the strengths of each.
Apify
2026-08-31
SerpApi countersues Reddit over API access restrictions
SerpApi has filed counterclaims against Reddit in response to Reddit's lawsuit, alleging broken promises of an open internet and unfair API pricing.
SerpApi
Apify compares its e-commerce scraping platform with Bright Data's site-specific APIs
Apify published a blog post comparing its full-stack scraping platform to Bright Data's site-specific APIs for e-commerce data extraction.
Apify
Scrapfly reviews nine Scrapy extensions and middlewares for 2026
Scrapfly tested nine Scrapy extensions and middlewares against Scrapy 2.18, covering rendering, TLS fingerprints, proxies, extraction, shared queues, and deployment.
Scrapfly
Scrapfly publishes 2026 diagnostic guide for blocked Scrapy spiders
Scrapfly released a walkthrough covering nine mechanisms that can block a Scrapy spider, from IP reputation to CAPTCHA, with evidence and mitigation steps for each.
Scrapfly
Cloudflare launches Adaptive Intelligence to shift bot economics
Cloudflare introduces Adaptive Intelligence, a new engine that autonomously learns from live traffic meta-signals to counter bot operators' economic advantages.
Cloudflare Bot Management
Scrapingdog launches MCP Server for AI-powered web scraping
Scrapingdog has introduced an MCP Server that integrates its web scraping and data extraction capabilities directly into AI-powered applications.
Scrapingdog
SerpApi publishes roundup of best web scraping tools for 2026
SerpApi has published a guide covering popular web scraping tools, from open-source frameworks like Scrapy and Crawlee to commercial scraping platforms and search APIs.
SerpApi
Google's GOTO link changes dramatically increase scraping difficulty
A study published on Growtika claims that Google has made web scraping 10 times harder in less than a year through changes to its GOTO link structure.
Hacker News
Builder spotlight: Goldmine automated outreach and won on Apify
Apify profiles a developer who built a LinkedIn scraper for his team and later became a top-rated Apify developer, winning an EMEA prize in the Apify $1 Million Challenge.
Apify
Zenrows publishes practical guide on web data for LLM fine-tuning
Zenrows outlines how to source clean web data for LLM fine-tuning, warning that a single bad extraction can measurably degrade a small seed set.
Zenrows
Zenrows blog walks through scraping 2026 FIFA World Cup data across three access patterns
Zenrows published a tutorial demonstrating how to scrape 2026 FIFA World Cup data from a JSON endpoint behind JavaScript, an API with per-session JWT, and server-rendered HTML using Python and its own scraping API.
Zenrows
Crawlbase shows how to build a web scraping pipeline with Zapier using async callbacks
Crawlbase published a guide explaining how to split web retrieval from Zapier's execution window by dispatching async crawls with a callback and catching the finished page in a second Zap.
Crawlbase
2026-08-30
Google introduces /goto redirect URLs in Search, SerpApi works on resolution
Google is rolling out new /goto redirect URLs across Search, and SerpApi is actively resolving affected links as the implementation evolves.
SerpApi
Apify compares Oxylabs and Bright Data for web scraping
Apify published a comparison of Oxylabs and Bright Data, focusing on their proxy services and recent web scraping investments.
Apify
Stagehand Python 4.0.3a0.dev1483 adds stagehand_facade tool surface for eval benchmarking
A dev release of Stagehand Python introduces a facade tool surface that allows evals to benchmark the exact byte-identical interface shipped to Claude Code, Codex, and Pi integrations.
browserbase/stagehand releases
SerpApi publishes guide on scraping Walmart product reviews with its dedicated API
SerpApi released a tutorial showing how to use its Walmart Product Reviews API to extract ratings, review text, feedback counts, and reviewer details in structured JSON and export to CSV.
SerpApi
2026-08-28
SerpApi explains HTTP 429 errors and rate-limit best practices
SerpApi published a guide covering the causes of HTTP 429 errors, how to read Retry-After headers, and strategies for retrying requests without escalating blocking.
SerpApi
Cloudflare launches BotBase for Operators with submission management and behavior modeling
Cloudflare has introduced a dedicated dashboard section where bot operators can submit, track, and edit their bot declarations, including a behavior model for content usage.
Cloudflare Ai Bots
Cloudflare launches BotBase for Operators to streamline bot directory submissions
Cloudflare has introduced a new dashboard interface for bot operators to manage their submissions to the company's bot directory, including status tracking, editing, and a behavior model for declaring content usage.
Cloudflare Bot Management
Zenrows integrates with smolagents to give AI agents production-grade web access
A tutorial shows how to swap smolagents' plain-requests VisitWebpageTool for a Zenrows-backed tool that bypasses bot checks.
Zenrows
Zenrows MCP brings live scraping to Cursor's AI editor
Zenrows released an MCP integration that lets Cursor users scrape JavaScript-rendered and bot-protected sites directly from the editor.
Zenrows
ScrapingBee publishes guide on MCP servers for web scraping, emphasizing control over data
ScrapingBee explains how a Model Context Protocol (MCP) server can improve scraping agent performance by keeping context lean and avoiding page bloat.
ScrapingBee
Crawlbase introduces Web Bot Auth as an identity layer for bots, coinciding with pay-per-crawl pricing
Crawlbase ships Web Bot Auth, a specification that treats bot identity as a verifiable signature rather than a self-declared claim, on the same day it launches pay-per-crawl pricing.
Crawlbase
2026-08-27
Scrapfly publishes 2026 guide to e-commerce scraping tools
Scrapfly's blog post surveys nine e-commerce scraping tools for developers, covering retrieval, extraction, browser automation, and discovery layers.
Scrapfly
Apify Builder Spotlight: Non-developer wins LATAM prize with niche Actor built via prompt and no code
Apify profiles Gui Stetelle, a salesperson who used a prompt, Apify docs, and Gemini to build a niche Actor and win the $2,000 LATAM prize in the Apify $1 Million Challenge.
Apify
DataDome launches hosted MCP server for AI agent access to trend reports
DataDome introduced a hosted MCP server that allows AI agents to query its Trend Reports data using natural language.
DataDome
Apify Builder Spotlight: John Cole wins $20,000 in Apify Challenge without a CS degree
Apify profiles John Cole, a self-taught developer who placed second in the Apify $1 Million Challenge with nearly 100 Actors and 15,000 users.
Apify
Zenrows shows how to build a web-aware agent by combining its fetch tool with OpenAI Agents SDK
A tutorial from Zenrows demonstrates registering the Zenrows Fetch tool as a function tool alongside OpenAI's WebSearchTool to let an agent choose between discovery and full-page retrieval.
Zenrows
Zenrows publishes tutorial on building a web research multi-agent system with AG2
Zenrows released a blog post showing how to integrate its Fetch tool as a typed tool in an AG2 multi-agent group chat to ensure clean content circulates among agents.
Zenrows
2026-08-25
Zyte argues EU AI scraping guidelines rely on outdated robots.txt standard
Zyte publishes a blog post arguing that Europe's new generative AI scraping guidelines, which lean on the robots.txt protocol, will harm users and entrench monopolies.
Zyte
SerpApi moves to dismiss Google's amended complaint in scraping lawsuit
SerpApi has filed a motion to dismiss Google's amended lawsuit after the court previously dismissed the original complaint in July.
SerpApi
Zyte pitches WebFetch as a drop-in replacement for coding agents' built-in fetch tool
Zyte published a blog post arguing that its WebFetch CLI tool outperforms the default webfetch tool in coding agents for research and coding workflows.
Zyte
Apify blog explains how to scrape Clutch.co and interpret its paid rankings
Apify published a guide on collecting Clutch.co data despite its blocking measures and explains how paid placements skew agency rankings.
Apify
Apify sets September 30 deadline for rental-to-pay-per-event Actor migration
Apify is sunsetting its rental pricing model for Actors, requiring developers to migrate to a pay-per-event model by September 30.
Apify
Puppeteer v25.9.0 rolls Firefox 154.0 and Chrome 152.0, adds shadow root support in MutationPoller
Puppeteer released version 25.9.0 of puppeteer-core, updating browser versions and adding support for shadow roots in MutationPoller and tool execution cancellation in webmcp.
Puppeteer
Apify shows how to build an AI research assistant in Slack using its MCP server and n8n
Apify published a tutorial on integrating its MCP server with n8n to create a Slack-based AI research analyst that answers team questions with live web data.
Apify
2026-08-24
Google Details AI-Assisted Rewrites of C/C++ Dependencies to Rust for Memory Safety
Google published a blog post on its bug hunters portal describing how it uses AI to assist in rewriting C and C++ dependencies into Rust at scale.
Hacker News
Scrapfly ranks six open-source YouTube scrapers by job, flags two failures
Scrapfly published a blog post comparing six open-source YouTube scrapers, including a GitHub snapshot from August 11, 2026, and noting two projects that failed in their tests.
Scrapfly
Scrapfly publishes guide to scraping Target.com via Redsky API and bypassing PerimeterX
Scrapfly released a blog post detailing how to extract product and pricing data from Target.com using the internal Redsky API while handling store-keyed prices and PerimeterX anti-bot defenses.
Scrapfly
Zyte report reveals retailers as second most aggressive sector in blocking AI crawlers
Zyte published a blog post detailing how major retail marketplaces deploy anti-bot technology to block AI crawlers and automated data extraction.
Zyte
Crawlbase publishes technical guide on scaling headless browser fleets to 10,000 concurrent sessions
Crawlbase's blog post details the architecture and capacity planning required to run 10,000 concurrent Playwright sessions across roughly 100 nodes.
Crawlbase
2026-08-21
Cloudflare launches Bot Preference Sync to align robots.txt with AI bot policies
Cloudflare introduces Bot Preference Sync, a feature that automatically synchronizes a site's robots.txt file with its AI bot management policies for search, agent, and training bots.
Cloudflare Bot Management
DataDome report finds AI agent requests hit 7.9 billion, scalper bots outnumber shoppers 10 to 1
DataDome recorded 7.9 billion AI agent requests in early 2026 and published a report highlighting widespread spoofing and visibility gaps in bot detection.
DataDome
SerpApi launches Markdown output for all search APIs
SerpApi now offers a Markdown output option for all 100+ of its search APIs, delivering the same data as JSON with roughly half the tokens and no additional configuration.
SerpApi
2026-08-20
Aaron Swartz prosecution contrasted with Meta's scraping impunity
A blog post argues that Aaron Swartz was harshly prosecuted for scraping academic articles while Meta scrapes data at scale with little legal consequence.
Hacker News
Zyte research argues web scraping faces pricing barriers, not outright blocking
Zyte's State of Web Access report, discussed in an interview with the researcher, finds that new economic barriers are making web scraping more difficult rather than technical blocks.
Zyte
Scrapy 2.18.0 ships with httpx2-based download handler and stable HTTP/2 support
Scrapy 2.18.0 replaces the Twisted-based HTTP/2 handler with httpx2, makes brotli and Zstandard compression always available, and changes late crawler attribute access to raise RuntimeError.
Scrapy
2026-08-19
SerpApi surpasses 1.5 million activated user accounts
SerpApi announced it has reached 1.5 million activated user accounts, marking a significant growth milestone for the search engine results API provider.
SerpApi
2026-08-18
SerpApi publishes case study on lead generation company scaling outreach with Google Maps API
SerpApi released a case study detailing how a lead generation company used its Google Maps API to scale lead discovery.
SerpApi
Apify migrates MCP server to new stateless specification
Apify has migrated its MCP server to the July 2026 stateless spec revision, detailing the changes and implications for hosted servers and existing clients.
Apify
2026-08-17
Apify blog shows how to build a Dun & Bradstreet data API without an enterprise contract
Apify published a guide on using its platform to create a custom API for Dun & Bradstreet company data, targeting users who want to bypass expensive enterprise agreements.
Apify
Zyte report finds fashion websites among the most heavily defended against scraping
A Zyte blog post examines the anti-bot and blocking measures used by fashion e-commerce sites, finding they are some of the most aggressively protected on the web.
Zyte
Puppeteer v25.8.0 adds followSymlinks option and Windows detached mode fix
Puppeteer released version 25.8.0 of puppeteer-core with a new followSymlinks option, a validatePath fix for computeSystemExecutablePath, and a change to launch browsers in detached mode on Windows to reduce flakiness.
Puppeteer
2026-08-16
PageSieve browser extension launches for Firefox-based web scraping
A new open-source browser extension called PageSieve enables users to scrape data from websites directly within Firefox.
Hacker News
PyScrappy brings self-healing selectors and an MCP server to web scraping
A new open-source Python library, PyScrappy, offers self-healing CSS/XPath selectors and an MCP server for resilient web scraping.
Hacker News
2026-08-15
Anthropic confirms watermarking concerns in statement
Anthropic issued a statement acknowledging all concerns about watermarking.
Hacker News
ScrapingBee publishes guide on building Python flight scrapers in 2026
ScrapingBee released a blog post outlining four methods for building a Python flight scraper, recommending a managed scraping API to bypass anti-bot measures.
ScrapingBee
ScrapingBee publishes tutorial on building a job aggregator scraping pipeline
ScrapingBee released a blog post walking through the full pipeline for a job aggregator, from sourcing and normalizing to deduplication and freshness management.
ScrapingBee
ScrapingBee explores how 'Agent Skills' can make AI coding agents more reliable
ScrapingBee published a blog post examining the Agent Skills repository by addyosmani, which aims to improve the reliability of AI coding agents by teaching them structured workflows.
ScrapingBee
2026-08-14
Scrapfly publishes tutorial on scraping Skyscanner flight prices with Python
Scrapfly released a blog post showing how to extract flight data from Skyscanner by constructing deep-link URLs and capturing itinerary JSON from the rendered page.
Scrapfly
Scrapfly publishes guide on scraping Airbnb listings and prices
Scrapfly released a blog post detailing how to scrape Airbnb search results, listing details, prices, reviews, and availability using Python and its own scraping platform.
Scrapfly
Scrapfly ranks five open-source LinkedIn scrapers on GitHub by auth model and ban risk
Scrapfly published a blog post evaluating five open-source LinkedIn scraping repositories on GitHub, ranking them by authentication approach, maintenance status, and real-world blocking risk as of August 2026.
Scrapfly
DataDome report finds European publishers face disproportionate AI bot scraping
DataDome recorded 7.9 billion AI agent requests in early 2026 and reports widespread spoofing and visibility gaps leaving organizations exposed.
DataDome
LuaCAD brings parametric CAD scripting to Lua with operator overloading
A new tool called LuaCAD lets users model solids in Lua instead of the OpenSCAD language, offering a CLI and desktop app with preview and editor.
Hacker News
Zyte blog post examines how AI and web scraping turn scattered personal data into security risks
Domagoj Marić explores the intersection of AI, web scraping, and OSINT to show how fragmented personal data is assembled into profiles, scams, and security threats at Extract Summit.
Zyte
ScrapingBee launches Agentic Employee Search API for natural-language profile discovery
ScrapingBee has released an Agentic Employee Search API that accepts natural-language queries to find relevant employee profiles, eliminating the need for manual filtering or complex search parameters.
ScrapingBee
2026-08-13
Scrapfly publishes guide on scraping Lowe's product data and bypassing Akamai
Scrapfly released a blog post detailing how to scrape Lowe's product, price, search, and store location data using embedded page state and their maintained Python scraper.
Scrapfly
Scrapfly Publishes Guide to Scraping DigiKey Data Past Cloudflare
Scrapfly's blog post details how to scrape DigiKey pricing, stock, and parametric specs while navigating its Cloudflare challenge, and compares this approach to using the official API v4.
Scrapfly
Zyte Audit Reveals Industry-Specific Bot Access Policies
Zyte published a large-scale audit of web access controls showing how different industries enforce different policies toward bots.
Zyte
ScrapingBee publishes tutorial on scraping Algolia via its hidden API
ScrapingBee released a guide explaining how to scrape Algolia search results by targeting the search endpoint directly instead of parsing HTML.
ScrapingBee
Crawlbase details the infrastructure behind 8,000 CAPTCHAs per second
Crawlbase published a blog post explaining the throughput math, Go-based control plane, and scaling challenges required to solve 8,000 CAPTCHAs per second.
Crawlbase
2026-08-11
Scrapfly ranks six open-source Instagram scrapers with notes on auth and ban risk
Scrapfly published a comparison of six open-source Instagram scrapers for 2026, covering auth models, ban risk, and maintenance status.
Scrapfly
Zyte blog profiles case study of $70 AI-coded app replacing $5,000 platform
Zyte published a blog post detailing how developer Fran Muñoz used AI coding and specification-driven development to build a production app that replaced a costly platform.
Zyte
Scrapfly compares five MCP servers for web scraping and browser automation
Scrapfly published a blog post comparing five MCP servers by their capabilities in protected-site scraping, browser control, debugging, static fetching, and cross-browser automation.
Scrapfly
Scrapfly compares six modern command-line tools as alternatives to cURL and Wget
A blog post from Scrapfly evaluates HTTPie, aria2, and other tools that address specific limitations of cURL and Wget, including a managed fetch tool for blocked requests.
Scrapfly
Scrapfly compares 8 Python HTTP clients for web scraping in 2026
Scrapfly published a blog post evaluating eight Python HTTP clients on async support, HTTP/2, HTTP/3, TLS impersonation, and maintenance, with runnable examples.
Scrapfly
Scrapfly ranks 7 lead scraping tools for 2026
Scrapfly published a ranked list of seven lead scraping tools covering no-code extensions and production APIs, with honest assessments of each tool's limitations.
Scrapfly
ScrapingBee publishes tutorial on building an AutoGPT agent with its API
ScrapingBee released a blog post showing how to use its web scraping API as a data source within AutoGPT's visual Agent Builder.
ScrapingBee
Study finds only 34.5% of free proxies work, thousands tamper with traffic
Crawlbase reports on two peer-reviewed studies that tested 640,600 free proxies, finding that just over a third were functional and many altered traffic.
Crawlbase
2026-08-10
DataDome CEO and Forrester Discuss Bot and Agent Trust Management Landscape
DataDome's CEO and Forrester analysts discuss the findings of The Forrester Wave: Bot and Agent Trust Management Software, Q2 2026.
DataDome
DataDome report finds 7.9B AI agent requests and widespread spoofing in early 2026
DataDome published a report detailing 7.9 billion AI agent requests recorded in early 2026, highlighting spoofing and visibility gaps that leave retail systems exposed.
DataDome
DataDome records 7.9B AI agent requests, warns of widespread spoofing and visibility gaps
DataDome published a report documenting 7.9 billion AI agent requests in early 2026, highlighting spoofing and detection blind spots.
DataDome
Kameleo blog post demonstrates watching browser fingerprinting calls via Chromium's Blink layer
A blog post on Kameleo's site describes a technique for observing browser fingerprinting calls by instrumenting Chromium's Blink rendering layer.
Hacker News
2026-08-07
Scrapfly publishes diagnostic guide for browser fingerprint testing tools
Scrapfly released a layer-by-layer guide covering fingerprint and bot detection tools, explaining what detectable results mean and how to fix each leak.
Scrapfly
Cloudflare shifts bot mitigation from risk scoring to continuous trust evaluation
Cloudflare announced a new approach to bot mitigation that moves from point-in-time risk assessment to continuous trust evaluation, introducing systems BotBase and Precursor to assess good and bad behaviors from bots and agents.
Cloudflare Bot Management
Vacation rental intelligence platform scales to 1 billion monthly crawl requests with Crawlbase Enterprise Crawler
Crawlbase published a case study detailing how a vacation rental intelligence platform processed 5.52 billion requests in six months with 99.96% success using its Enterprise Crawler.
Crawlbase
2026-08-06
Chrome's new navigator.cpuPerformance API opens a fresh fingerprinting vector
Zyte reports that Chrome 152 will expose a navigator.cpuPerformance property, giving sites a new way to fingerprint browsers.
Zyte
DataDome publishes guide on blocking AI bots and scrapers
DataDome released a guide covering methods to block AI bots, from robots.txt to advanced bot management.
DataDome
ScrapingBee publishes guide on scraping Uber Eats data with legal and technical caveats
ScrapingBee released a tutorial covering how to extract public menu items and prices from Uber Eats, noting the platform's JavaScript rendering and terms of service restrictions.
ScrapingBee
ScrapingBee ranks top sneaker proxies for 2026 drops
ScrapingBee published a guide ranking ISP, residential, and datacenter proxies for sneaker bots, focusing on checkout success during limited drops.
ScrapingBee
2026-08-05
Scrapfly publishes tutorial on scraping Google Jobs with Python
Scrapfly released a blog post showing how to scrape Google Jobs listings using Python and its own scraping platform.
Scrapfly
DataDome updates Account Protect with clearer visibility and stronger signals
DataDome has released updates to its Account Protect product, adding deeper visibility, broader detection across password-reset flows, and API control over protection status.
DataDome
Zyte publishes largest ever audit of web access control mechanisms
Zyte released a comprehensive audit of how websites regulate programmatic visits, revealing the current state of web access barriers.
Zyte
Zyte publishes tutorial on building custom fetch tools for AI agents with Claude Agent SDK
Zyte released a tutorial showing how to create a custom fetch tool using the Claude Agent SDK to help AI agents extract structured data from the web.
Zyte
2026-08-04
Cloudflare Wallets launch to give AI agents programmable payments and identity
Cloudflare announced Cloudflare Wallets, a programmable wallet that enables AI agents to make autonomous payments and verify identity on the web using the x402 protocol.
Cloudflare Ai Bots
Zyte releases scrapy-spidey-sense, a preflight CLI for Scrapy projects
Zyte has open-sourced a command-line tool that performs static analysis on Scrapy projects before a crawl begins, scoring production-readiness and linking findings to fixes.
Zyte
ScrapingBee publishes tutorial on using Playwright in Ruby
ScrapingBee released a guide covering installation and usage of the community-maintained playwright-ruby-client gem for web scraping and testing in Ruby.
ScrapingBee
2026-08-03
Scrapfly publishes guide on scraping Google Play app reviews and metadata with Python
Scrapfly released a tutorial showing how to extract full Google Play app reviews, ratings, and metadata using Python, bypassing the typical few-hundred-review limit of free libraries.
Scrapfly
Scrapfly ranks four open-source proxy scrapers still viable in 2026
A blog post from Scrapfly filters the crowded open-source proxy tool landscape down to four actively maintained scrapers and checkers worth using this year.
Scrapfly
Scrapfly ranks 7 AI browser agents for production scraping in 2026
Scrapfly published a ranked guide to the best AI browser agents for automation and scraping, evaluating them on production stability and anti-blocking capability rather than demo performance.
Scrapfly
Scrapfly publishes guide on scraping Marriott hotel data through Akamai defenses
Scrapfly released a tutorial covering how to extract Marriott hotel prices and availability using Python, including bypassing Akamai bot protection.
Scrapfly
Zyte blog post explores rendering JavaScript pages with Playwright and Scrapy
Zyte published a guide on using Playwright to render dynamic content within a Scrapy workflow.
Zyte
2026-08-02
Draco: A self-hostable, single-binary Firecrawl alternative built in Rust
A developer released Draco, an open-source, single-binary web scraping tool in Rust designed as a self-hostable alternative to paid APIs like Firecrawl and Browserbase.
Hacker News
2026-08-01
Amazonbot ignores robots.txt and falls for honeypot trap, user reports
A Hacker News user reports that Amazonbot aggressively scraped their personal website despite robots.txt directives and was caught accessing a hidden honeypot with fake HTTP endpoints.
Hacker News
DataDome publishes guide on AI agent access control, arguing traditional IAM falls short
DataDome released a blog post explaining how AI agents require dedicated identities, scoped permissions, and real-time oversight beyond traditional identity and access management.
DataDome
2026-07-30
DataDome report finds 7.9B AI agent requests and widespread spoofing in early 2026
DataDome published a report documenting 7.9 billion AI agent requests in early 2026, highlighting spoofing and visibility gaps that leave organizations exposed.
DataDome
ScrapingBee adds geolocation support to classic proxies, covering 40+ countries
ScrapingBee announced that its classic proxy offering now allows users to select a target country from over 40 options via the request builder or API parameter.
ScrapingBee
2026-07-29
ScrapingBee covers Ruflo, a multi-agent orchestration layer for Claude Code and Codex
A blog post on ScrapingBee introduces Ruflo, an open-source meta-harness that coordinates swarms of AI agents by wrapping around Claude Code or Codex.
ScrapingBee
ScrapingBee publishes guide on scraping TCGplayer, highlighting ToS restrictions and JavaScript challenges
ScrapingBee's blog post explains how to scrape TCGplayer while noting the site's Terms of Service discourage scraping and that JavaScript rendering complicates simple requests.
ScrapingBee
ScrapingBee publishes blog post on /last30days AI agent skill for multi-platform research
ScrapingBee's blog introduces /last30days, an AI agent skill designed to research topics across multiple internet platforms while navigating access restrictions.
ScrapingBee
Crawlbase argues AI agent failures are infrastructure failures, not code problems
Crawlbase publishes a blog post claiming that most AI agent failures stem from infrastructure issues like Markdown normalization, retrieval circuit breakers, and storage-backed memory.
Crawlbase
2026-07-27
Web scraper declares victory after court win: 'Google and Reddit do not own the Internet'
A web scraper has won a court case against Google and Reddit, asserting that the platforms do not have exclusive ownership over publicly accessible web data.
Hacker News
Scrapfly publishes guide to scraping Kayak flight data with its SDK
Scrapfly released a blog post walking through the process of scraping Kayak flight search results using its own SDK, covering JavaScript rendering and parsing internal poll JSON.
Scrapfly
Zyte launches 'Modern Scrapy for experienced developers' tutorial series
Zyte published the first part of a new blog series aimed at experienced developers building production-ready Scrapy projects.
Zyte
ScrapingBee publishes guide on using CloudProxy for web scraping
ScrapingBee's blog post covers setting up CloudProxy, its limits, and its managed API for lightweight scraping tasks.
ScrapingBee
2026-07-26
ScrapingBee publishes guide on bypassing CAPTCHA with Selenium in Ruby
ScrapingBee released a tutorial explaining how to avoid triggering CAPTCHAs when using Selenium in Ruby by reducing automation signals and using better IPs.
ScrapingBee
2026-07-24
Scrapfly publishes guide on scraping RS-Online for product data
Scrapfly released a tutorial on extracting pricing, stock, specifications, and datasheet links from RS-Online's North American listings and product pages.
Scrapfly
Scrapfly compares Browser Use and Playwright for web scraping
Scrapfly published a blog post comparing Browser Use and Playwright, covering architectural differences, speed and cost tradeoffs, silent failure risks, and a hybrid approach for production scraping.
Scrapfly
Scrapfly publishes guide on bypassing AWS WAF Bot Control for web scraping
Scrapfly released a blog post detailing how AWS WAF Bot Control detects scrapers across five layers and how to bypass it using their Scrapfly ASP product.
Scrapfly
Scrapfly rounds up top open-source Facebook Marketplace scrapers on GitHub for 2026
Scrapfly published a blog post listing the five best open-source Facebook Marketplace scrapers on GitHub as of 2026, along with repos to avoid.
Scrapfly