Remember How Painful Web Scraping Used to Be?

Ever been there?

You want to build your own search engine, only to discover that writing a crawler is way harder than writing regular code. Websites have anti-scraping measures, so you have to fake User-Agent strings, bypass captchas, handle JavaScript rendering. One wrong move and your IP gets banned. Worse yet, some sites load content dynamically, so plain HTTP requests won’t even get you the data.

I once spent an entire weekend trying to build a knowledge base for my AI assistant. The result? A pile of code with more bugs than features.

Just when I was about to give up, Cloudflare tossed me a new toy: the /crawl endpoint.

What Can Cloudflare’s Crawl API Do?

Simply put, it lets you crawl an entire website with a single API call, using Cloudflare’s browsers.

Think of it this way:

  • You call a food delivery app and say “get me some Kung Pao chicken”
  • The app doesn’t make you go to the market for chicken and peanuts
  • The kitchen cooks it and delivers it straight to your door

That’s exactly what /crawl endpoint does. You tell it “crawl this website,” and it sends a headless browser to visit, render, and package the content for you.

Supported Output Formats

Here’s the nice part. It supports three formats:

  • HTML - Raw page content
  • Markdown - Clean text, ready to copy-paste
  • JSON - Structured data with titles, links, content blocks, easy to integrate

The Markdown format is especially handy for RAG (Retrieval-Augmented Generation) systems. No need to write your own parser. The content comes out clean.

Configurable Scope Control

You can tell it:

  • How deep to crawl - How many levels from the homepage
  • Max pages - Prevent runaway crawling
  • URL filtering rules - Like only crawling a specific subdirectory
  • Auto-discover links - It finds URLs from sitemaps and page links

It’s like ordering food with options like “no spicy” or “less salt.” Fully customizable.

Async Processing: Get on With Your Life

This feature is asynchronous.

You submit a request, get back a job ID, and go do whatever else you need to do. Come back in a few minutes for the results.

Makes total sense. You wouldn’t stare at your phone waiting for delivery, right?

It also has incremental crawling:

  • modifiedSince - Only crawl modified pages
  • maxAge - Skip content still within cache period

This means you don’t have to crawl from scratch every time. Saves time, saves money on your Cloudflare bill.

A Fuel-Saving Mode

If you’re crawling static sites (no JavaScript rendering), you can enable “static mode.” It skips browser rendering and grabs HTML directly. Twice as fast.

Like choosing between grabbing a bento box (static mode) or having the chef cook fresh (full rendering). Different situations call for different approaches.

Does It Play Nice?

This is probably what everyone’s wondering: Is this going to be an out-of-control crawler?

The answer: Yes, it plays very nice.

It respects robots.txt rules, including crawl-delay directives. If a site says “one visit per hour max,” that’s exactly what it does.

The feature works on both Workers free and paid plans. Pretty generous.

Who Should Use It?

Official use cases:

  1. Training AI models - Need lots of web data
  2. RAG pipeline development - Building knowledge bases
  3. Content monitoring - Watching competitor sites for updates

I’d add a few more:

  • Personal blog backups
  • Building a small site search
  • Helping a kid with their “my first search engine” homework

How to Get Started?

The official docs are at Cloudflare Developers , and it’s already in open beta.

All you need:

  1. A Cloudflare account
  2. Enable Browser Rendering
  3. Call the API, and you’re good to go

Summary

Web scraping used to feel like a technical craft. You had to understand HTTP, parse HTML, handle anti-scraping measures. Now Cloudflare says: Stop struggling, let me handle it.

One API call and they crawl, render, and format everything for you. It’s like switching from cooking yourself to ordering takeout. Sure, the latter costs a bit more, but the time you save can be spent on so many other things.

As for whether it can completely replace building your own crawler, that depends on your use case. For simple content gathering, this is absolutely enough. For deep customization, you might still need to roll your own.

At least you’ll lose less hair over it.

FAQ

Q: Is Cloudflare’s crawl API free? A: Both free and paid plans have access, but there are usage limits. The exact quotas aren’t clearly documented, so check the docs before you start.

Q: Can it crawl paywalled sites? A: Technically, whatever it can access it can crawl. But please don’t do that. Respecting content creators’ work is basic decency.

Q: Can I use crawled data commercially? A: That depends on what site you’re crawling. Copyrighted content for commercial use is definitely a no-go. Your own sites or open-source projects are generally fine.

Update: After writing this, I discovered it can also handle batch processing tasks. Looks like the days of working overtime to write crawlers might actually be over.