AI Web Scraping in Python: When an LLM Earns Its Cost

Most "AI web scraper" tutorials answer the wrong question. They show you how to point a language model at a page and get JSON back, which works, and then they stop — right before the part that decides whether your scraper is viable at scale.

The real question is not can a model read this page. It is should it, on every request, forever.

This guide covers both extraction paths in Crawl4AI (Apache-2.0, Python 3.10+), the decision rule between them, and the hybrid that most production scrapers should actually run.

The cost model nobody puts in the tutorial

A CSS selector is compiled once and applied for free. A language model is billed per page, every page, every run.

That difference is invisible at ten pages and decisive at a hundred thousand. A crawl of 100,000 product pages at roughly 2,000 tokens each is 200 million input tokens. The same crawl with a selector schema costs nothing beyond bandwidth, and finishes considerably faster because there is no inference in the loop.

The last row matters more than teams expect. A broken selector fails loudly and points at itself. A model that silently starts returning a subtly wrong field produces plausible garbage that flows into your database unnoticed.

The rule: reach for an LLM when page structure varies across your URL set, when you need judgement a selector cannot express, or when it is a one-off job and your own time costs more than the tokens. Otherwise use selectors.

Setup

pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor

crawl4ai-setup installs the Playwright browser binaries the crawler drives; crawl4ai-doctor verifies the install before you debug a scraper that was never going to run. Skipping the setup step is the most common cause of a first crawl failing on a machine that looks correctly configured.

Baseline: no extraction strategy at all

Before either strategy, note that the crawler already produces clean markdown. For feeding a RAG pipeline or an agent, this is often the whole job:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        if result.success:
            print(result.markdown[:300])

asyncio.run(main())

If you only need readable text, stop here. You have spent nothing and called no model. A surprising number of "AI scraping" requirements are satisfied by this alone.

Path 1: deterministic extraction with CSS selectors

JsonCssExtractionStrategy takes a schema describing where each field lives. No model is involved, so the crawl is free and repeatable:

import asyncio, json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

schema = {
    "name": "Articles",
    "baseSelector": "article.post",
    "fields": [
        {"name": "title", "selector": "h2", "type": "text"},
        {"name": "url", "selector": "a", "type": "attribute", "attribute": "href"},
        {"name": "summary", "selector": "p.excerpt", "type": "text", "default": ""},
        {"name": "tags", "selector": "span.tag", "type": "list"},
    ],
}

async def main():
    config = CrawlerRunConfig(
        extraction_strategy=JsonCssExtractionStrategy(schema),
        cache_mode=CacheMode.BYPASS,
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com/blog", config=config)
        if result.success:
            for row in json.loads(result.extracted_content):
                print(row)

asyncio.run(main())

baseSelector picks the repeating container; each field selector is resolved relative to it. Field type accepts text, attribute, html, regex, list, nested, and nested_list — the last two for sub-objects and repeated complex objects. Setting default on optional fields prevents one missing element from emptying an otherwise good row.

Path 2: LLM extraction for structure you cannot predict

When layouts differ across sources, describe what you want instead of where it lives. Define the shape with Pydantic and let the schema drive the model:

import asyncio, json, os
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode, LLMConfig
from crawl4ai import LLMExtractionStrategy

class Product(BaseModel):
    name: str = Field(description="Product name")
    price: str = Field(description="Price including currency symbol")
    in_stock: bool = Field(description="Whether the item is purchasable now")

strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(
        provider="openai/gpt-4o-mini",
        api_token=os.getenv("OPENAI_API_KEY"),
    ),
    schema=Product.model_json_schema(),
    extraction_type="schema",
    instruction="Extract every product on the page. Omit anything you cannot find rather than guessing.",
    input_format="fit_markdown",
    apply_chunking=True,
    chunk_token_threshold=1000,
    overlap_rate=0.1,
    extra_args={"temperature": 0.0, "max_tokens": 800},
)

async def main():
    config = CrawlerRunConfig(extraction_strategy=strategy, cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com/shop", config=config)
        if result.success:
            print(json.loads(result.extracted_content))
            strategy.show_usage()

asyncio.run(main())

Four settings here are doing real work:

• input_format="fit_markdown" — the single highest-leverage change. Raw HTML is mostly markup a model does not need and you would be paying for it by the token. Filtered markdown carries the same information at a fraction of the size.
• temperature: 0.0 — extraction is not a creative task. Nothing about a product name benefits from sampling diversity.
• chunk_token_threshold with overlap_rate — long pages are split; the overlap keeps a record that straddles a boundary from being cut in half.
• show_usage() — prints actual token counts. Measure the cost before deciding it is acceptable.

The instruction explicitly permits omission. Without that, a model asked for a missing field will frequently invent a plausible one, and an invented price is worse than no price.

The hybrid: pay a model once, run free forever

This is the pattern most production scrapers should use, and it is the one the tutorials tend to skip.

generate_schema() uses a model to write the selectors, once. You cache the result and every subsequent crawl runs on the free deterministic path:

import json, os
from pathlib import Path
from crawl4ai import JsonCssExtractionStrategy, LLMConfig

CACHE = Path("schema.json")

def get_schema(sample_html: str) -> dict:
    if CACHE.exists():
        return json.loads(CACHE.read_text())

    schema = JsonCssExtractionStrategy.generate_schema(
        html=sample_html,
        schema_type="css",
        llm_config=LLMConfig(
            provider="openai/gpt-4o-mini",
            api_token=os.getenv("OPENAI_API_KEY"),
        ),
        validate=True,
    )
    CACHE.write_text(json.dumps(schema, indent=2))
    return schema

One model call for the whole project rather than one per page. The validate=True flag checks the generated schema against the HTML you supplied, so you learn immediately if the selectors do not actually match.

The failure mode to plan for is the site changing its markup, at which point the cached schema silently returns empty rows. Guard it: if a crawl returns zero records where it previously returned hundreds, delete the cache and regenerate rather than alerting a human at 3am.

Crawling many URLs

For bulk work, arun_many() handles concurrency through a memory-adaptive dispatcher that scales parallelism to available resources, rather than you guessing a worker count:

results = await crawler.arun_many(urls, config=config)
for r in results:
    if r.success:
        print(r.url, len(r.markdown))

Pair this with the cached-schema path and the marginal cost of each additional URL is bandwidth and CPU only.

What to check before shipping

Always test result.success before touching result.extracted_content. A page that returns without your content does not raise — it succeeds quietly and hands back nothing useful, which in a scheduled job means silent data loss rather than an error you would notice.

For JavaScript-rendered pages, CrawlerRunConfig accepts js_code for post-load snippets and page_timeout in milliseconds. An empty result on a modern site is far more often a rendering-timing problem than an extraction-strategy problem, so verify the page actually loaded before rewriting your schema.

And on the ethics, which are also the practicalities: honour robots.txt, rate-limit, and identify your crawler. A scraper that gets your IP range blocked has a much more expensive failure mode than a slow one.

---

Code in this guide follows the Crawl4AI documentation (v0.9.x, Apache-2.0). Verify parameter names against the current release before deploying — this library moves quickly.