Website Content Crawler

Turn any list of websites into clean, structured content. For each page the Actor extracts the title, meta description, headings, main body text (boilerplate stripped), word count, language and link/image counts — over plain HTTP, no browser and no login. Optionally crawl internal links to pull a whole section.

Ideal for LLM/RAG ingestion, content audits, competitive research, and building text datasets from many sites at once.

What you get per page

Input

{
  "urls": ["https://apify.com/blog", "example.com"],
  "crawl": true,
  "maxPagesPerDomain": 20,
  "maxDepth": 2
}

• urls — pages/sites to extract, one per line. (Or use startUrls.)
• crawl — follow same-domain links (BFS). Off = just the given URLs.
• maxPagesPerDomain / maxDepth — crawl limits.
• sameDomainOnly — keep the crawl on the start domain (default on).
• includeLinks — include the internal/external link arrays (counts always included).
• textMaxChars — truncate stored body text (word count uses the full text).
• maxConcurrency / maxItems — parallelism and a global page cap.
• proxyConfiguration — optional; enable for reliable access at scale.

Output (one row per page)

{
  "input": "https://apify.com/blog",
  "url": "https://apify.com/blog",
  "domain": "apify.com",
  "depth": 0,
  "title": "Apify Blog",
  "description": "News and tutorials…",
  "lang": "en",
  "headings": [{ "level": 1, "text": "Apify Blog" }],
  "text": "…clean readable content…",
  "wordCount": 812,
  "internalLinkCount": 47,
  "externalLinkCount": 6,
  "imageCount": 12,
  "error": null
}

Output schema

FAQ

Do I need an API key or login? No. The Actor works over plain HTTP with no browser and no login — just paste your list of URLs or domains and run.

How many pages can I get per run? Control it with maxPagesPerDomain, maxDepth and a global maxItems cap. With crawl on, it follows same-domain links breadth-first, so one run can extract from a whole site section or many sites at once.

Is crawling website content legal? The Actor collects only publicly available page content — the same pages any visitor can open. Public data is generally collectible, but you are responsible for respecting each site's terms and applicable law.

Does it collect emails or contact info? No. It extracts page content — title, description, headings, body text, word count and link/image counts — not contact details. For emails, phones and socials, use the Website Contact Scraper.

What's the output format? One structured JSON row per page with clean, boilerplate-stripped text and metadata — ready for LLM/RAG ingestion, content audits or text datasets. Export as JSON, CSV or Excel.

Can I connect this to other apps and services? The Website Content Crawler can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. It works with Make, Zapier, Slack, Airbyte, GitHub, Google Drive and many more, plus the Apify API, JavaScript/Python clients and MCP. Or use webhooks to trigger an action whenever a run finishes.

Pair content extraction with these web-tool companions:

• Website Contact Scraper — extract emails, phones and social profiles from the same sites.
• Google Maps Scraper — get local business listings by keyword and location.
• Google Maps Email Extractor — turn Maps results into contactable leads with emails and socials.

Notes

Every site is hard-capped in time so one slow page can never stall a bulk run. Only publicly available page content is collected; respect each site's terms.

You pay only for what the run returns. Set a maximum cost per run and it stops cleanly at that limit.