Substack Scraper

hipersoft/substack-scraper

Scrape posts from any Substack publication in bulk: title, subtitle, author, URL, publish date, likes, comment count, excerpt, full body text and paid/free status. Optionally pull top comments per post. Clean structured JSON, CSV or Excel.

Run this scraper →See the API call

Total users

Monthly users

Total runs

Per post scraped

$0.001

Last rebuilt

Oct 26, 2026

Overview

Scrape posts from any Substack publication as clean, structured JSON, CSV or Excel: title, subtitle, author, URL, publish date, likes, comment count, excerpt, full body text and whether the post is paid or free. Point it at one publication or dozens, cap how many posts you want per publication, and optionally pull the top comments on each post — all in one run, with no account to manage.

What does Substack Scraper do?

This Substack scraper turns any publication's archive into a structured dataset. Give it a handle (like bigtechnology), a Substack URL, or a publication's custom domain, and it walks the archive newest-first and returns one flat record per post — with the headline metadata, the full article text, and the paid/free flag. Turn on comments to also capture the top community replies under each post. Use it to build newsletter datasets, monitor writers and topics, feed research pipelines, or archive your own publication.

What data can you scrape from Substack?

Data point	Description
`publication`	The publication handle or domain the post belongs to.
`postId`	Numeric Substack post id.
`slug`	URL slug of the post.
`title`	Post title.
`subtitle`	Post subtitle / deck.
`author`	Author name(s) / byline.
`url`	Canonical URL of the post.
`publishedAt`	Publish date/time (ISO 8601).
`type`	Post type (newsletter, podcast, thread, etc.).
`likes`	Number of reactions/likes.
`commentCount`	Number of comments on the post.
`wordCount`	Approximate word count of the article.
`excerpt`	Short preview/teaser text.
`bodyText`	Full article body as plain text.
`isPaid`	Whether the post is subscriber-only.
`coverImage`	Cover image URL, if any.
`comments`	Top comments (author, body, date, likes) when enabled.

Use cases

Newsletter datasets — collect a publication's back catalogue for analysis or search.
Writer & topic monitoring — track new posts from the publications you care about on a schedule.
Research pipelines — feed structured post text into BI tools, notebooks, models or RAG apps.
Competitive intelligence — see cadence, engagement (likes/comments) and paid vs free mix across publications.
Personal archive — export your own Substack posts and comments to JSON, CSV or Excel.

How to scrape Substack data

Add Substack Scraper to your Apify account and open its input.
Enter one or more publications — a handle (bigtechnology), a Substack URL (https://bigtechnology.substack.com) or a custom domain (https://www.astralcodexten.com).
Set maxPosts per publication (or 0 for the entire archive).
Optionally enable includeComments and set maxComments.
Click Run, then export the results as JSON, CSV or Excel — or fetch them through the Apify API.

Input

{
  "publications": ["bigtechnology", "https://www.astralcodexten.com"],
  "maxPosts": 50,
  "includeComments": false,
  "maxComments": 20
}

Field	Description
`publications`	Publications to scrape — handles, Substack URLs or custom domains.
`maxPosts`	Max posts per publication, newest first (`0` = entire archive).
`includeComments`	Also fetch the top comments for each post.
`maxComments`	Max top-level comments to keep per post when comments are on.

What you get

One flat JSON record per post, ready to export to CSV, Excel, JSON or XML:

{
  "publication": "bigtechnology",
  "postId": 209279848,
  "slug": "when-artificial-intelligence-is-too",
  "title": "When Artificial Intelligence Is Too Valuable To Sell",
  "subtitle": "Just because the AI labs have sold models on a meter until now doesn't mean they'll always do so.",
  "author": "Alex Kantrowitz",
  "url": "https://www.bigtechnology.com/p/when-artificial-intelligence-is-too",
  "publishedAt": "2026-07-31T19:30:17.712Z",
  "type": "newsletter",
  "likes": 83,
  "commentCount": 0,
  "wordCount": 1139,
  "excerpt": "It's time to delete the assumption that the frontier AI labs will always license their best models…",
  "bodyText": "…full article text…",
  "isPaid": true,
  "coverImage": "https://substackcdn.com/image/…",
  "comments": []
}

Building a broader newsletter, news or research dataset? Pair this with:

Google News Scraper — headlines by keyword and topic.
Hacker News Scraper — stories, comments and users from Hacker News.
Reddit Scraper — posts, comments and users, no login.
Website Content Crawler — crawl any site into clean text for LLM/RAG.

FAQ

Do I need a Substack account or login? No. Just add the publications you want and run — no account, login or setup required.

How many posts can I scrape per run? There's no fixed cap. maxPosts limits how many posts are collected per publication (0 = the whole archive), and maxComments limits comments per post. Larger runs simply take longer.

Can I scrape comments? Yes. Turn on includeComments and the Actor returns the top publicly visible comments — author, body, date and likes — nested under each post.

Does it work with custom domains? Yes. Many publications use their own domain (e.g. www.astralcodexten.com). Enter the handle or the custom domain and the Actor resolves it automatically.

Is scraping Substack legal? The Actor collects only publicly available data. You are responsible for using it in line with Substack's terms and applicable laws — avoid collecting personal data you don't have a lawful basis to process.

What's the output format? A structured JSON dataset with one item per post (comments nest inside their post). Export as JSON, CSV or Excel from the Apify Console or API.

Can I use this with n8n, Make or Zapier? Yes. Run it from n8n via the Apify node, or connect it to Make, Zapier, Slack, Google Drive and many more, plus the Apify API, JavaScript/Python clients, MCP and webhooks.

Notes

Original clean-room implementation. Returns only public data; you are responsible for compliance with Substack's terms. Not affiliated with Substack.

Input

Field	Type	Description
`publications` required Substack publications	array	One or more Substack publications to scrape. Accepts a bare handle (e.g. "bigtechnology"), a Substack URL (e.g. "https://bigtechnology.substack.com") or a publication's custom domain (e.g. "https://www.astralcodexten.com"). Custom domains are followed automati
`maxPosts` Max posts per publication	integer	Maximum number of posts to scrape from each publication, newest first. Use 0 for no limit (scrape the entire archive). Default: `50`
`includeComments` Include comments	boolean	Also fetch the top comments for each post (author, body, date and likes). Adds one extra request per post and only returns publicly visible comments. Default: `false`
`maxComments` Max comments per post	integer	When "Include comments" is on, the maximum number of top-level comments to keep per post (sorted best first). Default: `20`

API

curl -X POST "https://api.apify.com/v2/acts/hipersoft~substack-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "publications": [
    "bigtechnology"
  ]
}'

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("hipersoft/substack-scraper").call(run_input={
    "publications": [
        "bigtechnology"
    ]
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('hipersoft/substack-scraper').call({
  "publications": [
    "bigtechnology"
  ]
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Pricing

Event	Price	Paid plans from
Post Scraped Charged per post scraped.	$0.001	$0.0005

You pay only for what the run returns. Set a maximum cost per run and it stops cleanly at that limit.

Related scrapers

Europe PMC Scraper

Bulk-scrape life-science and biomedical literature from Europe PMC (40M+ records: PubMed, PMC, preprints, patents): title, abstract, authors, journal, year, DOI, PMID/PMCID, citation count, MeSH terms, open-access flag and full-text/PDF links. Thousands per run. No setup.

325 runs · 5 users · $0.001 / article scraped

arXiv Papers Scraper

Search and extract arXiv papers: title, abstract, authors, categories, published/updated dates, DOI, journal reference, and direct PDF links. Filter by category and sort by newest, recently updated or relevance. Fetch by arXiv ID too. For research, lit reviews and AI/ML monitoring. No setup.

140 runs · 2 users · $0.0025 / paper scraped

Letterboxd Scraper

Scrape Letterboxd films with ratings, rating counts, genres, director, cast, runtime and synopsis. Feed it film pages, a user's films/watchlist/diary, any list URL, or browse pages. For film data, recommendations and research.

139 runs · 7 users · $0.0042 / film scraped

OpenLibrary Book Scraper

Search millions of books on OpenLibrary by keyword, author, subject or ISBN and get title, authors, publisher, year, subjects, ISBNs, edition count, ratings, page count and cover. Clean, no setup.

124 runs · 5 users · $0.002 / book scraped

Semantic Scholar Scraper

Bulk-scrape academic papers from Semantic Scholar's 200M+ corpus: title, abstract, AI TLDR summary, authors, venue, year, citation & influential-citation counts, fields of study, DOI/PubMed/arXiv IDs and open-access PDF links. Filter by year, field and citations. Thousands per run. No setup.

123 runs · 7 users · $0.001 / paper scraped

Wikipedia Scraper

Search Wikipedia or look up exact articles and get the full plain-text content, summary, categories, image, Wikidata ID and monthly pageview trends. Any language. Great for research, RAG and AI datasets. No setup.

121 runs · 1 users · $0.001 / article scraped

Need a source that isn't here?

Send the site and the fields. We build it and you get it first.

Request a build