arXiv Papers Scraper

Search and extract papers from arXiv.org as clean, structured JSON, CSV, Excel or XML: title, full abstract, authors, subject categories, submission/update dates, DOI, journal reference, and direct PDF links. Search by keyword, filter by category, sort by newest or relevance, or fetch exact papers by arXiv ID. No account, no API key — fast and reliable.

What does the arXiv Papers Scraper do?

The arXiv Papers Scraper searches the full arXiv corpus and extracts complete, structured metadata for every matching paper. Use plain keywords or structured field queries, narrow by subject category, and export thousands of papers with abstracts and PDF links in a single run — ideal for literature reviews, research monitoring, and building AI/ML datasets.

Features

• 🔎 Keyword & field search — plain terms or structured queries (ti:, au:, abs:, cat:) across the full arXiv corpus
• 🗂️ Category filters — narrow to fields like cs.CL, cs.AI, math.PR, stat.ML, and combine them
• 🆔 Fetch by ID — pass a list of arXiv IDs to pull specific papers directly
• 📅 Sort your way — by relevance, newest submitted, or most recently updated
• 📄 Full abstracts + PDF links — the complete summary text plus direct abs and pdf URLs
• 👥 Structured authors — clean author list, count, and a display string

What you get

Each paper is one dataset record:

{
  "query": "large language models",
  "arxivId": "2403.09676v1",
  "arxivIdBase": "2403.09676",
  "title": "Unmasking the Shadows of AI: Investigating Deceptive Capabilities in Large Language Models",
  "summary": "This research critically navigates the intricate landscape of AI deception...",
  "authors": ["Linge Guo"],
  "authorsList": "Linge Guo",
  "authorCount": 1,
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "published": "2024-02-07T00:21:46Z",
  "updated": "2024-02-07T00:21:46Z",
  "doi": null,
  "journalRef": null,
  "comment": "AI deception, Large Language Models, ChatGPT",
  "absUrl": "https://arxiv.org/abs/2403.09676v1",
  "pdfUrl": "https://arxiv.org/pdf/2403.09676v1"
}

Output schema

Input

{
  "searchTerms": ["large language models", "cat:cs.CL"],
  "categories": ["cs.AI"],
  "sortBy": "newest",
  "maxResultsPerQuery": 200,
  "includeAbstract": true
}

Use cases

• Literature reviews — pull every recent paper on a topic with abstracts in one run.
• Research monitoring — track new submissions in your categories, sorted newest-first.
• AI/ML datasets — build corpora of titles + abstracts for classification, search, or RAG.
• Citation & metadata — grab DOIs, journal refs, and PDF links at scale.
• Trend analysis — measure output by category, author or time to spot emerging topics.

How to scrape arXiv papers

1. Add the arXiv Papers Scraper to your Apify account.
2. Enter one or more search terms (keywords or field:value queries), or an idList for exact papers.
3. Optionally set categories, sortBy and maxResultsPerQuery.
4. Click Run.
5. Export the dataset as JSON, CSV, Excel or XML, or pull it via the Apify API.

Need more research data?

• Crossref Scraper — DOIs, citation counts and metadata from 150M+ registered works.
• OpenLibrary Scraper — book metadata, authors, subjects and editions.
• ClinicalTrials Scraper — study records from the clinical trials registry.

FAQ

Do I need an API key or login? No. There's no account, login or API key required — just add your search terms and run.

How many papers can I scrape per run? Up to 2,000 papers per query via maxResultsPerQuery, and you can pass multiple search terms in one run, so a single run can return many thousands of papers.

How does billing work? You pay only per paper you get, so small filtered runs stay cheap. See the Pricing tab for current rates.

What format is the output? Clean structured JSON — one record per paper — exportable as JSON, CSV, Excel or XML. Each record includes the title, full abstract, authors, categories, dates, DOI and direct PDF links.

Can I filter by category or fetch specific papers? Yes. Use categories to narrow to fields like cs.CL, cs.AI or stat.ML, combine them with keywords, or pass an idList of arXiv IDs to pull exact papers directly.

Is this only public data? Yes. It returns only openly available arXiv metadata and doesn't touch anything behind a login.

Can I connect this to other tools? The arXiv Papers Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. It works with Make, Zapier, Slack, Airbyte, GitHub, Google Drive and many more, plus the Apify API, JavaScript/Python clients and MCP. Or use webhooks to trigger an action whenever a run finishes.

Notes

Original clean-room implementation. Returns only public arXiv metadata; you are responsible for compliance with arXiv's terms. arXiv is a trademark of Cornell University; this Actor is independent and not affiliated with or endorsed by arXiv.

You pay only for what the run returns. Set a maximum cost per run and it stops cleanly at that limit.