AI SDK
AirtopBuilds AI agents that log in to websites, browse them and extract data, from a plain-English description of the task.
Data & analytics
Data scraping & pipelines: the tech stack of 38 real products
Scraping, ETL, data integration and enrichment.
What sets them apart
Picks at least twice as common here as among products overall, in decisions at least 10 of them show.
DuckDBDatabase6.0×as often as overall5 of the data scraping & pipelines that show database (19%)
LiteLLMLLM API4.4×as often as overall6 of the data scraping & pipelines that show LLM API (40%)
MySQLDatabase3.6×as often as overall10 of the data scraping & pipelines that show database (37%)
LlamaIndexAI SDK & Agent Framework2.9×as often as overall5 of the data scraping & pipelines that show AI SDK & agent framework (42%)
AstroFrontend Framework2.9×as often as overall5 of the data scraping & pipelines that show frontend framework (23%)The typical stack
The leading pick where at least 8 of them show the decision and the leader has at least a quarter of it.
| Decision | Most common pick | Share | At default usage |
|---|---|---|---|
| Database | MySQL | 10 of 27 · 37% | — |
| Frontend Framework |  React | 13 of 22 · 59% | — |
| Backend Framework |  FastAPI | 9 of 17 · 53% | — |
| LLM API | OpenAI API | 9 of 15 · 60% | $10/mo · GPT-6 Luna |
| AI SDK & Agent Framework | LlamaIndex | 5 of 12 · 42% | — |
| Database Access & ORMs |  SQLAlchemy | 7 of 10 · 70% | — |
| Hosting |  Cloudflare | 4 of 10 · 40% | $12/mo · Workers Paid |
| Transactional Email |  Resend | 3 of 9 · 33% | $20/mo · Pro 50k |
Monthly bills add up to about $42 at the calculators' default usage, list prices. Set your own usage →
What they chose, decision by decision
Among the data scraping & pipelines that show each choice, from makers' products and open-source code alike.
Small samples, fewer than 8 products: Authentication (Clerk 3, Auth.js 2) · Background Jobs & Cron (Celery 3, RabbitMQ 2) · Payments (Stripe 5, Paddle 1) · Vector Database (LanceDB 4, Qdrant 3) · Image & Video Hosting (sharp 3, Cloudinary 1) · Product Analytics (PostHog 4, Mixpanel 1)
Data scraping & pipelines we track
21 makers' products and 17 open-source projects. Makers' products first.
AirtopBuilds AI agents that log in to websites, browse them and extract data, from a plain-English description of the task.
DataFuel.devScraping made easier for developer!
Parseflow.ioThe Data Automation Platform for Every Business
PulpMinerConverts Any Webpage Into Realtime JSON API 🟢
SaturnTurn Japan's public data into an AI-ready spreadsheet WS
fileAIClassify, extract, enrich, and validate any file 
SliqAutomated data cleaning, in minutes, not hours or days.
AgentQLA query language and API that lets developers select and scrape data from web pages with natural-language-like queries instead of XPath or CSS selectors.
Supametas.AIStreamline unstructured data into LLM RAG-ready datasets
Tilores entity resolutionOne trusted view of every customer, company, transaction
+9
+4
+7
+4
+4+3
+3
+1
+2
+1
