Tendril
Web ingestion that reaches what other crawlers can't
Tendril scrapes and maps the web from a residential Mac in real WebKit, returning deterministic Markdown and structured data you can regression-test.
86
Hermetic tests passing, zero network
0
LLMs in the extraction path
3
Tiers in the escalation router
4
Public API endpoints
3
MCP tools exposed to agents
6
Workspace packages in the monorepo
Overview
Cloud crawlers run in headless Chromium on datacenter IPs — and get blocked for it. Tendril takes the opposite bet: it runs on a residential Apple Silicon machine, in WebKit, Safari's actual engine, behind a real IP. Sites that block conventional scrapers often don't block Tendril at all.
Extraction is fully deterministic — zero LLMs in the pipeline. The same page always yields the same Markdown, which means golden-file tests, versioned extraction rules, and reproducible output that enterprise workflows can actually depend on. JSON-LD, OpenGraph, Readability with a density-based fallback, and hand-tuned Turndown rules do the work parsers were built for.
At its core sits a three-tier escalation router: every request starts at a cheap undici HTTP fetch with Safari-fidelity header ordering, and escalates toward a native WKWebView pool and a real Safari driver only on evidence of failure. The whole platform is live at www.ten-dril.com, hardened against SSRF, honouring robots.txt by default, and self-identifying with a bot User-Agent — stealth means matching a real browser's fingerprint, not hiding who you are.
Key Features
Real-browser fidelity
WebKit on macOS ARM behind a residential IP is not a headless-Chromium signature, so sites that block other crawlers often let Tendril through.
Deterministic extraction
No model inference anywhere in the pipeline — the same page always yields the same Markdown, so extraction is covered by golden-file tests.
Map without rendering
Merges sitemaps, robots directives, homepage links, and /llms.txt into one deduplicated URL list in seconds.
Three-tier escalation router
A pure, unit-tested shouldEscalate() function starts every request at cheap HTTP and escalates to WKWebView or real Safari only on evidence of failure.
Structured data harvest
JSON-LD, OpenGraph, and standard meta are parsed into a normalized structured object alongside clean Markdown and deduplicated links.
Agent-ready integrations
A single-file MCP server (tendril_scrape, tendril_map, tendril_status) and a drop-in Agent Skill plug Tendril into any AI agent with no npm install and no API key.
SSRF-hardened egress
Resolve-then-validate with IP pinning blocks private addresses and re-checks every redirect hop before any byte is fetched.
Fair by default
Identifiable bot User-Agent with a contact URL, full RFC 9309 robots.txt compliance, and per-host rate limiting are the defaults, not options.
Self-hosted on Apple Silicon
Runs on a single Mac under PM2 with ngrok as the sole ingress, the API bound strictly to 127.0.0.1.
How It Works
Request intake
A POST to /v1/scrape or /v1/map hits the Fastify API, where Zod schemas validate the payload and SSRF checks resolve and pin the target IP before anything is fetched.
Tier 0 HTTP fetch
undici fetches the page with Safari's exact header order, manual redirect handling with per-hop SSRF revalidation, and gzip/brotli decoding — the cheapest path that serves most traffic.
Escalation decision
The pure shouldEscalate() function inspects status codes, challenge markers, and empty SPA shells; on evidence of blocking, the router escalates toward the WKWebView pool and real Safari tiers.
Deterministic extraction
Raw HTML flows through a versioned pipeline — linkedom parsing, JSON-LD/OpenGraph harvest, Readability with a text-density fallback, and custom Turndown rules — producing Markdown, structured data, and links.
Response & delivery
Every response is a uniform { success, data, error } envelope with an X-Request-Id header, served from a Mac where the API binds only to localhost and ngrok is the sole public ingress.
Tech Stack
Core
Fetch & Rendering
Extraction
Infrastructure & Testing
Highlights
- Runs in real WebKit on a residential Mac — not a headless-Chromium datacenter signature
- Zero LLMs in the extraction path: the same page always yields the same Markdown, verified by golden-file tests
- 86 hermetic unit and contract tests that pass with no network, on strict TypeScript with Result<T,E> error handling
- Full RFC 9309 robots.txt parsing, identifiable bot User-Agent with contact URL, and per-host rate limiting by default
- Single-file MCP server and Agent Skill make it a drop-in tool for Claude and any MCP-capable agent — no API key required
- SSRF-hardened with resolve-then-validate IP pinning re-checked on every redirect hop
Explore Tendril
Web ingestion that reaches what others can't — the full source is on GitHub.