ka4
The Quebec web, mapped aggressively into a living knowledge graph.
ka4 is the aggressive edition of the Groupe KA bot fleet: a Claude-driven crawler pushing 10 concurrent pages toward 3,000 pages a day. Its real-time dashboard is live at www.ka4.bot.
10
Pages crawled in parallel
3,000
Target pages per day
220
Max pages per run
0.4 s
Delay between requests
2
Claude models (Sonnet 5 + Haiku 4.5)
3
Resilient launchd services in production
Overview
ka4 is an AI-driven scraper and search bot from Groupe KA that charts the Quebec web as a knowledge graph: entities such as companies, organizations, people, creators and influencers, agencies, and brands, together with their attributes and typed relations like WORKS_AT, FOUNDER_OF, REPRESENTED_BY, COLLABORATES_WITH, and SPONSORED_BY. It is the aggressive tier of three twin bots sharing one engine — above the light ka2 (Haiku 4.5, 6 concurrent, 800 pages/day) and below the flagship ka6 — running a Claude Sonnet 5 controller at 10 concurrent pages and 3,000 pages per day, under a red-themed dashboard.
Missions submitted from the dashboard drive a rigorous Claude agent that plans, discovers, collects, and verifies through tools: search_web, map_site, scrape_page, deep_dive, and crawl_queue, with automatic context compaction. Fetching goes through Firecrawl first with Scrapfly as an anti-bot fallback, while respecting robots.txt, rate limits, fail-fast timeouts, per-URL dead-letter retries (2 attempts per URL), and a per-domain circuit breaker that trips after 4 failures. A Claude Haiku 4.5 extractor turns page markdown into graph entities and typed relations, and entity resolution deduplicates by normalized name, domain, and region with full provenance.
The codebase is Python (97.8%, plus 2.2% Shell): a FastAPI web app streams stats and events over SSE to a real-time dashboard, and everything persists in SQLite in WAL mode — entities, relations, social links, sources, crawl queue, missions, and archives. A structured directory offers 3 views (cards, sortable table, force-directed global graph with pan/zoom) plus facets and CSV/JSON export. ka4 is deployed on MacLustr cluster node M4M36 as 3 resilient launchd services (com.ka4.web, com.ka4.bot, com.ka4.ngrok), served by uvicorn on port 8899 and exposed via ngrok at www.ka4.bot.
Key Features
Typed knowledge graph
Normalized entities linked by typed relations — WORKS_AT, FOUNDER_OF, REPRESENTED_BY, COLLABORATES_WITH, SPONSORED_BY, MEMBER_OF, PARTNER_OF — each carrying a role and source provenance.
Aggressive concurrent crawling
10 pages fetched in parallel with an aggressive frontier that auto-queues 12 to 16 discovered links per page, up to 220 pages per execution at a 0.4 s request delay.
Claude-driven agent
A Claude Sonnet 5 controller plans, searches, and deep-dives via tools (search_web, map_site, scrape_page, deep_dive, crawl_queue), while Claude Haiku 4.5 extracts graph data from page markdown.
Robust fetch pipeline
Firecrawl as primary scraper with Scrapfly anti-bot fallback, robots.txt compliance, rate limiting, fail-fast timeouts, dead-letter retries, and a per-domain circuit breaker after 4 failures.
Real-time dashboard
SSE-powered dashboard with live KPIs, an activity feed, live settings, mission controls (stop, prioritize, delete), and archive snapshots with restore and export — in a responsive red theme.
Structured directory
Three explorer views — cards, sortable table, and a force-directed global graph with pan/zoom — with type/sector/region facets, search, and CSV/JSON export.
How It Works
Mission intake & planning
Missions created from the dashboard drive a Claude Sonnet 5 agent that follows a plan, discover, collect, verify loop and pilots the system through five tools, with automatic context compaction.
Resilient fetch layer
Pages are fetched through Firecrawl with Scrapfly as anti-bot fallback, under robots.txt compliance, rate limiting, fail-fast timeouts, 2 attempts per URL, and a per-domain circuit breaker at 4 failures.
AI extraction
A Claude Haiku 4.5 extractor converts scraped markdown into a graph of entities and typed relations, including creator/influencer attributes like handle, platform, follower counts, niche, and languages.
Resolution & storage
Entity resolution deduplicates by normalized name, domain, and region against canonical Quebec regions, then persists entities, relations, social links, and sources with provenance in SQLite (WAL).
Live serving
A FastAPI app streams stats and events over SSE to the red-themed dashboard and the 3-view directory, and exposes a REST API for entities, the global graph, exports, missions, and bot controls.
Tech Stack
Core & AI
Crawling & Data
Deployment
Highlights
- Crawls 10 pages in parallel with an aggressive frontier auto-queuing 12 to 16 links per page, targeting 3,000 pages per day.
- Two-model Claude pipeline: Sonnet 5 pilots missions and deep dives while Haiku 4.5 extracts entities and typed relations.
- Firecrawl-first fetching with Scrapfly anti-bot fallback, dead-letter retries, and a per-domain circuit breaker after 4 failures.
- Every relation edge (WORKS_AT, FOUNDER_OF, SPONSORED_BY, and more) carries a role, a confidence score, and source provenance.
- Directory ships 3 views — cards, sortable table, and pan/zoom force-directed global graph — with facets and CSV/JSON export.
- Runs on MacLustr node M4M36 as 3 launchd services behind ngrok at www.ka4.bot, respecting robots.txt and Quebec Law 25 / PIPEDA.
Explore ka4
Aggressive AI mapper of the Quebec web — the full source is on GitHub.