ka6
The flagship of the Groupe KA fleet — the Quebec web charted at 9,000 pages a day.
ka6 is the top tier of three twin bots sharing one engine, running 16 concurrent pages under a Claude Sonnet 5 controller. Watch it work in real time at www.ka6.bot.
16
Pages crawled in parallel
9,000
Target pages per day
320
Max pages per run
0.35 s
Delay between requests
2
Claude models (Sonnet 5 + Haiku 4.5)
3
Directory views (cards, table, graph)
Overview
ka6 is the flagship AI-driven scraper and search bot of Groupe KA, mapping the Quebec web as a knowledge graph of entities — companies, organizations, people, creators and influencers, agencies, and brands — with their attributes and typed relations such as WORKS_AT, FOUNDER_OF, REPRESENTED_BY, COLLABORATES_WITH, and SPONSORED_BY. Among the three twin bots sharing the same engine, ka6 sits at the top tier: where ka2 runs 6 concurrent pages at 800 pages/day and ka4 runs 10 at 3,000, ka6 pushes 16 concurrent pages toward 9,000 pages per day, under a purple-themed real-time dashboard.
A Claude Sonnet 5 agent takes missions from the dashboard and works through a rigorous plan, discover, collect, verify loop, piloting the crawler via tools — search_web, map_site, scrape_page, deep_dive, crawl_queue — with automatic context compaction. Fetching runs Firecrawl-first with Scrapfly as anti-bot fallback, respecting robots.txt and rate limits at a 0.35 s request delay, with fail-fast timeouts, 2 attempts per URL into a dead-letter queue, a per-domain circuit breaker after 4 failures, and up to 320 pages per execution. A Claude Haiku 4.5 extractor then distills each page into graph entities and typed relations with provenance.
The project is Python (97.8%, plus 2.2% Shell): a FastAPI application streams KPIs and events over SSE to the dashboard and a structured directory with 3 views — cards, sortable table, and a pan/zoom force-directed global graph — with type/sector/region facets and CSV/JSON export. All data lives in SQLite in WAL mode, from entities and relations to social links, missions, and restorable archives. ka6 is deployed on MacLustr cluster node M4M36 as 3 resilient launchd services (com.ka6.web, com.ka6.bot, com.ka6.ngrok), served by uvicorn on port 8999 and exposed via ngrok at www.ka6.bot.
Key Features
Flagship throughput tier
The highest tier of the three twin Groupe KA bots: 16 concurrent pages and 9,000 pages per day, versus 10/3,000 for the aggressive ka4 and 6/800 for the light ka2.
Typed knowledge graph
Normalized entities connected by typed relations — WORKS_AT, FOUNDER_OF, REPRESENTED_BY, COLLABORATES_WITH, SPONSORED_BY, MEMBER_OF, PARTNER_OF — each with a role and source provenance.
Two-model Claude agent
Claude Sonnet 5 plans missions and deep-dives via tools (search_web, map_site, scrape_page, deep_dive, crawl_queue) with auto-compaction, while Claude Haiku 4.5 extracts the graph from markdown.
Resilient fetch pipeline
Firecrawl as primary with Scrapfly anti-bot fallback, robots.txt compliance, rate limiting at 0.35 s delay, dead-letter retries per URL, and a per-domain circuit breaker after 4 failures.
Real-time dashboard
SSE streaming of KPIs, activity feed, and live settings in a responsive purple theme, with mission controls (stop, prioritize, delete) and archive snapshots that can be restored or exported.
Structured directory & exports
Three explorer views — cards, sortable table, force-directed global graph with pan/zoom — plus type/sector/region facets, search, and filtered CSV/JSON export through the REST API.
How It Works
Mission intake & planning
Missions from the dashboard drive a Claude Sonnet 5 agent through a plan, discover, collect, verify loop, orchestrating the crawler via five tools with automatic context compaction.
High-throughput fetch layer
16 concurrent fetches run Firecrawl-first with Scrapfly anti-bot fallback, respecting robots.txt and rate limits, with fail-fast timeouts, 2 attempts per URL, and a domain circuit breaker at 4 failures.
AI extraction
Claude Haiku 4.5 converts scraped markdown into entities and typed relations, capturing creator/influencer fields such as handle, platform, per-platform follower counts, niche, and languages.
Resolution & storage
Entity resolution deduplicates by normalized name, domain, and region against canonical Quebec regions, storing entities, relations, social links, and sources with provenance in SQLite (WAL).
Live serving
FastAPI streams stats and events over SSE to the purple-themed dashboard and the 3-view directory, exposing a REST API for entities, the global graph, exports, missions, archives, and bot controls.
Tech Stack
Core & AI
Crawling & Data
Deployment
Highlights
- Flagship tier of the Groupe KA fleet: 16 pages crawled in parallel toward 9,000 pages per day, up to 320 pages per run.
- Two-model Claude pipeline pairs a Sonnet 5 mission controller with a Haiku 4.5 extractor that builds the knowledge graph.
- Firecrawl-first fetching with Scrapfly anti-bot fallback, dead-letter retries, and a per-domain circuit breaker after 4 failures.
- Typed relations (WORKS_AT, FOUNDER_OF, SPONSORED_BY, and more) each carry a role, confidence, and source provenance.
- Purple-themed SSE dashboard with live KPIs, mission controls, restorable archives, and a 3-view directory with CSV/JSON export.
- Runs on MacLustr node M4M36 as 3 launchd services behind ngrok at www.ka6.bot, respecting robots.txt and Quebec Law 25 / PIPEDA.
Explore ka6
Flagship AI explorer of the Quebec web — the full source is on GitHub.