Trouve-KA
A real search engine for the Quebec web — its own crawler, its own index, its own ranking.
Trouve-KA crawls, classifies, and ranks the Quebec web with zero dependence on Google, Bing, or Brave — live at www.trouve-ka.com. A fetched page becomes searchable in about 2 to 4 seconds.
59,073
Pages indexed in the first ~6 hours
7,898
Quebec domains discovered from 64 seeds
76,198
Pages fetched per hour
20–95 ms
Measured production search latency
~2–4 s
From fetch to searchable
57
Unit tests passing
Overview
Trouve-KA is an independent, Quebec-first web search engine by Groupe KA — not a metasearch wrapper. It runs its own crawler, its own OpenSearch index, its own ranking, its own API, and a public web app, with no external search provider behind the results. Starting from 64 high-authority Quebec seeds (government, municipalities, universities, media), it discovered 7,898 Quebec domains and indexed 59,073 searchable pages within roughly six hours of its first crawl, growing at about 35,000 pages per hour.
The pipeline runs crawler, frontier, fetcher, parser, Quebec classification, deduplication, indexer, and ranking. Every page receives two deterministic Quebec scores — page and domain — computed from TLDs, a toponym gazetteer, postal and area codes, known organizations, and structured province mentions, with no LLM in the hot path. Indexing happens inline, so a fetched page is searchable in about 2 to 4 seconds, while enrichment (authority, entities, embeddings) runs asynchronously. Ranking blends bilingual BM25, Quebec scores, domain authority, freshness, and a locality boost.
The monorepo pairs a Next.js 15 / React 19 / TypeScript / Tailwind web app with a FastAPI (Python 3.12) API and an async Python crawl pipeline built on httpx, selectolax, and Protego, backed by a PostgreSQL 16 frontier, Redis 7 coordination, and OpenSearch 2.17. It deploys via Docker Compose on MacLustr cluster node M2M32, exposed through an ngrok tunnel at www.trouve-ka.com, with satellite crawl workers on M2M32b and M2M32c coordinating without any orchestrator. Fifty-seven unit tests and a strict no-fake-data rule keep it honest.
Key Features
Quebec-first ranking
An OpenSearch function_score blends bilingual BM25, dual page and domain Quebec scores, inlink-weighted domain authority, freshness, and a locality boost — 'plombier Gatineau' surfaces documents with geographic proof.
Searchable in seconds
Workers run fetch, parse, score, and index inline with a 1-second OpenSearch refresh interval, so a crawled page is live in search results roughly 2 to 4 seconds after fetch.
Distributed crawl, no orchestrator
Three cluster nodes coordinate purely through the Postgres frontier (FOR UPDATE SKIP LOCKED) and Redis politeness locks over the internal LAN — never more than one request per site per second, cluster-wide.
Transparent, polite crawler
A declared TrouveKABot user agent with a public bot page, robots.txt parsed via Protego with 24-hour caching, respect for noindex/nofollow and Crawl-delay, and a 2-second default per-host spacing.
Bilingual FR/EN search
French and English analyzers with cross-language synonyms at query time (thermopompe matches heat pump), so Quebec's bilingual web is searchable in either language.
Hardened fetching
An SSRF guard blocks private, loopback, and cloud-metadata IPs and revalidates on every redirect, with 3 MB response caps, redirect limits, timeouts, and detection of crawler traps like session IDs and infinite calendars.
How It Works
Seed and frontier
The crawl starts from 64 high-authority Quebec seeds; discovered URLs (433,999 pending on day one) queue in a PostgreSQL 16 frontier that workers claim safely with FOR UPDATE SKIP LOCKED.
Polite fetch
Workers fetch under the declared TrouveKABot identity, honoring robots.txt via Protego, holding a per-host Redis politeness lock, and passing every request through SSRF guards and crawler-trap detection.
Parse and classify
Pages are parsed with selectolax, then scored for Quebec relevance with two deterministic signals — page and domain — built from TLDs, toponyms, postal codes, area codes, known organizations, and JSON-LD, with no LLM involved.
Inline indexing
Fetch, parse, score, and index run in a single worker pass into OpenSearch with a 1-second refresh interval; content-hash and ETag change detection drive adaptive recrawl intervals.
Ranking, API, and web app
A FastAPI service exposes search, status, and admin endpoints over an OpenSearch function_score query, and the Next.js 15 app proxies /api/* internally so a single port sits behind the ngrok tunnel.
Tech Stack
Frontend & API
Search & Data
Deployment
Highlights
- Indexed 59,073 pages across 7,898 Quebec domains within about six hours of the first crawl, starting from just 64 seeds.
- Pages go from fetch to searchable in roughly 2 to 4 seconds thanks to inline indexing and a 1-second OpenSearch refresh interval.
- Measured production search latency of 20 to 95 ms, comfortably under the p50 < 100 ms target.
- Three cluster nodes fetch 76,198 pages per hour while never hitting any single site more than once per second.
- The Images tab hotlinks each page's representative og:image with attribution — no images are ever crawled or stored.
- The public /status page reports real counters straight from live tables — a strict no-fake-data rule, backed by 57 passing unit tests.
Explore Trouve-KA
An independent Quebec-first web search engine — the full source is on GitHub.