A technical deep-dive into the architecture, detection engine, and design decisions.
ThreadWeave is an open-source system that connects to the tools your organization already uses and builds a searchable memory of the knowledge flowing through them. Everything runs on-premises. No cloud dependency. AGPL-licensed.
ThreadWeave is a FastAPI server with a modular pipeline. Every piece of content passes through the same stages:
[Connectors: Gmail, Chat, Drive, M365, IMAP, .eml]
│
▼
POST /api/v1/ingest
│
┌─────────┴──────────┐
│ 1. Dedup │ SHA-256 of content + subject + sender
│ 2. Detect │ Regex-first classifier (LLM optional)
│ 3. PII Gate │ Strip emails, phones, identifiers
│ 4. Store │ In-memory + MemPalace (ChromaDB + BM25)
└─────────┴──────────┘
│
┌─────────┼──────────┐
│ Relevance Rank │ Semantic + org proximity + freshness + authority
│ Confidentiality │ 7 sensitivity levels, auto-detected
│ Audit Log │ Every read, write, and access decision
└─────────────────────┘
│
▼
/api/v1/search
| Module | Purpose |
|---|---|
| detector.py | Regex classifier: ANSWER / DECISION / QUESTION / CHAT / REFERENCE |
| llm_detector.py | Optional LLM-based classification with regex fallback |
| mempalace_client.py | Hybrid BM25 + vector search via ChromaDB |
| confidentiality.py | 7 sensitivity levels, auto-detected, enforced at read |
| org_model.py | Temporal knowledge graph — who was on which team when |
| relevance.py | 4-factor ranking for search results |
| profiling.py | Latency percentiles, throughput, Prometheus export |
ThreadWeave wraps MemPalace — the highest-scoring open-source AI memory system — for hybrid semantic + keyword retrieval and structured storage.
MemPalace organizes knowledge as a building:
The Palace is the entire organization. Wings are departments and teams. Rooms live inside wings — specific projects, regulatory domains, client engagements. Drawers hold raw, verbatim content. Closets hold compressed summaries.
This structure mirrors how people actually think about organizational knowledge. Search isn't a keyword match — it belongs to a specific wing, room, and moment in time.
Results are ranked by semantic match, organizational proximity, freshness, and author authority. The result isn't a document list — it's an answer that feels like institutional instinct.
MemPalace tracks relationships automatically — who created what, which project it belonged to, which team was involved, when it happened. Hallways connect knowledge within the same wing. Tunnels connect knowledge across wings — so the tax team's 2018 solution surfaces when M&A encounters the same problem in 2026.
Connections that get used grow stronger. Connections that go unused gradually fade. The knowledge graph is temporal — it knows when things were true.
Every piece of content gets classified into one of five types:
| Type | Example | Save? |
|---|---|---|
| ANSWER | "The reason we use Postgres is JSONB support" | ✅ |
| DECISION | "We decided to migrate to GraphQL for the new API" | ✅ |
| QUESTION | "Does anyone know why we chose Redis?" | ❌ |
| CHAT | "ok thanks, sounds good!" | ❌ |
| REFERENCE | Newsletters, HR digests, system notifications | ❌ |
The primary detector uses weighted pattern dictionaries:
ANSWER patterns (+0.15 each): "the reason", "because", "due to", "this is why", "root cause is", "turned out that" DECISION patterns (+0.15 each): "we decided", "decision:", "we chose", "we will use", "applied the fix", "released as", "resolved"
The regex detector outperformed the LLM on real-world test data (88% vs 76% on a 25-email set). It correctly handles technical emails with ticket numbers and version strings — patterns that cause LLMs to over-classify everything as "reference."
21 patterns catch newsletters, HR digests, and system notifications before type classification. 3+ matches forces REFERENCE at 0.90 confidence. 1-2 matches halves answer/decision scores.
THREADWEAVE_LLM_BASE_URL=http://localhost:11434/v1 THREADWEAVE_LLM_MODEL=llama3.1:8b THREADWEAVE_LLM_PROVIDER=ollama
If the LLM call fails, the system gracefully falls back to regex. Nothing breaks.
Carefully tuned. At 0.30, too many borderline saves. At 0.40, legitimate terse technical answers were missed. 0.35 is the sweet spot.
dedup_key = f"{content}|{title}|{author_id}"
content_hash = sha256(dedup_key)
Same email ingested twice → same hash → correctly deduped. Same body, different subject/sender → different hash → both saved. The dedup cache is in-memory; restarting the server clears it.
Strips emails, phone numbers, and personal identifiers before storage. Content triggering PII detection is rejected entirely.
Entries stored in both an in-memory dict (fast retrieval) and MemPalace/ChromaDB (persistent, vector-enabled). Searches work even if MemPalace is unavailable.
| Level | Description |
|---|---|
| Public | Safe for external sharing |
| Internal | Within the organization (default) |
| Confidential | Restricted to specific teams |
| Restricted | Senior leadership only |
| HR-Privileged | Personnel matters |
| Client-Confidential | Specific client data |
| Legal-Privileged | Attorney-client material |
Auto-detected at ingest through pattern matching. Enforced at read — you can only see what your role and team permit.
| Connector | How | Auth |
|---|---|---|
| IMAP | Any email provider | Password or app password |
| Graph device-code | M365 inbox via Graph | Browser sign-in, MFA supported |
| Outlook COM | Classic Outlook | None (local) |
| .eml files | Batch import | None |
| Google Workspace | Gmail, Chat, Drive + offboarding harvester | Service account |
| Graph external connector | Push to Copilot & Microsoft Search | Azure app registration |
The Google Workspace connector's offboarding harvester drains institutional knowledge when someone leaves — every decision, process, and client context from their email, chat, and documents.
No data leaves the network perimeter. No vendor lock-in. Air-gapped ready. Bring your own LLM or don't use one at all.
Docker Compose includes an optional Ollama profile that auto-pulls a model on first start. Or point it at any OpenAI-compatible endpoint — vLLM, LiteLLM, corporate API gateway.
GET /api/v1/metrics # JSON GET /api/v1/metrics/prometheus # Prometheus scrape target
Per-stage latency (p50/p95/p99), dedup hit rate, LLM vs regex ratio, throughput, memory, uptime. Full audit log at /api/v1/audit/recent.
# Docker (recommended) git clone https://github.com/PowerLooming/ThreadWeave cd ThreadWeave docker compose up # Or native Python bash setup.sh threadweave serve
API docs at http://localhost:8000/docs. Health check at /api/v1/health.