0sint
An OSINT triage system for an operational intelligence desk officer. It ingests a high-volume Telegram-style feed, clusters raw messages into distinct real-world events using a domain fine-tuned sentence embedder, scores each event on veracity, relevance and urgency, retrieves comparable past incidents, and generates a grounded situation card with a recommended action set.
Fine-tuning as a response, not a default
Off-the-shelf embeddings plateaued at ARI 0.574 — two incidents in the same city at the same hour merged into one cluster at 0.788 centroid similarity. Contrastive fine-tuning moved the same window to 0.883.
Hard negatives mined from an audit
~600k candidate pairs → 10,204 above 0.70 similarity → ~25 sharing time and place → the 5 most confusable held out of training entirely as the eval set.
Guardrails moved into code
The model kept inventing agencies that don't exist. Prompt instructions reduced it; a recipient whitelist generated in code eliminated it. Battery: 6 cards, 0 violations.
The problem
A desk officer monitoring open Telegram channels faces a specific failure mode: a thin trickle of genuinely relevant information buried inside overwhelming noise. Keyword filters fail in both directions — missing events described in unexpected wording, flooding on incidental matches.
Three harder problems sit underneath. The same event is reported by many accounts in wildly different registers, from official bulletins to panicked eyewitness posts. Two different events in the same city at the same hour look nearly identical to a general-purpose embedder. And an LLM asked to summarize raw intelligence will confidently invent agencies, units and protocols that do not exist — which in this domain is worse than no summary at all.
Architecture
Telegram-style feed (11,774 messages / 1,102 events)
│
▼
0sint-event-embedder ....... MiniLM-L6-v2 fine-tuned
(384-dim, 22.7M params) MultipleNegativesRankingLoss
│ 15,293 mined triplets
▼
Agglomerative clustering ... average linkage @ 0.50
│
▼
Event scoring .............. veracity / relevance / urgency
│
┌────┴────┐
▼ ▼
Similar Constrained generation → grounded card
past what happened · does it matter ·
events action set · recipients
│
▼
Gradio Space ... 4/8/24h window · V-X triage marks
world map · semantic search
Telegram PDF export
The hardest part
Two events, one city, one hour. An airport hostage incident and a bus-station explosion, reported in the same window by overlapping accounts, sat at 0.788 centroid similarity — well above any threshold that would still merge genuine duplicates. No threshold tuning could resolve it, because the geometry itself was wrong: the base model encoded "something serious happening here" and discarded the distinction that mattered.
Fixing it required changing the embedding space, and the eval had to be built from held-out confusable pairs rather than a random split — a random split would have scored well while leaving the actual failure untouched.
The honest caveat: 0.883 is same-window clustering performance for the production model on an 8-hour, 466-message window at average linkage 0.50, measured against the 0.574 baseline on that same window. The held-out confusable-pair evaluation is a separate exercise reporting centroid similarity and separability, not ARI. Keeping those two numbers distinct matters more than having one headline figure.
Results
- Fine-tuned embedder published — 22.7M parameters, 384-dimensional, trained on 15,293 triplets in 7.5 minutes on a single GPU.
- Corpus of 11,774 messages across 1,102 events, category structure verified from the parquet rather than assumed: security events average 8.78 messages against 3.58 for irrelevant ones — a ~2.45× coverage ratio that empirically justifies the breadth term in the veracity score.
- 6 cards, 0 rule violations under an automated six-rule battery — after a bug was found where an early return meant only the first rule had ever run.
- Deployed with windowed filtering, V/X triage marking, an interactive world map, semantic search, and Telegram PDF export of marked events.
What I took from it
Measure the baseline's failure precisely before reaching for a bigger model, and let that failure define the eval set. Move constraints from prompts into code wherever the constraint can be expressed deterministically. And design for the user's flow rather than exposing the architecture — the technical layer is something to answer questions about, not something to put on screen.