aesop_intell

Module XXX — OSINT Intelligence Pipeline & Synthesis Service

aesop_intell — Internal Architecture SOURCES RSS Websites Facebook X / Twitter API / Webhooks Manual Upload STEP 1 — COLLECTORS app_step1_rss app_step1_website app_step1_facebook app_step1_x app_step1_flot app_step1_capture app_step2_osint — 6-Stage Pipeline Ingest S1 Extract S2 Language S3 Relevance S4 NER S5 Classify S6 app_intell_embedding pgvector — vector generation app_graph Apache AGE knowledge graph SYNTHESIS LEVELS L1 app_step4_intell Rolling Draft Cache L2 app_step5_level2 Aggregation: organic / temporal / dimension L3 app_step6_level3 Strategic: weekly + structural reports SUPPORT APPS app_step3_situation app_map app_chat app_settings app_account Situation Room Leaflet Map Chat UI Config Users REST API Layer (Django REST Framework) aesop_modelisation MIDAS — :8001 aesop_simulation AESOP — :8002 entities, alerts, synthesis scenario context STACK: Django 5.x PostgreSQL 17 Apache AGE pgvector Redis + Celery Mistral AI Leaflet.js HTMX DRF PORT: :8000 APPS: 28 Django apps STATUS: Implemented

All 28 Apps

App Purpose Key Models Celery Tasks
app_step1_rss RSS/Atom feed collector RSSSource, RSSEntry poll_rss_feeds
app_step1_website Website scraper (Playwright / requests) WebsiteSource, ScrapedPage scrape_websites
app_step1_facebook Facebook public page collector FacebookSource, FacebookPost poll_facebook
app_step1_x X / Twitter collector XSource, XPost poll_x_feeds
app_step1_flot API / Webhook ingestion endpoint FlotSource, FlotMessage —
app_step1_capture Manual upload (PDF, images, text) CaptureDocument, CaptureFile —
app_step2_osint 6-stage OSINT pipeline (ingest, extract, language, relevance, NER, classify) OSINTItem, ProcessingStage, Pipeline run_pipeline process_stage
app_step2_extract Content extraction & normalization ExtractedContent extract_content
app_step2_language Language detection & translation LanguageResult, Translation detect_translate
app_step2_relevance LLM-based relevance scoring & gating RelevanceScore score_relevance
app_step2_ner Named Entity Recognition (LLM) Entity, EntityMention extract_entities
app_step2_classify Taxonomy classification Classification, TaxonomyTag classify_item
app_step3_situation Situation Room dashboard SituationView, Alert, Bookmark —
app_step4_intell L1 synthesis — rolling draft cache DraftCache, TokenAccumulator, L1Report check_accumulator refine_draft
app_step5_level2 L2 aggregation (organic, temporal, dimension) L2Report, AggregationRun aggregate_l2
app_step6_level3 L3 strategic synthesis (weekly + structural) L3Report, StrategicBrief generate_l3_weekly generate_l3_structural
app_intell_embedding Vector embedding generation (pgvector) Embedding, EmbeddingBatch generate_embeddings
app_graph Apache AGE knowledge graph GraphNode, GraphEdge (AGE) update_graph
app_map Leaflet.js geospatial map layer MapLayer, GeoEntity, MapMarker —
app_chat Chat interface for analyst interaction Conversation, Message —
app_settings Application configuration & provider settings Setting, ProviderConfig —
app_account User accounts & access control Account, Profile, Role —
app_api REST API endpoints (DRF) — (serializers only) —
app_taxonomy Classification taxonomy management Taxonomy, TaxonomyNode —
app_cost LLM cost tracking & budget enforcement CostRecord, Budget, ProviderLimit check_budget
app_celery Celery configuration & task monitoring TaskLog, PeriodicSchedule —
app_log Structured logging & audit trail LogEntry, AuditEvent —
app_export Report export (PDF, DOCX, JSON) ExportJob, ExportTemplate generate_export

Intelligence Synthesis Patterns

Rolling Draft Cache (L1)

The L1 synthesis layer uses a three-tier cost strategy to produce continuously updated intelligence drafts without wasteful LLM calls.

  • Token Accumulator (free): Every 10 minutes, a Celery beat task checks if new items have been ingested since the last draft update. If the token delta is below threshold, no LLM call is made — zero cost.
  • Incremental Refinement (cheap): When the accumulator threshold is hit, Mistral Small is called to merge new content into the existing draft. Fast, low-cost, runs as needed.
  • Weekly Finalization (quality): Once a week, Mistral Large rewrites the draft into a polished L1 report. This is the only expensive call, amortized across the entire week.
no-LLM check: 10min Mistral Small: on threshold Mistral Large: weekly

Cost Control

LLM usage is tightly governed to keep the entire platform under a monthly budget ceiling.

  • Relevance gating before NER: Stage 4 (Relevance) scores every item before Stage 5 (NER). Low-relevance items are dropped, saving the most expensive LLM calls for content that matters.
  • Monthly budget ceiling: $22/month hard cap enforced by app_cost. When the ceiling is reached, pipeline pauses and alerts the operator.
  • Per-provider limits: Each LLM provider (Mistral, OpenAI, etc.) has its own configurable sub-ceiling to prevent any single provider from consuming the entire budget.
$22/month ceiling per-provider limits relevance gating
Implementation Status: aesop_intell is the only microservice in the AESOP platform that is already implemented and operational. The full OSINT pipeline, all six Step 1 collectors, the L1/L2/L3 synthesis layers, embedding generation, knowledge graph, Situation Room dashboard, and cost control are all functional. The remaining work is the REST API layer needed for inter-service communication with aesop_modelisation and aesop_simulation, which are currently in the architecture/design phase.