SpeakesQuery – Splunk-style search over local Parquet, with LLM pipes

Local-first · zero telemetry · zero cloud dependency · AI when you want it, budget-gated when you do

Quick Start • Features • Docker • Query Syntax • Application Guide • Email Setup

One query language - SPQL - over local Parquet and SQLite. Ingest anything on a schedule, search it with pipes, rank it semantically, hand it to an LLM mid-pipeline, and get analyst briefs in your inbox. Everything runs on your machine: no accounts, no telemetry, no cloud dependency. (The optional AI features call the Anthropic API or a model on your own GPU - your choice.)

index="indexes/polymarket/active_markets/*" earliest=-7d

| nearest "surprise fed rate decision" topk=20

| llm model="claude-haiku-4-5-20251001"

prompt="One line: why could this move markets this week?"

max_cost_usd=0.10

| table question, yes_price, volume, _llm_output

One pipeline: pull a week of prediction-market data, rank it by meaning (local embeddings, no API), ask an LLM to explain the top hits - with a hard 10-cent ceiling - and lay it out as a table.

Project ethos - SpeakesQuery is intentionally designed as non-rent-seeking software. It exists to be transparent, inspectable, and useful on its own merits. It survives only through correctness, clarity, and trust - not artificial restrictions or gated capability.

Prerequisite: Docker installed and running. That's it.

git clone https://github.com/13alvone/speakesquery.git

cd speakesquery

./install.shOne command. install.sh verifies Docker, generates a .env with secure defaults, builds the image (all Python deps, C++ components, system libraries), starts the container, and opens SpeakesQuery in your browser (it prints the tokenized URL that authenticates your session). Your data persists across rebuilds in indexes/ and lookups/.

Query something in your first minute. Every install ships 30 days of sample app logs at indexes/sample/app_logs/ - the Query page shows five one-click starter queries (raw events, error rates by service, a 30-day timechart, latency buckets, and a regex field extraction). No connectors, API keys, or waiting for schedules required.

./install.sh --stop # Stop SpeakesQuery

./install.sh --status # Check container status

./install.sh --rebuild # Force a full rebuild (no cache)

./install.sh --port 8080 # Use a different portUpdating a running deployment

./update.sh # stop + remove + install.sh

./update.sh --pull # git pull --ff-only, then stop + remove + install.sh

./update.sh --rebuild # stop + remove + ./install.sh --rebuild (no cache)

./update.sh --dry-run # trace the plan without executing anythingupdate.sh autodetects whether sudo is needed, treats a missing container

as "already cleaned up" rather than an error, takes a pre-update backup of

user data, and forwards any extra flags verbatim to install.sh - so

./update.sh --pull --rebuild --port 5112 does the full

pull → stop → rm → rebuild cycle in one invocation.

Local development (without Docker)

For contributors or advanced users who prefer a local Python environment:

Prerequisites: Python 3.12 - 3.14 (3.14 recommended; it matches the Docker image).

./setup.sh --recreate-venv

source env/bin/activate

python desktop_app/main.pysetup.sh creates a virtual environment, installs all dependencies, and generates a .env file. See ./setup.sh --help for all options.

To launch all services (server, query engine, ingestion engine):

./run_all.shQuery Engine - Execute queries against local Parquet and SQLite indexes using SpeakesQuery's custom language. Results are displayed in a paginated table with CSV/JSON export, directory tree browsing, and native file dialogs.

Semantic Search - | nearest "query text" and | dedup_semantic SPQL pipes backed by a local sentence-transformers model (all-MiniLM-L6-v2, 384-dim). A background sweeper pre-computes embedding sidecars for every index so semantic queries stay fast; everything runs locally - no embedding API calls. See Semantic Search.

Notebooks - A cell-stream workspace where each cell's output feeds the next (SPQL, LLM, markdown, chart cells), with reactive content-hash caching so iterating on a prompt only re-runs what changed. The promote_to_alert_group cell renders a dry-run preview in place and deploys to a production alert group with one explicit click. Exports to HTML/PDF. See Notebooks.

Visual Builder - A drag-drop pipeline canvas backed by the same SPQL grammar with lossless round-trip to query text - the on-ramp for teammates without SPQL fluency. See Visual Builder.

Lookup Management - Upload, download, preview, and delete reference data files (CSV, JSON, TSV, Parquet) through the UI. Lookups augment index queries via | lookup directives.

Script Library - A curated, tested collection of 131 premade ingestion scripts (97 need no API key at all) spanning markets (Polymarket, Kalshi, Manifold, Metaculus), economics (FRED, BLS, Treasury yields), securities (SEC EDGAR, options chains), crypto (CoinGecko), news/events (federal register, GDELT, Hacker News), and more. Each script is preview-able from the UI and deployable with one click; trust-tier _pro variants opt into scipy / scikit-learn / rapidfuzz for heavier statistical analysis. The Massive.com options suite (IV rank, term structure, skew, earnings-implied move, unusual activity) anchors the Options Edge Brief.

Data Ingestion - Create, test, and schedule Python3 ingestion scripts from the UI. Scripts run in a RestrictedPython sandbox with a curated module allowlist (json, re, hashlib, base64, collections, io, bs4, lxml). Output is written atomically (.tmp → rename) to gzip-compressed Parquet files. A periodic maintenance job compacts small files and enforces disk limits. Per-execution resource budgets (max rows, max bytes, timeout) scale with the schedule interval - shorter intervals get tighter limits, longer intervals get more headroom.

Web Scraping - Ingestion scripts can use BeautifulSoup and lxml to scrape web pages in addition to REST APIs. Scripts that import bs4 are automatically classified as scrapers and enforce a minimum 4-hour schedule interval to protect both your machine and remote servers. The HTTP response cache deduplicates repeated fetches within a single execution.

In-Browser Python Linter - The ingestion script editor includes a live Pyflakes linter (via Brython) that validates syntax keystroke-by-keystroke with no server round-trips. Autocomplete covers the full sandbox allowlist. The "Test Code" button runs the script in the actual RestrictedPython sandbox and returns structured, actionable error messages - no raw tracebacks, no information leaks.

Credential Vault - API keys are Fernet-encrypted at rest in credentials.sqlite. The master key lives outside the repo (~/.speakes-query/master.key, 0600 permissions). Keys are decrypted only for the duration of a single script run. Credentials are also manageable from the Settings page in the app.

LLM Pipes & Model Registry - | llm, | llm_batch, and the advanced pipes (llm_route cost cascades, llm_refine drafter/critic loops, llm_ensemble voting, llm_until convergence) make LLM calls composable SPQL stages. A YAML model registry + provider-agnostic router dispatch to the Claude API or to self-hosted models (Ollama, LM Studio, llama.cpp - any OpenAI-compatible Chat Completions server) - so a GPU box on your LAN runs your daily briefs at $0 marginal cost. Every billable pipe supports max_cost_usd= budget gates and dry_run=true cost previews, and a content-hash cache makes idempotent re-runs free. See LLM Pipes.

Alert Groups - Combine up to ten saved search results into a single Claude API dispatch with a reusable boilerplate prompt template. Results are serialized, row-capped, token-estimated, and delivered as an analyst brief via email. Supports cron scheduling, per-group timezones, and manual triggers. When a dispatch fails (missing API key, Claude outage, SMTP failure) an admin gets a plain-text failure email - routed to a dedicated admin_error_email so paid mailing lists never receive operational notices - and the alert-groups page surfaces a "Last run" pill with the specific error. Disabled groups remove their scheduled job AND short-circuit at dispatch time (defense-in-depth against billing leaks). See the Alert Groups Guide.

Options Edge Brief (OEB) - A once-daily options-trading brief that surfaces 5–10 picks per dispatch across five signal classes (IV rank, term structure, skew, pre-earnings implied move, unusual activity). Each pick is rendered at three difficulty tiers (BEGINNER / INTERMEDIATE / ADVANCED) on the same underlying thesis, and every pick computes the minimum account size it fits at ≤2% sizing - so a small account never ends up oversized on a high-premium contract. A deterministic mark-to-market tracker grades every closed pick against the rules-as-they-existed-at-entry (no hindsight), and a weekly Claude review aggregates outcomes into hit-rate, calibration, and rule-tweak observations. Picks + closures + reviews land in a protected indexes/IMMUTABLE/ tree that's never garbage-collected and never permitted to drop a column - a decade-horizon trading record by design. See the Options Edge Brief Guide.

Claude API Test Button + Cost Audit - Settings has a Test Claude button that fires a minimal probe to verify your key, network, and SDK wiring. Every Claude call (from anywhere in the app) flows through a single wrapper with retry, hard timeout, and dual logging. A dedicated claude_api_history.sqlite keeps full request + response payloads forever (you manage its retention manually), and a lightweight indexes/logs/claude_api/*.parquet stream lets you SPQL-query costs in real time: index="indexes/logs/claude_api/*.parquet" | stats sum(cost_usd) by model. See Claude Analyzer.

Scheduled Searches - Promote ad-hoc queries into recurring search alerts with cron schedules, configurable lookback windows, and email notifications. Configurations are stored as YAML files. Deleted searches are archived in last_chance.sqlite for 30-day recovery.

Email Alerting - Scheduled searches send email notifications via aiosmtplib when results are found. SMTP credentials can be configured from the Settings page in the app or via environment variables. Gmail with an App Password is the default and recommended setup for most users. See the Email Setup Guide.

Email Groups & Macros - Reusable @group_name mailing lists resolved by every email-send path, and user-defined SPQL macros (YAML) with parameters for query reuse. See Macros.

Schedule Operations - A day-of-week × hour heatmap of every cron-scheduled job (firing counts + expected data volume), recent-activity charts, and a one-click branded PDF operations report with per-alert-group feeder health and anomaly buckets (failing runs, never-ran, empty output, latency outliers). See the Application Guide.

Logs Index - Config changes, scheduled search runs, alert group dispatches, Claude API calls, ingestion tasks, and system lifecycle events all land in indexes/logs/<category>/*.parquet - queryable with SPQL like any other index. The logs tree has its own max_logs_size_gb budget (default 5 GB) independent of the main indexes/ budget, so noisy logging can never evict your ingested data. See Logging.

Global Settings - All tunables (disk limits, timeouts, retry counts, cleanup intervals, allowed API domains, SMTP configuration, Claude retry + timeout + history retention, logs budget) are configurable from a Settings page in the app. Settings persist in global_settings.yaml.

SpeakesQuery is designed to run via Docker. The recommended path is ./install.sh (see Quick Start), which wraps Docker Compose with preflight checks and environment setup. Volumes persist indexes/ and lookups/ across rebuilds; the port is configurable via the PORT environment variable (default: 5111).

Localhost-only by default, token-gated beyond it. The container's host port mapping binds to

127.0.0.1: a fresh install is reachable only from the machine it runs on. On top of that, every Docker install is protected by a generated access token (the Jupyter model) -install.shprints the ready-to-open?token=URL, and requests without the token get a 401. SpeakesQuery is a single-operator app (one token, full control - no multi-user roles), so exposing it on a network is an explicit opt-in (BIND_ADDR=0.0.0.0in.env). Read Network Exposure and LAN Access first, and never expose it to the public internet.

Manual Docker usage

Run

./install.shat least once before using the compose file directly. The compose file bind-mounts a dozen state files (global_settings.yaml,*.sqlite) and data directories from the project root.install.shcreates them; if they don't exist whendocker compose upruns, Docker creates each missing file mount as a root-owned directory, which corrupts the SQLite stores and leaves the container in a crash-restart loop.install.shalso creates.env, which the compose file requires atuptime (all of its content is optional - see.env.example); if you skipinstall.sh, runcp .env.example .envyourself first. After the first./install.sh, the manual commands below are safe for day-to-day control.

docker compose -f desktop_app/docker-compose.yml up --build -d

open http://localhost:5111Or without Compose:

docker build -f desktop_app/Dockerfile -t speakesquery .

docker run -d --name speakesquery-desktop -p 127.0.0.1:5111:5111 \

--env-file .env --restart unless-stopped speakesquerySpeakesQuery is a single-operator, local-first tool and its threat model says so out loud: RestrictedPython is treated as a hardening layer, not a security boundary; the opt-in _pro script tier is arbitrary code execution by design (an honest trust label instead of a pretend sandbox); and the credential vault's encryption key lives on the same machine as the app, so script provenance - not containment - is the load-bearing control. Defaults are hardened accordingly: loopback-only binds, an auto-activating access-token gate on any non-loopback bind, an outbound domain allowlist for sandboxed scripts, and secret redaction on every LLM call recorded to history. Read the full layer-by-layer analysis, including what each defense explicitly does not stop, in docs/lang/24_threat_model.md.

flowchart LR

A["🌐 APIs · feeds · scrapes 131 connectors (126 core)"] -->|"sandboxed ingestion (RestrictedPython)"| B[("🗄 Parquet + SQLite <code>indexes/</code>")]

B --> C{{"⚙️ SPQL engine 57 commands · DuckDB pushdown"}}

C --> D["🖥 Desktop UI notebooks · visual builder"]

C --> E["⏰ Scheduled searches email alerts"]

C --> F["🤖 Alert groups Claude or local-LLM briefs"]

Directory layout

desktop_app/

main.py Native pywebview desktop application

server.py Flask server for headless/containerized use

ui.html Single-page interface (includes Brython-based Python linter)

Dockerfile Container image (Python 3.14-slim)

docker-compose.yml Compose config (context: project root)

query_engine/

QueryEngine.py Scheduled search executor

Alert.py Async email delivery via aiosmtplib (STARTTLS, certifi CA)

Scheduler.py APScheduler-based cron driver

Database.py Search history and result storage

scheduled_input_engine/

engine.py Background scheduler (APScheduler, 4-thread pool)

executor.py RestrictedPython sandbox with safe import/getattr

subprocess_runner.py Isolated process execution with resource budgets

parquet_writer.py Atomic gzip-compressed Parquet writes

cache.py Per-execution HTTP response cache with budget tracking

credentials.py Fernet-encrypted credential vault

cleanup.py Single-pass disk enforcement and file compaction

store.py SQLite CRUD for ingestion task configs

script_library/ Premade ingestion scripts (JSON metadata + Python3 code)

handlers/ Query command handlers (search, eval, stats, strings, lookups, charts, …)

functionality/ Shared infra (DuckDB index calls, datetime parsing, log writer, atomic writes, embeddings)

lexers/ ANTLR4 grammar and generated parser

saved_search_store.py YAML-based scheduled search CRUD with last_chance.sqlite recovery

global_settings.py Thread-safe YAML-backed settings singleton

Measured on an Intel i7-8809G (8 logical cores, 30 GB RAM, Python 3.14.4, Linux) against a synthetic 1 GB corpus of realistic application-log data: 5,704,248 rows across 17 gzip Parquet files (103 MB on disk). Every pipeline runs end-to-end through the real SPQL engine (ANTLR parse, handler chain, DuckDB predicate pushdown); 3 runs each, median reported.

Honest readings, good and bad:

- Time bounds are the fast path. The 2.6 s time-bounded scan vs the 12.8 s full scan is DuckDB predicate pushdown doing its job - always constrain earliest=/latest=when you can. Most real usage (scheduled searches, alert feeders) is time-bounded by construction.

- rexover millions of rows is the known slow spot. Regex extraction runs row-wise in pandas, so ~5.7M rows costs minutes, not seconds. Filter first (- search,- where, time bounds) so- rexsees thousands of rows instead of millions - or use- | sqlwith DuckDB's vectorized- regexp_extractfor large-corpus extraction. This number is published rather than hidden because surfacing it is the point of the harness.

- Designed scale: one person's accumulated data streams on one machine. There is no distributed story and no pretense of one.

Reproduce on your own hardware with one command (generates the deterministic corpus, runs the benchmark, prints this table, removes the corpus):

python -m tools.benchmark_corpus --generate --size-gb 1 --run --cleanupThe corpus is seeded and epoch-fixed, so two machines running the same command benchmark identical data. Pass --json results.json to keep the full machine-context report.

Environment variables load in order: ENV_PATH → ./.env → shell environment.

Email: the easiest way is the Settings page in the app (fill in SMTP credentials, click Save, then Send Test Email). For environment-variable configuration (useful for Docker or CI):

SMTP_USER="your-gmail-address@gmail.com"

SMTP_PASSWORD="your-16-char-app-password"

Optional overrides: SMTP_SERVER (default smtp.gmail.com), SMTP_PORT (default 587), SMTP_STARTTLS (default true), SMTP_FROM (defaults to SMTP_USER). Missing SMTP variables never block startup - errors surface only when a send is attempted. Step-by-step Gmail App Password instructions live in the Email Setup Guide.

Server binding: by default SpeakesQuery binds to 0.0.0.0:5111 (reachable on your LAN). Set HOST=127.0.0.1 in .env to restrict to localhost only; PORT changes the port.

source env/bin/activate

pytest -vv # 5,600+ tests

flake8 --exclude=env

bandit -r .ci_setup.sh handles dependency installation and component builds for CI environments.

Regenerating the parser

The grammar is defined in lexers/speakesQuery.g4. To regenerate:

antlr4 -Dlanguage=Python3 -no-listener -visitor \

-o lexers/antlr4_active lexers/speakesQuery.g4Note: You need ANTLR 4.13+ installed. On macOS:

brew install antlr. On Linux: download from antlr.org and alias accordingly.

Versioning policy

SpeakesQuery follows Semantic Versioning 2.0 (MAJOR.MINOR.PATCH[-prerelease]):

- PATCH (e.g., 0.9.1): Bug fixes, documentation updates, no behavior changes

- MINOR (e.g., 0.10.0): New features, backward-compatible

- MAJOR (e.g., 1.0.0): GA release, potential breaking changes

- Pre-release tags: -alpha,-beta,-rc.1

The current version is stored in the VERSION file at the project root and served via the /api/version endpoint.

Current status: v1.0.0-rc1 - release candidate for the 1.0 GA. The full strategic roadmap - four bets, six phases, ~24 months - lives in ROADMAP.md.

SpeakesQuery is authored infrastructure, not a productized service. The core engine is open, inspectable, and intended for real-world use. Attribution matters - professional credit enables accountability and future work. Commercial use is welcome; erasure of authorship is not.

Apache License, Version 2.0. See LICENSE and NOTICE for full terms and attribution requirements.