Mosaic Eval Harness
Does a room of AI models beat the best single one? A harness that answers with real statistics.
7 LLM Providers
- Evaluates 7 distinct cloud and local model providers via unified adapters
- Features 10,000 paired bootstrap resamples for statistically valid capability uplift
- Field-level AES-256-GCM envelope vault for sensitive evaluation completions
# What it is
A local-first, production-grade LLM evaluation platform and statistical harness designed for biosecurity research. It evaluates whether combining multiple frontier and local LLMs produces statistically significant capability gains over standalone models across biosecurity proxy tasks.
# Why it exists
No single LLM dominates across virology, protein engineering, and chemical synthesis. Standard evaluations suffer from judge position bias, vendor self-preference, and OOM issues on multi-gigabyte benchmark datasets.
# How it works
A unified adapter mesh normalizes 7 providers (OpenAI, Anthropic, Gemini, Kimi, MiniMax, Mistral, local engines) behind one completion interface. Experiment runs persist to SQLite WAL via Drizzle; a Next.js UI orchestrates evaluations while a statistics layer computes paired-bootstrap confidence intervals and FDR-corrected uplift. Sensitive completions are sealed in an AES-256-GCM envelope vault.
# Decisions worth explaining
- 01
Non-parametric paired bootstrap over parametric significance tests
Model-comparison samples are small and non-normal; 10,000 paired resamples with FDR correction yields defensible capability-uplift claims without distribution assumptions.
- 02
Blind multi-judge ensemble with isotonic PAVA calibration
LLM judges exhibit position bias and vendor self-preference; blinding order and calibrating scores removes the synthesis-score bonus that skewed early runs.
- 03
Streaming JSONL ingestion with <50MB bounded memory
Multi-gigabyte benchmark corpora OOMed naive loaders; a streaming pipeline keeps the whole harness runnable on a laptop instead of requiring a server.
# Details
- 7-provider unified adapter mesh across OpenAI, Anthropic, Gemini, Kimi, MiniMax, Mistral, and local engines
- Non-parametric 10,000-iteration paired bootstrap confidence intervals and FDR multiple-testing correction
- Blind multi-judge ensemble with Isotonic PAVA calibration eliminating synthesis score bonuses
- Zero-OOM streaming JSONL ingestion pipeline with bounded memory footprint (<50MB)
# Built with
TypeScript · Next.js 16 · Drizzle ORM · SQLite WAL · LLMs · AI Safety