My implementation focuses on a local, open-weight specialist for short English text, trained on an RTX 3070 (8 GB of VRAM). Here is what I built and how I trained it:

System One accepts a state and a set of typed questions. Each question uses one of three primitives:

- choice: select an option and return its probability distribution.

- score: evaluate an ordered rubric and return an expected score.

- noul: return the probability that a statement is true/false.

The runtime produces structured responses directly from model scores. There is no autoregressive decoding or generated text to parse into JSON.

Starting with a pretrained encoder

I used ModernBERT-base (149M-parameter pretrained backbone), with LoRA adapters and a trainable decision head.

Each input contains the question, candidate options, and state. A [MASK] marker identifies each option. The decision head reads those positions and produces a logit for each candidate.

For multiple questions, the runtime builds a separate sequence per question and processes them together in one batch.

The complete sequence is capped at 512 tokens, including instructions and options. That constraint shaped the project around messages and bounded decisions.

Training in two stages

I first trained on human-labeled examples from three datasets:

- BANKING77 for banking intent classification.

- SMS Spam for spam detection.

- SST-5 for sentiment on a five-level scale.

This stage used cross-entropy, with an additional ranked probability score loss for ordered outputs. For sentiment, that gives the training objective a way to account for how far a prediction is from the correct level.

The configuration used LoRA rank 16, batches of four, and eight gradient accumulation steps. The recorded first training phase took approximately 35 minutes and used about 2.5 GB of VRAM.

Next came teacher distillation.

I used a frozen Qwen2.5-7B-Instruct model in 4-bit AWQ format to score support and email questions. Instead of asking it to generate answers, I extracted logits for tokens representing the available options.

That produced 13,480 teacher-labeled question–state pairs. I also generated 4,756 stress examples to expose the student to variations in the inputs.

The second training stage combined gold supervision, teacher probability distributions, and stress data. KL divergence trained the student to match the teacher’s distributions on soft-labeled examples.

Everything ran sequentially on the same GPU. I unloaded the student before teacher labeling and unloaded the teacher before student training.

Checkpoint achieved:

|---|---:|

| BANKING77, 77 intents | 88.5% |

| BANKING77, 8 coarse categories | 95.4% |

| SMS spam detection | 98.9% |

| SST-5 sentiment | 55.9% |

BANKING77 finished below my original 90% target.

The selective prediction result was more encouraging: when retaining the most confident half of BANKING77 predictions, accuracy reached 99.0%. At 80% coverage, it reached 95.9%.

That suggests a useful direction for automation: handle confident cases locally and escalate the rest. These are benchmark results; deployment thresholds still need validation on the actual workload.

In my constrained scoring setup, Qwen achieved 56.2% on a 500-example BANKING77 sample. The student achieved 88.5% on the full 3,076-example test split. Those are different evaluation sizes, and this does not establish a general ranking between the models. It does show why task-specific human supervision matters: the student had direct training on the target labels.

Teacher quality depends on the task and scoring setup. A larger parameter count does not automatically make its labels a better training signal.

That's all!

Code: GitHub

Weights: Hugging Face