This model makes calibrated, typed decisions about an image plus optional text. It answers choice, score and noul (yes/no probability) questions in one forward pass, with no text generation.
It adds image input to Laya by replacing Laya's ModernBERT encoder with SmolVLM-256M-Instruct. Laya's predict(state, questions) API, proper-scoring-rule training and temperature calibration are unchanged.
- Code: github.com/r33drichards/laya-vision
- Demo: thaitea/laya-vision-demo
- Status: experimental. This is an independent research fork, not affiliated with Convai Innovations, the authors of Laya.
git clone https://github.com/r33drichards/laya-vision && pip install -e ./laya-vision torchvision
import laya
from PIL import Image
agent = laya.load_vlm("thaitea/laya-vision-smolvlm-256m")
result = agent.predict(
{"image": Image.open("photo.jpg"), "note": "customer says it arrived broken"},
{
"damaged": {"type": "noul", "instructions": "Does the item in the photo look damaged?"},
"category": {"type": "choice", "instructions": "What kind of item is this?",
"criteria": ["electronics", "clothing", "furniture", "food", "other"]},
},
)
result["answers"]["damaged"]["noul"] # calibrated P(true)
result["answers"]["category"]["choice"] # top option; see ["probabilities"], ["confidence"]
Scores are on the full validation splits. "Calibrated" uses the per-type temperatures stored in vlm_agent_config.json, which are applied automatically.
* The VQAv2 train/val split is a re-split of the official VQAv2 validation set by image (the only official split with answers), so these numbers are not comparable to published VQAv2 results.
- Latency: about 71 ms for one image question on an NVIDIA L4 (bf16). The image is encoded once per predictcall and shared by every question.
- Option order: across 4 rotations of the A-OKVQA option order, accuracy varies by 0.7 points. Averaging over orders (n_permutations=4) doesn't help.
- Backbone: SmolVLM-256M-Instruct. The vision tower is frozen, and the language model and decision head are trained, about 150M parameters.
- How options are read: the question and state come first and the options are listed last. Each option is scored from the hidden state at the end of its own line.
- Loss: soft cross-entropy plus a strictly proper scoring rule (log + spherical), with the option order shuffled at random.
- Data:- A-OKVQA: 17k questions, choice
- ScienceQA image subset: 6k questions, choice
- VQAv2 yes/no: 50k questions, noul
- Datasets were sampled equally, and no dataset was repeated more than 6 times.
- A-OKVQA: 17k questions,
- Schedule: 6,780 steps at batch 32 on one A100 (about 33 minutes). Head LR 1.4e-4, backbone LR 2.8e-5, 3% warmup, then cosine decay. The checkpoint with the best mean validation accuracy was kept.
- Calibration: per-type temperatures fitted on the last 300 training records of each dataset, which were held out of training: choice 3.33, noul 1.69.
- training_metrics.jsonhas every evaluation from the run.
- scorequestions are untrained. There was no ordinal image data, so their outputs are meaningless.
- A-OKVQA overfits. Training accuracy reaches 97.6% against 61.8% on validation, and raw confidence is too high there, so rely on the calibrated outputs.
- Narrow domain. The model was trained on everyday photos (COCO) and science diagrams. Expect to fine-tune on your own data for other domains.
- One image per state. Each image is resized to a single 512 px tile.
- Text-only states work but are unevaluated. This model is not a drop-in replacement for Laya's text checkpoint.
The weights are released under CC BY-NC-SA 4.0, because they were trained partly on ScienceQA, which uses that license. The base model, SmolVLM, is Apache 2.0. A-OKVQA is Apache 2.0. VQAv2 annotations are CC BY 4.0, and its COCO images carry Flickr terms. The code is Apache 2.0.