Conversation
This was referenced Sep 6, 2026
evandhoffman
added a commit
to evanwtf/local-llm
that referenced
this pull request
Sep 6, 2026
antirez/ds4#991 puts the fork our whole ds4 stack stands on upstream. #141 closed as superseded by it -- its premise had already been retracted once, and two issues watching one PR is one too many. #120 goes back below the line: promoted this morning when #149 unblocked it, demoted the same day, which is what a queue with a fixed size does. The A/B read-out was still iterating a hardcoded ("on", "off") when collecting transcripts, so #146 -- whose arms are legacy and sandbox -- printed a conditional table with a heading and no rows under it. An empty table reads as "no failures", not as "wrong arm names", which is the kind of quiet wrong answer this script's tests exist to prevent. The arms now come from the manifest. Also records the #146 batch: 2 of 4 runs, one per arm, stopped by its own 09:15 cutoff.
evandhoffman
added a commit
to evanwtf/local-llm
that referenced
this pull request
Sep 7, 2026
A paired three-pair A/B, six sweeps and 90 rows on 2026-09-07, put the Q4_K
imatrix stack on ivanfioravanti's ds4 fork 16% faster than the same model on
llama.cpp at identical reliability:
paired pass, by TASK: 0 down, 0 up, 15 tied of 15 sign test p=1.000
wall: n_pairs 15 ratio 0.84 (95% CI 0.76-0.92) win/loss/tie 12/1/2
pair1 0.814 (ds4-led), pair2 0.806 (llama-led), pair3 0.862 (llama-led)
90/90 passing on both arms, 0 deaths
The rule that decided this was written down before the third pair ran, so the
threshold was not fitted to the result: CI must exclude 1.0, reliability must
not regress on the paired-by-task comparison, all three pairs must agree in
direction, and no void condition may stand. All four held.
Not a token-rate claim. ds4 took MORE work per trial -- 11 turns against 9,
33% more input tokens, similar output length -- and still finished sooner.
Two corrections to earlier drafts of this file:
- The memory penalty was written "~105 GB against 84 GB", comparing decimal
GB to GiB and overstating the gap by half. In consistent units it is 98 GiB
against 84 GiB. 32 GB of the ds4 side is the PLE sidecar, which is easy to
miss when downloading.
- The median-based reading published mid-run had llama.cpp ahead. A median
over fifteen unmatched tasks moves with which tasks sit near the middle;
the paired per-task ratio and the sweep totals agree with each other and
disagree with it.
llama.cpp keeps a named place in the note and its own install section as the
mainline fallback, because this stack loads only on a fork and antirez/ds4#991
is still open -- the durability risk #141 raises, and this file has already
had one stack withdrawn by its author. The fast row now carries an install
section of its own; recommending a stack with no instructions is half a
recommendation.
tasks.toml: the kimat backend description pinned commit bd9cfbc, which is not
what any of the 45 rows ran on. The branch is force-pushed, so the built
commit belongs in each row's env, not in a description that silently rots.
evandhoffman
added a commit
to evanwtf/local-llm
that referenced
this pull request
Sep 8, 2026
…wo stacks "This stack is one force-push from needing attention" was written as a hypothetical. It happened: ivanfioravanti/ds4-metal moved 149 commits ahead of the ba01f5d this project builds from, and ba01f5d is not an ancestor of the new head (#228). Which stack survives turns on one string. The Q4_K imatrix build declares general.architecture qwen4exp and loads on the new head; the Q4_0 fast-pack declares qwen4-exp and is refused. Each build refuses the other's file with the same deepseek4.block_count error, because the loader matches exactly one architecture string and validation falls through to DeepSeek otherwise. It fired in favour of the stack this file recommends. On the new head the imatrix stack is +6.2% decode and +7.7% prefill over the ffd85d42 these rows were measured at -- three runs, 0 of 8 frontiers against it in any of them -- and bit-exact: identical selected tokens and top-20 logits over 128 steps at 2047 and 16380 prompt tokens. Faster, same model, nothing about quality moves. For the Q4_0 build it is the end of the line rather than an inconvenience. It was already withdrawn by its author; it now also cannot follow the head without a re-quant or a re-declared architecture string. The "loads only on ds4-ivan-qwen38fn" line was already too narrow and is now wrong: it loads on ivanfioravanti's trees generally, including the newer ds4-metal head. What it does not load on is mainline ds4, which has no PLE sidecar support at all -- that is the fork dependency that actually binds, and it is the one #141 and antirez/ds4#991 are about. The advice added for a reader: record the commit you build, and check your weight file's general.architecture against the loader before assuming an upgrade is free. That check cost minutes and would have cost a day. Also committing the two verify-posts logs from today's source verification, which is where the #151 quotes were checked.
This was referenced Sep 11, 2026
Implement qwen4exp loading, tokenizer and chat formatting, PLE sidecars, Metal text and vision graphs, built-in MTP, and recurrent KV persistence. Integrate the CLI, agent, server, download targets, and conversion tools. Preserve the native Q4_1 encoder, Q4_K accumulation guard, and shared quantizer build target. Keep exact-MTP rewind and vision-cache row-width fixes. Persist valid logits and recurrent state at completed prefill chunks, and reuse Qwen text-answer history when tools clients omit reasoning. Retain exact reasoning replay and test continued checkpoints, cancellation, cache eviction, and restart reuse with and without reasoning_content. Include reproducible charts and measurements comparing throughput before and after the checkpoint fixes on M3 Ultra. Add Qwen trunk steering and prompt activation capture, including GDN layers. Keep the MTP predictor outside the 48-row bank and preserve prompt dumps through one-token prefill tails and multi-row speculative verification. Recover Chrome target creation and retain its last page target. Preserve full agent transcripts in stripped-session format when KV no longer fits, while retaining the newer automatic compaction and continuation logic. Add steering/MTP/capture, Chrome CDP recovery, and full-session save/reload regressions. Adapt the intended changes from fork PRs #13 and #7. Reconcile with antirez main bd66c40, preserving DeepSeek v4.1 model IDs, tool syntax, session state, and Metal kernel dispatch. Exclude superseded engine code and prototypes. Co-authored-by: Andreas Parusel <aparuael@7170c824-6152-4cf1-985e-9503832ce673.fritz.box> Co-authored-by: david <contact@davidberard.fr> Co-authored-by: Michael Kuebbeler <cropduster@users.noreply.github.com> Co-authored-by: Ogilthorp3 <bertrand@nepveu.name> Co-authored-by: anemll <realanemll@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.