I have a small project where I needed to label commits as either “maintenance” or “new development”. The obvious way to do it is with a cheap but relatively capable llm, like gpt 5.6 Luna. I tested it on a small set of commits and manually verified its labelling, and it emitted the same label as I would have for the entire test set. That was good enough for me to roll out on a wider scale.
If we have Simon Willison’s llm cli tool installed (and you should – it’s
great!), we can call it in a pipe from Perl, and read its response. My script
had a loop that retried the request a few times1 I have experience of
models sometimes failing to heed output format instructions, which is usually
solved by retrying once or twice. My loop bailed at five attempts, and I’m not
sure it was ever needed. Luna is a more capable model than those that have had
trouble following output format instructions., but without that bookkeeping,
the code for this is simple enough.
sub classify {
my ($msg) = @_;
my $pid = open2(
my $output,
my $prompt,
'llm -m openrouter/openai/gpt-5.6-luna'
);
print $prompt prompt_template($msg);
close $prompt;
# Slurp the entire response and chomp off the
# trailing newline.
chomp(my $result = do { local $/; <$output> });
waitpid($pid, 0);
return $result;
}
The drawback of this approach is that I wanted to label some 21,000 commits, and this shells out to the llm for every single commit. Since the prompt is designed to contain mainly input tokens and not many output tokens (see Appendix C for details), these calls are cheap in money, but the latency is no fun, at around 1.5 seconds per call.2 If I actually cared about latency, I would not use Perl to call a Python cli that asks OpenRouter to send my request to OpenAI, but make the request directly to OpenAI. In this article I will pretend to care about latency, but it’s really more about sharing this cool technique.
Some commits are obviously new development, or obviously maintenance. It would be nice if we could classify those through a more primitive – and faster – way, and only ask the llm about the more difficult cases. Here’s the pseudo-code for such an algorithm:
sub labeling {
my (msg) = @_;
# Take the word frequencies of the commit message
# as the features for the fast classifier.
my $features = bag_of_words($msg);
# Use the fast classifier when it has seen enough
# training samples, but skip it sometimes to
# prevent accidentally learning p=0 or p=1 based
# on long runs of a single label.
if ($fast_classifier->{samples} > 50 && rand() < 0.95) {
my $p = $fast_classifier->predict($features);
# Use the labels predicted by the fast
# classifier only if it is very confident.
return 'maintenance' if $p < 0.08;
return 'new development' if $p > 0.92;
}
# If we have insufficient samples or fall through,
# use the LLM classifier to get a label.
my $label = classify($msg);
# Since we got a real label, use it to train the
# fast classifier.
$fast_classifier->refine($label, $features);
return $label;
}
The question is what object implements the interface we have assumed for the fast classifier. We have called two non-trivial methods on it:
- predict($features): return a probabilistic prediction of the features being labelled as new development.
- refine($label, $features): add the association between the features and the label to the training set of the classifier.
An obvious choice is naïve Bayes. The drawback is that it assumes independence between features, which generally won’t be the case for us. Another alternative is a logistic regression.
I learned for this project that a logistic regression can be trained by streaming it one label at a time, and using gradient descent to nudge its weights toward a better fit. I haven’t fully worked through the derivation, but this is not that complicated as far as these things go.3 I mean obviously, yes, it’s complicated. Just look at the maths on that page! But in terms of closed-form gradient descent derivations, this is probably one of the simpler ones. And the final result is just 3–10 lines of code depending on how you count. I wish I knew this years ago! It’s so elegant.
package LR {
# Constructor creates new logistic regression with
# default learning rate, no regularisation, and
# an empty set of initial weights. All three
# are overrideable by passing them to the
# constructor.
sub new {
my ($class, %opt) = @_;
my $self = bless {%opt}, $class;
$self->{weights} //= {};
$self->{learning} //= 0.1;
$self->{regularisation} //= 0;
$self->{samples} = 0;
return $self;
}
# Compute the log-odds of the positive label.
sub score {
my ($self, $features) = @_;
my $z = 0;
# Default to zero weight for unseen features.
$z += ($self->{weights}{$_} // 0) * $features->{$_}
for keys %$features;
return $z;
}
# Compute the probability of the positive label.
sub predict {
my ($self, $features) = @_;
# Convert log-odds to probability.
return 1 / (1 + exp(-$self->score($features)));
}
# Gradient descent on one label.
sub refine {
my ($self, $label, $features) = @_
# The coefficient of dLL/dw_i for any weight.
# The negative of this is used to compute the
# gradient of the log-loss.
my $dlldw =
($label eq $self->{positive} ? 1 : 0) -
$self->predict($features);
# Descend the gradient of log-loss at learning
# rate speed, defaulting unseen weights to zero.
$self->{weights}{$_} =
($self->{weights}{$_} // 0) -
$self->{learning} * -$dlldw * $features->{$_}
for keys %$features;
# Regularise by pulling all weights toward zero.
$self->{weights}{$_} -=
$self->{regularisation} * $self->{weights}{$_}
for keys %{$self->{weights}};
# Record that we have seen more samples.
$self->{samples}++;
}
};
And that’s it! The pseudo-code from before becomes actual, working code once we
instantiate this class and store it in $fast_classifier.
Let’s step back and see what happened:
- We have an expensive classifier that gets labels right.
- We have a cheap classifier that stands in front of the expensive classifier and gets dibs on calling the label first.
- When the cheap classifier is confident, we accept its label.
- Otherwise, we ask the expensive classifier for a label.
- Then we also use that label to train the cheap classifier.
This took less than 40 lines of code for all of it, logistic regression and everything. Without any advanced maths libraries. I did not expect us to get away with so little code when using plain Perl.
A logistic regression is generally calibrated when fully trained, but some things can knock it out of calibration, such as regularisation (which my instance had), or small data (which it also has early on in the process).4 When I checked, this showed up as a 0.2 log-odds bias toward the positive label, and log-odds 10 % more extreme than desired. These are small effects in the grand scheme of things, and we probably shouldn’t care about them, but since we’re having so much fun anyway we might see what we can do.
A common way to improve the calibration of classification models is through Platt scaling, which means fitting another logistic regression on top of the output of the first one to neutralise the bias and noise. Doing this complicates the labelling loop a bit, because the logistic regression will be overfit on its training data, so we need an independent judge (a separate stream of data) to train the scaler. We can implement this by siphoning off, say, 10 % of the training samples to the Platt scaler instead.
sub labeling {
my ($msg) = @_;
my $features = bag_of_words($msg);
my $z = $fast_classifier->score($features);
my $p = $platt_scaler->predict({_INTERCEPT => 1, z => $z});
# Use a lower threshold of Platt scaler samples
# because by the time the scaler has seen this many
# samples, the classifier has seen even more.
if ($platt_scaler->{samples} > 10 && rand() < 0.95) {
return 'maintenance' if $p < 0.08;
return 'new development' if $p > 0.92;
}
# If we have insufficient samples or fall through,
# use the LLM classifier to get a label.
my $label = classify($msg);
# Start by feeding training data only to the fast
# classifier. Once it has seen a few samples, use
# every tenth label to train the Platt scaler
# instead.
if ($fast_classifier->{samples} > 10 && rand() < 0.1) {
$platt_scaler->refine($label, {_INTERCEPT => 1, z => $z});
} else {
$fast_classifier->refine($label, $features);
}
return $label;
}
Both of $fast_classifier and $platt_scaler are instances of that LR type
we made! The feature observed by the Platt scaler is the raw log-odds emitted by
the fast classifier. (This was why the score() function was exposed separately
in that class.)
As written, this runs like a black box, so we’ll likely want to add a routine that regularly prints statistics about what’s going as the script is working. When we do, we’ll find that it performs well. Some of the diagnostic output I added assigns predictions into ten evenly spaced buckets, and prints the error for each, i.e. the difference in average assigned probability and actual average probability for each bucket.5 This loses us some more training data, unfortunately, because we don’t want to evaluate the final predicted result on either regression’s training data!
0.05: -0.02 0.15: 0.03 0.25: 0.10 0.35: 0.04 0.45: 0.04 0.55: 0.01 0.65: 0.01 0.75: -0.08 0.85: 0.00 0.95: 0.02
This tells us e.g. that the predictions in the range 20 % to 30 % are on average 10 percentage points too high, meaning the corresponding events happen slightly less often than predicted. On the other hand, the opposite is true for predictions in 0 % to 10 % range, which are on average 2 percentage points too low, i.e. the corresponding events happen a teensy bit more often than predicted. However, all of these numbers are within the margin of error that would be expected for however many samples these calibration numbers were drawn from, meaning there’s no significant deviation from calibration.6 Had I been less lazy I would have made the diagnostic print also show the p-value or something, for each bucket alone, and the combination of them.
The fast classifier is used for roughly 46 % of the classification tasks in my case, so it roughly doubles the speed at which classification runs. Not amazing, but it was fun building it anyway. This is definitely a technique to keep in the back pocket for when there’s something that’s easier to classify, or when the expensive classifier is more expensive. It could easily speed up (and cheapen) the process by an order of magnitude or more.
Continue reading past comments for the appendices.
I enjoyed this article. I understand the main point is to show off the fast/slow classifier technique, but considering the occasional output format issues, I was wondering if you’ve seen Jev or the on-device equivalent Qwen-2.5-1B-rlcd?
Jev is an rlcd (reinforcement learning for calibrated decisions) model, so it’s been trained to produce a decision alongside a confidence score in a typed schema (rather than raw text). Qwen-2.5-1B-rlcd isn’t trained like this, it’s backed by a regular rlhf’d llm that calculates the token probabilities, but it still does constrained decoding to output json that fits a schema.
These models have cheaper api/compute costs (no quadratic autoregressive decoding), are faster to run, and you can consume the output as a structured json object rather than having to instruct an llm to output only a specific label (which doesn’t always work).
Unfortunately, there are no independent benchmarks yet as these are super new tech. But the training approach, plus schematic generation, seems promising.
For this use-case, you might ask for a choice:
{
"state": "|feat: add email notifications on new direct
messages|",
"questions": {
"commit_type": {
"type": "choice",
"instructions": "Is this commit new development
work or maintenance work?",
"criteria": {
"new_development": "Things that are considered
new development: adding features, adding cli flags,
adding api endpoints, removing limitations on usage of
features. New development is anything developed because
it was requested by a user to help them perform their
job to be done.",
"maintenance": "Things that are considered
maintenance: bugfixes, performance improvements,
removing features, adding diagnostics, stability
improvements, observability improvements, security fixes.
Maintenance is any development that needs to happen in
support of the features the user need to perform their
job to be done."
}
}
}
}
Then the model would respond with a JSON response with the same structure as the question schema. This output taken from a test run at the Jev playground:
{
"model": "jev-1.13.0",
"answers": {
"commit_type": {
"type": "choice",
"choice": "new_development",
"confidence": 1,
"probabilities": {
"maintenance": 0,
"new_development": 1
},
"stats": {}
}
},
"usage": {
"input_tokens": 423,
"output_tokens": 40
},
"request_id":
"playground_1f164a52e7a8fbc489d99873e20ea1f69ab",
"evaluation_time_ms": 109.0331659943331
}
I’m not sure how the additional information (confidence and probabilities) would impact the logistic regression. I suppose you could weight each observation in the update by the confidence (i.e. confidence of 0.5 means you only update the classifier half as much as a data point with confidence 1), and the probabilities would be treated as soft labels. But I am no statistician!
I think another interesting augmentation would be to move from a binary to a multinomial logistic regression. For the git commit classification schema, I’m thinking of conventional commits, which suggest a breakdown of commit types like: feat, fix, refactor, perf, style, test, docs, build, ops, chore.
Anyway, this article was a great read! The fast/slow classifier approach is super cool, I’ll have to keep it in mind in case I ever need to use machine learning tech at work.
(Julian Ferrone writes occasionally at their website.)
I thought it was interesting to peek into the fast classifier to see how it discriminated between maintenance and new development commits. It was trained on commit messages as bags-of-words, so it’s basically associating individual words with the probability of that word occurring in the commit message for new development work, and vice versa.7 Though unlike naïve Bayes, the logistic regression accounts for some interdependence between words.
Here are words commonly used in maintenance commits:
- fix (or fixes, fixed)
- use
- move (or refactor)
- cleanup (or remove)
- bug
On the opposite side, these words are often used in commits with new development:
- add (or implement)
- support
- allow
- command (or directive)
- parameter (or option, api)
For my project, I trained the classifier separately on multiple open source repositories, and it learned these same words for all of them. It seems like developers, even across different projects, use many of the same words when indicating something is new development or maintenance. Which also means training jointly on all repositories would have given me a better and more robust classifier, but that would have required rewiring some of the code I had already written and I can’t take the time.
Another optimisation, which probably had a greater effect than the fast
classifier in this article, is that all of those rand() calls in the code
above are actually calls to a custom function called commit_rand($commit)
which turns the commit hash into a uniform probability. This means the
randomness is always the same for each commit. This is particularly useful in
the loop that samples which commits to label from each repository, because it
will always sample the same commits every time it runs.
This, in turn, means it becomes meaningful to cache the labels generated by the
expensive classifier. The part of the labelling loop that does this replaces the
call to classify($features) with
my $label = $cached_labels{$commit->{hash}};
if (! $label) {
$label = classify($commit->{msg});
open(my $cache, '>>', "labels.csv");
printf $cache "%s,%s\n", $commit->{hash}, $label;
close($cache);
}
This stores the label for a commit immediately when it has been retrieved. When the script starts up, it reads all the cached labels from before into a dictionary.
my %cached_labels = ();
if (-r "labels.csv") {
open(my $cache, '<', "labels.csv");
while (<$cache>) {
$cached_labels{$1} = $2 if /^(.*),(.*)$/
}
close($cache);
}
printf STDERR "Reclaimed %d cached labels.\n",
scalar(keys %cached_labels);
Couldn’t be simpler!
This was the prompt I used for the llm classification. It is indicative of my typical prompting style, and I performed no separate evaluations of prompt variants since the first one I thought of was good enough.
You will be presented with a git commit message. Your job is to determine whether the commit is new development work or maintenance work.
Things that are considered new development: adding features, adding cli flags, adding api endpoints, removing limitations on usage of features. New development is anything developed because it was requested by a user to help them perform their job to be done.
Things that are considered maintenance: bugfixes, performance improvements, removing features, adding diagnostics, stability improvements, observability improvements, security fixes. Maintenance is any development that needs to happen in support of the features the user need to perform their job to be done.
==START OF COMMIT MESSAGE==$msg
==END OF COMMIT MESSAGE==It may not be clear whether this is new development or maintenance. Yet you must pick one label, whichever you feel fits best. Respond only with the label, either “new development” or “maintenance”. No other text is allowed.