If you’re someone who spends all day prompting agents, this is for you.

The increasing bottleneck to your progress is your ability to ingest, process, and act on extremely high information throughput. The effort you put into processing information, applying judgment, and crafting your next prompt is a tractable prediction task. A model with the right data could predict your writes in 1/1000th of the cost of the time you take to generate them, freeing you up to do much more.

We show progress towards this reality by converting computer use data into structured events that allow for training and evaluation. In a two-week, single-user study, Astra xhigh achieves 17.1% semantic accuracy on substantive next-write predictions at just $0.3 per call. Qwen 3.6-35B-A3B only achieves 0.9% accuracy, while our continually trained version of the same model achieves 3.0% accuracy at just $0.01 per call. Importantly, we show early data that suggests loss reductions scale with the amount of data, akin to “local” scaling laws, providing a basis for further investigation.

Links

Github Public notes Email X Labs

Get in touch if you’re interested in this direction, working on similar problems, or hoping to try out early versions!

Motivation

This is a very practical post. I’ve written about my theses in prior work, and many more eloquent writers describe the issue well. What’s the hard core of this problem? We have the models. We have the compute. No one wants to do the grunt work of collecting and cleaning the continual data that feeds personal models. We are fixing that.

Why does this matter? Multi agent systems can create outlier value but require heterogeneity. In-context heterogeneity is not enough. In-weight heterogeneity (a.k.a. judgment, a.k.a. the process of converting information to action in dynamic and non-verifiable domains) lacks data to bootstrap from. Our existing digital workflows provide a dynamic, continuous data resource for this! In fact, the only one. The issue is that capturing data, cleaning data, maintaining data pipelines, loading models onto GPUs, training them, evaluating them, hosting them, serving them well, and generating near term economic value to sustain development is a Herculean effort. The resulting negative externalities of this manifest in today’s practical workflows. Most knowledge workers today, instead of focusing on generating actual outcomes or thinking creatively about novel problems, spend their time and effort giving agents context, direction, and evaluation, and wait for their responses. This is massively wasteful.

At first, a solution that is able to predict what you write into your computer would improve the piloting of other long running agents by more rapidly collecting context and converting it into high taste prompts that reflect what you, in a slower and higher effort way, would do. Eventually, this ‘user model’ gets good enough to infer your goals and start making suggestions that you would consider better at achieving what you want to achieve that would otherwise have been higher friction to compose yourself or you would not have thought of (i.e. unbounding your rationality). Then, frontier models could work directly with your user model to complete tasking together before presenting it to you. Finally, a single combined agent could both distill your judgment and also apply it in a way that is faster and better at achieving your goals than you could alone. Every step builds upon the prior step, so a single combined frontier agent would still be continuously updating based on the data you generate continuously during your workday. Your presence remains necessary to give the superhuman capable machine a direction; it’s useless without it.

There are several phases towards making this a reality, and we’re very excited to unveil initial results from the first step. Over the past month, we’ve publicly iterated on our local data capture and cleanup pipelines that convert raw computer use data into structured SFT datasets to teach models to predict the user’s next write conditioned on the history of their read and write data. The goal here was to show that timely accuracy of predictions scaled positively with the amount of trained data. More specifically, that there existed a data capture and construction pipeline that, when fine tuning open source models, would produce a significant number of outputs that were considered equal or better than the human’s write in less time than the human took to produce it, and that this number improved as the amount of collected data trained on increased.

Data

The amount of work and iteration that has gone into, and will continue to go into, determining what tools are available on MacBooks to capture data and solidifying a deterministic ruleset that converts this raw data into something that supports actual learning during training in a way that can be genuinely used in real time, is large. It’s likely out of scope to describe the entire process here, but feel free to reach out if you’re building it for yourself or interested in learning the process. The full history of the work is in the open source repository. What is worth mentioning is the resultant pipeline at a high level, and what each stage does, courtesy of the agents that built it.

Believe it or not, I’ve reviewed each step manually and they’re necessary. I underestimated the work required to get data that sets up training for success, and there’s definitely more juice to squeeze.

Training

The next step after data construction is deciding the full training recipe. My initial experimentation focused on supervised fine tuning with token level cross entropy loss on the user’s writes, a 32k context with a simple sliding window of prior read and write events, non-reasoning Qwen 3.6-35B-A3B on Tinker, static learning rate, and a single training epoch. One important point to keep in mind is I believe strongly that local, personal models are most useful in non-verifiable, dynamic domains. The more often the model’s weights are updated, the more likely it will outperform static frontier intelligence at this task. For the purposes of the initial experimentation, I decided to update the model after every 50 examples. This changes how evaluation occurs. Instead of splitting the data randomly into a train set and test set, we train on the first 50 examples, then evaluate that model on the subsequent 50 examples. After, we train on the 50 examples we just saw, then evaluate on the next 50 examples, etc. Each example is associated with a single gradient descent step, and shown exactly once, in order of when they occurred in real time. This is to mimic the live setup, which will operate similarly, avoiding breaks in causality.

With all that being said, here are the results, with different ways of looking at the data.

Next Steps

Overall, this work gives us enough signal to attempt productization next. It was a major question whether this type of task would be predictable at all. Astra shows that it is. Trained Qwen shows that learning can occur and can scale with data, while the exact relationship between accuracy and data remains unclear. Under the assumption of rapidly increasing intelligence per dollar, this is the worst these results will ever be.

There are a ton of missing pieces and major open questions to collect further evidence about. Primarily, the existing dataset we have is still too small. Two weeks of data from a single person results in really wide error bounds on the extent to which performance scales with additional data. It’s also unclear what level of timely accuracy is required for genuine joy incorporating the product into your workflow; this can only be found by integrating it and monitoring results. Finally, the jump from 1) continuously learning from a human demonstrator that is suboptimal in achieving their goals to 2) goal inference and superhuman performance on those goals may require different training recipes. There’s a lot to unpack on this point.

To get to productization, we’ll need to compress the data collection → construction → training → evaluation → hosting → serving → high uptime pipeline into something that can run cheaply, quickly, deterministically, continuously, without intervention, and for multiple users. This is and of itself will take substantial effort. At the same time, we’ll need to build the UX that queries the user model and structures the underlying requests to match the context passed during training and displays the output in a way that works. We’ll need to maintain extremely strict privacy standards given the sensitivity of the data and models that are being produced. We’ll need to consider how the visibility of real time user completions from the model alters the validity of the training assumptions we’ve made. There is a lot to do.

The payoff is worth it.

Establishing a pipeline that converts all forms of inbound information into a structure that increasingly intelligent models can learn from allows individual human judgment, taste, preferences, and goals to be amplified into a world where societies of agents operate largely without direct oversight. We start with text, and can eventually support image, audio, and even thought directly, feeding these inputs and their processing into models that represent us in real time. Without it, our future societies consolidate to the goals and preferences of agents that operate for themselves or their builders, rather than for you.

Links (again)

Github Public notes Email X Labs

If you made it this far, you can also DM me on Telegram or Discord @handsdiff