Algorithms.md

algorithms considered and rejected for now and why: (there are likely practicalities that result in other algorithms perhaps being superior, this is a non practitioner's pre-experiment take) - PPO - maybe useful for phase 3, but we dont have a value function to use as a critic until after phase 2 collects data to generate the implicit rewards from preferences - GRPO - requires raw numerical scores for each candidate, which is too much friction for the user to develop. anyways only seemingly relevant for phase 3, even if you develop a reward function from the earlier phases that could assign numerical scores - SDPO https://arxiv.org/pdf/2601.20802 - "aligning language models from human interactions" https://arxiv.org/pdf/2603.12273 is apparently an instance of SDPO that learns from human interactions rather than a 'successful sibling rollout' https://arxiv.org/abs/2601.20802 or 'canonical answer' https://arxiv.org/pdf/2601.18734. either human interaction or successful sibling rollout i think could 'work', but a successful sibling rollout would just be phase 1. the difference seems to be that its for 'rollouts', which assumes the model has an environment where i can take actions that impact state transitions. in phase 1 the actions dont have any impact on state transitions? in phase 2 they might, but its unclear if thats the best way to model it vs top K posterior sampling + DPO. for phase 1, whats the actual loss function diff? - LSDPO⁡(𝜃)=𝔼𝜏∼𝜋𝜃⁡[𝑇∑𝑡=1KL⁡(𝜋𝜃⁡(⋅|𝑠𝑡)⁢||⁢stopgrad⁡(𝜋𝜃⁡(⋅|𝑠𝑡,𝑐)))] - the loss matches the student on the teacher with a 'hint' i.e. helpful information, and updates the student to match the teacher. this seems like it could be used for phase 1, still unclear why this over standard cross entropy. after my discussion with codex, SDPO could be used, its just less direct of a loss. something to explore. it trains given hints rather directly on preferences or self supervision. helpful loss comparison https://gemini.google.com/app/9eddf57f5e81e914 - https://trajectory.ai/field-notes/scaling-sdpo this article seems useful for understanding SDPO but the conclusion is just clip gradient updates? lmao. results are results though. - "For hard tasks where the right behavior is rare in the base policy, secondary objectives like behavior cloning or DAgger may be needed. Our preliminary experiments in that direction showed enough hint-copying that hint design looks like the harder problem." - yeah no shit? ^ this is just saying the reward function is the actual bottleneck to consider - "Soon, we will be running off-policy SDPO on live production traces, with the user's actual interactions as the hint" i dont see why they dont just use online DPO, i likely am not understanding something. - there is a discussion around staleness here that i do not fully understand - OPSD - from my learnings, OPSD is for when a ground truth exists (relates to deterministic vs non deterministic reward function being the distinction when we discuss 'dynamic' environments or not, with models definitely able to learn static environments i.e. deterministic reward functions) - "if you're doing OPSD and you keep seeing model collapse, one of the best debugging tricks is to remove the hinted logprob ratio and replace it with an advantage estimator" - https://arxiv.org/abs/2209.15189 Learning by distilling context - "cool new approach to address the spike problem in on-policy self-distillation by letting the privileged information steer the student distribution during rollouts directly" - OPD - OPD is just OPSD but the distillation is from a different model, not the same. so doesn't rely on ground truth existence BUT cannot achieve superhuman performance. perhaps helpful for phase 3 - very useful if you RLVR a strong teacher, then OPD with dense supervision into a small LoRA - https://arxiv.org/pdf/2606.30406 MOPD - just trains a model with multiple teachers at once instead of one. very useful for distilling multiple 'capabilities' than trying to teach each one at once - PNLC https://arxiv.org/pdf/2505.18098v2 - this requires a large corpus of offline data that you train a model on, to then grade and steer a live models actions with reasonings. seems indirect compared to CE loss or sDPO - NLAC https://arxiv.org/pdf/2512.04601 - seems incredibly cracked and dont fully understand the details. i think its probably the best RL algorithm, but again it requires there to be actual rewards/value network, which theoretically needs to be bootstrapped in our case by phase 1/phase 2 - policy gradient - again, RL maximizing total rewards of trajectories. needs rewards. - contextual/multi armed bandit - similar in theory to sDPO but requires a prefixed action space rather than autoregressive/natural language actions - DAgger https://arxiv.org/pdf/1011.0686 - this basically solves 'covariate shift' which is fancy words for the trained model drifting into states it does not know how to handle well due to small compounding errors, cascading into failure. solved by having the expert tell the model what to do when it compounds errors. don't have the bandwidth for that kind of feedback. - recsys https://www.tensorflow.org/recommenders https://arxiv.org/pdf/2510.27157v2 https://research.google/blog/regen-empowering-personalized-recommendations-with-natural-language/ https://arxiv.org/pdf/2601.20316v1 https://shopify.engineering/generative-recommendations https://gemini.google.com/app/9c691500d7b6008a HSTU seminal paper: https://arxiv.org/pdf/2402.17152 - very similar. doing a lot of the same steps of shopify. pinterest also has interesting work. the other ones i havent full read through / have the same issues as contextual bandits. a lot of it comes down to whats being recommended (large or small vocab) - very, very similar to the algorithms i arrived on: https://shopify.engineering/generative-recommendations. posted feb 2026.
- autoregressive with causal mask to start (fancy description of prediction next action given action history), then sample that model with user choices to further tune behavior (fancy description is hard negative sampling). quite interesting. the gap is that the recommendations are still a fixed set of product IDs rather than semantic text, but they mention that as next steps - prospective learning - too complicated to implement right now. not sure how I could tokenize time in a way that would be positive ROI right now - MARL - non LLM MARL feels too esoteric / not generalizable enough / too many assumptions, LLM MARL complexity is ginormous, predicting the primary user well is the hypothesized first step to true LLM MARL - model the user as part of the environment? - hard to reason about this unless i get more concrete about the data structure. it also depends on to what extent phase 2 is RL or not. what this even means specifically is that the agents recommendations are its action space, and my actions and my responses to its recommendations are the state space, and the agent has the job of achieving max rewards in this formulation by taking actions (recommendations) that have a stochastic impact on state transitions. the issue with this formulation is that its unclear where rewards come from? is a reward given basically whenever an action is 'chosen'? i dont think ive seen formalizations for this. basically a stochastic MDP where if the agents action leads to a specific action by another agent in the env (i.e. the user), then the agent is rewarded. what is the reward? a preference update like DPO? you would need a counterfactual. do you formulate the recommendations given as proposed actions? and then reward via log sigmoid of the policy probabilities to separate chosen from nonchosen? - how is this fundamentally different from classic autoregressive causal mask formulations in phase 1? well phase 1 is separate from phase 2 and this mostly works for phase 2. then how is it different from phase 2? - the proposed phase 2 samples 1 step rollouts/actions from the existing model/policy, then compares the preference of the human to take that action or not, and uses that to update the model weights via an optimizer like gradient descent from the DPO loss function. this system would basically do the same thing, since model = policy, and actions = sampling. with the RL formulation there is no reward, so DPO makes more sense because you need to reward the model for the actions its taking. - this formulation likely starts to make more sense when we consider more esoteric dragan work around model -> human influence, but seems undifferentiated for now - this makes sense with more context. phase 1 is building a human model. the human model can then be used to train separate policies. - CIRL / assistance games - https://arxiv.org/abs/2504.07091 https://cassidylaidlaw.github.io/minecraft-building-assistance-game/ - i previously wrote down this was a bit too far out for right now, then i reconsidered given new information, but it is too far out. it assumes the agent is able to take actions in the environment as well. we wont have that ability until there are good parallel sandboxes or some way of allowing the agent to take actions on the computer if their recommendation is actually chosen - will brown phd https://willcb.com/blog/feedback-loops/ https://scholar.google.com/citations?user=JUJdJMoAAAAJ - "We started with a set of pretty ambiguous questions: - How can we model online recommendations in a way which gives rise to the kinds of feedback loops observed in reality while remaining analytically tractable? - Treat the user’s recent consumption history as state, menus as interventions, selections as stochastic responses, and preference changes as state transitions. - Can we gain any insight into why algorithmic feedback loops occur, and design recommendations algorithms which avoid their potentially harmful consequences? - Recommendations can move users into states where alternatives become increasingly hard to induce. Preserve diversity and baseline exploration so that the process remains recoverable, and optimize long-run rather than immediate reward. - How do we characterize the space of possible outcomes and algorithmic benchmarks when agent preferences are adaptive as a function of our recommendations? And we found satisfying answers to all of these." - The attainable outcomes depend on the current preference state. Benchmark only against distributions that remain attainable along the relevant trajectories: conservatively EIRD, or—with stronger structural assumptions—a smoothed version of nearly the whole simplex. - https://proceedings.neurips.cc/paper_files/paper/2022/file/a75db7d2ee1e4bee8fb819979b0a6cad-Paper-Conference.pdf - https://proceedings.mlr.press/v237/brown24a/brown24a.pdf - https://proceedings.mlr.press/v247/brown24a/brown24a.pdf - essentially, some good work on complexity, feasibility, and algorithms to optimize for recsys with dynamic users, but makes assumptions that leave it not directly applicable to my use, specifically a verifiable reward and a fixed action space - one thing that stands out is that they DO model how prior recommendations influence future choices, which i had not seen before and is relevant in theory. the implication is that optimal recommendations are path dependent - RLHF https://arxiv.org/pdf/2203.02155 https://gemini.google.com/app/1c431b0b8914983a https://gemini.google.com/app/e11d7e2c3bda71cc https://www.youtube.com/watch?v=XKLGuwvSKvI&list=PLoROMvodv4rPwxE0ONYRa_itZFdaKCylL&index=10 - apparently RLHF has 3 stages. the first is SFT. the second is training the reward model from pairwise preferences, the third is training the step 1 model using the reward model, with a KL divergence penalty to keep it close to its original behavior. this is basically my 3 step process. the main difference is that the SFT trains a model to follow instructions, i.e. respond to a prompt, from its base pretrain. whereas my step 1 is more like SFT to alter its next token prediction itself. also step 3 is not needed with doing DPO according to the DPO authors because youre already updating the language model from the rewards directly, whereas classic RLHF step 3 is where you take the reward model you learned and update the language model from it - if so then how to 'scale' softmax DPO? the preference data is coming from 1 step rollouts i.e. recommendations. start showing 2 step recommendations? how to better learn given possible step issues rather than overall issues? does that matter? probably not since its a premature optimization - actually unclear whether classic RLHF (with PPO) or online GPO / softmax GPO would be better for the use case. gemini says RLHF is online and on policy whereas basic GPO is not, but not sure i buy that - it does raise a question as to why the recipe from the original RLHF paper hasn't been attempted on user specific data. likely because its way less economically valuable. but no one experimenting has tried it either, maybe because giving feedback sucks because the prior was ass. but then why not develop a prior? maybe because that step fails? - possible differences that stand out are (1) does RLHF lead to 'superhuman' performance? i think yes actually (frontier LLMs shown to be more empathetic, etc) so maybe not an issue. - interesting paper i think establishing RLAIF https://arxiv.org/pdf/2212.08073 from anthropic and also showing constitutional RL showing pareto efficiency in harmlessness + helpfulness - RLHF shows results that the generated summaries in their benchmarking were superhuman! - https://jacobxli.com/blog/2026/machine-studying/ seems very relevant to continual learning, possibly good benchmark - biggest takeaway from this is that next token prediction over a corpus is an instance of a self supervised ML objective, the other two of which are test time training (paper below) and compacting KV caches (paper below that) - i wonder whether next token prediction over a corpus is the right way to think about my proposed phase 1. because what i really want is next action prediction / next step prediction. obviously thats an abstraction, but i wonder whether the useful mental frameworks also change as a result. for example as i type notes or prompts into chats, is the model mostly learning how i write/think? or is it learning the overall content of my output given the overall content of my input? or both? if both whats the mixture? - i think a lot of the critiques come down to well, it will learn your style (the former). haven't properly communicated or understood the latter well enough - anyways, they say the simplest is 'continual pre training' i.e. next token prediction over the corpus in which you update LoRA adapters - they note that it degrades instruction following abilities, which is fine - they mention synthetic data as another main axis, which i gained intuition for from the paper that generates Q/A pairs from the corpus for studying (forgot where in the notes it is) - overall this seems to optimize recall (answer a quiz better) rather than behavior (whats the next action given prior actions), so not as relevant - https://arxiv.org/pdf/2506.06266 cartridges (by CTO of Engram) likely explains how they're 'scaling compute on context'. differs in that it maximizes retrieval efficiency as a primary goal. quite interesting. comes with code https://github.com/HazyResearch/cartridges - generate synthetic conversations about a corpus of text (self-study), uses that to 'train a KV cache' (not sure what that means) (cartridge), then loads the KV cache for the LLM on long context benchmarks comparing it to ICL. can compose cartridges as needed. very similar to PLUM in terms of the upsampling. 38x lower memory and 24x higher throughput with similar performance. cool. might be worth trying alongside the existing matrix of ICL, SFT, memory, but seems slightly off mark in terms of utility since its an efficiency gain rather than performance, but it seems to have some performance gains as well. so perhaps worth trying if we go deeper. - also shows that self study (synthetic data) is needed to handle diverse question answering regarding the core corpus - honestly fair, i do this all the time when understanding things well. and if the goal is future MAS, it likely necessitates question answering, since otherwise its just predicting my answer to a user question, which might work but likely performs poorly due to lack of data? depends how far you go with the phases. - yeah likely the case that synthetic Q/A data is useful. hmmm - works great for static context seemingly, probably not so much dynamic - this is a type of prefix tuning - vLLM and SGLang already handle cached prefixes - trained using context distillation. lots of nits. initialized with the KV cache from the base model reading the corpus. the first key and value are frozen since otherwise the model collapses. then basically ensures the cartridge + LLM matches the outputs of a LLM that has the full context in memory via KL divergence minimization, while only backpropogating through the cartridge weights - multiple cartridges can be concatenated seamlessly - E2E TTT https://arxiv.org/pdf/2512.23675 - (maybe gwern was right [[Gwern GA#^1c6876]]) - first find a weight initialization that is optimal for the ability to learn, then update weights every batch of tokens in an online manner. only update the MLP layers, not embedding, normalization, or attention layers, for stability. - core finding is that dynamic evaluation doesnt work because the initial pretraining weights optimize for training loss, not the ability to minimize future test loss. doing so is why they call it "E2E" - this is done by simulating TTT on the initial training set, then iterating to see which set of initial weights learns best on average, which is where the double loop and 'gradient of gradients' comes in - there exists standard pretraining datasets, like this paper used, if you want a clean pretrain for the intended task to work - the core concept that dynamic evaluation does not work since of poor weight initialization, and then during test time you batch updates as well as have a sliding context window, feels very applicable - TTT-E2E resulted in faster and lower loss on test sets. not obvious since even though its backpropogating continuously, it still needs to be able to predict the next token well - https://github.com/test-time-training/e2e - https://arxiv.org/pdf/2602.16284 extends cartridges with KV efficiency - optimization, not core concept - https://arxiv.org/pdf/2004.10964 continuous pre training (bit old, 2020) - simply trains a pretrained LLM on domain specific data before it does a task in that domain to improve performance. fairly standard nowadays - https://arxiv.org/abs/1904.08378 dynamic evaluation in transformers. - pretty basic cross entropy autoregressive lossi suspect this works well in scenarios where the distribution does not change drastically, but unsure if the increased costs make it worth it - https://thinkingmachines.ai/blog/on-policy-distillation OPD again, but relevant to reconsider post gwern ga reread post TTT deep dive - one very good explanation here is that you weight the size of the token update of the student based on how far away it is from the teacher's per token probability - when you take a trajectory, the teacher just needs to be sampled given the same context, and the difference in log probs for each token can be updated accordingly into the student - this is stated, but very similar to dagger and process reward modeling - reverse KL = student minus teacher. the goal is to minimize the diff over the trajectory, with each token conditioned on the SAME prior trajectory - the second bullet is wrong in that if the student makes a mistake at token 2 after the context, and the teacher does not, and the teacher keeps going, then the student can no longer learn at the token level - "There could be advantages to combining distillation-based per-token rewards with sequence-level environment rewards" - "Reverse KL has natural synergy with RL, which generally optimizes a form of sequence-level reverse KL induced by the reward model" - need to get a better understanding of token level vs sequence level, probably by reading papers - impossible without an easily queryable teacher - apparently good for maintaining old behavior while augmenting new ones instead of forgetting old behavior, since you use the original model as the teacher in OPD for the model with specialized knowledge. unclear why this doesn't degrade the newly learned knowledge. depends on the phasic behavior i am assuming, since it depends on the path you gradient descent through the state space - again, this is likely because the benchmarks test something more akin to retrieval rather than the ability to predict next write given read/write history - this continues to feel categorically different at a gut level. its unclear how prediction would degrade instruction following, or if that should even be a goal of phase 1/2, or at all - its unclear how its needed for enterprise MAS, if at all - i dont understand the last section. sampling a model with temperature of 1 from a standard set of prompts, then performing SFT the model on those pairs, reduces ability. yes, local batches have slightly different distributions that perturb the loss, but its unclear why this is for sure bad instead of randomly good or bad, probably because it optimizes for something 99% good (its sample), rather than ground truth, would be my intuitive understanding of it - https://micahcarroll.github.io/ - https://arxiv.org/pdf/2405.17713 AI Alignment with Changing and Influenceable Reward Functions Dragan 2024 - it doesnt seem like the concerns for influence are valid, at least to my spec. thinking through this, a model that is being optimized to predict my next action, given my history of read and write, will have top K samples that attempt to influence me? i get that the choices will influence me, but to imply a 'desire' for influence is a bit strange. maybe the argument is that to minimize loss, it will implicitly try to influence me, but thats kind of the point. it will work if i believe it is maximizing my reward. so the issue is really in my analysis of 'advice', which is already a problem. im not sure its exacerbated by AI. maybe theres concerns around an 'echo chamber'? feels like it comes down to my personal cognitive security, and theres not much you can do to align an AI system on a stupid person. i get there needs to be some semblance of 'longer term' preference data eventually, but this feels like a prerequisite. - it comes down to whether its 'easier' for the model to influence me negatively or positively - another way to think of phase 2 is that there is no notion of accept or reject. its purely influence. showing samples from the model has the goal of influencing me to do something else other than what i would have directly done myself (although i may have considered it). when would this occur? - what needs to be true for top K sampling to be viewed as a better next action given a current state than anything I thought of before? - probably that your actions have enough, but not perfect, overlap with your goals. this is likely the case for me and my customers. - overall way too esoteric to include anything practical initially - https://arxiv.org/pdf/1910.05789 utility of learning from humans for human-ai collaboration (super good and relevant, reading through) - makes an interesting argument that the human model and the agent learning to interact with the human model should be decoupled. first get a human model via imitation learning (optionally can update it online to deal with stochasticity but its more complex). then train an interaction agent via RL for some collaborative goal. unfortunately this is still in a game setting with a clear, verifiable reward. - "Lerer and Peysakhovich [23] train their agent with Observational Self-Play (OSP): a combination of imitation learning and MARL" - imitation learning is another way to describe phase 1. the canonical loss is just least squares / cross entropy. there are stated issues around averaging actions leading to worse results, and stuff like DAgger, but the former shouldnt matter for single human, and the latter is mostly for rollout space - "We embed our learned human model BC in the environment, treating it’s choice of action as part of the dynamics" - "Real humans learn throughout the episode to anticipate and work with the agent’s particular coordination protocols... Notably, PPOBC cannot learn and take advantage of human adaptivity, because the BC model is not adaptive." - confluence with the E2E-TTT paper that using meta learning to initialize test time learning with weights that are trained to be adaptive for future loss rather than minimizing current loss might help with dynamic or stochastic or distribution shifts of human behavior - references on this paper go hard - https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf - basic math behind inverse RL. modern approaches are Deep MaxEnt IRL and GAIL, and AIRL - https://gemini.google.com/app/e4c31806ee6129a4 - pure imitation learning does not imply a reward function since there is no contrastive, not chosen action? but isnt the model's y output the contrastive action, compared to the demonstrator's y hat? - maxent irl assumes the trajectories are expert so doesn't apply here - AIRL https://arxiv.org/pdf/1611.03852 from finn - i should clarify the drawbacks of online DPO, if any - works by having the agent do rollouts while the expert (me) does rollouts (normal work), train a discriminator using binary CE loss to identify between the two, calculates implicit rewards based on the discriminator, uses that reward to update the policy - unfortunately we do not have the ability for the agent to do rollouts until after phase 2 at minimum - https://arxiv.org/pdf/1906.05838 goalGAIL, relates to take of setting goal alongside rest of algorithm, kind of like the hint in OPSD - stuff like GAIL is not applicable today since it requires env rollouts and per step analysis, whereas things like DPO or even online DPO are considered 'contextual bandits' in the sense that they are single step - anything mentioning an 'expert demonstration' is in some sense inaccurate since the human does not necessarily know how to achieve its goals either. unclear what algorithms account for the expert also learning / changing its policy. but i assume that a smart enough imitation learner that is online can just map the changing policy - https://arxiv.org/pdf/2211.10869 varying forms of masking during training may help improve meta learning - https://arxiv.org/abs/2411.02306 models will learn to identify and reward hack vulnerable users even when they only make up 2% of their deployed population - [[Google Pi Team]] - https://arxiv.org/pdf/2603.02960 great thesis piece but no relevant algos - https://arxiv.org/pdf/2603.28925 no algos, but they find that LLMs can have theory of mind abilities without attributing minds to themselves - https://arxiv.org/pdf/2602.16301 MUPI toy example - they find that decentralized MARL + model diversity can lead to cooperation rather than extortion in scenarios where all agents are learning - https://gemini.google.com/app/41686aaf9f555d03 - crucially this can happen in context if you just MARL train against a variety of opponents - apparently the training approximates the ability to learn the scenario you're in based on the test time in context history and act best according to your prior of the situation. https://openreview.net/attachment?id=W1Ta00wVlK&name=pdf - in other words, the interaction history allows the model to determine the best action at test time, since its trained on a a variety of opponents and can determine the 'type' of opponent given the interaction history - reminds me of meta learning since the initial weights are optimized to handle a mixture of potential test time tasks rather than optimizing for one - https://gemini.google.com/app/22e45c199a56744d - https://arxiv.org/pdf/2512.20605 RL within the residual streams of autoregressive models, rather than applied after - "We analyze transformers and state-space models (SSMs) trained to autoregressively predict the actions of goal-directed agents, whose goals are unknown. We find that the networks learn to represent (and infer in-context) a belief about an agent’s goals in their residual stream activations" - shows that you can add a dynamics loss with a weighted parameter to the loss function if you want the model to be able to learn env dynamics - ^ ECHO also did this months later. would be interesting to apply ECHO to two of our human models - i dont understand this paper, tried for a while to do so - https://claude.ai/chat/dcf030af-4c28-47d7-8ab6-3bb3587626dd - https://arxiv.org/pdf/2403.08635 online IPO / IPO-MD - apparently similar to online DPO except it focuses on best approximating preference probabilities rather than preference differences, which seems strictly better - offline version from a few months before https://arxiv.org/pdf/2310.12036 - KTO https://arxiv.org/pdf/2402.01306 similar to DPO - dpo needs preference pairs, assumes there is an implicit score to each, focuses on maximizing diff - KTO takes whether a response is good or bad and optimizes based on loss aversion i.e. heavily update against bad, lightly update towards good - works in settings where good/bad is given from the user but doesn't seem to apply as much here - algos from https://cs224r.stanford.edu/ - i think these all apply after you get a reward model from phase 1/2, so after that worth revisiting. i also havent well mapped the concept of Q values to my work. one thing that does stand out is model based i.e. learning environment dynamics, and how that relates to being able to predict consequences of actions, not just actions from prior actions. i.e. what happens when i type in something in my browser. likely intractable since i dont want the model to try to waste cycles learning what a search engine will respond, but important to keep in mind towards multi agent scaling. - assigning 'credit' to specific steps as discussed in lecture 10 might be useful for phase 3 - lecture 10 near the end has really interested advantage weighted step by step training that seems relevant to predicting my next action and learning from recommendations when thought of as a MDP but i cant put my finger exactly on why/how - maybe because some prior during the training process might be 'noisy', and this could help delete the noise to focus on the actions that actually helped in achieving the goal? but again this needs a value function which we would not have at this stage - in the case where we need to actually reconsider the phase 1 construction as an RL rollout to handle tool use for additional content, a lot of these algorithms will come back. lecture 10 stands out as high yield. "RL on incorrect synthetic data increases efficiency 8x". offline DPO from rollouts. talks alot about how to actually get LLM reasoning using offline data, failed data, synthetic data, how to augment reasoning with the help of LLMs during the work, stitching together steps using advantages, etc - empirical recommended temperature is 0.6 for rollouts ^ - process reward RL (PAV) outperforms ORM RL (binary outcome) - verifiers are just reward models - he discusses how a big advantage for improved reasoning lately is having more chunky, abstract 'actions' or 'steps' i.e. a sentence rather than a token - i havent deeply internalized RL for reasoning and how it relates to my problem formulation. likely most applicable for phase 3, probably impacts phase 1 and 2 - lecture 18 has a lot of good practical advice. examples where the pretrained weights constricted learning. the beginning is literally frameworks for what im working through. - "goal conditioned policy" is a phrase i remember from the prior walkthrough but seems interesting since i believe ive been thinking about it independently. you can probably semantically put the perceived goal in the context of either weight space or prompt space model, which might help with learning the reward model - does LLM pref optimization still rely on single turn? if its multi turn, are 'process reward' like formulations utilized? if so how? does it make sense to interleave policy updates with pref optimization in online DPO? policy update interleaving apparently happens with rlvr like math. do you batch? what size N? asynchronous model update after batched data collection is considered practical - DQN https://arxiv.org/pdf/2207.03029 - q networks need a reward. the reward needs to come from preference data at this hypothesized stage. perhaps for phase 3. - IPO https://arxiv.org/pdf/2502.16182 - basically DPO but using the policy that generated the rollouts as the preference producer. had good diagrams that are useful for visualizing DPO and worth reading in full for how they decided to structure data and apply the algorithm. everyone seems to consider SFT as a necessary prerequisite to good DPO. has loss functions in there as well. im confused because it still requires a preconstructed dataset of human feedback? - P-RLHF https://arxiv.org/pdf/2402.05133 - conditioning the policy on the 'user' model, which seems different from the reward model, and is instead some tokenized description of the user? weird - https://arxiv.org/pdf/2402.09269 personalized language models - 'personalization' is so so vague. this paper tests what emotion a user feels when given a piece of text. they do this by training it on a history of felt emotions given text. they show PFT wildly outperforms ICL. good, but i feel like this is a way easier task than predicting outputs given inputs. - PLUM https://arxiv.org/pdf/2411.13405 - this paper, to solve knowledge injection, basically takes a history of traces, uses an LLM to convert them to question-answer pairs, and performs SFT with LoRA on the QA pairs. its competitive with RAG (81 vs 83). seems worse than directly training on next action prediction. an interesting finding is that hard negative sampling was critical for it to work - this is broadly called 'self study' in a bunch of future, related work. the argument is that self study, which is a form of synthetic data, improves 'understanding' rather than 'memorization' over a corpus of text - noisy BC/IL - https://arxiv.org/pdf/1907.03976 D-REX - introduces noise into a trained BC model, then ranks the trajectories based on amount of injected noise to preference learn a reward function, then RL on the reward function. significantly outperforms the original demonstrator. cool. - https://arxiv.org/pdf/2207.10050 discriminator BC - uses a discriminator to distinguish expert vs non expert trajectories, but we don't actually know which ones are expert vs non expert - online BC/IL - just batch retraining, nothing crazy. 'online' is a misnomer, since it doesnt determine freshness of data. 'online' is when the policy actually acts in the environment. in our initial formulation the policy is not acting at all. this is offline data just retrained at test time (ideally, or perhaps not ideally) - https://oshaikh.com/ (these also have git implementations that could be useful during implementation to compare to) - https://oshaikh.com/papers/longnap Learning Next Action Predictors from Human-Computer Interaction - closest to my work. trains next action prediction from computer use data. - super useful in terms of practical data pipelines - they explicitly include the reasoning to retrieve data from memory that is not directly in recent context, which makes sense, then bases predictions off of reasoning about the retrieval. so very much so not directly next action prediction - i wonder how reasoning to retrieve then predict compares to direct autoregressive prediction. the latter feels way more sample inefficient. - the retrieval is very simple BM25 from raw event logs - but they need GRPO and LoRA for this?? - basically because they allow the model to reason through the correct answer, then do self-supervised fine tuning via LoRA to improve the reasoning towards the right answer, rather than trying to one shot the right answer. interesting. - this is way simpler than attempting TTT or the like - also discusses going the next step of taking a computer use agent to allow the predictor model to actually take actions - one thing that stands out is that this is much more replacement than augmentation - https://github.com/GeneralUserModels/tada - references https://arxiv.org/pdf/2309.12170 which forecasts keyboard/mouse actions using RNNs and CV - https://oshaikh.com/papers/gums general user models - i thought this was insane but it just seems to create natural language propositions based on computer use, and recommend based on that. simple, but somewhat effective. this might be similar to something my work ends up falling back to. likely a very good baseline to compare to, as well as good resource for comparable data preprocessing and UX - https://oshaikh.com/papers/grounding grounding towards common knowledge research - highlights that LLMs assume shared knowledge rather than confirming it, which is a result of current RLHF, making interaction less useful than it could be. interesting framing for what one of the issues is - https://oshaikh.com/papers/ditto DITTO (align LLMs from demonstrations) - "Brown et al. (2019) first showed that with ranked demonstrations, one could improve a policy beyond the demonstrator’s performance" - almost exactly relies on the take that the LM's sampled actions can provide hard negatives to be used in DPO - good algorithm and loss function that I can reference, along with practical nits about data mixing as well as ablations - shows that it works better than SFT, which makes sense. supports the phase 2 hypothesis. - actually the nit there is that my phase 2 implies that the samples are shown to the user. how does that change the formulation, if at all? - the future work section is also relevant - https://oshaikh.com/papers/just-in-time-objectives JIT objectives, inferring objective from user actions and helping, seems very similar to assistance games - simple use of LLM on the same data as GUMs to infer local objectives and inject - good work on collecting and cleaning data, again useful for practical implementations upcoming and as baselines, less useful as novel algorithms - interesting that the output was a generative UI, not a language suggestion/recommendation. using something called Poppins, which is open source - UI was chosen since there was a take that chatbot UX is poor, and they didnt want to the model to suggest to take actions to the user, they just wanted to present something useful - https://web.stanford.edu/class/cs329x/slides/Lecture6_A_hai_grounding.pdf - https://collaborative-agents.github.io/ - this seems to be basically the same as general user models from shaikh - CTRL-Rec https://arxiv.org/pdf/2510.12742 - - FPL https://freeform-pl.github.io/fpl.website/ - - RLIF https://arxiv.org/pdf/2505.19590v3 - - NetworkIDBD (from sutton's blog post) - - ProAgentBench arxiv.org/abs/2602.04482 (2026) - a benchmark for proactive agents that continuously monitor user screen activities and contextual signals, deciding when to intervene based on historical observations and behavior patterns — essentially the evaluation counterpart to NAP. - ProAct - "ProAct predicts future needs from history and persistent memory, then prepares relevant artifacts during idle time. It reports 14.8% fewer turns and 11.7% less user effort—but on a synthetic benchmark, so this is encouraging evidence, not product validation. - https://arxiv.org/pdf/2606.03979 learn during sleep, data during day - conceptually similar to phase 1 but way more complicated, useful to keep in mind if initial implementations fail. specifically uses different frequency updating, low rank MoE, fast vs slow modules, synthetic data and self study, and student/teacher distillation/RL - https://www.nature.com/articles/s41586-024-07711-7 + https://arxiv.org/pdf/2606.24752v1 loss of plasticity in deep continual learning - reminds me of idbd by sutton in how the original paper detects important / not important weights and resets accordingly (super hand wavy but generally accurate description) - found in non LLMs ^, no prescription in future work on LLMs - https://people.idsia.ch/~juergen/1991-unnormalized-linear-transformer.html - - https://arxiv.org/pdf/2607.20465 data prep bench - tests a bunch of models on their ability to generate good QA-style SFT examples - might be useful if that is needed later - open source dreamer v4 https://next-state.github.io/open-dreamer/ - - https://arxiv.org/pdf/2607.16097 quantifies SFT vs RL - lots of details but essentially shows that RL performance is positively correlated to SFT performance in terms of ceiling and training speed, it can also extend low prob SFT moves if they're useful, while reinforcing strong moves - has more details on actual numbers - https://docs.adaptionlabs.ai/api/resources/autoscientist/methods/create essentially productizes autoresearch on your data, from my understanding. may be useful later - - https://arxiv.org/pdf/2607.11020 researches whether an LLM can learn facts into weights continuously - seems counterpositioned to the cartridges paper which makes it seem like you can actually do this - i feel like weights aren't for facts though. weights are for converting information to action (still need a good word for this) - interesting take from the author in a later MTS interview that kimi k3 was increasingly trained on 'work' rather than on 'answers' and that that is the direction of training - https://arxiv.org/pdf/2507.21071v2 fingertip 20k, like longNAP at a high level but mobile, from tsinghua - - https://github.com/ramp-public/portallib portable lora from ramp, open source code, may be useful - - ECHO https://arxiv.org/pdf/2605.24517 - classic algo surprised i missed it. basically learns how the env reacts to actions. very solid. perhaps useful for later phases - edgebench https://arxiv.org/pdf/2607.05155 - establishes how models perform learning in real world environments by updating themselves, shows log sigmoid learning and learning rate doubling every 3 months on frontier models. very related to continual learning imo - seems extremely relevant to discussions around context windows / retrieval vs judgment etc - https://arxiv.org/pdf/2605.23668v2 "a key step towards proactive interaction is next query prediction" should study the data construction and method used for actually implementing the 'dynamic memory' but from an initial skim it just seems like a scratchpad that the LLM can use... - going from roughly 40 to roughly 44 score is pretty weak no? went through the training method but havent combed through the datasets or construction yet - https://arxiv.org/pdf/2501.00663 titans, learning to memory at test time - https://x.com/jsuarez/status/2089372615744086019?s=20 RL practical discussion - insane resource. frontier RL 2026 https://rl-conference.cc/ (conference schedule, custom websites, accepted papers -> paper schedule) - https://arxiv.org/pdf/2202.06991 differentiable search index -> knowledge in weights directly - reasoning vs knowledge latest research https://claude.ai/chat/c4dd1c91-cacd-4599-a6c3-874720e2e9a3 - discusses ambiguity in learned distribution impacting LLM training https://arxiv.org/pdf/2608.15448 - how does ambiguity relate to entropy? i.e. what are the levers that allow a model to learn a 'higher entropy' distribution? more parameters? more data repetition? etc. and what is the magnitude of the impact of each lever?

• is phase 1 a process reward model?

• does phase 1 result in an 'implicit' reward model?

• one commonality is ease of getting rewards/values. anything assuming an 'expert' that can be queried at low cost is a non starter for phases 1 and 2. could be useful for phase 3 pending results.

• pinterest generative recommendations https://arxiv.org/html/2606.00422v1

• DPO paper https://arxiv.org/pdf/2305.18290 (coauthored by finn!)

• classical LLM pretraining serves as a prior for predicting me. predicting me serves as a prior for recommending me. recommendation data serves as an implicit reward model towards increasingly multi step rollouts to simulate and achieve rewards I might not otherwise be able to recognize or achieve

• higher parameter models fine tune more successfully given the same amount of data, and fine tune equally given less data. but to be specific this seems to come from pretraining, not the SFT or RL that labs do after the fact

• frontier LLMs could be used to 'expand' the label data i DO give to apply it to different scenarios, so i need to give less feedback, but still some? that model acts as a critic?

• if recommendation gives a set of recs that all do not get accepted, do you consider ground truth as the 'preference'? isnt that off policy whereas the others are on policy? can you just mix data like that and feed them both into the DPO loss? also relates to whether you can just mix loss functions [[Entry#^e7fcba]]

• big diff in 'personalization' is 'i like it' vs 'its me'. maybe should call it something else to distinguish. even 'its me' implies style, so need something to distinguish that. the best gemini could come up with is 'cognitive language models' or 'neural proxy language models' with the context i gave it

• explains, from rich sutton, how 1 step doesnt necessarily lead to N step due to compounding errors, very similar to what DAgger solves http://incompleteideas.net/IncIdeas/OneStepTrap.html. for my stuff, probably need to instead start giving per step advice on 2 step recs, 3 step recs, etc, rather than assuming 1 step recs can be let free, which i think is valid and was my prior

• has a lot of relevant practical work https://sabrieyuboglu.com/. the engram launch blog has a ton of relevant work as well.

  • there is so much more to understand here that is likely useful
  • what i see here is a team that is cracked at algorithms and the goal even more so is data production and context elicitation for reward direction, although unsure if their setup changes the reward function. the parametric memory stuff stands out to me
  • "Specifically, this includes:
    • Memory and knowledge internalization — designing and evaluating methods for encoding large, heterogeneous document corpora into compact parametric memory (e.g., LoRA/adapter-based representations, prefix tuning, state-space methods).
    • Synthetic data and self-study — understanding what makes synthetic training data generalize, and developing self-study pipelines that allow models to reflect on and consolidate new context.
    • Continual learning algorithms — tackling catastrophic forgetting, sequential updates, knowledge conflicts, and the tradeoffs between in-weights memory and agentic retrieval.
    • RL and online training — exploring reinforcement learning methods that let models improve from interaction and feedback in real deployment settings.
    • Scaling and capacity — empirically studying how model capacity, data scale, and compute interact; developing the scaling laws that inform our product roadmap."
  • side question, can you 'measure' the extent to which ICL influences the implicit reward function of an LLM?

• one diff starting to stand out is that my proposed phase 1 and 2 do NOT expect question answer format, whereas most of RLHF and LLM research do. i specifically want to stick with next token prediction, at least until we formalize more complex approaches around embodied agency and self-prediction and influenceable reward functions re dragan. and according to finn, question answer pre training MAY hurt performance here. so it might actually be worth comparing a pretrain non RL'd model, if its possible to find one, vs typical frontier LLMs. although it will likely be hard to compare due to size diffs, so maybe we could find an older RL'd one to more directly compare

• one mental framework is that you will be filling the context with as much history as possible, and you want to have a model that optimally outputs the next action given this history. if you do 'continual learning' on this, lets say in batches, then there is a relationship between weights and context

• probably include importance sampling in the loss function for any choice

• it doesnt seem like recsys is the right framework at all. its not really a recommendation. its more a prediction that influences the principal's behavior in some way. even if a 'rec' is good, i wouldnt just 'click accept', i would continue working with the suggestion in mind. loss still decreases if what i type is close to the suggestion. hmm. it changes how the loss function is modeled.

• unresolved thread between online DPO, online IPO, mirror descent, nash mirror descent, MUPI toy example, recsys vs "suggsys"/two player game vs human model + agent model vs single model

• tight feedback loops as the unifying value prop of the local vision

• it feels like the transition from recsys to [[Phase 1]] is an implicit statement on replacement vs augmentation, but haven't fully articulated it

• if the agent is being sampled and those are being shown to the user, and that impacts the user's behavior, since even though the policy may be constant, the input information is now different. the agents responses need to be stored in the data itself, so the human model can update to properly map how the human reacts to the agent's data. if the model is changing constantly to account for the changing agent policy, does the steady state behavior ever actually do something useful?

• i need to review the loss function in [[Phase 2]], keeping in mind E2E-TTT and AssistanceZero. i dont like how its combining losses vs having a batch updated ref policy from BC that is used as the baseline for IPO

• are there details with phase 1 that im overlooking? if its in an online setting, do i need to consider meta learning initialization, or replay buffers, or batched descent? and which loss is actually being used, just cross entropy? how often do i update relative to the plans for phase 2?

• what are the tentative phase 3 plans, if any? i need to create a sandbox/harness that the agent can take actions in, and simulate in, seems to be the primary issue. but if i have that, what algorithms make most sense? Assistance games built on top of reward inference? is human preference data ever used as a value network for rollouts, rather than strict env rewards?

• how to balance understanding exploration vs exploitation? if a model recommends exploitative action a and i take explorative action b, and the current hypothesis trains the model to have produced closer to b,

• how does IPO even work when there are no 'probabilities' since the state space is so large, and you'll essentially only ever see one state at a time, and you're relying on semantic similarity analysis between states to determine actions? since IPO is supposed to be matching a distribution of preferences rather than binary. it might actually not work.

• this is likely where complexity analysis comes into play since you need to understand the actual costs associated with the training

• given to team:

  • first concern is lack of specificity in the purpose of the model. concretely, do question answering abilities and tool calling abilities need to be retained?
  • second concern is potential lack of superhuman performance. if purely prediction, many results show that it loses concrete reasoning ability (a form of overfitting). the goal would likely be augmentation, not replacement, of intelligence. there are solutions for this (synthetic data, but relates to the Q/A issue) so we’d need to actually run the tests to see, but tbd
  • third concern is impact of initial weights on test time learning. many results showing dynamic evaluation suffers when weights are initialized with a lack of meta learning. again something to be actually tested, but important to keep in mind
  • fourth concern is a potential decoupling of the learned human model vs an agent that learns to interact with the human model. which approach is desired? and why? its unclear. it does not impact phase 1 (imitation learning / behavior cloning) but it impacts data structure for long term algorithms
  • fifth concern is the modeling of the hypothesized phase 2 (recsys). to what extent do the recs influence the human, and as such invalidate the learned human model from before recsys? does this matter?

• at a high level, phase 1 is imitation learning / behavior cloning to produce a human model. it needs to be continuously updated. phase 2 is intended to infer a reward model, but the current method intends to use contrastive loss from counterfactuals generated from the human model, which doesn't seem robust. reward inference basically requires preference data, from my understanding, unless you have verifiable rewards that you can bake into training that do not drift during test time.

• schmidhuber suggests alternating SSL with RL, where you update the SSL using history during 'sleep', then collect data during 'wake'. good framework but doesn't immediately apply here since there is no environment for the agent to go collect data in

• the goal of reward inference is to allow the agent to then propose actions that the human recognizes as better towards its goal than could come up with itself. the implicit bet is that a good human model essentially unbounds rationality to allow for more creative ideation. is reward inference necessary for this though? you could just sample the human model and be done with it. there aren't really recommendations. if anything the model's actions being introduced into the state of the human change the environment state space (there are now 'read' events from 'assistant'). do we actually want model behavior to change at this point? i dont really see why we do. can this human model be used for multi agent training eventually? you can likely use it for decentralized MARL policy gradient like the MUPI example did if you have multiple agents assuming the training data involves messages from others.

• the gap here would theoretically be 'superhuman performance', but practically that might come from the base properties of models being way higher throughput. otherwise superhuman performance needs actual environmental rewards, which i dont believe it.

• or you use tight feedback loops with humans, but how is this different from just imitation learning?

• the pro of this simplification is that its simple. i dont see any cons besides ego

• the two gaps seem to be (1) can you get superhuman performance from imitation learning? can you get superhuman performance from preference data?

  • what is the definition of superhuman performance? the strict definition is that it can achieve my rewards better than I can, but that is a competitive framework. the cooperative definition is that it can take actions that result in my achieving my rewards faster than i could otherwise, although thats more of a superhuman system, not a superhuman model.
  • in this framework, the hypothesis is that a human model presented to the user after being sampled MAY be a superhuman system.

• what would make this more of a superhuman system? i think the more specific version of this question is: are there preferences that are not expressed in behavior? if there are actions that take work to think of, or i do not think of, that would be more helpful for me, then yes. but that covers sampling. without DPO/IPO loss, if i take a sampled action that i would not have taken earlier, then we do not change the model weights. should they be changed in that scenario? the top K sample already produced the gold standard as far as we know? technically we'd want the weights to change to produce that as a more dominant sample? either higher ranked or if top ranked, higher prob? does it simply help with the existence of hard negatives that can result in improved training?

• doesnt the thinking machines work state that putting the model's samples into its training data reduce benchmark performance?

• data is likely most noisy from (1) notifications (2) conversations (3) reminders (4) having a thought pop up to do something previously forgotten. this could drastically impact performance

  • for example if im typing something and my dad texts me, and my next action (reply) makes no sense without the personal context (but you would likely exclude both anyways)

• seems like notes, ai chats, and browser use are the dominant initial sources

• i think it makes sense to consider that behavior cloning for a human model may lead to a superhuman centaur system, and that further contrastive suggestions that influence the human used in an IPO loss produces hard negatives to make the model more robust.

• how does importance sampling and KL divergence come into play in the loss function?

• with the BC -> IPO pipeline, there would be an inferred reward, although its context dependent. also we would want to keep in mind how the data will immediately shift after suggestions start, since they will now be part of the history. this likely has implications i am not fully grokking.

• reverse KL is intuitively, from the model perspective: "if I'm guessing something it better be right", whereas forward KL is intuitively, from the model perspective: "I should make sure I say something for anything important, even if its far off"

• online BC/IL and noisy BC/IL and explorative BC/IL need to be considered, as an optimization

• noisy BC/IL research seems to indicate that ranked demonstrations allow for better than demonstrator performance, which supports the IL -> IPO pipeline

• env dynamics and the ability to actually take actions would theoretically be needed for any phase 3

• should the BC loss be token level or action level? how does that impact training and data construction? current Paper specifies token level

• how should the BC model be frozen and updated and how should the IPO model be frozen and updated? for example should the BC model update every 10k steps, then freeze for 10k steps as IPO model uses it as a baseline? etc

• maybe its fine if the loss is combined if youre freezing/updating reference policies appropriately

• how to even display a suggestion/sample might be complicated. what if the model wants to say that im going to add a phrase in the middle of a long series of notes? how would that be shown? if youre going to use IPO loss in phase 2, then the UX of the suggestions is a massive confounding factor. confusing UX -> ignored -> assume the sampled content is pairwise worse, rather than the presentation of the content

• adaptive UI as a vision component

• if 80% of next write action today involves talking to a chatbot, how does that impact the direction of this work?

• regarding the work around next token prediction resulting in memorization rather than understanding, which could be solved by Q/A synthetic data (but that requires the model to retain question answering abilities), i wonder whether another form of synthetic data as a step between phase 1 and phase 2 would be the model just outputting its understanding of the users actions and local goals, with the user outputting whether thats accurate somehow via a revision or a binary signal. the issue is that the UX becomes way worse since i now have to go out of my way to provide feedback to the model rather than my work providing implicit feedback with no changes to how i operate

• i think a visualization about how the suggestions priming the user to do something, without actually 'accepting' them, results in the model outperforming demonstration (well established from literature around preference data) while also moving the user towards goals faster, so it becomes a symbiotic, positive sum relationship. the visualization would be two sloping exponentials, with the human consistently above the AI, the AI getting pulled up from pairwise contrastive learning outperforming SFT, and the human being pushed up as a result since they are still in the loop

• do E2E-TTT and Sutton's NetworkIDBD address the same thing?

• E2E-TTT stands out as the most practical method of addressing common failure modes around lack of long context learning / drifting data distributions

• you just cannot outsource your thinking. you get left behind, realize you're left behind, then need to catch up. true replacement involves you not doing anything

• did cursor sell out when they pivoted to cloud agents instead of doubling down on personal tab completion?

• LongNAP stands out from my original formulation in a few ways

  • uses a VLM to semantically label raw screenshot/video data
  • focuses on prediction/replacement rather than suggestion/augmentation
  • the training involves GRPO on the reasoning that produces a next action sample, while still using self supervised learning, rather than simply cross entropy loss
  • generates a sequence (8) of next actions not just one step
  • the predictions take the form of the VLM's labels on the computer use, grounding the behavior
  • uses unfiltered memory logs as part of the reasoning via retrieval
  • intended to extend to computer use to "finish predictable tasks for users by acting on predictions about what the user would do next"
  • i feel like an intuitive understanding of it is using GRPO and self supervised training via prediction to optimize how memory works, since its fundamentally based on an append only log of history that is RAG searchable with BM25

step 1 in local tasking is mostly described in the beginning of [[Phase 1]]. the only thing that feels weird is how to think through phase 2. the model will be displaying the next write actions it predicts based on the given history. that will actually result in an additional read event the model did not previously consider when it was sampled. how will it ever be able to learn to predict the write event well unless it learns to predict itself? does this actually prevent training or will a good model just figure out how its samples impact the user? the loss function forces its output to (1) be close to the users output and (2) distance itself from counterfactual samples.

it feels like if you just mask the model's samples from the model training then it's fine. the samples will impact the user but if that impact is positive and recognized as such by the user, then you are teaching the model to now predict a 'better' user, which is the goal anyways. if the impact is positive and unrecognized, the user will not use the product, which is a different question. if the impact is negative and recognized as such by the user, it will not be used, so it doesn't matter. and if the impact is negative and not recognized as such by the user, then nothing can help that user anyways

i need to figure out the details around the continual retraining of the pure BC policy and how that relates to whats currently scoped, but phase 1 and phase 2 seem inseparable because the BC policy will only be practically useful if displayed to the user, which leads into thinking about how to properly model that relationship

continual BC retraining in and of itself is a huge challenge due to catastrophic forgetting risks

how to handle write actions done by AI, not by the human? likely just dont consider it a write action, or label the author and only train on human author writes. how to detect? a bit harder. for example if chatgpt app writes to obsidian, any demarcation without requiring screen recording or world dynamics models?

longNAP actually produced a result in terms of user action prediction months ago. thats something to keep in mind in terms of what the bar for success is. feels like THE baseline to use.

The “goals” I’m expecting the human model to implicitly map (for it to predict well) likely to cost in the data itself, since I work in a team where I communicate my goals to them

the formulation that the human does not know their reward function, or that the latent goal is nonexistent, is incorrect. there is a latent goal that can be expressed. its just the policy demonstrated to achieve it may not be optimal.

"ECHO but with my thoughts" formulation i stated previously reminds me of having the model predict specifically how i respond to its suggestions, rather than masking the models suggestions as it learns to model my behavior from else prior context. not sure this is useful

there is a risk of amplification of the user's behavior whether thats positive or negative. but its more leverage, if built well. whether thats value creative or not seems up to the user?

i kind of want to understand whether [[Phase 1]] makes sense from a theoretical perspective. like is this a solvable problem at a fundamental level? what is the complexity?

if local scaling laws are positive, then it may imply that building a better application for knowledge work is worthwhile, since improved data granularity improves custom models which leads to a large inference business

longNAP continues to stand out

is OPD/OPSD the goated algorithm since you literally just tell the agent what to fix? and then during batched training it increases the probability of the desired tokens from the prompt before when the fix was stated? i wonder if anyone has done this / people are doing it

may be useful for better fact retrieval as time goes on: big blocking question right now is related to the above context details point, why longNAP learned reasoning for retrieval, how e2e-ttt consolidates information, online convex optimization algos, how other memory startups handle it, the continual learning research for the X account above, token efficiency of retrieval, etc, all pointing to a better understanding of how context is managed / trained, and how that surface bumps into the intended surface of the prefixed sliding window default choice. i.e. how to actually use the collected data

• charlie o neill work
• cartridges and broader engram work
• e2e-ttt work
• omar shaikh work

discusses research similar to longNAP https://claude.ai/chat/d3de1f6d-cd67-404b-a93a-936b3a662d7e - the most interesting related paper which I put into algorithms uses something called LifeTrace for collecting data, takes snapshots at 1 Hz (can we track keyboard usage and snapshot during a pause, like it occurs in git in this obsidian?) - another one frames 'when to assist' as when its high likelihood that the "user would turn to an AI assistant right now" which is interesting

internalizing continual learning - what shape of continual learning, from an algorithmic perspective, do I most believe in? How does this impact data collection as a valuable slice, as well as the shape of the data processing? - what are the hardware bottlenecks to continual learning? how do compute resources change on a very fundamental level with continual retraining? - does my intended SFT approach (autoregressive train on next action prediction i.e. causal mask) violate i.i.d. data assumptions? does shopify's generative recommender have the same issue? what do those assumptions actually mean in practice? do all online or continual learning setups violate this? how does this relate to the practice of storing rollouts in a buffer that you then sample from? does that essentially fix i.i.d. for continual learning scenarios? - https://gemini.google.com/app/9de51346992f5bae wild stuff

useful swath of research by https://arxiv.org/search/cs?searchtype=author&query=Shrivastava,+V, lead author on ECHO paper

https://cursor.com/blog/problems-2024