- How does Thinking Machines’ interaction models impact inference architecture? Consumer use cases?
- https://thinkingmachines.ai/blog/interaction-models/
- https://thinkingmachines.ai/news/interactivity-research-grants/
- https://thinkingmachines.ai/blog/ probably should review the old ones too ^f94b64
- do interaction models solve theory of mind for the given interactor? what does the data for interaction models look like?
- theory of mind according to blaise is modeling other minds. modeling your own mind is consciousness according to blaise [[Thoughts#^963ac3]]
- need interaction because too dynamic and complex to statically hardcode preferences
- https://arxiv.org/abs/1606.03137 CIRL https://gemini.google.com/app/bb201e6013c0056c ^2bff57
- what is the relation between interaction and MARL? how does that relation explain karten's take that "realtime envs with agent actions" is the new paradigm? seemingly relates to 'prospective learning' which relates to 'dynamic preferences'
- prospective vs retrospective learning??
- it does feel like predicting behavior of user you are engaging with would be drastically economically beneficial for modern labs. are they not doing reward modeling on simply predicting what the user says? like the conversation version of echo? they must no? try it as [[Experiments]]?
- i defined an interaction model to jakub as a model that optimizes for a dynamic reward that is clear when hit but hard to codify before hand. so in some sense the model learns a function approximation of the reward model of its user in some ways superior to the user (it cannot necessarily produce ground truth but it may predict superhuman-ly)
- conceptually, it makes sense to prove that you can find a local or global minimum on a static loss function because trying to find a local or global minimum on a dynamic loss function (which theoretically is what continual learning is)
- Corrigibility might be a overlapping set of interaction
- https://arxiv.org/pdf/2307.15217
- https://arxiv.org/abs/2504.07091 https://cassidylaidlaw.github.io/minecraft-building-assistance-game/ assistance games from head of AI safety at deepmind
- probably need an intuitive understading of POMDPs
- slow response times make it much much harder for "interaction" to occur
- https://arxiv.org/pdf/2606.03237 another paper as a call for cooperation paradigms during training. considered solipsistic. referenced in "from agi to asi"
- agi to asi paper https://arxiv.org/abs/2606.12683 called it 'interactive learning'
- https://willcb.com/blog/feedback-loops/
- "We started with a set of pretty ambiguous questions:
- How can we model online recommendations in a way which gives rise to the kinds of feedback loops observed in reality while remaining analytically tractable?
- Can we gain any insight into why algorithmic feedback loops occur, and design recommendations algorithms which avoid their potentially harmful consequences?
- How do we characterize the space of possible outcomes and algorithmic benchmarks when agent preferences are adaptive as a function of our recommendations? And we found satisfying answers to all of these."
- "We started with a set of pretty ambiguous questions:
- https://trajectory.ai/field-notes/scaling-sdpo one of the algorithms that potentially enables continual learning, but SDPO feels categorically different from CIRL / assistance games, unsure why/how.
- https://www.youtube.com/watch?v=GwSl1OH1i4w anca dragan interview
- 'legible' motion planning -> acting in a way that observers can easily understand and predict. prerequisite for collaboration (and trust) as well.
- you cant just make predictions in isolation because it depends on what others will do and what others will do depends on what you will do
- LLMs currently will give you an answer for the sake of giving you an answer. whats likely better is multistep collaborative interaction that focuses on helping the human even understand what their goals are
- evals guide everything
- nothing in the evals rewarding the model for admitting not knowing
- (relates to randall take, i need to find it)
- her safety focus is A) closing the loop. reason about the effect of your actions on the world and B) holding uncertainty about what the objective actually is
- reward hacking is the result of the reward model being confident when it should not be
- gave a dog trainer example where you actually want to introduce randomness to the reward signal to induce caution in reward hacking/overfitting
- A good continual learning system will tease out orders of magnitude more context from its users and be orders of magnitude more retentive
- https://arxiv.org/abs/1701.07570 how does this dynamic regret paper (one of will brown's favorites) relate to inverse RL or reward inference more generally?
- https://jacobxli.com/blog/2026/machine-studying/ seems very relevant to continual learning, possibly good benchmark
- https://www.datocms-assets.com/104802/1781805778-baseten-research-sft.pdf discussion on dataset construction, hyperparameters, etc for fine tuning ^f5c205
- https://arxiv.org/pdf/2405.17713 AI Alignment with Changing and Influenceable Reward Functions Dragan 2024 ^2e1db5
- an example given here is if someone is trying to lose weight, should the model optimize for losing weight even if they get higher short term reward for eating candy? if the model says no candy the user might be mad. if the model says candy the user might be mad. not sure how they reconcile but the way i'd reconcile is always optimizing for long term rewards, and choosing short term rewards to the extent by which they increase intrinsic motivation to continue pursuing long term rewards.
- probably relates to research around intrinsic motivation / laziness in models. there is likely an actual term for this in human psychology
- https://people.eecs.berkeley.edu/~anca/publications.html worth exploring. lots of relevant information
- https://claude.ai/chat/90f6570c-41ea-4cd4-8e2e-f54ef8df4197
- https://claude.ai/chat/8d0ee16d-491f-4f94-b416-626b7c42b745
- https://gemini.google.com/app/f44fe68a684ec176
- seems to relate to [[AIXI]] since the agent manages a set of possible 'true' reward functions and adopts a policy based on its observations + coupled with its environment a la MUPI if the fear of persuading the human to change to make its own job easier is well founded
- git history as the history over which the agent learns in the [[Product]] sense feels directionally correct but overall lacking in context (what i read, what i see, what i conversate, etc)
- https://gemini.google.com/app/f44fe68a684ec176 early part of this topic. it eventually degrades
- https://gemini.google.com/app/d3409327dab2a45f explanation for PNLC https://arxiv.org/abs/2505.18098 vs NLAC
- can you apply the step from PNLC -> NLAC to PPI? think i had a claude chat somewhere about this. the take seemed to be yes its possible since LLMs are fundamentally the same structure as the GRUs that were tested. again also seems related to SDPO
- is NLAC similar to continual/interactive learning if you replace the critic with a human? starting to feel like this vague idea doesn't actually make sense because what are you even learning/predicting?
- how to deal with states that truthfully reward the user but the user doesn't recognize as such? this is probably the basis for sycophancy. probably similar to P vs NP. i can verify that i like something after i have it but i cannot tell you or codify it before hand.
- https://arxiv.org/abs/2601.20802 how does SDPO relate to interactive / continuous inverse learning? seems relevant
- https://www.forethought.org/research/design-sketches-angels-on-the-shoulder ^4c96d9
- possible counter weight paper but might be stupid https://arxiv.org/pdf/2408.16984 interesting paper that seems, from the abstract, to conclude anthropic's approach is superior, but then says that this leads to pluralism?
- technical possible how to explanation nitty gritty RL work https://x.com/SemiAnalysis_/status/2066941079920791760?s=20
- agency arises when reward signal is peer approval in humans (forget where I read this)? how to set a dynamic reward signal of peer approval in LLMs? relates to CIRL. perhaps relates to (non)assistant training paradigm
- probably need to go through these (recent papers by ECHO author) https://arxiv.org/search/cs?searchtype=author&query=Shrivastava,+V
- https://www.mdpi.com/1099-4300/28/6/596 genewein and hutter explore the extent to which LLMs approximate AIXI and what the specific challenges are https://gemini.google.com/app/e5723b735ee76668
- seemingly a gap between append only agent turn logs as some vague 'memory' solution vs use as a formal interleaved dataset where the agent can learn causal loops, which opens up multi agent systems which opens up collective intelligence. again, ECHO seems to be the first version of this
- prospective learning vs retrospective learning?
- but when agents do next token prediction that's considered an 'action', no? whats the actual difference
- https://x.com/kalomaze/status/2065498921443438928?s=20 data mixing to improve learning
- https://x.com/dwarkesh_sp/status/2062353335529935114?s=20 10min OPSD explainer
- seems useful as a beginner initiation https://github.com/gvkhosla/pi-tinker but perhaps slop
- another example end to end training run https://docs.primeintellect.ai/hosted-training/end-to-end-run
- https://substack.com/home/post/p-197387291
- Feeling like this post makes arguments that could be usefully extended by well analyzing the nvidia tech report and microsoft tech report recently and coming to novel conclusions about scaling complexity
- This also seems to indicate that the karpathy hire on pretraining is due to the fact that pretraining was paused rather than saturated, but incoming compute will continue to deliver major scaling gains
- https://www.goodfire.ai/research
- long horizon Q learning from chelsea finn https://arxiv.org/pdf/2605.05812
- nemotron training stack https://github.com/NVIDIA-NeMo/Nemotron/tree/main
- may be a relevant algo for CIRL / personalized training, by Levine. unsure how diffusion-only it is https://q-guided-flow.github.io/
- https://x.com/emollick/status/2064395281903346013?s=20 good discussion on what fable feels like. important to internalize and imagine. as it relates to the thesis of data, reward models evals, and especially programmability in prompt space vs weight space
- parts are similar thesis https://www.a16z.news/p/institutional-ai-vs-individual-ai coordination as first pillar here very similar to my multi agent take. The signal part feels like what im trying to do with the future version of these notes and my listed problems. Unprompted is also a novel thought ive been exploring, similar to proactivity per the randall takes.
- increased sparsity helps code a lot but not necessarily other domains? https://x.com/eliebakouch/status/2064736476995146014?s=20 from MAI report. i wonder whether this is evidence of differing capabilities
- another relevant thesis piece https://x.com/RyanPGreenblatt/status/2065185280295100481?s=20. for all the theses pieces, where do i agree and where do i differ?
- another relevant thesis take
- 5: What observation would make you say, “this model has taste”, rather than “this model has learned to imitate Gwern’s evaluations or predict Gwern’s past preferences”?
- G: The most convincing thing would be if a model was right, but I thought it was wrong and eventually realized that it was right, and had just seen deeper and better than me.
- super relevant analysis of algorithms, compared to prompt space, and data tricks/tips needed to juice performance at least in that specific domain https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/ ^72b341
- unfortunately the implicit conclusion is that prompt space algos get 78% accuracy and most RL algos get below that. only when you combine advanced knowledge and implementation of multiple different RL algorithms over the course of two months can you get 85% accuracy. does RSI lead to frontier models that can develop these algorithms for you? does that imply that you can always achieve better than frontier performance? how does mythos's ML research ban change that? is that easily jailbroken?
- relevant nit on RL training for PPO https://x.com/MatternJustus/status/2072477044932551099?s=20