speaker-0: My guest today is the brilliant Stéphane Ralef, an assistant professor at RPI where he runs the BaseOps lab. He's also the creator of Baseflow, the Python library for Amortize simulation-based inference that we first covered back in episode 107 with Marvin Schmidt. Since then, Stéphane and his team have pushed the framework forward considerably and have been a big fan of the whole amortized inference paradigm ever since. So yes, we'll dig into what amortized inference is, why it's particularly well suited to messy, noisy, low-resolution data, the kind you get in psychology and neuroscience, for instance, and what the sim-to-real framing is all about. You train on simulations, you deploy on real, unlabeled data. But the really exciting part is that Stefan is going to do a live demo. on screen, so if you're listening, you'll want to jump to YouTube for this part. And that's going to be a live demo of an AI agent skill, him and I co-developed to guide you through a state-of-the-art amortized inference workflow. We'll run it, go through diagnostics, parameter recovery, calibration coverage, and actionable next steps. All automated, all open source. This is Learning Vision Statistics, episode 157, recorded April 24. 2026. Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible. I'm your host, Alex Andorra. You can follow me on Twitter at alex-underscore-andorra. like the country. For any info about the show, learnbasedats.com is Laplace to be. Show notes, becoming a corporate sponsor, unlocking Bayesian Merge, supporting the show on Patreon, everything is in there. That's learnbasedats.com. If you're interested in one-on-one mentorship, online courses, or statistical consulting, feel free to reach out and book a call at topmate.io slash alex underscore and dora. See you around, folks, and best Bayesian wishes to you all. Hello, my dear Abasians! Just a few words to let you know that our soccer factor model is now available as an app on the App Store for iPhones in 175 countries. It's all thanks to Maximilien Goebel, who's been working on that for month. Check it out if you want to see what Abasian model looks like on your phone and also sounds smart during the coming World Cup, starting June, from some messy code on a Jupyter Notebook. to slick looking app, who would have thought? So if you wanna see what a production based model looks like in the wild, check out the website, I put it in the show note. Of course, check out the app on the Apple App Store. I also put that in the show notes. And if you want more details about the soccer factor model, well, you have that on the website, of course, but you can also listen to Max Goebel on this show. I put the link to his episode in the related episodes for that one. And I think it's a good idea to have a refresher with the World Cup coming up. We'll probably have some even better football news for you coming up in the coming weeks, but I don't want to spoil it for you. So in the meantime, let's listen to Stephanie Radeff and talk about amortized patient infants. Stephane Raef, welcome to Learning Patient Statistics. Yeah, it's great to finally have you on the show. We've collaborating for a few months, maybe even some years now. don't know, it's bit blurry in my memory. yeah, I've been, of course, talking with you and the rest of the Bassflow team since I discovered about Bassflow in episode 107 with Marvidge Schmit.

speaker-1: See you, Alex.

speaker-0: And yeah, since then I've been a really big fan of Baseflow and Amortization inference in general. I think it's also a great setting to learn more about neural networks and their power and how they can help for a type of inference that is dear to our heart here on the show. And so yeah, it was long overdue to... to have you on the show and actually I'm very happy that it happened now because we have something really cool to share with people today which I was also super happy to collaborate on with you but before that, as usual, let's start with your origin story. What are you doing nowadays and how do you end up doing that?

speaker-1: So let's keep the long story short. I'm currently an assistant professor at RPI where they let me set up my base lab, the base ops lab. I didn't start out as a Bayesian. Most people don't. I guess I started out as a psychology student in Heidelberg University. I had no idea what I wanted to do, but I really liked neuroscience. So I joined. I managed to join an unpaid internship job at the neuroscience lab in Mannheim. And there, my mentors were really kind and smart people that they let me participate in different experiments. though I was not of much help at that point, but because I showed some motivation, one of the, my advisors there offered me a job on the condition that they learn how to program over the summer. So I took her up on the challenge. I got a book, Zero to Hero, in Matlab.

speaker-0: ⁓ okay.

speaker-1: which is not a typical language to do any kind of basic analysis to begin with. But I managed to learn a lot during the summer. grabbing a book, going to the beach with the book. And when I came back, suddenly I found out you can do a lot of things with programming. And this was the pre-AI era. So coding skills were still important. And of course, I introduced a lot of books and so on. The usual ones I didn't crush any space shuttles in the process, but I fell in love with programming. So I even decided to add a parallel bachelor's degree in computer science. And at that time I also started another student assistant job at the department of quantitative research methods in hydrotech. That's where I met a PhD candidate. Ulf Mertens, was already towards the end of his PhD. He was working on something called adaptive design optimization, which is back then was very, ⁓ it was very hyped and it's a cool idea. Basically says, have your experiment as part of your model and try to have informative measurements as you go. And the way they were selling this idea was suppose you have a neuroscience experiment which takes four sessions each session, it's two hours for a participant. That's a lot of time. What if you could do much better? What if you could ⁓ gain the same information for just 20 minutes? That's how they were selling. ⁓ We're reading a lot of papers by Jay Mung from Ohio State University. I only knew from papers, I had the pleasure of meeting him actually last year. at the conference and we started implementing this in C++. Yeah, eventually we found out about Stan, so we looped in Stan and it was very slow, as you can imagine, because this is also the pre-neural network era in basic statistics. At some point you already had amortized inference and all that, but enthusiasm has waned about adaptive design optimization, but I digress. So I figured out that's what I want to do at that point in time. And I decided that I have to try to avoid the labor market as long as possible. And I happened to be this graduate school about statistical modeling with a focus on psychology. I applied, I got the PhD and then I met during that time, I met two of my mentors Ulrich Kötter from Heidelberg and later Paul Bjorkener. So everything I know about deep learning I know from Uli Kyote. Everything I know about PESO statistics I know from Paul. So it's important to have smart and open-minded mentors, I would say. So I was rather quick with my PhD. Took two and a half years. Then transitioned to a postdoc, the usual story. And a colleague of mine just randomly forwarded me an email about my current job description. and said, okay, maybe it's a good idea to start applying something beyond the postdocs. I okay, let's do a practice run. So I applied and I ended up getting the job. And now I'm here.

speaker-0: Wow. Okay. Yeah. Yeah, I love it. So that's, ⁓ that's definitely, ⁓ somewhat, senior space, but I like how it sounds like you've been, you know, conscious of all the choices you've made throughout the journey. I think it's, yeah, it's, it's beautiful to see that it's really something you were really interested in and passionate about since the beginning. and how So that's the beginning and how you were drawn to patient inference and deep learning in particular. Is that when also you started being interested in in amortized methods and you know, like yeah, basically ABI and the themes we're going to talk about today.

speaker-1: I would say Bayesian inference happened to me in a way that at the time where I was transitioning from my master's to my PhD, the so-called Bayesian Revolution in the social science was happening, partially in response to the so-called replication crisis. Now, for you viewers who don't know what the replication crisis is. It was the precarious situation that a lot of psychological effects tend to not reproduce in independent studies. And there were some very big papers, about 2015-16, showing that, okay, one-third, two-thirds of the effects cannot be reproduced. there were, it's still not clear what caused the crisis, whether it was a theory crisis or a methods crisis, but... methods crisis camp proposed as a solution, swapping out p-values for base factors, basically. And there was an outpour of gazillion of papers on like how to do any classical analysis in a Bayesian way. There was a lot of excitement about Bayesian analysis and parallel to adaptive design optimization, we also started fiddling with approximate Bayesian computation, which is the sort of the brute force precursor of our ties based on inference where your model is too complex to formulate as a set of analytical equations. So what you're doing is you're using simulations and you're trying to compare the simulations to your real data and you keep the simulations that are as close as possible to your real data and then the remake simulations and corresponding parameters to check the signatures. The signatures are easier are your posterior space. And we implemented a lot of these algorithms. I even implemented at that time a graphical user interface for the basic methods. And at some point, think, during the lab launch, my psychology advisor suggested, well, can't you just train on your network on this data? said, actually, that's a good idea. Let's try it out. And it worked. In retrospect, the idea is very obvious. I think the idea was also developed independently in various areas. For example, neuroscience, particle physics, population genetics. I was myself a bit oblivious to a lot of the related work when we were starting out. We mostly driven by excitement because something big is happening with these neural networks. AI was already cool. It was not super cool like today, but it was starting to get very cool.

speaker-0: Hmm. Okay. Yeah, yeah, yeah. I mean, yeah, all of that makes sense. it's, it's like, it does sound like it was the, you arrived at the intersection of both fields at a very good moment and a very ripe moment where there was a lot of very interesting work to do and, probably also computing power that was now enabling to do that work, ⁓ which is something, something you need definitely, in So we've talked on the show, as I was saying, about amortizing friends, with base flow. I will put in the related episodes, show notes of this episode. Episodes 151 with Jonas, Jonas Arruda, and episode 107 with Marvin Schmidt. I will also put episode 18, just because it's about the replication crisis with Daniel Lackins. And also episode 51 with Aubrey Clayton, about exactly that author of the book, Bernoulli's Fallacy. Very good episode. that's for folks who want to dig into that. Now, to get back to Amantize the Influence, we don't need to recover that ground. So don't worry. We can just pick up where episode 151 was. But I... do want to understand your specific training to it, which is I've seen you write and talk about something you call seem too real in the sense that you train on simulations and you deploy on real data. I think this is quite a change of mindset for lot of listeners. So what do you mean by that concretely?

speaker-1: question. So I say if you're working in the deep learning realm, are two major ways in which you can use synthetic data. The predominant approach is what's called synthetic for ML. The goal here is to create a bunch of fake data that ideally resembles the real data and you use the step as data augmentation, especially in cases where real data is scarce. Think again, a lot of brain-computer interface research, neuroscience, typically deals with low participant counts. We're talking about a few thousands. For example, that's not a great starting point to do deep learning. So wouldn't it be dreamy if in these settings you had a simulator that can produce infinite streams of training data? and even better if you could annotate this data. ⁓ And this is a very fruitful approach. Some recent reviews show this for particular niche cases, for example, I recently came across a review on data meditation for EEG studies on automatic diagnosis of major depression. And the results there were that state augmentation. This can increase downstream classification accuracy anywhere between 1 % to 40%. So it really depends on the application, but it's a cool approach. And in these settings, generally speaking, you're training on purely on simulation data. You're in a semi-supervised setting. have basically two domains, real data, simulated data, and try to... extract as much information as possible from both domains. Now the less popular approach is where our research also falls into. It's a simulation-based inference approach where you use simulations to identify mechanistic models. The simulation itself is an epistemic tool to gain insights into some process. A lot of the work on SPI is seen to be real. You basically train your network on gazillion of labeled simulations. And then you deploy the network on your real data, which is unlabeled for construction because you don't know which parameters generated your data. So that's the basic idea, but I have to say the lines are getting very blurry. For example, you have nowadays world models, right? So world models. about neural networks that can simulate physics and state spaces from, by just training on video sequences. In this case, have a Rio to SIM, but then you can use world models as simulators for other downstream neural networks. So you have a sort of loop, Rio to SIM, SIM to Rio. And there are also a few SBI papers, SBI simulation based inference, which in fact, do try to plug in the real data in the loop as well. We're also guilty of a few papers in this realm as well where the claims basically, the real data provides additional signal. Not only that, it can robustify the networks against cases where the simulation is an imperfect representation of reality. But I think we'll come to talk about this in more detail.

speaker-0: Yeah, yeah, this is really, really fascinating and I love how it complements the modeling workflow. ⁓ Before, so you have something to share with us today live where you'll do a demo of what we've been up to. But before we do that, ⁓ maybe one good way to establish again, What all these methods are about is, as you were saying, you started working on cognition and psychology and so a lot of your applied work is in these fields. So that means messy models, noisy data. What makes amortized inference particularly well suited to that domain?

speaker-1: Yeah, great question. ⁓ Unfortunately, I wish I did more psychology than I actually did. Even towards the end of my PhD, there were some questions. so your thesis was supposed to be about at least a little bit about cognitive modeling, but all you did was deep learning. ⁓ but psychology remains a great test bet for our methods. As you say, we're dealing with a lot of noise there. And thanks to the big data revolution, we now also have access to gazillions of data, okay, from participants taking online experiments, for example. But let's take a step back. Not only is this data very noisy, it's also a very low resolution, as you might think. You don't have a window. into the mind. You are confined to very indirect measures like response times, like mouse tracking data. If you're lucky, eye tracking data. So you can get closer and closer to the brain, but generally speaking, you're dealing with very impoverished data, but you have a lot of it. And if you were doing model-based reasoning, or model-based inference, which is still not the default approach in psychology. In many instances, in psychology and social sciences, you wouldn't think about mechanistically modeling how this behavioral data comes about. You would rather fit a statistical, classical statistical model with regression, mixed effects models, et cetera. In the still in frequent cases where you do have a claim on the underlying processes that generate the behavioral data, you have a model that you want to apply to a lot of data sets, right? And each data set in this case could be a participant, but it could be a data set of multiple participants. And you may have multiple data sets of multiple participants at different sites, right? So you have the same model. You want to apply it many, many times. If you're coming from a classical, I hesitate to say the classical, Bayesian perspective, you applying your favorite Markov chain mode, the Carlo MCMC sampler, you would be applying it from scratch to each and every participant to obtain the underlying cognition. With amortized inference, you train once. You simulate the cognitive process, you train your network to predict the parameters from the data. This takes some time. When we're starting out, for example, we got our hands on data from the project implicit. This is an extremely cool repository of data from years of response times experiments in psychology. And the data is ginormous. We have more than 5 million participants. So we pre-processed around 1.2 million participants, which at that time count was probably the largest model-based analysis that one has done in this realm. And it took us around a day, I think, to train the network. And inference took a few hours. If you repeat this... analysis with the infrastructure we have nowadays and we actually did it in an upcoming paper now we did it with five million people instead of different questions it's going to take training is going to take less than an hour and inference is going to take also around that time we're talking about enormous gains of compute power back back in the days we we extrapolated the time it would have taken you to do this naively with MCMC without any kind of parallelization and it was something in the order of a year. That is not a great timeline if you want to graduate and especially if you then get prompted to fit other models etc etc. So this is what we call amortization. The cost of inference amortizes across repeated evaluations of the model.

speaker-0: Yeah, yeah, I think it's ⁓ really great way to explain it and illustrate what the advantage is. And probably we'll talk a bit more about the nuances of amortized inference. have still a few questions for you about that, but I want to make sure you have the time to demo what you wanted to demo today. So do you want to get into that? ⁓ So basically, you're going to show us an agent skill we've been working together to help people to do amortized inference ⁓ in the real world in production with a very serious and validated process because it was validated by you. So who better than you to do that? yeah, basically a way to teach the AI agents to do state of the art amortized inference. On that note, this is merged by the way on the Beijing skill repo folks and it's all open source. So if you want to see that, the link is in the show notes and you can just ask your AI agent to install the skill from the link. You just give it the link and it will figure out how to install the skill. And now, Stefan, sorry, Stefan. You're gonna you're gonna show us You're gonna show us. ⁓ Yeah, what what these all is about in chariose screen? So if you are if you're listening instead of watching that's a portion of the episode you might want to Tune into YouTube and use the chapters to to jump in there Very easy

speaker-1: Alright, let's get into it. So do we want to first give a short overview on what a skill actually is?

speaker-0: ⁓ no, I think that's fine. I think people know what, what this is. ⁓ so yeah, elevate your pitch folks. think the easiest way to understand that is if you, if you code with an AI agent with them, which I'm sure everybody did here. ⁓ sometimes the agent is not, ⁓ it's not good enough on what you're trying to do and amortized inference. It's something usually it's not good enough at because it's new. There is not enough training data in the and the whole internet and so making sure you train and you teach your agent is very important. It's basically what a skill is doing. If it's still blurry to you, ⁓ I will refer you to the blog section of the LearnBasedTest website which is new now but I've written a few blog posts exactly about that, about the different skills I've developed and I will write one for sure with the Stefan2. to go through the Armatize Inference skill. So yeah, basically that's that and you can take it from there, Stefan.

speaker-1: Excellent. Thank you for the elevator picture Alex. So what we have here is the entry point of the skill. And this is in the context of a skeleton of the skill. This is the description which should express user intent. So when you make a request to your agent, it's going to see if the request matches any of its skills. This saves you a lot of

speaker-0: Hehe.

speaker-1: tokens basically because skills follow what we call in computer science, progressive disclosure, right? You don't load the whole thing into memory. Only load things on demand if certain conditions are met. Right? So the way progressive disclosure is going to work here, I would ask the agent the following prompt, and I have already prepared this for the podcast because I didn't want it to learn to run in the background for five to 10 minutes, is usually the time it takes and it's also interactive, right? It's gonna ask you if you wanna run the script and so on. I'm just going straight to the essence. My prompt was this. I want to calibrate the Heston model in Heston.py using simulation-based inference. I don't have real data yet. I just want to set up, and interpret an amortized Bayesian workflow. Okay? Now this prompt triggered the intent. And this is the first step in validating any skill. Of course, you have to make sure that the intent is clearly not experienced. The skill is triggered by what you know, by the relevant problem space. Now, what is this model all about? The so-called Heston model. I took something that I myself by no means an expert in, but this is something that people in my lab are fiddling in. So I said, Why not? This is also a great demo where I myself am not an expert in the modeling domain itself. The Hestad model is a model from the 90s and it's a way to simulate how asset's price evolves over time with volatility is not constant. So it's a popular model used in options pricing. I'm not really sure if it's that useful. in practice, but it's an interesting toy model. And the way this model works is basically there are the two processes. There is the asset price. So you're modeling the asset price as a geometric Brownian motion. And there is also a process for the variance. This is called the volatility process. Okay. So you have volatility over the variance of the geometric Brownian motion. And there is a correlation. But this model has six parameters, okay? We don't have to understand all the parameters. These parameters mean something to quads, okay? For example, we have the annualized drift of the asset, which tells us how fast the price tends upward on average. Okay, then other interesting parameter is, for example, omega. This is the long run mean variance of the process, right? Where does variance settle in the long run? And so on, we have four more of these. But this model is easy to simulate. That is, it takes probably on this computer, takes around five milliseconds to generate this whole ensemble of paths, of price paths, for a given trading day. And the model has access to the simulator, the simulator is in this file. Now how does the skill work? Now the skill tells the agent what are the essential steps of an amortized puzzle. So this is basically an outpour of my brain, my expertise in fitting amortized models edited by Alex. And we're not going to go over the whole skill, right? But the skill basically has hard rules. These are the musts and the nevers. harnesses that we're using to constrain the LLM. It has installation instructions. It has also instructions on how to set up the networks. So for that, we have prepared a set of quick references. So references are important to skills. Think of these as quick checklists, just as a pilot has. These quick reference checkbooks, are ways for the agent to not... try to guess or guesstimate certain parameterizations, but immediately just fetch predefined configurations. Here, for example, based on experience, I have defined different model sizes for the people of neural network, right? For transformers, for example, the agent can choose between a small and an XL model depending on different problem settings. Okay? And there's also lot of built-in heuristics here as well. Now going back to the skill.md.

speaker-0: Yeah, I think it's a great, just as you go through that again, ⁓ I mean, as you get back to the skill.md, I think it's a great way to understand a skill is basically that like, not trying to, it's basically being a manager, think. ⁓ So it's trying to give guidance without being micromanaging and over controlling so that you know that the agent goes into the right path. but you don't tell it what the right path is because it might be a new one that you didn't even think about. So basically it's trying to make sure it doesn't go the wrong way and then letting it be creative enough to find a good way.

speaker-1: That's an excellent way to put it. It's automation with opportunities for creative expression. Okay? Now, the body of the skill basically specifies the outline of a full end-to-end amortized workflow. Now, this is using all the base law interfaces. So, as you can see, there is also a minimum amount of code, which is also part of base laws. core design philosophy will get you from idea to inference in the least number of lines of code. Now, here is another interesting aspect of the skill. Now, we're also going to ask the agent to produce a publication ready report, format it in an aesthetically pleasing way for you to look at, interpret for you the main diagnostics in an advertised space and workflow. and suggest next steps for improvement. Now, one important heuristic that I've built in here is being parsimonious and frugal with your time. So for any new problem, the agent is instructed to first do a pilot run. Okay. So instead of ⁓ spending a lot of time in simulating training, decision things, just to figure out that the model was not properly set up. I'm advising the agent, this is something also I glean from personal experience, fit me with this. For us, it's always good to just have a development set, do a quick run, iterate for less than five minutes. That's kind of my personal benchmark. If you're iterating for longer than five minutes during the development phase, you're doing something wrong. So try to call this preference to the agent too. So it's always going to first pre-simulate. a few thousand simulated datasets, depending on how fast simulation is. Right. Now there's maybe also be the case where you don't have access to the simulator at all. You're just giving a pre-simulated simulation back. The agent could deal with that as well. It knows based on the problem characteristics, how to choose the best possible dev set, right? To get you very quickly from your idea to the first report. All right, so let me now show you what this agent generated. And this is available here. So the run proceeds as follows. You can actually, just to show you that this is genuine, right? So here is the reasoning trace for when I run the model. It gives me a little summary, but everything I need is now in this has to an SBI folder. The agent is also instructed to collect neatly everything into its own folder to say, prepend also version names and so on so you actually can keep track of the different runs. So let's look at this by first run. Here is the report mask down. Let's render this. And the way this is currently structured is like that. We have the training and network configuration, always handy for you to look at. In this case, we had a flow matching inference network. This is Frontier Generative AI families, which we're using to sample from the posterior. We have the summary backbone. In this case, it correctly selected a time series transformer because our data is price paths, or more precisely log differences of the price path. It trained 400 epochs, batch size 32, some general deep learning information. The first thing it did was inspect the convergence, right? Convergence and deep learning. We look at training and validation loss trajectory. We look at differences between training and validation. If you're a deep learning person, you know all that. If you're coming purely from a statistics background, this is new. You're used to interpreting your MCMC trace plots. Think of this as the trace plot. If this looks bad, you don't proceed. Just as if your trace plots don't look good. You just cycle back Okay. No, here's the model successful. That's an interesting thing. This text is here scripted which explains what the loss should say But this is the creativity part the model Judge the training looks healthy the loss decreased steadily from 1.8 to around 1 over 100 epochs no non spikes Evaluations is lost very close to the training loss. There's no over fitting It also spotted that the loss was still marginally decreasing towards the end. And it may be helpful to train for maybe 50 more epochs to reap these last performance droplets that you may want to reap from this application. Next, we see the parameter recovery. ⁓ This is something that is typically very expensive to do. It's non-amortized methods because it requires you to fit your model on hundreds of simulated data sets. For amortized inference, this is just a nice side effect. So we're seeing here the ground truth versus the estimate, and we see that I would say three of the parameters can be estimated somewhat precisely. Two of the parameters show signs of recovery, but not impressive, and one parameter role to this is correlation. between the two stochastic paths is completely unrecoverable. And these things, if I talk to my group, these things are actually expected. So this run, would say, did not produce anything surprising in this case. Let's continue to some more than that.

speaker-0: So here, can you just tell listeners why that would be... why that's not surprising in this case?

speaker-1: Because in Sweden, in some of the papers that, no, we have to note here that this way of fitting these options pricing models is absolutely non-standard. In the finance literature, goes model calibration, and they have other tools, and it's just some other papers already indicated that it's really, really hard to estimate the correlation rule. And now I personally, I haven't done any mathematical analysis on this model. to prove that it's the case.

speaker-0: But that's something you're like you always seen these kind of models.

speaker-1: Yeah, or you expect to see unless you have let's say much longer time series Or some other information for some reason you make it for some in some way you measure the volatility or you get an indicator of the voltage Because all the motor is seeing here is a price path, right? The path of the volatility is latent. Yeah, you know things it can be very hard to estimate a correlation like that. Yeah Okay

speaker-0: Yeah

speaker-1: Calibration coverage, think we don't need to go into detail here. These are more comprehensive diagnostics, which basically... ⁓

speaker-0: Yeah, I definitely recommend people to use these plots. I use them all the time now. They are in the new RVs 1.0 and well in base flow they have been there since the inception which is something I always really loved in the package and that's also why I've bugged Osvaldo, Martin, I think to really have them also in RVs because honestly they are extremely useful. And very important also, like they give you a lot of information about how good your model is at recovering parameters when you're developing and also how good it is at actually predicting. So very, very important. And I love it. can do them for out of sample data for in sample data. You'll see some very interesting patterns of hierarchical models, which usually, you know, they, they under fit in sample. So it can be like. So if you compare models only in sample, can be like, ⁓ my god, the hierarchical model is bad, actually. ⁓ And then out of sample, the hierarchical model almost always becomes the best one, because thanks to the underfitting in sample, it actually becomes much better out of sample. So this is a very interesting pattern.

speaker-1: Yeah, I would definitely absolutely recommend never ignore these diagnostics, especially if they can be computed handily.

speaker-0: Yeah, exactly. Now they can be done very easily, both when you're using Placeflow or RVs. So yeah, definitely do not skip them. If you're using the Bayesian workflow skill from the Bayesian skill repo, it will do that automatically for you. So it will make sure you don't forget that because I made sure of that because I love these plots.

speaker-1: Yeah, there's no way around it.

speaker-0: Yeah, yeah, exactly. Keep going, Stefan. Sorry, I derailed you with my weird passion for coverage plots.

speaker-1: This is great. So what we also get in the end is also the same information in the form of a numerical summary. ⁓ Because to each of these diagnostic plots, you can produce a single number summary, which loses, of course, some information, but gives a general idea of how the model is doing. You can also verify things are reasonable. see calibration errors. They're all in the order of point O1 to point all two fives. is a good result. ⁓ The model also currently does a little bit of qualitative interpretation here, which also matches what we see it saying for, let's go to the correlation row, poor recovery, excellent calibration, which of course is something that is something that throws off ⁓ newcomers that this can happen. But of course you should remember the prior is always well calibrated. So even if you learn nothing, At least you recover the prior, which is per construction. Well calculated. All right. And now the most interesting part is this segment over here, which contains the suggested next steps. Let's see if we agree with what the agent suggested. And the first suggestion is to extend training 150, 200 at most. Right? And the reasoning here is that the loss curve still had some remaining slope towards the end. I would say that's a correct interpretation at this point. Another suggestion is to augment with option price data for raw and theta. This is actually making a suggestion here on how to make the model more identifiable. That's interesting. See, this is not something that you would get from a plain automated workflow. It's also suggesting tightened priors for theta and raw if domain knowledge supports it. fourth, it suggests to switch to online training as a refinance step, which I think is also reasonable to now that we've seen we can get recovery for five out of six parameters. The question is, what is the maximum performance that we can squeeze out of this work felt? So online training is the natural next step. In this case, it's also something that I would have done, especially if the model is so cheap to simulate, right? So you basically basically profits enormously if you can quickly simulate. It literally means an inference stream of training data. And finally, proceed to real data inference when price data are available. Okay, well, thank you, agent.

speaker-0: No, that's that's yeah, that's really awesome. That's really something you've added in the in the second iteration of the scale that we just merged and Yeah, I think this is super super valuable to have these kind of Harness on the on the report and making sure the report is actionable. I really love that. I'm actually gonna Gonna tell you that idea for the other ⁓ skills like the Colzor Rainfront skill and the Bejeweled Warflow skill I think they would both ⁓ benefit from that so yeah you'll probably see a PR from me coming up in the coming days ⁓ where I implement that and I adapt it obviously to the previous skills because I think it's making the agents work even more actionable for the human and I think this is basically what we want to do here with these kind of skills which is like okay do that and and and then come back to me and give me the diagnostic but also what's next you know what are the next steps what do you recommend based on the state-of-the-art research and I think this is really something Akive Tari and I have been talking about on the show last time he came where he has been very focused on ⁓ I don't remember how he was calling that at the time, but today you would call that an AI assisted patient workflow. And his idea was really that like trying to basically multiply the number of people who can recommend you state of the art research without having them with you. So here I'd be doing amortized inference. with you looking over my shoulders and giving me advice, ⁓ which is extremely valuable. Fantastic. Well, thank you. Thank you so much, Stefan, for walking us through that and also contributing that skill. think it's really amazing and super helpful. Is the notebook you just presented available somewhere already?

speaker-1: It is not, but I can definitely just contribute it. Yeah, we should. It's my pleasure because this is, I think this is a really going to be a continuous development process. Yeah. ⁓ During the skill development, I noticed that we lack a scientific way to construct these skills. Right. We're completely reliant on our sometimes tacit knowledge and experience. to first verbalize it, verbalize it. Also, in a way, it's sort of an exercise for us to see if we on. ...explicate some domain expertise, right? But we don't have a scientific way to generally discuss, right? To act as scientists and say, okay, let's systematically vary certain aspects of the skill, randomize orders and see what works best.

speaker-0: Yeah, yeah, yeah, exactly. Yeah, something I do also to test them is making sure I'm testing them on something I really know or a model I've worked on and that test that I've worked on and then see what the skill does. And especially if it does something really weird or recommends something that is not good and that helps me to stress test it. So yeah, but yeah. I agree with you. is my, this is also the super fun part, which is you have to be very explicit about your knowledge and basically write that in the in the markdown files. And this is super valuable. So, yeah, let's do that. Once you have the notebook available publicly, ⁓ let me know and I will add that to the show notes or you can do that also since you have access to the document. So that will be in the show notes and also ⁓ I'll be working on the blog post to announce and explain the skill to ⁓ put that on the website of Learn Based Dance. well, I will of course run it by you and in the blog post, we'll add a link to the notebook for people who want to see and read the demo. And of course, link to your episode. That way people have different ways of... ⁓ understanding what's going on and what they can do with Amplified Pageant inference with this skill and I think it's going to be a very nice ecosystem. Let's say they have the episode, the blog post and the notebook to dig even deeper. So I'm super excited about that. Anything you want to add about this section or can I ask you other... conceptual question about the Beijing inference that I've had for a while and that I'm very happy to be able to ask you today.

speaker-1: Absolutely, we can move on. That was an excellent summary and maybe just add that we want people to use these things, right? Stress test them on your own problems and let us know when they fail. That's how we can make them better.

speaker-0: Yeah, definitely. Yeah, please do that. When you use any of the skills that is on Beijing skills, if there are issues, you can always contact me, but I have a lot of work, so my bandwidth can be limited. If you open an issue on the GitHub, ⁓ I'm not the only one to see it. Stefan will see it, and especially if it's about the ABI skill, ⁓ he'll be even better than myself to answer you. I definitely encourage you to do that. actually, the V2 of the skill, Stefan was was inspired by your own workshop that you taught with Paul Berkner ⁓ a few days ago and actually stress testing the skill with students. ⁓ do that, folks. And also, if like Stefan, actually, you're an expert of ⁓ your field and think that you could be contributing a skill to help your workflow, but also everybody else's. ⁓ Please open an issue on the GitHub repo or even a PR for me to review. That'd be even better. And or contact me ⁓ by email or LinkedIn and we'll get that going. always happy to welcome new folks on this open source project. That's super fun. On that note, I need to ask you, but hierarchical models Stefan because that's that's been Hard to do with amortized inference Historically, so I'd like to see where we are right now because you've you've worked on that a lot a lot especially on compositional amortized inference for large hierarchical models so yeah, what makes hierarchical models hard to amortize and How does composition help now?

speaker-1: Yeah, now we're getting into the hard questions. Now, hierarchical models, you can treat them as the reward at the end of the final level of SPI. Almost all SPI, with very few exceptions, is dumb and flat, aka non-hierarchical models. There's a good reason for that. Hierarchical models are super challenging. To begin with... What is the simplest hierarchical model that you can have? It's a two-level model. What is your favorite two-level model? What example should we have here?

speaker-0: Well, you know, one I really like and I started working with was something doing electoral forecasting in France. here the hierarchy is very interesting because you've got cities inside things that we call departments, but they are like, you know, regions. And then you've got the whole country. So you have that pyramid of three levels. I think it's an interesting one.

speaker-1: Yeah, so let's stick with that. So we have to simulate this model across three levels, which are now, well, one level is always there. That's a flat model. But the new levels that you added, let's say the region and the country, now you have two more dimensions that you have to simulate. may not seem very problematic if your simulator is fast, but suppose that your simulator is slow, like it's already pretty hard to simulate even one instance. ⁓ Suppose you're modeling the brain, right? And you have a brain emulator that takes a few minutes to generate one sequence, right? Now suppose you want to model hundreds of brands at same time, have hundreds of these to simulate. And this is just one training instance. Okay. So there's no way you can train this model efficiently if you proceed like that. Okay. And this is also bearing memory issues and the need to design specialized networks. Now, what do I mean here? Right? When you have hierarchical models, you modeling two different at least two different categories of parameters. They've got the local parameters, which vary, for example, stick with your case again by location, but just for global practice, that capture what is shared among the locations. And you don't want to it separately, but what you really care about, if you are a proper basin, you want the joint distribution of all these parameters conditioned on all the available data. to get this precious shrinkage that you're after in the hierarchical model. This puts you in a tricky situation when designing your neural networks because now you can't just say, I'm gonna take all my parameters and put them as a single vector and say, okay, that's a high dimensional parameter space and neural networks can deal with higher dimensional parameters spaces, images. No, no, no, you can't do this. because your problem has a symmetry. So this joint posterior factorizes in a particularly nice way if you think about it. So you can actually ⁓ pose this problem as estimating each of the local parameters conditioned on each of its local data, but also conditioned on the global parameters. So you have, let's say, any of these problems, And then you have still to solve one big problem estimating the global parameters, keep on all the data. You can do it. We have a paper on the others have paper on that. You can do it by chaining different networks together. Right. And the insight here is the so-called inverse factorization. This is something that's not commonly, let's say discussed in hierarchical modeling because MCMC always gives you the joint distribution. I know it's little bit technical, but for neural networks, you need to consider how this joint distribution factorizes. And these different factors, when you have repeated factors at the same level, you just use one shared neural network that only knows how to estimate the parameters for one factor. because you want this network to generalize to different numbers of factors. That's the main idea. So it's the good old story of inductive bias, right? You're trying to encode the probabilistic symmetry into the network instead of hoping that the network can learn it out of the box. No, this is very, very hard. Actually, for the two-level model, it's easy to imagine for three, four-level models, with many factors, the inverse factorizations are not unique. In fact, even for a two-level model, the inverse factorization is not unique. You have two possible inverse factorizations. One of these is nicer to amortize than the other. And for three and four and other, generally speaking, graphical models, there are some papers that have already looked at which of these inverse factorizations are particularly favorable for amortization. Okay, so theoretically you can build a heuristic algorithm that already gives you the inverse factorization that requires the minimum number of networks chained together. But even if you could do this, you still have to simulate the whole thing at once. And you have to do it multiple times for simulation-based training. So, now if you put your computer science glasses on and you ask, is there a way maybe to... Think about divide and conquer. Can you solve the whole problem by partitioning it into multiple easier problems? Sort of in a way, can I solve this in a non-hierarchical way by just simulating one unit of the hierarchy, training a network that is competent in estimating the parameters for this one unit, and then somehow aggregate after the fact. So I can use all the existing infrastructure, but maybe just train two networks now, right? One for individual units, one for global units. And I never ever have to simulate exhaustively the full hierarchical model. That is what we are currently extremely excited about, right? We have, so this is what we call compositional score based modeling. And we took the idea from earlier papers for this, you need a diffusion model that estimates a score, right? So it estimates the gradient of the law of density instead of the density itself. This makes everything that factorizes additive its score space. So, and it lets you aggregate this through some very simple mathematical reformulations. And this is great. This has already been shown for simple exchangeable models. where you don't even have a hierarchical model. You have a stream of data coming one at a time. If you tackle such problem in a typical way, you will exhaustively simulate all sequence lengths and then induce some kind of a transformer that generalizes all sequence lengths. And this way with composition, you enable proper base in a data. The same idea has been applied to complete pooling. So basically aggregating information. different experiments without hierarchical structure and we showed it for the first time for when you have a hierarchical structure. And we also tried to scale it up to really big problems where you have hundreds of thousands of hierarchical groups. Now, this works somewhat, but there is a caveat. ⁓ We're still not at a point where we can aggregate hundreds of thousands of groups without loss of categorization. So there's a lot of work to be done there, but I'm confident in saying this is already now implemented in the version of Base Loan. For up to a few thousand groups, you can now use Amortize inference and treat the problem as if it were not a hierarchical problem. So feel free to test it out. It's going to be out in the next release on Base Loan, but there's still some unsolved issues there that we're currently working on. in making this fully feature complete and fully scalable with the really big problems. I think it also makes sense that this is a hard problem. If this were to work, it addresses most of social sciences, most problems.

speaker-0: Okay, yeah, I mean that that's already extremely good progress. ⁓ what so yeah take maybe to summarize what's the current state of of base flow because that's the state of the art package to do amortization inference. So can listeners who need to try API with a hierarchical model can they already Do that right now in Baseflow or do they have to wait for the next release? And what are the limits of what is currently available? Like which kind of hierarchical models will not work well with what's currently available and what's coming in the very next release?

speaker-1: Yes, the interfaces are available on the development branch on GitHub. There is also a full end-to-end tutorial, which Jonas Saruda is also on the podcast. He was the main engine behind this paper, right? It's this do-where-it's-do. So I can say based on the current state, if... you have a moderate hierarchical model with around thousand groups. So you have thousands geographical locations, data available in each of these and you wanna aggregate this, go ahead. It's most likely it's going to work. And of course, if it doesn't, this comes with all the diagnostics that are available anyways. So at least it's very easy to diagnose when things fair in this case. when you have a lot of data, right? It is still going to work in the sense that it's going to give you an approximation that is probably not very sharp and not very well calibrated and you will notice it in the diagnostics, right? So we are ourselves unsure at which point the method just fails silently in the sense that you keep aggregating information but your estimate it's not getting sharper in any way. And this occurs for different models, around a few thousand. So this is still in the works, I would say, but feel free to use it. There many cases where we have in social sciences, your typical, most typical model is we have participants. Each participant is a data set you want to aggregate over different participants. If you have a couple of hundred participants, this is ready to go. If you have something crazy, crazy high dimensional, then just break it. Let us know already.

speaker-0: Yeah, exactly. Always let us know, please. And please make sure to add this paper and this example in the show notes for this episode because I am sure people are going to want to check it out. There's already a to Bass Flow, but I think the link to that paper in particular and to that tutorial in particular from from John S.'s work is going to be very... interesting and also very practical for people to apply that. So let's do that, please. And so I'm going to start to have to start winding us down here. But a practical question I have for you in that you started touching on with hierarchical model is in general, when does a monetized inference give you fast but wrong answers and you would not necessarily know it? And what are the failure modes that people don't talk about enough and need to be aware of?

speaker-1: You really want to put our hands into the conceptual beehive now.

speaker-0: Yeah, yeah, exactly. I I love that and I think it's very important. Like, you, so you know, that choice very practical. So I want people to be able to know, okay, so that method is really cool. I can use that for this case, but in that case, it's not appropriate. And I need that one instead.

speaker-1: Yeah. So let's start with actually how simulation based inference sort of matured. When we were starting out, was very exciting. The fact that two networks produced anything that's useful and resembles a posterior, then more and more people entered the field and we showed, okay, we could actually get fully based in inference under ideal conditions. Now what happens in less than ideal conditions, namely when you're working with a misspecified model. Okay. now model misspecification is a very dangerous ground to threat because there are many different definitions, right? And depending on which field you're coming from, people understand different things. But suppose basically the situation where you have your single editor, you've trained the networks on simulations. But your real data is super atypical under the simulator. What do you expect to happen there? There is no general answer here. As a proper basin, you expect to get the posterior that you would have gotten from an Oracle MCMC satellite, from a Converged MCMC satellite. This is what we're calling the correct posterior under the wrong model. Okay? Now this may raise some eyebrows. I've been drawing a lot with physicists, right? So some physicists, especially particle physics, ⁓ they don't like the idea that you should be interpreting an object, that even though if that's the proper object that's based through implies, if the model is wrong, like why do you want to deal with this? posterior. No, we should improve the model. Now, you talk to social scientists. Well, nobody's making an ontological claim that we have the right model, right? Well, actually, the world assumption is the model is wrong anyway. We still care about the posterior to get from MCMC. And unfortunately, neural networks are not guaranteed to give you that posterior that an Oracle MCMC would have given you if your real data is very rare under the simulator. And for that, we have diagnostics. So we first notice, for example, you can use known techniques from machine learning, from out of distribution detection or domain generalization, where basically you construct something like a representation space of your simulation, which is interpretable all dimensional. and you look at where the real data lands in this space. If it's an outlier in this space, you probably shouldn't trust your inference, right? In fact, all guarantees just drop out. In this case, and we've shown it on the very different stress test, misspecified priors, wrong likelihoods, unmodeled noise that the degree of misspecification correlates to the degree of deviation between. the neural network implied and the NCMC implied posterior. There are ways to mitigate this. We've proposed a way to diagnose it. Others and we have also looked at different ways to align the neural network's estimates with what you would expect as a proper patient. Now, an unfortunate consequence of the inherent conceptual murkiness of the field is that people have used different terms in different ways. or use synonyms, but not really synonyms. We are also ourselves guilty of this in our first paper. We didn't get model specification right in this case. We actually looked at differences between distributions, but it turns out it's important to look at differences in only one direction, not in the case where, for example, your model is over dispersed relative to reality. It's more probably under dispersed relative to reality. Et cetera, et cetera. And when people talk about robust inference, it's really important to specify what they mean. What do they mean? You're now robustifying the neural networks so that they give you the expected estimate under the assumption of proper Bayesianism or whether you're robustifying the underlying Bayesian model, which actually entails changing the model, right? As in, for example, swapping out the Gaussian likelihood for student T like. in a regression case or assuming a mixture between a noise process and a signal process. So in practice, by the way, it turns out a very simple way to at least have a high probability of a lion is to simply noise the data in some ways. For example, for human data, very easy. Assume some sort of a random guessing process, simple uniform process, it's already suffices to bring you in line with what MCMC would give you for, of course, a small loss of accuracy, right? When the models misspecified, I mean, this this is a fundamental trade off for plus methods, but there's also no way around this, right? Then again, other approaches which try to, for example, do some kind of post hoc correction to the neural networks. trading off amortization for some sort of a semi-amortized method with post-hoc optimization. This is also promising. Then again, others are saying, let's go in direction of generalized based, right? So we learned something like a power scaled base and model, power scale posterior, which deviates from the correct posterior under the proper base and model, but has higher predictive file under the specification. I would say this is one of the frontiers now of unorthodox inference, simulation, base inference. mean, bringing all these methods, there's some conceptual work to be done, bringing them under some conceptual framework, noticing the differences and the similarities with other deep learning fields like domain generalization. For example, if you think about it in SBI, we're doing domain generalization. ⁓ So it's good that you mentioned that because it's still one of the itchy. problems of the field. But the bottom line is if you're a practitioner, don't just plug your data into the network, get the results at face value and write your paper. Always diagnose. And so I about workflows, not analysis.

speaker-0: Yeah. Yeah. No, exactly. Exactly. I think, I think people, I think people are aware of that on that show. Um, actually, you know what, Stefan, I think so it's time to call it a show, but, um, I do still have a lot of questions for you, as you can see from the Google doc. And I think these are very important questions and very interesting questions to not only me, but the whole, uh, um, the whole audience. So you know what? Let's do a. Let's do part two of this discussion and record that later and actually do a part two of that discussion because I think it's very important. And as a cliffhanger for people, I will not ask you the last two questions I ask every guest at the end of the show right now. I will ask you that at the end of the second part. And so that way you folks have to tune in for the second part. How does that sound to you?

speaker-1: Excellent. Thank you for the invitation, which I happily accept. And sorry for talking too much. This is a side effect.

speaker-0: No, that's perfect! Yeah, I think it's great. I mean, you're very passionate about what you're doing ⁓ and you love to explain and educate and teach, so I think it's really perfect. You make my job easier, to be honest. So that's perfect. So let's do that, folks. This is the end of part one. We'll see you very soon for part two. Stefan, thanks a lot for taking the time and being partly on this show. This has been another episode of Learning Bayesian Statistics. Be sure to rate, review and follow the show on your favorite podcatcher and visit learnbaystats.com for more resources about today's topics as well as access to more episodes to help you reach true Bayesian state of mind. That's learnbaystats.com. Our theme music is Good Bayesian by Baba Brinkman. Fit MC Lass and Meghiraan. Check out his awesome work at bababrinkman.com. I'm your host,

speaker-1: you

speaker-0: Alex and Dora. can follow me on Twitter at Alex underscore and Dora like the country. You can support the show and unlock exclusive benefits by visiting Patreon.com slash LearnBasedDance. Thank you so much for listening and for your support. You're truly a

speaker-1: the show. good basey and change your predictions after taking information in and if you're thinking I'll be less than amazing let's adjust those expectations let me show you how to be a good basey change calculations after taking fresh data in those predictions that your brain is making let's get them on a solid foundation

speaker-0: Yeah