speaker-0: Welcome to part two of my conversation with Stefan Radeff and I promise you, won't regret staying for it because this is where we go deeper into the ideas that I find most intellectually exciting in the whole amortized inference space. We start with the role of simulations across the full-vision workflow and from there we get into priori-citation. How do you translate the domain expert's vague intuitions into a probability distribution? And can generative AI help? Then we tackle foundation models for Bayesian inference. What would one actually look like? And then the part I found most practically useful. Sensitivity analysis and multiverse analysis. And how amortized inference can make running a full multiverse analysis nearly as cheap as running one. So yeah, again. Welcome Basion Statistics. ⁓ a podcast about patient inference, the methods, the projects, and the people who make it possible. I'm your host, Alex Andorra. You can follow me on Twitter at Alex underscore Andorra, like the country. For any info about the show, learnbasedats.com is the place to be. Show notes, becoming a corporate sponsor, unlocking patient merge, supporting the show on Patreon, everything is in there. That's learnbasedats.com. If you're interested in one-on-one mentorship, online courses or statistical consulting, feel free to reach out and book a call at topmate.io slash alex underscore and dora. See you around folks and best patient wishes to you all. Hello, my dear patients. We set up a new way for you to support the show at no extra cost to you. Because yes, we have a Friends of the Show page now. And on it, you will find a bunch of discounts for you and products that I actually genuinely use myself, starting with Cloud Code, my AI pair programmer in the terminal, the tool I reach for to keep the podcast website evolving, Riverside. for podcast recordings. That's the software I use for the show itself. Wwise for international transfers and multi-currency spending. I've been loving them for years. They have super low fees and instant wire transfers. Rove Mives to get the most out of credit card points and airline mines. And a lot more, honestly. Each one comes with a perk for you. So make sure to check them out at learnbasedats.com slash friends of the show. In the show notes, I swear, no fluff, no sponsored posts, just tools I genuinely use and recommend and think that you'd like to. Now, let's tune in for second part of our episode with Stefan Ravich. Stéphane Ralef, welcome back to Learning Belgian Statistics.

speaker-1: Good to see you again, Alex.

speaker-0: Yeah, it's always great to have you on the show, honestly, so that's a delight. You're actually... So I've had recurring guests. Usually I don't do two-part episodes. It's not really a hard world, it's just like how it ended up being. But I did do that at the very beginning of the show. So eight years ago, something like that, Colin Carroll had... a two part episode also that was the third episode at the time. ⁓ It was numbered 3.1 and 3.2, know, of course software convention. But this one, no, I will do like, I will do the normal counting integer counts. That's actually just easier for people. ⁓ So yeah, like I showed you the second time this happens. ⁓ But this is a very rare. So the Poisson rate on second, on two-part episodes is very low. ⁓ So thank you for taking the time again.

speaker-1: I'm starting to get used to this but I don't know what I'm going to be doing with my morning next Friday.

speaker-0: So, yeah, let's get back into it because so first part, ⁓ folks, obviously encourage you to listen to it because that's going to be the whole context of this new episode. This one, we're just going to jump in where we left off at the end of first part. So if you want all the background about Stefan, about amortized patient inference, the agent skill with done together and a live demo of that and then the limitation of these workflows. That's first part. Now, I'd to talk a bit more about that's going to be a bit more theory oriented, at least at the beginning, because you do that a lot. And something I'm very curious about is you have a paper with a friend of the show, Paul Burkner, on simulations. in statistical workflows. And I think it's a very interesting one. We definitely need it in the show notes. What's the core argument for that paper?

speaker-1: We do have a lot of papers with Paul. is just one of these conceptual slash theoretical papers. And the core argument is basically Bayesian statistics has never been more computational than it is now. And we parcelated basically the minimal Bayesian workflow into four stages. And then we discuss how simulations can accelerate bootstrap or help in various ways in each of these stages. So let's start first with stage number one, which we call model specification. And everybody is familiar with this stage, can call it model development. And this is where all existing base of workloads recommend doing prior predictive checks, okay, which you can do. rather informally, but just running a few simulations from the model and ensuring that what comes out meets a set of expectations that you have. could do it also very formally in Michael Betancourt style. So, prior push forward checks where you're transforming the outputs of your model with their high dimension and transforming them into a low dimensional, more interpretable representation, which you can then... also formally check is it under or over dispersed relative to reality or your domain etc etc right so you can use this would be or what if worlds not all the simulation to basically validate your model before you see any data but it's actually surprising how many models you could simply discard by doing these types of prior predictive checks and I still think they're despite all these workloads, they're sort of underutilized in practice from what I see of this in lot of papers. Another aspect of this is prior dissertation, which we may touch upon later down the line, but we also have another set of ideas on how you can translate non-probabilistic as expert knowledge into probabilistic knowledge, into distributions. also using simulations, right? And perhaps it's, of course, a lot of this paper is about simulation-based inference, which is unsurprising given that this is our bread and butter of research. ⁓ But still, we should remember that Markov chain Monte Carlo methods were also termed simulation-based methods, right? A couple of decades. a goal simply because you're using random numbers to simulate the distribution. Also, this notion has changed a little bit nowadays. When you talk about simulations, we talk about simulating from the model, not simulating the posterior as it used to be called. So this is stage number one, model specification. Stage number two is what we term model verification. And this is another part of workflow where you have to Well, now we basically want to check if your inferences are informative, if your inferences are well calibrated. So this contains classical procedures like simulation-based calibration, parameter recovery, checks, all very important, all traditionally associated with very high computational costs. Because these checks are basically brute force. you simulate ⁓ sometimes thousands of synthetic data sets and you run inference on each any of these data sets. ⁓ And this of course is also the reason why you also don't see a lot of these checks in papers. in research papers, even doing such a check, a parameter recovery study for a new model is a valuable contribution simply because you had to pay a price, a certain computational price to get it. So this is also one aspect, by the way, is just greatly encompassed by amortized methods and simulation-based inference because all these checks, they just come for free if you're doing amortized inference. It's literally a few seconds for something that for obtaining millions of posterior samples to something that traditionally used to maybe take a few days depending on your whole. So after the stage, small model verification, we have the model inference part. And this is basically a simulation-based inference. The somewhat more modern idea that you now use simulations for model identification, right? Starting with approximate Bayesian computation, Bayesian projection methods, and into the model realm of neural simulation based inference. By the way, we also talked about some frequentist approaches, We should not also disregard these, they exist. And the full stages, there's something everybody is familiar with, right? Posterior predictive checks, model checking, as we call it. There's also a nice aspect of simulation-based inference being embedded into the general Bayesian workflow because you also just inherit all these best practices that you as a Bayesian are doing anyway or at least you were taught to do.

speaker-0: Yeah, yeah, yeah. No. ⁓ And I think the paper does a very good job at summarizing all these steps. So ⁓ let's definitely add that in the show notes, the link to that paper. And since you touched on prior elicitation a bit, this, as you were saying, this being a classy headache in Bayesian workflows, as well as all the checks just talked about, I do think the agent skills help a lot with that because they will be able to just automate doing that. And this is basically a very repetitive part of the workflow of our job, which is not, you know, the most creative one where we really need to be in the loop and it's a lot of checks. ⁓ So I think actually this can be much more automated. I've been doing that myself in my own workflow with the patient workflow skill, for instance, where I've added a lot of that. The new RVs 1.0 Plus helps a lot because they have been starting to ⁓ write functions that are actually giving you back text reports, which are perfect for LLMs to parse through and basically see if all the steps passed. ⁓ So I think this is going to be much more helpful because also you can just, if that doesn't look good, you can just tell the agent what to do, what to change, and then do another. another another pass on all the checks and it just runs in the background. You don't have to supervise that. I it's going to be going to be very helpful. It's already been in my experience. ⁓ But so, like you you work a lot also on this particular part of simulations, which is prior elicitation. ⁓ And you've worked on simulation based approaches to it, if I understood correctly. So. What to you what does that look like in practice now?

speaker-1: We did two papers on this. This work was led by a graduate student by O'Bugner, Florence Okhting. She is now a research software developer for acubectoring. The logic of this work was as follows. Prior elicitation techniques have been around for decades. ⁓ Prior to the citations, it's a fancy term to say, okay, we ask a bunch of experts to tell us what are meaningful ranges, let's say, for parameters. And the realization in a lot of large body of this work has been that typically you can't just ask experts, give me a prior distribution. Okay, because that's not how human knowledge is typically represented. that are presented in a strictly formal probabilistic way. It's represented sometimes in the tested structures. so elicitation is a translation process, basically. And so the perennial problem there is translating this tacit knowledge from non-probabilistic knowledge into prior distributions that conform to the expectation, which may not be obvious, but...

speaker-0: and

speaker-1: An expert can even tell you what these are the priors that you should use but then these priors when you actually do Prior checking or model checking they lead to surprising results. For example your model diverges a lot. Okay So the idea here was to automate This with the simulation based procedure in which okay, you still ask your experts or expert elicits certain quantities. Let's say that expert gives you quantiles, a mean, and that's all you got. So what you can then do is you can parameterize your primes with what we call hyper parameters. For example, instead of fixing ⁓ the mean and the standard deviation of the Gaussian prior, you let them vary and you start simulating. from this hyper model. And when you simulate this, also compute the model implied at least quantities as it comes. We compute the model implied contents and means, let's say. And we basically compare them to the ones given by the expert. Initially, they'll be very far apart. And so we need a way now to propagate this error, the divergence between the model implied and the expert implied. quantities and if you can keep your pipeline differentiable end to end, that's nice, you use gradient descent. You can do various reparameterization tricks to make cycling differentiable, et cetera. If you can't keep the whole pipeline differentiable, you can use black box optimization, something like Bayes optimization. Yeah. And you run the situative procedure and you end up with ideally with the hyperparameters to the prior that most closely match what the expert had in mind. So that was the idea of the first paper. And we showed that this can be very useful for a bunch of case studies. Now in the follow-up paper, we said to take this one step further. In the first approach, it's true that you have some flexibility, but you're still confined to a certain parametric pattern. Yeah, you may optimize for the hyperparameters of a Gaussian, but you're still in a Gaussian world. So we asked here, can you do better? Can you have some sort of a non-parametric procedure that is going to give you the prior direct without assuming its distribution of family a priori? And to do this, of course, we turn to generative AI. So it would basically replace the prior with an untrained generative network. We use normalizing flows for the protocol concept and we applied exactly the same procedure now, but with a normalizing flow as a surrogate for the prior. ⁓ And this sounds really cool and all, but of course there are some caveats, right? Because ⁓ this prior is very flexible, there is a lot of non-identifiability issues. cases where you end up with a lot of possible solutions. And you have to deal with stability issues. So, but I think that the general idea is still very viable because if you can take this approach now and add some further constraints, you end up with an end-to-end pipeline for finding priors based on non-prog-listic expert knowledge, which... in 2,000 binary sinks can also be theoretically followed by an agent. So that is the basic idea.

speaker-0: Yeah, this is super interesting and practical, think. ⁓ Do you guys are doing still research on that or what is the state of what you're doing right now on these topics?

speaker-1: I have personally drifted away from these topics, but Paul Buechner is still very interested in this and he has a couple of works lined up. Luigi Cherubi from Aalto is also doing some research in this topic. This is from my club surrounding, I'm pretty sure others are also interested.

speaker-0: Yeah, okay. Well, sounds like I should have Paul back on the show and maybe Luigi. That sounds like it be good episodes. I think Luigi never came on the show, so it could be cool to have him for first time. And so getting back to what you do, something I wanted to ask you is, and we talked about that a bit with Jonas Aruda in episode 151 because of course there is a lot of excitement about foundation models for Bayesian inference which if understood correctly the main idea is you have one network that works across many models. My question for you is first can you remind listeners what foundation models are why they would be useful in this case And if you think what I just described is a realistic near-term goal or if it's still an off-tar.

speaker-1: Yeah, that's a great futuristic question. A foundation model is basically a general purpose neural network that has been typically trained with self supervised learning without labels and that can then be plugged into various downstream tasks. So it's basically a general purpose multi-task model. is a foundation model, right? Gemini is a foundation model. So these are the large frontier foundation models. Typically it is understood that the foundation model also works with different modalities. So they're a multi-modal. Meta recently published a foundation model for in-sci-to-go neuroscience, right? This is called the Tride V2 foundation model, which processes text, audio, or videos, sequences of images, and emulates brain activations. They train this on 700 brain scans. We already jumped on it, by the way, so we show that you can be using BASOL, you can invert such model for doing in silico decoding. But that is the idea. Now, a foundation model for Bayesian inference, sounds like... ⁓ a goal that you may have, but I think we need to make clear what we mean by that, or in what sense existing foundation models are not already foundation models for base inference. So let us imagine maybe what would be the ideal inference engine. So suppose, I think the ideal, at least in my perspective, the ideal engine would be I take my data, I have a prompt, and I tell it, Okay, I want you to test these models on this data. Give me all model-implied quantities, like posteriors, diagnostics, et cetera. So we plug this into the engine, out comes an ensemble of posteriors and diagnostics. So that will be maybe a foundation model based on inference. And now we have to ask the question, is this even desirable? Right, so should we... Should we take, say, an existing multimodal LLM and fine tune it to be able to do something like this, to solve a numerical task like that, to work with numerical data? ⁓ I think the question is no. I think the question is similar to say, shall we have a foundation model of chess? Now you know, if you're not trying to play chess with with any of the frontier LLMs, occasionally they will teleport pieces, right? Or they will just invent a new position. So you may think, shall we make an effort to fine tune this model to be able to play chess? Or should we just tell the model when to call Stockfish or a specific chess engine? And I think... the same is going to apply to Bayesian inference. We're going to have some kind of layer, a semantic layer, which we should interact. And this semantic layer is going to then call on specific procedures, like an MCFC sampler or an unorthized sampler, which is going to do the job. And I think skills are already a step in this direction. And now if you start thinking about adding various harnesses to this, then you get a... You get a NeuroSymbolic engine, which I think NeuroSymbolic is what the field is now excited about, is... It's a neural network, a lot of harnesses, which are auditable, which are more interpretable. So I think if we're being very futuristic, very... general here, I think this is what practitioners will eventually be using. And this also now asks the question, existing labs, they can already write perfect stand-gold in my experience. ⁓ I taught a class which used a lot of Bayesian inference this semester and there was almost no model that GPT-5 cannot just write given a good given a good description, which I think also this comes back to what Stephen Wolfram defined as computational thinking. He has a very beautiful definition of computational thinking that he gave in one interview. Computational thinking is the ability to express ideas clearly enough so that an arbitrary smart computer can follow them. And in a way, we are now in this situation where we have approximately arbitrary smarts and one big computers. with LLens. So again, it's about the clarity of ideas that maybe separate us from the end goal. Not all that being said, I think there's still a lot of value in having amortized engines that generalize across large spaces. this idea has already been explored in a certain depth and breadth in the field of SBI. We also contributed the paper recently, which is an amortized engine that can generalize to different GLM type models, which with a lot of regressors and a lot of ⁓ changing assumptions. So what we induced is a model that tries to power set of configuration, power set. scales as 2 to the power of the number of configurations. So the number of configurations which you want to scale grows very large. And this is a case where simple regression without interactions. If you want all possible combinations of predictors, it scales as a power set, the size of the power set. Now, if you want to have a simple regression with all possible predictors, and all possible interactions, two-way, three-way, four-way, n-way interactions, you're looking at the scaling of the power set of the power set, which is a ginormous number. So I think there's a lot of value in having simply more general amortized engines that you don't need to retrain when a small assumption changes, when a practitioner now comes and says, ⁓ I actually didn't want to use this prior. So maybe this engine can adapt to the prior. ⁓ We have worked on that as well. ⁓ Maybe the practitioner says, OK, let me fix this parameter. Suddenly I know I want this parameter to be zero. I don't want to retrain the whole thing now. I want to get an immediate result. I want this to reconfigure itself. This, think, is very interesting direction. This is where also we are going with all of our research.

speaker-0: Yeah, yeah, this is definitely super exciting. ⁓ And to make sure I understand what you're saying, basically your point is that, well, your bet is that the way this is going to look like is, yes, we can have some foundation models for dubation inference, but they would be specialized in a way to dubation inference. So this wouldn't be like you just go to ⁓ chatgpt.com or Claude.ai and you just ask for the ad this would probably look more like a coding agent but where if I understood correctly if I think where it's different from what we already get with agent skills is that you're saying that the specialization invasion inference would happen upstream during training of the foundation models and not downstream which is what skills do which is more like the model is already trained genera generalistically and now which is oriented more towards that part of the knowledge and that's also you're also not saying this would look like fine-tuning because this would still be post-training you're you're saying that these specialization would happen during training of the models themselves. Did I understand that correctly?

speaker-1: would be my bet. Yes, is an excellent way to summarize it. So it's going to be something more than skills but closer to skills than closer to fine-tuning.

speaker-0: And... you mean closer to fine-tuning?

speaker-1: Closer to skips, then to fight.

speaker-0: Closer to skills. Okay. But so in, but that would happen at training. So like I would not have, I mean, unless, unless I'm the one training the model, I would not have control into what gets into that training. And why do you think this is a more interesting approach? Because to me, what I'm thinking is But if we already have these generalistic models, why do we spend time training them on a sub-part of the knowledge instead of just specializing them afterwards, whether it's with fine-tuning or skills? Why do you think it's going to be more interesting to do it that way? What would we get by doing that?

speaker-1: So that is certainly also a possibility and I think ultimately it's going to be an empirical question of what works best. There already exists certain specialized GPTs, for example, ⁓ FinGPT for FinTech, TimeGPT, Parameter, TimeSeries. So I'm mostly observing and astutely what is happening. in this ecosystem and you can certainly see signs of bold approaches now, ultimately it's going to come out on top. It's very difficult to predict. I think ⁓ what is going to happen is ⁓ an evolutionary selection of the model that just works best. It's going to establish itself.

speaker-0: Yeah Yeah, no, and I mean beyond that prediction step here. I think my question is more towards what would you like ⁓ to do if you had this possibility and it sounds to me like it's something that you think is promising to try that and so my question is How come you're saying that like how come you think that doing that a training would be would be more interesting? What would we get from that?

speaker-1: Simply because we, as researchers, we like playing with things. We like having control and to be at the source. We have a certain disliking towards black boxes. I think, I can't speak for everyone, but I think I would much rather be the trainer than the user. Of course, there is a possibility that simply we don't have in academia, we just don't have the resource to train anything like that. that there may even be, right, the ecosystem may parcel itself into different solutions. Because I'm not even sure if we should not just wait out for a few more generations. may already be just perfect based on code generators. then ⁓ our job is going to be ⁓ fine tuning skills, finding out, ⁓ as I think I mentioned during our last conversation, like the science of skills, what makes a good skill and what are the factors that can make it so that we have in harnesses.

speaker-0: Mm-hmm. Yeah. Yeah, yeah. But I'm definitely curious, like, if we could do that. Basically train a specialized model to help us do Beijing inference, like, basically a Beijing caught. That's the idea, right? Or Beijing chimney knife, something like that. Okay, yeah. Yeah, that'd be super fun to do for sure. ⁓ Let's see how it goes, you know, like maybe we'll have these opportunities at some point. We'll see. ⁓

speaker-1: I'll be happy to see it in any case.

speaker-0: Yeah, yeah. Yeah, definitely super and very fascinating topics. You also another another paper you have is on self consistency. And we didn't talk about that yet. So then I think if I understood correctly, that's you see that as a way to make a multi-faceted inference more data efficient. So can you tell us what self consistency is and What the problem? What problem it is solving?

speaker-1: Absolutely. We actually have a few papers on self-consistency. You may even say that we really like to be self-consistent in that regard. Now the idea started out as a question. In many simulation-based inference applications, you still have access to the likelihood, Be it the analytic likelihood or a synthetic likelihood of sorts. The way more simulation-based inference pipelines work is you don't consider this information in your training. You just use the simulations as training data. But you don't use likelihood information. So can we in these cases, what we have, the likelihood, can we devise better loss functions that simply use this information as additional training signal for the networks? And what we came up with was almost trivial. So you take Bayes' you have the posterior on the left-hand side, and you have the evidence on the right-hand side, like the normalizer of the posterior, and just swap their places. So now we have the evidence on the left-hand side and the posterior on the right-hand side. That is not a familiar picture for Bayes. You practically never do this because the posterior is never available. It's the object which trying to get. in the first place. But the game changes completely if you now training a neural network that is going to become a surrogate for the posterior. Now, in fact, you do have this object. And you can evaluate its density if you're using models with closed-圈 density, with normal Isaac flows, flow matching. So if you can do that, then this new object, this new right-hand side, is a calculator for the evidence. So you have something that is independent of your parameters in theory. This is a cool property. you plug in any parameter value, the effects cancel out. So you get the same x. That is interpretation here. But now, if you have an imperfect approximator for your posterior, you start having variance. You treat this now as a marginal likelihood or an evidence estimator, basically, which has an RS. This variance is a direct proxy for an approximation error. It's differentiable. So you can turn this into a loss function. You can also plug into a network, can differentiate to it, you can propagate gradients back into your network. And these gradients use more information than simply your simulations. traditional simulation based lobs. So that's the core idea here. And at first we just tested this for cases with limited simulation budgets. Like we're talking about a few hundred and we saw improvements. Later on when we started thinking about semi-supervised approaches, okay, so how can we use both synthetic and real data during training? We realized that such self-consistency lobs us. give you the perfect proxy to just plug in the real data in there because this is an object that is in theory completely independent of any ground truth products. So you have now your two sets, have a set of labeled simulations, these go into your canonical simulation based laws, you have your set of unlabeled real data, this goes into your self consistency laws. So you have yourself a really nice semi supervised objective that trains your entire pipeline and it uses information from reality and simulations. What's the catch? The catch is you still need the likelihood or you need a very accurate likelihood approximation. So think where our money is now is finding a way to have something like this, which is just another loss function. but doesn't depend on the light. So now this could be, you can port this into the light with three settings. And by the way, we also, we're also trying this out. The student of Paul is trying this for model comparison as well as somewhat neglected setting in simulation based inference. But this very same logic applies to any conditional distribution, which is subject to base rule. So that's why we're now talking about this as a family of self-consistency laws. I think there's some really interesting theoretical research. We did some theory on it in the second paper, but you can do much more. You can start thinking about deriving error about different types of self-consistency estimators. this is, I think this is very interesting theoretical strand of this world. There's also the second paper was about robustness. Because we also know this if now you start using a real data Your network is no longer surprised by the real date It has seen the real data and it's consistent on the real date in sense that it you're directly minimizing The residual error on the real date if you're doing this so you bring your neural estimator in line With the theoretical oracle MCMC supper if you're doing this you you minimize or mitigate these so-called extrapolation errors that we talked about last time. So it's also a really nice way to make ⁓ neural simulation based inference more attractive to people who more familiar with the traditional MCMC work thought.

speaker-0: Yeah, okay. This is... Yeah, this definitely needs to be in the show notes because it sounds like it's super helpful. ⁓ And yeah, I mean, you guys have so many papers that I found super practical that need to be distilled into some... into either some functions in Baseflow and or some agent skills that we should work on together. Stefan, to make sure we have all these... great knowledge and guidance distilled to practitioners even without them realizing it, know, but at least their models will be much, much more robust and powerful. another, so I think, I don't think we talked about that yet. It's another paper you have is on sensitivity aware, I'm a test inference. So that's not the same thing as self-consistency. So what does sensitivity mean in this context and why does it matter for practitioners?

speaker-1: This is not a very exciting topic. It's sort of orthogonal to self-consistency. So self-consistency, just view it as a way to justify your training or make it a bit more simulation efficient. Now, sensitivity analysis came as an inspiration from a Nature of Opinion paper. The paper was titled, One Statistical Analysis Must Not Rule Them All. Those Who Get It Will Get. The core argument of the paper was really straightforward. Any analysis, be it Bayesian or Frequentus, hides an iceberg of uncertainty. Typically, this uncertainty is not explicated enough. The paper starts with a great example from 2020, the early days of the COVID pandemic, where the UK modeling group asked nine analysis teams to carry out an analysis of the basic reproduction number as an indicator of the rate of spread. If you remember, this was the most famous number for a while on the news. Just as a reminder, at base revolution number one, this is a stationary point, if it's below one, the pandemic is receding. If it's above one, means we're on an exponential trajectory. These nine teams came up with very different estimates. And if you looked at the lowest estimates, the credible interval of the lowest estimate contained the possibility that the pandemic was receding. If you looked at the highest estimate, it contained the possibility that one person infects more than one and half people on average. Highly raging trajectory in this case.

speaker-0: key.

speaker-1: The key observation was that the uncertainty across teams was higher than any uncertainty within any team. But it's just pause and ponder for a moment to realize what this means. Every single team underestimated the epistemic uncertainty, simply basing their analysis on an implicit set of assumptions. Like the authors of this opinion paper used this example also to motivate what is now called in the field many analyst teams, where you task different teams to analyze the same data. You don't prescribe how the data should be analyzed, you just let them romp freely and then report the results and then you see what was kind of a heterogeneity in results. So we looked at this paper and realized, well, isn't amortized inference? the perfect method for carrying out such analysis precisely because, ⁓ well, you distill, you can have a single model that distills all these possible configurations. Now, to give you a practical example of what this might look like, when I was analyzing neural data and I started scripting, I noticed that I have have to choose among gazillions of settings. Like how to filter the data, how to do artifact rejection, how to do other sorts of pre-processing, how to aggregate the data. The universe of choices was just vast. And what I also noticed, because I was writing scripts and I can very quickly run different analysis, was I noticed a somewhat startling observation, which we have as a student, that you can bend the analysis. one way or another depending on the settings you choose. And this may seem obvious now after the fact, but and it was obvious because many methodologically are fine people in the field who also noticing this which eventually gave rise to the idea of multiverse analysis. The multiverse analysis, you got to love these fancy terms. Multiverse analysis is simply a way to run all of your analysis for any different configuration. Basically, take a bunch of configurations, you run the analysis with any configuration, and suppose you're a null hypothesis tester, you compute the p-value for your final effect under any of these configurations. A p-value is usually stochastic, though it's not treated that way. But now you have a distribution of p-values, right? Every configuration gives one p-value. And now you take for example, some sort of a threshold here and say, okay, if 99 % of these settings give me a significant pivot, that I could be more confident in my statistical conclusion than I can be under any of these individual configurations. Of course, there are much smarter ways to aggregate these things because not all settings are equal, right? Some produce junk results. Also, you can have adaptive weighting and so on. There's research on that as well. ⁓ There's also research on how to aggregate results from amortized estimators. There's a great paper by Yuling Yao ⁓ and colleagues, which also shows different methods to aggregate results. This is generally a fascinating field, the way, simply aggregating results. Simple question, very complex answers. So anyway... You have this situation where you want to run multiverse analysis, but you have to consider that this is very costly. If you're in a Bayesian setting and you're already running one configuration, takes you a few hours. Well, you don't want to be doing this for thousand times, times four hours. This would be frankly annoying. Again, you'll see a lot of these sensitivity analysis unless... in research unless reviewers ask for them. And then you have to do it, but you are not required. So what we propose is first let's taxonomize different sources of sensitivity in basic analysis. We found four basic sources. Source number one is the prior. That's the obvious one, prior sensitivity analysis. You almost always get asked the question in an analysis. What if you had a different prior? Like how would that skew your results? Right? So that's one of the start. Second target, the observation model itself, right? What if now suppose I have a different noise model? How would that change the conclusions? The third source is data pre-processing. Basically, what if I tweak the data a little bit, right? Would that also give me the same? And the same answer or are my results simply contingent on the fact that I have this one very influential data point in my data? And because now this was all conceptualized in a simulation based in France setting, we also asked, what about your approximator? What about your neuro estimator? And this is also a determining factor. What if you change something, you change different, you use different architecture, right? Do you expect to get the same posterior or not? said? Amortized inference now gives you a way to simply expand the scope. Well, this is sort of precursor to our GLM like train ones for many models work. But the key empirical finding that you have to have if you're proposing something like this, so it doesn't sound trivial is to say, okay, if I want to generalize over 10 different priors with one neural net. Do I have to simulate 10 times as much as I would for one prior? So in other words, you want to have sub-linear scaling for this. You want to generalize over 10 priors by, let's say, simulating a bit more than what you would do if you're only amortizing a single configuration. So that's what we showed in this paper, basically. you see sub-linear scaling, which we attribute to weight sharing. Weight sharing is this observation that if your network is learning about different problems, learning about how to infer the posterior under prior A is beneficial for learning how to infer posterior under prior B. We think the worst case would be you have your training a network for completely independent. unrelated product, which case the network has no choice but to somehow internally split itself into two networks with a selection. But in practice this never happens because you're analyzing the data in a given domain, so all models will be similar among each other, Or they'll be more dissimilar to other models. So weight sharing always kicks up in this situation. For example, one of our findings here, apart from toy examples, exploring, scaling, et cetera, was if you take climate change forecasting models, in this case, and you subject to such kind of prior sensitivity analysis, and not just prior sensitivity analysis, sensitivity analysis in general, you see that they're relatively insensitive to the choice of prior. They're more sensitive to the type of model parameterization. And also relatively insensitive to the kind of overall scenario that you're assuming, suppose, let's say business as usual versus middle of the road. Your forecasts for the critical threshold not change so much. They change much more when you vary the underlying single model assumptions. So you can do all this with amortized inference. for a sub-linear cost. We actually advocate this as a default approach. But there's also one last point here, which is what about the different neural networks? So we proposed a somewhat brute force approach here, like train an ensemble of neural networks. And then look at disagreement between members of the ensemble. If they disagree a lot, there's some points to some deeper problems. One of these problems in a complex model comparison could be, for example, so-called dancer-based factors, which is the observations that base factors change to dance around a lot if you do small changes, small perturbations to the data. So we noticed something similar in our analysis, but now you can actually, you can reveal this within one analysis. You don't have to perturb the data more. any sort of cross-related. You can just see it by the fact that your ensembles, your ensemble members suggest different models for the same data set. Yeah, and I think now in Basel with the new version, it's very easy. We have an ensemble workflow, which you can specify, basically list the different networks you have. You can do this all that in a single workflow. Again, not for the time that it would cost you to train this. sequentially then backs because we have also different ways to reuse data. I do it batching so yeah so this is all available and we recommend this as a as a default basically because it's just so natural to do in our asset.

speaker-0: Hm. Okay. Yeah, so basically, as a default, whenever you do... Amortize Inference?

speaker-1: Yeah, I would stress just go for an ensemble of networks. It's also way to cheaply do a naive hyper-parameter optimization. Maybe as we wrote in the skill, right? Check models of different sizes as a heuristic to see how much performance gains can you reap by increasing your model size.

speaker-0: Hmm. Hmm. Okay. Yeah. And so, is it already the default in... in... in Bassflow or... or not?

speaker-1: This is not the default. So as a user, have to choose. It's called an ensemble workflow in Baseball. And now if you want to do different kinds of prior sensitivity or likelihood sensitivity, you still need to simulate under these conditions. But from the perspective of the library, the only thing that you need to change is you need to tell the network that these different configurations exist. like a simple case would be explanatiated price. Right? Data Gaussian, you add an exponent, an alpha exponent that shrinks or widens your price basically because it has an effect on the variance and just let the network know that this factor exists. Now you train the network, during inference, you just do a sweep over a range of alphas, right? And you report now an ensemble of Osterias and you just visualize or quantify if you want. If you don't want to be stuck with qualitative analysis, you just quantify the difference for me. See, okay, this is a difference that makes a difference really or not.

speaker-0: Mm-hmm. Yeah, yeah, Okay. So, yeah, so that's why you don't have that as a default on Baseball, right? Because they...

speaker-1: Yeah, because you can really anticipate what factors of sensitivity are going to be particularly interesting for this user group.

speaker-0: Mm-hmm. It seems though that we... Do we have that already in the skill or not?

speaker-1: No, but we should.

speaker-0: Yeah, we should. Sounds like we should. Because I think it's something where, so basically if we need the users, here I think we need the user's ⁓ input, right? Before setting that up. But we can definitely do that in a skill where, for instance, I did that in the causal inference skill where it won't kick off its work before it validates with you the DAG. the assumptions that are absolutely necessary for the causal interpretation to be valid, but that these are assumptions that are not in the data. So the agent cannot figure them out. It has to ask you for your like this literally prior recitation where the agent will hate. So I think the best method for that is don't know, Diff in diff. The core assumptions of Diff in diff are this and that. Do you think this is true in this case or not? If it's not, then it will ask it. It will suggest another method if there is one or otherwise it can say, well, we can do that, but the causal interpretation will be lowered. So I think here it's something we can definitely have in the skill where basically if the agent sees this ensemble is useful and it sounds like it's going to be the case for almost all the cases from what you're saying. Then it's going to going to prompt the user with a few pointy questions. You enter yes or you enter whatever you need and then the agent goes and do its work. I think the skill framework for that is very useful.

speaker-1: Yeah, there goes my weekend.

speaker-0: Yeah, and mine too. But I mean, this is, this is a good way to spend the weekend. know, that's funny. ⁓ but, I mean, for sure. Let's, let's add that, ⁓ when, when you have time and also when I have time, can, we can definitely add that in the scale because I think it's going to be, ⁓ going to be making it even more helpful and practical for people.

speaker-1: Yeah, I think that the key consideration there is also manageability. From my experience, everybody agrees sensitivity analysis is good. And you have these papers on meta-science that they all tell you sensitivity analysis is good. But many practitioners are simply overwhelmed. They don't have to deal with

speaker-0: Mm-hmm.

speaker-1: one result, they have to deal with an ensemble of results. So they don't teach you that at school. Like what to do now if I have now two competing explanations suggested by my results. So perhaps it is on us to just push these as defaults.

speaker-0: Mm-hmm. Mm-hmm. In my experience, yes. mean, it's, that's what you also, what you see from, you know, ⁓ behavioral economics, literature, where the power of defaults is extremely important. And so if we think this is a better, this is a better behavior for ⁓ users to have in their models, then having it as the default is going to be much, much more powerful because they'll just use it by default without even knowing it because they don't really need to and it's just it gives them better model better predictions better inferences for free. I think it's a great it's a great deal. And so yeah the power of default and never never yes never to be underestimated. And so yeah. Cool. So you've heard it, listeners, now we have to do it. we'll do it. We'll get to it at some point. It's pretty fun to work on these skills and even more fun when we're at least two of us working on that. Yeah. That's cool. So let's start playing this out here because we're approaching the hour already. Yeah, I'm curious. Basically, after everything we've said in this episode and the previous one, if you had in front of you someone coming from PyMC or Stan, know, classic PPL, when should they actually consider switching to amortized methods? And I'm saying switching here, but it's like, it's not really a switch, like, especially with the future work you guys have on the deck. And we'll talk about that on the podcast ⁓ with other guests. That's a teasing. ⁓ so yeah when when should they actually consider switching traumatized methods and when should they just stick with what they know

speaker-1: That's a great question and I think is also the way you pose it, think. It also calls for a criterion in a SKU itself. That if you're a user who doesn't really care about any of the methods, you're just facing an only problem. A SKU can recommend. Go for amortized or go for a non-amortized approach. But I think in the way we are also positioning our work, in base law is in direction of complementarity. I think modelers, petitioners, statisticians, they should have a multi-signal tool kit basically as a complementary set of methods whenever the classical tool kit is infeasible for a variety of reasons. Like we even have a paper that now has a dedicated website and a repo, it's put an amortized Bayesian workflow, paying homage to the Bayesian workflow. This is joint work with Paul Aki and Numi Giaccheri, again, very strong student from Aalto. In this work, we actually outline a step-by-step workflow that uses both amortized neural inference and MCMC and it tries to, it tries to balance basically the Pareto frontier of accuracy versus speed. I think the main trade off, if you're in a business, main trade off here is speed versus accuracy. And if you would attach utilities to speed versus accuracy, at some point, suppose MCMC is a viable option. But the cost of applying MCMC grows linearly unless you're paralyzing things, etc. Most people are not paralyzing a lot. So the cost grows approximately linearly. The cost of amortized inference is sub-linear. So if you approach it as economist, that is a breakeven point. Even in a case where MCMC is feasible for which you have a stream of data coming in and you have to update your inference as well. Think of a pandemic monitoring system where data is arriving ⁓ from different locations, different time points. So you need to base on updating. So some point you just cannot afford to wait for more than an hour to get the new result, right? Amplified inference gives it you instantly. We have to worry about other things. Like, can I trust the result basically? Complementarity simply means that you start with amortized first in a setting where suppose you have 100 data sets and you need to apply one more to this data set. You run an amortized pipeline, you get your results and then you diagnose basically which of these results are trustworthy or not. It's putting emphasis on trustworthiness proxies. Suppose that this diagnostic flags stand up these results. We can't trust. So what we propose is try to do a correction first. Post-hoc correction, correct to smooth the deporting cycling or something similar, which basically tries to for the bias that you have. Okay, now you ruin your diagnostic again. Eight out of the 10 data sets now pass in the diagnostic. You're left with two problematic data sets. What do you do? Well, this is where again you enter the MCMC world. but you don't have to start from scratch. What you show is that you can now use the neural network to initialize the soundwaves. So, you can do that. It's now, again, another aspect of complementarity. You use the neural network outputs to have a very informative guess on where MCMC should start, which then lets you run massively parallel or partially parallel MCMC, where you do a lot of short chains. Because in the ideal case, you skip the initial exploration phase of the sample, right? You basically landed in a typical set. So you start reaping benefits in terms of reducing the approximation error. And that's how you close the loop on the pipeline. So we're envisioning a lot of these, right, hybrid workflows, which we also need a skill eventually. But generally speaking, ⁓ I can't give a good rule because it depends on expertise. And in some notion in which you need to approximate currently in your mind, the quarter the speed accuracy trade-off, trade-offs, how much is it going to take to run a classical analysis? Now, if a classical analysis is not feasible at all, right, then there is no choice to be made. It just basically go for SBI.

speaker-0: ⁓ Yeah, yeah, and I think I love your answer because it shows that this is not an either or case but actually this is a case where we would love to have a hybrid workflow where ABI would come and help the classic MCMC ⁓ workflow and so yeah as you were saying There is a bunch of things we need for that first one probably a PMC baseflow bridge which ⁓ I heard he's coming very soon.

speaker-1: already exists yeah but I won't say more about it

speaker-0: Yeah, we'll talk about that on the show. And then, yeah, probably an agent skill that basically is the orchestrator here between these two workflows and can make the choices in between both. And yeah, I think this would be super powerful and helpful for a lot of models and people.

speaker-1: Yeah, yeah, I think that the future is very exciting.

speaker-0: Yes, yeah, definitely. ⁓ This is really, really, really super fun to work on. If anybody wanna come and help us, please contact me and ⁓ or Stefan. And ⁓ we'll for sure appreciate the help that will just make things go much faster because otherwise Stefan and his team has a limited amount of time per day and as it turns out, I do too. So that's a shame. So Stephen, ⁓ looking forward, actually, what are your plans for Baseflow?

speaker-1: We have a lot of plans for this. I would say maybe let's start with the long term plan for this is to become the workflow layer for our guys. Like fully AI ready, agentic ready. Like this, should be the, the gold standard workflow layer for amortized inference that interacts very well with, with agents. It's fully interoperable with bi MC and Stan. We're working at all these.

speaker-0: Mm-hmm.

speaker-1: directions, but this is also working in a changing ecosystem. we also, it's continuous development effort that is happening. I think a huge strength of that is our positioning in Keras 3, the multi-backend Keras 3, because this makes us also interoperate with the three most popular deep learning backend tracks, Desert, Loam, Hydro, Shunjax. So you can plug in your own custom network in there. and it works, can download a network from Keras Hub and it just works out of the box. So the workflow layer is this long-term goal, now more short-term, we're looking at inducing these very generalized, amortized engines and piloting them in different fields. For now, we're with time within cognitive modeling because this is still one of... wow, but they say one fourth of my heart still belongs to cognitive modeling. And these models are excellent testbeds because they keep changing their running targets. Researchers keep changing the assumptions that they have. So it's just a great set of testbeds for our methods. having this also enhanced generalizability, which doesn't come at the cost of. In a capitalization gap or robustness gap is the way to go. We still have a bit of homework to do in terms of robustness. This is a thing where small theoretical work is needed. Also more implementation work. Because as you said, we have all these scattered ideas in different papers, even in our own papers, which is they still haven't made it into the software simply because either they're not production ready yet, there's still more in the realm of or because there's not enough energy to put in. But since BASOL is now a community effort, I think this is happening a lot. So it is just that when you're trying to uphold good software development standards, it just takes some time to make sure that this is not just an artifact of research, but it's actually something that can be deployed. So also bridging this existing gap between research on SBI and the deployment of SBI is also a goal of baseball. And we'll keep working on this, we'll keep expanding, we have a lot of ongoing research. also, in a third strand of our research, we are targeting more more challenging applications in our recent review paper on diffusion models. for simulation-based inference, we categorize problems according to difficulty. And now we're getting more vicious targeting the very difficult problems, high dimensional problems, expensive simulators, not a lot of data. I this is where SBI is currently completely underutilized. these methods, so these applications also provide great challenges for the methods themselves, because we just can't. continue benchmarking on our favorite toy examples and pretend that everything works.

speaker-0: ⁓ Yeah, I love that. really love how you guys are approaching these topics in your lab. Well, I think it's in Bean Park thanks to you. You lead the lab. But I really love that you're doing... And that's also on brand with Paul Berkner, know, that's really his kind of research too. No, exactly. And I love how all of you guys think about it as, yeah, we're doing state of the art research, which is...

speaker-1: It's strictly a team effort, yeah.

speaker-0: extremely complex and fascinating, but we never lose the sight of, but how are people going to even use that? And is that even useful or should we maybe work on something else? ⁓ And I think this is the most impactful research there is. And I really love that. And it's also the brand of the show. So ⁓ that's why you've been here twice already in just two weeks. ⁓ Amazing! Well, Stefan, I think it's time to call it... Finally a show! So that means I'm gonna ask you the last two questions I ask every guest at the end of the show. First one, if you had unlimited time and resources, which problem would you try to solve?

speaker-1: Yeah, so I didn't think about this a lot. Well, let's confine ourselves to local research problems and not big world problems, I would say. But if we're very, very locally, I would just solve the problem of automating statistics, democratizing statistics. For everybody, it's just taking the strain of dealing with computational issues and unlocking creativity.

speaker-0: Mhm. Yes, love that.

speaker-1: simply free of computational considerations.

speaker-0: Mm-hmm. I love that. Yeah, of course. I resonate with that. And I'm sure a lot of people will. Second question, if you could have dinner with any great scientific mind, dead, alive, or fictional, who would it be?

speaker-1: That's also a very difficult question, but I would probably pick Ed Torp, the mathematician who developed card counting strategies in Blackjack. Blackjack then went on to start the first hedge fund and basically demonstrated what it means to think independently and think outside the box. He's still alive. 93 going strong. So this is also very inspirational. So that will be my aspirational peer.

speaker-0: of that yeah sounds like a fun dinner for sure well ⁓ Stefan I think I think that's it so yeah that that was of course two dense shows I hope ⁓ you folks liked it and please make sure to to add the different papers and content we just mentioned today to the show notes Stefan for people who want to dig deeper and all Thank you again for taking the time and being... again on this show.

speaker-1: Thank you again for the invite, I'm already getting nostalgic.

speaker-0: This has been another episode of Learning Bayesian Statistics. Be sure to rate, review and follow the show on your favorite podcatcher and visit learnbayestats.com for more resources about today's topics as well as access to more episodes to help you reach true Bayesian state of mind. That's learnbayestats.com. Our theme music is Good Bayesian by Baba Brinkman, fit MC Lass and Meghiran. Check out his awesome work at bababrinkman.com. I'm your host. Alex and Dora. can follow me on Twitter at Alex underscore and Dora like the country. You can support the show and unlock exclusive benefits by visiting Patreon.com slash LearnBasedDance. Thank you so much for listening and for your support. You're truly a good Bayesian. Change your predictions after taking information in. And if you're thinking I'll be less than amazing. Let's adjust those expectations. me show you how to be a good daisy Change calculations after taking fresh data and Those predictions that your brain has made Let's get the on a solid foundation