speaker-0: My guest today is Luigi Ancherbi, Associate Professor of Artificial Human Intelligence at the University of Helsinki, where he leads the Machine and Human Intelligence Lab. I've been following Luigi's work for years, and it sits right next to a bunch of episodes we've done recently. Luigi is, above all, a tool builder. He came to Bayesian inference from theoretical physics and computational neuroscience. Chasing one idea that inference and prediction might be the thing that brains, science, and machines all have in common. Out of that came methods like VBMC and BADS and the Pi VBMC and PyBADS packages that a lot of scientists actually use. More recently, Luigi has fallen in love with neural processes, and we get into why Transformers Turn out to be such a natural fit for them. We talk about his amortized conditioning engine, active data acquisition, changing priors at runtime, and where all of this is heading. Foundation models for inference. We even get into small local LLMs and Bayesian decision theory for agents. This is Learning Bayesian Statistics, episode 161, recorded May 27, 2026. Let me show you how to be a good day and change your predictions after taking information. And if you're thinking they'll be less than amazing, let's adjust those expectations. What's a Bayesian is someone who cares about evidence. Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible. I'm your host, Alex Andora. You can follow me on Twitter at Alex underscore Andora. like the country, for any info about the show. Learnbase stats.com is Lab Plus2B. Show notes, becoming a corporate sponsor, unlocking Bayesian merch, supporting the show on Patreon. Everything is in there. That's LearnBaseSats.com. If you're interested in one-on-one mentorship, online courses, or statistical consulting, feel free to reach out and book a call at topmate.io slash Alex underscore andora. See you around, folks, and best Bayesian wishes to you all. Gia Cherby, benvenuto a Learning Vision Statistics.

speaker-1: Grazie.

speaker-0: Yeah, I I would love to do the the episode in in Italian. my Italian is very, very rusty. I I used to speak pretty ⁓ pretty good Italian but then I went to Argentina and had to learn Spanish and ⁓ the two languages are so close that ⁓ like it just like my brain just ⁓ confuses it. So I need I guess I need to go live in Italy again.

speaker-1: True. Yes. I mean actually I noticed that my my wife is Spanish, so let's say, so we ⁓ we can ⁓ we we speak English. The official language is English in the house, but ⁓ we we ⁓ kinda you know, so we pick up the each other's language, so let's say so I speak a bit of Spanish and she speaks ⁓ actually she speaks way better Italian than than well than me than my Spanish. So anyways.

speaker-0: Yeah, okay. yeah, I mean that's now I'm the weird thing is that so now when I come to Italy, my Italian comes back after a week, more or less, but the first week I'm like the on the classic proverbial French that ⁓ people hate ⁓ who thinks you know they who think they speak Italian but they actually speak Spanish. And and I know Italians hate that and I don't do it on purpose so I'm

speaker-1: I don't mind. I don't mind. It it's close enough.

speaker-0: also. Well, really, really happy to to have you on the show because I've been following your work for a few years now and you do super interesting stuff related to ⁓ to a few episodes we've done now on the show, especially with ⁓ with Marvin Schmidt and ⁓ even more recently with ⁓ Stefan Kalef. Episodes one hundred seven, one fifty seven, one fifty eight. These are gonna be in the related ⁓ episodes for this one. But so for today, ⁓ let's start. As usual with your origin story, Luigi. What ⁓ what are you doing nowadays and how do you end up there?

speaker-1: Right, yeah. Well thanks thanks for having me. Really really happy to be here talking to you. And okay, so where am I? So well we're now physically I'm in Helsinki and ⁓ so I'm ⁓ professor, associate professor of artificial human intelligence. ⁓ the work we've been doing ⁓ recently in the past few years focuses on using modern machine learning tools to do Bayesian inference. That's that's kind of the the broader area and ⁓ we apply these tools sort of to various ⁓ domains. One goal is to do just predictions and inference. ⁓ but let's say one application is obviously ⁓ model fitting in the sense that you have a model, you want to you know infer the parameters of that model, ⁓ you want to make predictions with that model. And nowadays let's say there are powerful you know neural networks that allow you to kind of do that. ⁓ fast and there anyways that that's the direction that we are ⁓ following recently but how I got here is a much longer path and so I'll try to compress it like literally how you know modern agents when you do compaction so I'll try to compact a very long string of tokens into a few sentences. Originally say as we were discussing a while ⁓ like a little ago ⁓ yeah, so I mean I grew up and studied in Italy, ⁓ but physics, that that's kind of where I come from. ⁓ theoretical physics. But then ⁓ for my PhD I moved to the UK, Edinburgh, and ⁓ where I studied ⁓ was researching computational neuroscience. ⁓ at the time it was called computational neuroscience and neuroinformatics. And the brain is very confusing, ⁓ especially if you don't know much biology, and I didn't definitely know enough biology at the time. No, no. And ⁓ Neuroscience is even more confusing because it's also I mean it's very interesting, but it's also very, very confusing. And as a physicist, always, you know, one of the things that teach you is that you know you want to simplify things as much as possible. So let's say and trying to simplify the brain is hard. ⁓ possibly let's say ⁓ you know there are also let's say disciplines or currents that say that you shouldn't try to simplify too much. But anyways, but you know, that's the kind of the temptation. You want to kind of come to the simplest possible description in a way. And ⁓ I was kind of flailing to try and find ⁓ to make trying to make sense of it. And then I read a few influential things and I studied a few influential things. ⁓ one first I think it's the one one one book that kind of influenced me at the time and actually convinced me to go to computational neuroscience was ⁓ on intelligence and ⁓ and the idea that ⁓ what the brain is doing is prediction. And that seemed like a a beautiful so everything that the brain is doing, everything like in the cortical columns in the brain are doing is just, you know, kind of predicting, ⁓ predicting inputs. And I found that you know a very powerful idea at the time. And then you I studied well studied slash ⁓ started researching in computational neuroscience. Then I was exposed exposed at the time to Bayes. And ⁓ it that was in a course. ⁓ as a physicist, I had absolutely no idea of what Bayes was, because it's a ⁓ that's not taught. It wasn't taught at the time. ⁓ so let's say I just had you some kind of sprinkling of ⁓ you know classical statistics. And then you when I was kind of exposed to Bayes theorem, it's like, okay, okay, this explains everything. And ⁓ and then essentially not just base that that just was based around in a machine learning course, ⁓ probabilistic machine learning. But the and then let's say the idea that the the brain does based on inference and what the brain, you know, essentially what does in a nutshell, ⁓ everything in the brain does ⁓ I mean not everything, but say a a big chunk of what the brain does is being a sort of statistical inference engine. as kind of trying to make sense of the external world by building model, you know, assigning probabilities, you know, computing posteriors. And I find that idea ⁓ very ⁓ alluring. And then essentially then we that essentially that's kind of everything kind of came together. So it's just starting to study this version of ⁓ kind of neuroscience, cognitive science. Then of course then I became more practical with the tools of Bayesian inference. But then again, I was kind of studying Bayesian inference because the system I was studying was using Bayesian inference. So I was kind of trying to style a system, you know, but then you know you can use the tools of Bayesian inference to study the system which uses Bayesian inference. So I was doing kind of sort of a Bayesian inception in which you know I was using Bayesian tools or Bayesian models to understand a Bayesian system. And ⁓ People have called this a double Bayesian. I argue that what I was doing then became tri triply Bayesian because it's say then I kind of I started using tools which themselves used Bayesian inference to perform their task, like and I will get to to those later. So anyway, so I base everywhere in my life. And ⁓ and then essentially so I started building tools to do model fitting. And then I kind of like that part. ⁓ these models are complicated. ⁓ requires know complicated tools to fit the parameters. And and then essentially I started developing those tools as part of my job. Because anyways, Jammer is the same. You have base and you know whether, you know, how you apply it is up to you. So everything became essentially yes, you know, building tools. And then eventually I kind of moved even more into machine learning because again, as as I tend to say, I'm interested effectively, you know, again in this ⁓ this idea that inference is kind of kind of some sort of a foundational, fundamental ⁓ concept in the universe or ⁓ you know that's ⁓ that's useful you know say in in ⁓ in biology is useful in ⁓ in ⁓ in science and ⁓ so so I'm studying it you know in all these different areas. And then so essentially so it was kind of natural to move to machine learning to do you know the the the things we are doing now. ⁓ because again ⁓ studying you know say the the the end system in a way is relatively relevant. What what kind of the the core element that stays the same is exactly this this idea of ⁓ inference and prediction.

speaker-0: Yeah. Yeah. ⁓ makes sense and it's really interesting to have the to hear you about the the the you know the the whole arc of your your career. so you've touched on it a bit, but maybe ⁓ even more specifically. What if is there anything in particular or someone in particular that drew you to patient statistics specifically and and then to amotize the inference and and neural processes?

speaker-1: So the path so yes, so let's say so I had a few ⁓ professors at ⁓ when I was still let's say a master's student and a PhD student ⁓ in Edinburgh. ⁓ at the time there was ⁓ I mean say still there is ⁓ a very, very vivid and interactive ⁓ machine learning and computer science and also at the time say a neuroscience, computational neuroscience community. And ⁓ so there was a lot of interconnection between ⁓ our cohort of computational neuroscientists and the machine learning cohort. In fact, let's say people work on those who ⁓ all the time. The I think I said there was one course which I took at the time, which was ⁓ something called like probably something like ⁓ probabilistic quality modeling, which is similar to the course I'm currently giving. And and ⁓ so yeah that introduced essentially among other things introduced ⁓ idea of the Bayesian brain or the idea that you know the brain does probabilistic inference. At the time it was taught by ⁓ Peggy Sergez. She was ⁓ lecturer ⁓ anywhere at the time and ⁓ so yeah so that was quite inspiring and then kind of I I took ⁓ that route and then I I kind of know never really ⁓ left it. And ⁓ coming to today you're asking about well you know what I'm doing today you're asking about the neural processes So actually, that is kinda kind of funny because neural processes are essentially ⁓ neural networks that ⁓ learn to do inferential and prediction tasks. And I stayed away on purpose from a neural network for most of my life. That was a conscious choice, possibly wrong, actually, surely wrong. And and the reason is that they say when I when I was a PhD student at the time, say neural network ⁓ deep deep nest, well actually at the beginning they didn't really exist. Deep Nest exploded, let's say, in two thousand, you know. famously people mentioned you know Alexnet at twenty twelve, ⁓ and so as the year in which you know deep learning started working. And I started my PhD in twenty ten. So let's say so when I started it was it wasn't quite working yet. And ⁓ but still let's say at the time for a number of reasons, let's say I wasn't interested in neural networks and that was a mistake, obviously, ⁓ in hindsight. ⁓ but so let's say so so I kind of over took me a while to kind of get back to neural network, let's say I did it very late in my career, like I did it like probably a few years ago. And what exactly what kind of convinced me to come back were neural processes. ⁓ so neural processes, so there were these beautiful papers ⁓ in ⁓ 2018. I kind of discovered them maybe a couple of years later, one two years later. And these ⁓ so the the first author, ⁓ Marta Garnello, I think she's at the time she was a deep mind, ⁓ I think. ⁓ now I'm not sure. But let's say and you know this was yes anyway. So ⁓ Group at ⁓ Deep Mind, if I'm not mistaken. And the idea of the original neural processes paper, there are kind of two papers. One is called Neural Processes, the other conditional neural processes, but yeah, they're similar in ⁓ in scope. The idea is that you can train a neural network. At the time, it was kind of like a standard ⁓ multi-layer perception, so like you know, a relatively simple ⁓ network. So, normally how we train a network is that ⁓ you have a data set and you know you train on a single data set. I mean, that's people who People you used to do it. So you will train on a single data set. Maybe you have a held out data set that you use for validation. But you you you train on a data set and then you have to perform a task. And you have a single data set to train on. The idea that they had at the time is that instead of training on a single data set, you train on a data set of data sets. So say you have a lot of you know kind of small data sets that you're training on, and for each one of them, you're training your network given the small data set ⁓ to do a task like you know predicting. ⁓ for example the val you know it's giving us a small ⁓ data set of points like you know an aggression problem I give you x and y value for each so that's my data set then you know I want to predict you know the value of the function at other points other values of x. So what you do you take your data set you split it in a a training set and a test set and in the ⁓ in the neural process ⁓ literature you would call them context and targets and by the way context will come back and And you know, and idea is that okay, so now I'm training my neural network to given the context to predict the targets. You do it once, twice, one thousand times, because you know you have this a lot of small data sets that you can chop, you know, ⁓ you chop in context and target and you repeat this procedure a lot. So now what you're doing is you're you're training your network that you know to do this task, what you this prediction predictive task, and then you know, at at runtime, now you give it another data set, which you call. context and you give it a bunch of targets to predict and your network has learned to solve this prediction task. Okay. So but the point is of course it's done it by it's done it by learning by seeing a lot of different data sets. And of course the the idea is that you would use ⁓ for example some generative process or a data set of data sets to train your network in such a way that it kind of learned the statistics of the task Such that when you present a new instance, it can just resolve it immediately. And nowadays, or already at the time, but you nowadays, you we would call this a form of amortized inference in the sense that you can't do this training on many, many, many data data sets, and then you know, you just deploy it and then at runtime you can do your predictions on new data points, you know, kind of instantly. And I found it it was beautiful because it's a, for example, when you condition on a bunch of points, that's a set, typically. So it's I mean, unless it's time series, but let's say, you know. ⁓ typically it's a set. ⁓ that's you know also a property. So kind of you want to satisfy some prob some properties or probability of stochastic processes, like for example, you know, if you if you change the order of the points, you know, your prediction shouldn't change because it's a set. The data points are exchangeable. ⁓ if you say that's the term the term. And they they use a structure which ⁓ is called the d a deep set to represent this set with a neural network. And so essentially so it's a mix of you know Bayesian statistics. ⁓ you know stochastic processes like Gaussian processes for example. ⁓ but let's say anyone uses deep networks, but let's say it's kind of also mathematically beautiful because let's say you have, you know, you have these properties that gets, you know, ⁓ so which are preserved. These are these the kind the fact that you can now move you know permute your your nice small set of data. So anyways, and then you know so you have something that learns a task. And I found that actually really, really interesting and promising and kind of you know very elegant in a way, as as opposed to maybe you know kind of just okay, I I train a network to do a bunch of stuff. Like, you know, this is ⁓ there is some elegance to the to the concept of of a new process. And I fell in love with that. And then you know, I started digging and you know, and then we get to today.

speaker-0: Super cool, yeah, yeah. Dim. Thanks for ⁓ thanks for all these details and and actually so one of the things you you've been working a lot is ⁓ is a concept we haven't touched down yet really on the show, which is called ⁓ VBMC for variational patient Monte Carlo. And so you describe it as like patient optimization but for finding the full posterior instead of just the optimum. Wha what does that mean? What is the core trick, let's say, and the core method that makes that possible?

speaker-1: Okay. Cool. Okay. So yes. So to connect a bit to what they just said. So before I kind of mention in pass in passing Gaussian processes. So let's say so that was kind of my intermediate phase, which is still ongoing, let's say. So so before I move to kind of I I came back to neural networks. Well actually I mean, not I came back, I I came to neural networks. so I was kind of still in the camp of kind of traditional Bayesian non parametrics, which is users cando, yeah, and and the the workers of that field is the Gaussian processes. And ⁓ so I mean I may I will assume that you know your audience might have heard of of ⁓ Bayesian optimization, which is the idea is that you have ⁓ essentially you want to optimize a function. ⁓ this function might be expensive. so let's say so you know you have a a limited number of evaluations you can take to find the optimum of this function. So what you would do, ⁓ say kind of traditionally, is that ⁓ one approach is called is called Bayesian optimization, and the idea is that you take your your evaluations. you can use some form of ⁓ of regression or surrogate model that you're fitting to these points and then you use this surrogate to decide okay where to sample next. And and the goal of this ⁓ of this procedure is that of course you want to find points which are promising in the sense that they tell you where the optimum might be and they kind of guide your sampling procedure so that you know you can find, you know, a good candidate for for the optimum. And ⁓ yeah the the as I said, the workers for this is a Gaussian process, which is again is a Bayesian kind of mathematical object, which has a lot of nice, very, very nice and pretty properties. Everybody kind of loves Gaussian processes when they can use them because they're really, really mathematically beautiful. ⁓ They have a bunch of kind of downsides. But ⁓ otherwise I say yeah they're they're they're really practical and ⁓ they have again so one of the sample some of the properties they have is that they they behave very well with with little data. So let's say because essentially they're they're they're a Bayesian kind of construct. So they come up with a strong prior for what ⁓ a function looks like. For example, your function, depending on some choices you made, but typically you assume the functions are so smooth. So it's a sort of your function is not going to do some something crazy, but it's kind of you know it interpolates between points nicely. And ⁓ so people use it a lot in ⁓ in optimization. And that's kind of beige optimization is kind of you know it's a it's a field in itself. It's a big field. And there is a a beautiful book by Roman Garnett ⁓ recently ⁓ came out ⁓ beige optimization and you know I I really recommend it's a it's a Really, really awesome book. And anyways, so that's optimization, but ambasion. I don't want to just optimize, I want to find the full posterior. And so let's say so. The idea I had ⁓ many years ago ⁓ was well, can so because you know when when you're building your your third model using a Gaussian process, you're kind of representing a function. And of course, at the end you only care about optimum, but I say no, you're kind of representing the shape. So can we use that? But you know. where the function is not an arbitrary function, but it's is effectively a posterior, and more specifically the a log posterior, so the logarithm of the posterior. And you know, it's it's a function. So can I use something similar to Bayesian optimization to kind of you know sample this distribution? By sampl when when I mean when I say sample, I mean evaluate at arbitrary point, not sample like as in MCMC. So you know so I'm just you know evaluating this function at a few points. And then you know I use these very powerful interpolator kind of build kind of you know the entire shape and then instead of optimizing it what I want to do with this shape is no no I I just I keep all the shape and that's the posterior. So I want to learn this shape of the posterior by a few carefully selected points. And the reason to do that, well of course if your function like if if you say if you're working with a model that you can write in ⁓ in Stan or PyMC, probably that's not very expensive so you can run MCMC. But you know there are lots of models in ⁓ in main disciplines where you can write ⁓ you know, your your model, your log likelihood. But evaluating it is expensive. So I say, so you can calculate it, but I say it's expensive. So I say, so, you know, so you and you know, and that's essentially the idea of V BMC that's essentially kind of does kind of the same job as basic optimization. So it lets you get the shape of the function. But you know, instead of, you know, being expensive and taking, you know, that maybe you know MCMC might take, you know Tens of thousands, hundreds of thousands, even million evaluation of your function, which is your function even takes one second per evaluation. You know, you need to wait one million second to to do a full MCMC. While VBMC typically, you know, with a few hundred evaluations, you can, you know, kind of get a a good reconstruction of the shape of the posterior. It's not perfect. ⁓ you know, we can discuss it. But let's say so it's ⁓ again, it's a very good approximation typically. in low dimension. That's that's the cave. The essentially the big caveat is that you know this works well in low dimension because you're building this surrogate. And as anybody, you know, you work with GPs, from production processes, ⁓ yeah, they work well, you know, in low-ish dimension. Let's say, you know, if you go above, you know, 10, 20, 30, let's say it's kind of gets hard because just you you don't have enough point to kind of interpolate ⁓ the shape of the function. So it works for modulars that have up to I mean five, ten, fifteen parameters.

speaker-0: Damn. Okay. Yeah. So that means how would you in which cases would you use that concretely? ⁓ I think it's gonna be interesting for for listeners to understand. and while preparing for the show, I was actually looking at your Pi VBMC package. which looks super cool. I I encourage people to to go in there and and give it a try. so actually Can you can you tell us when should we reach for for that package? When is it not appropriate? I think it will give us an idea of ⁓ when these kind of of methods are most powerful.

speaker-1: Yeah, yeah, thank that's that's a good point. So so one thing I want to say that so I always the reason why I moved to to kind of doing these things exactly because I came from the practitioner side. So I was building models and fitting models and it was slow. So let's say so I say, okay, I had an actual problem I wanted to solve. And many of the users of VBMC or of the other ⁓ our other tools like BADS are are practitioners. They they work in cognitive science, computational neuroscience, and now you know it's kind of spreading to other fields. You know, kind of but let's say so. So yeah, so if you if you look, for example, at the at the website, ⁓ there is always, you know, I've I've big FAQs that are gonna tell you, you know, when it works, when it doesn't, because exactly we kind of want to ⁓ be clear for the user and say and you need to be very honest about what ⁓ what a method does and and doesn't. So in the case of ⁓ PyBMC, so when should you wish it out? First of all, let's say so if you're if you can code up your model in ⁓ in Stan, in ⁓ in PyMC, ⁓ you know, everything is working fine. you're good. So you know there is no need for for prime MNC. ⁓ so the point is, you know, whet say if you know that you so if you have a problem and you know you have a problem, you know, in the sense that okay, so for example, you know, you have a model which for a number of reasons, doesn't you know, y sometimes you can't even code it in a in ⁓ say in Py ⁓ C or in ⁓ or in Stan because say there are some aspects which are non differentiable. So I say sorry in trouble because ⁓ you you say these methods you know would need to compute the gradients, I mean automatically ⁓ to to do MCMC. So for example, you know, if you're for for a number of reasons, your your function is not an easy function, but say there's some sort of a complex process that you need to solve. And this emerge in ⁓ in the computational sciences. So for example, you know, if you're building a statistical model from scratch, this is rare because say by definition, you you think in terms of easy, nice distribution that you can fit. But let's say ⁓ if you come from the science, it's kind of the opposite in the sense that you know you don't think in terms of statistics or distribution pack, you know, you think in terms of the process that you're trying to model, and that might not kind of behave like ⁓ any of our nice ⁓ distribution that you can get out of a package. Let's say so strategically so I would say most of the usages come from science and engineering, where you're kind of trying to model something which is kind of has its own process that you're trying to model, and then you know, say then ⁓ And the the kind of symptom that you have, you know, it's not that you know the process complex or not, but the symptom that you have in practice is that ⁓ for example, it takes time to compute. So I say so just you know evaluating one ⁓ point, one param, you know, the log likelihood or log density at one point, you know, takes you know, the order maybe you know, one second or something. One second might not seem a lot, but I say it is a lot, because I say again, if you need to sample like if you evaluate your function in again, tens of thousands of points, yeah, ⁓ that's you start feeling it. So say speed typically is kind of the first symptom that you know people get aware of. And this might happen, let's say, even if you have a classical kind of easy model that you know you could fit with ⁓ you know, with Stan, et cetera. But maybe for some reason, let's say either you you have a lot of data or there is something that makes this ⁓ particularly nasty. ⁓ and then let's say, so again, maybe the time factor. So I would say the time factor is the kind of the first one that people would would notice. So if you're in that case. Then there are a few kind of other check ⁓ check marks that you need to check because for example, since we use this Gaussian process as as a surrogate, and you know, so essentially you're doing interpolation in well, essentially you're doing interpolation in this in these spaces. ⁓ so we we kind of know, let's say, from you know, from we using Gaussian processing regression, invasion optimization. So these things work well. They up to order of ten parameters or Which means you know maybe you can push them a bit farther, you can maybe get to fifteen, et cetera. But say so if you're, for example, you know, if your model has you know fifty parameters, I would not recommend ⁓ Pi BMC in its current form at least. You know, who knows, maybe you we'll we'll we'll push it to high dimensional. ⁓ but again, it's again again, it's like other problems in ⁓ invasion optimization. The so there is a so why, for example, there is that limit? Because in bridge optimization ⁓ in the end, you know, using the surrogate just to kind of to guide you to find a good point. But anyway, then you're evaluating your your function. So optimization is is an easy problem because it's self-verified in the sense that you can evaluate a point, you know, you measure it, and you can just check whether it is a a better point than the previous one. Then you know you know whether you know you're kind of you're improving your optimization. It's kind of self ⁓ you know ⁓ verified. That's not true for Beijon for ⁓ Beijing inference, because say you're you have a distribution and it's absolutely non trivial to diagnose whether you know your distribution is improving. What does it even mean? Let's say so you just get a different distribution. But let's say ⁓ there is no there is no immediate ground truth that tells you whether something is correct or not. So anyways, so it's a way harder problem. So let's say so there's a lot of things that V BMC does, Pi BMC does, ⁓ to make sure that inference is going correctly. So let's say there is a bunch of diagnostics. but anyways, so let's say so this this is ⁓ check check the speed of your evaluations. And check the dimensions. These are kind of the starting point. Then you know there are other things that you can start looking for. So if you are in this case, then you know, then it might be useful.

speaker-0: Yeah, okay. Very practical, thanks. And and that's indeed what I really love from your research is that it's not ivory tower research. It's it's really okay, I have this problem, how do I solve that? Not only for me but for ⁓ for everybody. So so I think I think that's what makes what you're doing and and also what the whole group at Helsinki university and alto ⁓ mm is making is just always very very practical and helpful. And so accompanying these papers with ⁓ with open source packages as you do is is always a great way to make sure papers actually go through two practitioners' workflows, which is in the end what I'm guessing is the most useful. ⁓ And so you have you have another package actually that's called Pybats B A D S Can you tell us what the difference is, like what is bad in comparison to V BMC? And and when is that helpful? You know, which cases does it help?

speaker-1: Yeah. Yeah, thanks. By the way, I didn't mention what V BMC stands for. That's ⁓ it's it's it's a mouthful. It's a Vagational Beijing Monte Carlo. So that's it was ⁓ I didn't I didn't check any any PR or any it wasn't to to find like ⁓ a good name. But anyways, ⁓ but is ⁓ it's probably nicer is ⁓ Bayesian adaptive direct search. And ⁓ what is it? So I kind of came earlier and ⁓ I I think it's a yeah it's quite ⁓ widespread now. and ⁓ It's an optimization method. So let's say so unlike so VBMC is, you know, me trying to do Bayesian inference properly. So PyBuds or bads came earlier. So it was kind of okay, I I'm trying to solve these module 15 problems. ⁓ let's treat them as ⁓ an optimization problem. So let's say you just you maximum likelihood estimation. So you won't just want to find the parameter that maximizes, you know, your the probability of your data density of data, the your log likelihood, your likelihood. And ⁓ so And you if you strip of everything, it's just an optimization problem. So you can use any optimizer you want to do maximum liquid estimation. You you you you take your favorite optimizer, you know, if you can compute the gradient, you compute the gradients, and you and you you go. ⁓ so the problem is that most of the problem they say of the models that we had at the time, still nowadays, you know, in the sciences, ⁓ for example, yeah, they're not nice, you you don't have gradients, etc. etc. So I was looking, so you enter the realm of ⁓ derivative free optimization in the sense that you need to do optimization without having gradient. That's hard because then you know, given a point you don't know exactly where to go because the degredient helps you know knowing where the minimum is, as it's you know, a good a promising direction is where I say without the gradient, you need to figure it out. You know, and that's actually I I had ⁓ kind of habit home for one year, kind of studying optimization, ⁓ you know, ⁓ studying all sorts of different methods. But anyway, so I was very unhappy with existing methods at the time. I was still kind of still doing you computational neuroscience and ⁓ model fit thing. So I said, okay, I need to I need to write a better method. And Which works, right? Because there were many methods which were published will look very, very good. And then I any any method I tried broke on on our problems. Like, you know, there nothing works. And it was very frustrating at the time. Then I realized how, you know, okay, maybe I shouldn't say it, but I realized how you publishing machine learning works. And ⁓ okay, so one thing is what you say in the paper, one thing, you know, then trying to use the method in practice. And you know, literally mostly papers I tried like broke on, you know, first impact with a real problem. So I said, okay, you know, I want to solve this. And I mean well, solve it was ⁓ is a big word, but saying I want to produce something that you know is usable. Okay, let's let's say it that way. And ⁓ yeah, so that's essentially BADS. So BADS we we talked about Bayesian optimization before. So Buds is a hybrid Bayesian optimization method. And the reason why it's hybrid is because if you use Bayesian optimization in a in a vanilla sense, like you know, just you know, kind of out of a textbook, it doesn't work. So that's the secret. It doesn't work. So you need to ⁓ use a lot of other tricks which you know even modern packages use kind of you know behind the scenes. ⁓ but even so let's say many packages would break on our problems because they're kind of very nasty and the Gaussian processes would break in various ways. Gaussian processes unfortunately are very nice, but also break all the time. And ⁓ so what's hybrid? So hybrid effective, you know, you want a method that so when your surreal is good, like you know, your version process is good, then you trust your Gaussian processes, Gaussian process, and you know, and that can allow you really to kind of reach the minimum very fast. So if you have a good approximation, that can be very powerful. And you can really jump through the landscape and find the minimum. The problem is that many times your surround is not good. And ⁓ because you know your function might be very nasty, you don't have enough points to really represent what's going on. And trusting the surround would be a terrible idea then. Then essentially you want some sort of a fallback ⁓ methodology to kind of you know to still do optimization in the situation, maybe that get out of that region which is kind of nasty, and then you know go back to kind of trying to fit things with ⁓ with your surrogate. ⁓ now what I said look like it looks like, but there is a framework which is called mesh adaptive back search, ⁓ which is you know it's it's a paper, it's a very highly cited paper. The journal. of optimization if not mistaken. But anyway, so it's ⁓ so that gives you a framework for how to do this. So what I did essentially I took the framework, it's a framework so you can kind of play around with different components. And I kind of you know I kind of built my version, which is again ⁓ buds. And the short answer is that essentially so again it's it's it has limits because of course metals have limits. So say so for example works up to you know 20 ish dimensions because of this idea that uses discussion processes. This is you can use it with higher than 20 dimensions, but then that kind of that part of ⁓ building a a surrogate a nice circuit breaks down pretty much all the time. So you end up only with the kind of the other part, which is the kind of slower and not fast way. So you get you know a a bad method. But if you are in a situation in which your model, you know, sorry your your function can be modeled in in some parts, not in all of it. So say so you can have bad parts and you know and the method gets out of those. But then you know once in a in a good part ⁓ then you know you can you get this kind of acceleration because now you can ⁓ build ⁓ the threat and then you so it it can be very efficient. And there's this nice thing which is kind of the you know ⁓ I think ⁓ Andrew Gellman ⁓ mentions it in ⁓ in his blog so there's this thing that when a model is good like ⁓ and when you know near the like near the optimum, near the maximum maximum likelihood ⁓ value, things are nice, you know, because I mean if this come actually you know from ⁓ the Sorry, ⁓ Bernstein von Mises theorem. ⁓ in the sense that, you know, things eventually, you know, kind of all converge to a Gaussian. And so I said so near the the kind of the the maximum like solution, maximum posterior solution, everything kind of looks like you know it's nice and quadratic and Gaussian. And so I say so. If you're in a good region, things are nice and then you you can model them nicely. If you're farther away, things can be nasty. But the point is, you know, you don't particularly care because those regions are just BAD. So you know, you kind of want to e exit those regions quickly and get to the good ones. So anyways, so that's that's why in a way why ⁓ BADs I think also works very well for optimization of especially for models.

speaker-0: Okay. Okay. Damn, this is this is also super interesting. I feel like it it works in in a smaller number of cases, right, than than V B Dm, but if you enter that case that use case it's gonna be extremely helpful to use Pybets.

speaker-1: Yeah, it's actually more the other way around. Like Pi Buds is why is more of a universal in the sense that a lot of people in our field I know I know I know no I'll say I think it's a you you of course say you come from Bayesian statistics, so you think everybody does Bayesian statistics and that's what I wish. But no, that's not it the reality. The reality in ⁓ in many scientific fields, at least you know, in ⁓ in cognitive science and ⁓ especially in neuroscience, people are not the level yet of doing Bayesian inference and they they are totally okay doing maximum lack estimation. For a number of reasons which are also understandable because the models are very complicated, so doing Bayesian inferences would be completely hellish. So I say so yeah, so people are totally happy with getting, you know, half it. So let's they don't, you know, they don't have any pretense of trying to find the full posterior distribution. So that's that's what's kind of my attempt with the VBMC. ⁓ so that's why buzz you know is way more used as well, because it's essentially in the end it's just an optimization method, ⁓ with you know, with some tricks to make it work, etc. in you in in harder cases, etc. ⁓ so yeah, so I mean I would say a a lot of people are using ⁓ buzz and it is a smaller kind of community that uses ⁓ V B ⁓ C.

speaker-0: Yeah, yeah. Okay. Interesting. ⁓ that's actually a good point. ⁓ and I don't know if you already have a mascot for ⁓ bats, but it obviously needs to be a bunny 'cause that way it's ⁓ it's a bad bunny, you know, and ⁓ and I think it's it's a perfect marketing tool, you know, so

speaker-1: Yes, that's that's a good point.

speaker-0: Yeah, we need to we need to get them there. ⁓ that's how we'll win the battle. so okay, so that's BBDM z that's bads. ⁓ and and and running a running theme, a running topic in your work, especially lately, is neural processes. You you said at the beginning you really love them. ⁓ what was the unlock for you? What's so fascinating about them?

speaker-1: Yeah, so let's say so it w it was interesting exactly because I said, you know, I was kind of wary of ⁓ of deep nets. ⁓ because again, you know, I mean just honestly, just because, you know, in a way I did I I didn't understand them. I mean, let's say I mean yes, okay, I understand them, but you know, I didn't get them because it looks like okay, you just have a problem, you throw a neural network at it, and then it kind of solves it, but that's a bit ugly. And and then, you know, I mean of course it's a I mean I I I I'm talking like a physicist, you know, who likes to see you know symmetry and and you know and ⁓ And then you let's say yeah, neural processes to to me at least, ⁓ they spoke to me ⁓ exactly because it's a they were not completely arbitrary, but it's a there was this idea that first, you know, ⁓ first of all, they did the very nice idea of this kind of meta-learning in the sense that you're training on a data set of data sets. So it's a so you're kind of learning what the task is by seeing a lot of instances of the task. And so and each it it's nice because each kind of so. Each instance is kind of like a micro problem that the network needs to learn to solve. And then you're just given another micro problem, and then another micro problem, and then you know, you kind of ⁓ and it's kind of learning to possibly generalize, you know, across different ⁓ data sets. And that's kind of I find that's okay, that's interesting. There's something going on there which is which is interesting. And ⁓ and then they say I specifically liked this idea that okay, so since again, you are conditioning on this available data and this is a set. You want your the architecture of the network to to respect this ⁓ the symmetry. And and this is done, let's say, with the old kind of ⁓ conditional neural processes or neural processes originally, with a deep set, which was proposed in a previous paper, ⁓ a deep set. And ⁓ which essentially takes it's a way to encode ⁓ set based information in a neural network. Again, it kinda you it's it's quite elegant. And ⁓ it tells you, you know, how it I mean the original paper tells you how you can do it, what are the constraints that you can have, you know, on your architecture to do this. And ⁓ yeah, it's ⁓ I mean I say and it's actually quite simple. Like you know, the b the basic version is quite simple. Essentially the idea is that you have a bunch of data points, each data points get embedded separately, and then you get a representation and then you just sum all the representation. And since, you know, the sum operation, you know, is commutative, you can swap all the orders and you know, I say the the end up the end representation essentially just, you know, ⁓ it doesn't change if you change the ⁓ the order of your data because say that just you know you embed them and then you sum them. And so I found that you know it was ⁓ it was kind of very, very pleasant. Then you know you can have ⁓ deeper ⁓ deep sets, but say for example the original neural processes and many other papers essentially can't they stop at one level in the sense that you know you kind of get these data points, you embed them, and then you know, and then you just sum the embeddings and you get this representation. And then you know you have a kind of a readout that kind of makes predictions based on that embedding.

speaker-0: Okay. And so that's why you're saying that transformers are a natural process architecture. A natural sorry a natural neural process architecture? Like I mean because I I've seen you write that and I'm wondering what that means precisely and what that implies for how we f should think about prediction in context learning. I'm a type basis and inference in and all the the other topics you work on.

speaker-1: Yeah, yeah, yeah. That's that's a good point. So so that's why that's that's for me was the bigger unlock. When I when I especially after kind of learning about neural processes, and then a few years later, again, ⁓ you know, took me a while. ⁓ you know, everybody was talking about these transformer things. And, you know, twenty seventeen, of course, ⁓ tension is what you need. But then, you know, Transformer became very big, ⁓ as we know. And ⁓ and then yes, so let's say so what I described before. ⁓ I mean I use that language, but so it was a language at the time. It's like you know, you have data sets, you embed them, you have a representation, and then you work with that representation. ⁓ nowadays we would call them tokens, and but it's exactly the same thing. So you have each kind of data point ⁓ is is a token. ⁓ you embed it in a representation, which is you know your whatever ⁓ and you know D-dimensional vector, and Now you wouldn't sum them because summing is what you know what we were doing before in in a kind of standard neural processes in ⁓ in modern you know in the transforming architecture, you allow each token to kind of look at each other using the attention mechanism, which is just you know it's a is a more complex operation. And the beautiful thing about attention is that attention again is a set operation. So let's say so there is no ordering in a transformer. ⁓ since transformers were proposed for Language modeling, even originally. So actually they had to break the permutation invariance or equivariance. So they have to introduce ⁓ what's you know position embedding, which is something like, okay, this you know is the first position, this is the second position. So it's something that you need to add to a transformer to make it ⁓ aware of the sequence or that it's a sequence. But otherwise, you know, a transformer as is the architecture is ⁓ you know permutation, say equivariant slash invariant, depending on what you do. And ⁓ so they say, so yeah, so you can just shuffle around your data points and you know, and the result doesn't change. So I say so it's a very, very natural ⁓ kind of architecture to represent the idea of getting information ⁓ from a data set, a set of ⁓ data points. Each one is a token. These tokens you know talk to each other by attention. You do it multiple blocks, they keep talking to each other, and then in the end you have a representation. And now you can use that representation to do things. And If you think in Bayesian terms, what's going on here is that your network is conditioning on this data set. And now whatever so it's kind of learning, you know, is kind of given this data set, then you know you can take, you know, the for example the last layer or whatever, and then you know, do prediction based on this layer. But say what you're doing is you're conditioning on that information to predict other stuff. And that again, you know, it's it's ⁓ that's why I I was saying that yes, so transformers are very natural because say ⁓ In a way they kind of they supercharge, you know, what, you know, kind of deep sets we're already doing, but say no, deep sets are a less efficient architecture, you know, transformers do that, you know, you know this very kind of because essentially all tokens kind of can talk to each other and it's a very powerful representation.

speaker-0: Hmm. Yeah, yeah. And such a such a universal one also, so that that makes it extremely, extremely powerful. ⁓ and actually, if I understood correctly, you work on an amortized conditioning engine. It's called Ace. ⁓ and another paper that you have which is called Alin. ⁓ let's spend a bit of time on these papers, but they're from last year, so Just a bit of time and then afterwards we'll we'll go into what you're actually doing right now and your your current work. But I think these two papers and also the blog post that you have where you just you talk about like you you make the case that you can just predict the optimum. All this work is related and and that's all from last year. So if you can go first through through ACE, amortized conditioning engine, to to give to give listeners an idea of of what you've been doing last year and maybe there's something useful in there in there for them. Okay.

speaker-1: Yeah. Yeah. So like ⁓ yeah, so essentially a couple of years ago, maybe a bit yeah, two, three years ago, so we started this kind of this agenda in my group, which is ⁓ we used to call half jokingly the amortize everything ⁓ agenda. So I say we want to amortize everything. ⁓ now, you know, I c I'll kind of make a correction later to this, but at the time, you know, that's what we were we were aiming to do. And again, you know, we realized that okay, so what we just need to do is, you know, build a big transformer that can condition on things. And Essentially, yeah, the the 2025 paper, this ⁓ amortized conditioning engine. The idea there is okay, so neural processes and similarly say ⁓ prior fitted or prior data fitted networks, PFNs, which are also ⁓ transformer neural processes, ⁓ they have I'll say I mean architecturally they're essentially the same. ⁓ they are kind of minute differences. ⁓ But let's you know, essentially they enter this kind of this the same world of ⁓ what we call the transformer probabilistic models. So the idea is that yes, we have a transformer, you can condition on data points and all the methods until ours. Let's say so what they were doing is that they were conditioning on data points, and then you know you have some target points that you want to predict at, then you know they would give you a prediction as you know, a distribution ⁓ expressed in some way, a Gaussian or something more complex. But anyway, so you would predict it's a devalue of the function at at these other points. But we wanted to generalize that because say I don't always only want to predict ⁓ values of things, say or or functions. So say because say so there was at the same time, you know, there was this other kind of nearby field of amortized inference where you want to predict your data points and you want to predict ⁓ the model parameters that generated those data points. That's you know, kind of classical amortized inference. So you you train your model ⁓ to give it some data points to predict. ⁓ the parameters that generated those. And the kind of the field ⁓ you know is ⁓ is a subfield of simulation based inference in the sense that you use this data that you generated from these ⁓ simulators to train your network to you know predict the parameters from ⁓ from the data. And then you know of course at runtime you give it another data set and now you you know you predict the parameters. And for me, not not enough for me, but this is It's the same thing. Like, you know, so the whole point is that you want to condition something and predict something else. So let's say whether that's a parameter and you know, again for and for Bayesian statistics, it doesn't matter. Like that's just a a a random value that you want to predict. You know, I don't care whether you know it's ⁓ the value of a function or of a new data point or it's a latent variable of your model, like you know, a parameter. It's just, you know, you just it's a random variable that you want to predict. And so let's say so the idea was to kind put everything together ⁓ so that ⁓ the model doesn't care whether something you know is a data point, like literally a data point or ⁓ a parameter of your model or anything. So you want to conditional random variables to predict the random variables. That's kind of the mathematical idea. And and that that's pretty much it. And then you also say, so we we built it. And the idea then, you know, I mean, say you need to train the usual way, which you know you have you generate your data, ⁓ your you're for say the only thing you're doing is that, you know, again, we now when we train our network, sometimes we give it only data and we ask it to predict. parameters that generate those data. Sometimes we ask you to predict other data. Sometimes we ask you to predict both. Sometimes we condition only on data. Sometimes we condition ⁓ on data and parameters. So it says so we can say, okay, ⁓ this data were generated by these parameters. Now you predict these other ones. And so on and so forth. So it says you kinda you a mix and match of what you predict on. But again, you know, just you you have this mix of random variables. But you know again, condition on something, predict something else. That's that's kind of the core idea. And then the other core idea is that A lot of tasks and this this kind of communic back to to my origins in the sense that, you know, everything is prediction. Everything everything is is you know inference. So let's say so a lot of tasks in in machine learning are literally just conditioning on something and predicting something else. And now not all the tasks, but let's say ⁓ an overwhelming majority of tasks. So let's say so again, you know, we have a few examples in the paper. ⁓ you know, say image classification is prediction. You want to predict like a label, which is you know kind of like a ⁓ a big latent variable, you know, kind of ⁓ coarse latent variable. You in MNIST, you're predicting this latent variable, you know, let's say you zero, one, two, three, or whatever. And yeah, that's ⁓ that's just you know conditioning on the image and predicting. ⁓ for example, in painting is conditioning on part of an image to predict other parts of the image. ⁓ for example, you know, optimization is I condition on the points I've seen so far and I want to predict. ⁓ the location slash value of the optimum. And ⁓ and so on and so forth. So I say so, you so a lot of tasks is just, you know, ⁓ you just need to kind of formalize in and figure out what what you're conditioning on and what you want to predict. And that's your task. So let's say so that that's kind of the idea, which you know it goes beyond, you the say the specific example we make in the paper, but say kind of it it's broader and you know I'm I still believe in it. So that's just kind of paper number one. And ⁓ the the blog post, ⁓ just I'll talk about it briefly. So the problem with this paper, I'll be honest, let's say it was a bit of a on an issue, let's say, and we put too much stuff in the paper. Let's say that was actually it's it's kinda like a capital scene in ⁓ in academia because say people will read the Read the title, maybe maybe scan the abstract, but say, of course, obviously, but this I I mean it's in a good way, that says that yeah, yeah, nobody's going to check, you know, whatever our appendix to see what what else we did. And so that was ⁓ honestly our mistake. So I say, so there are too ideas in this paper, too many things. So I say, so for example, the blog post I wanted to get to highlight some of the stuff which kind of went, you know, a bit ⁓ lost in the message. And ⁓ which is a guy, for example, you our application. So w as a side kind of minor thing we we did, we know we had this whole section on Bayesian optimization. How overall, let's say you when you do optimization, let's say you can think of it as yes, you know, I want to condition some data to predict ⁓ the location/slash value of the optimum. And the point is if you if you put it that way, a lot of existing methods that you know, for example, you know, there are methods that ⁓ so sometimes you know you for example you have some hunch of where the locate the where the optimum might be, so you might want to put that as a prior over the location of the optimum. Now that's incredibly hard to do using Gaussian processes. So let's say so so you can define the shape of the of the EGP. ⁓ you can tell you know it's smooth, not smooth, etc. But tell telling a Gaussian process, okay, I really want the minimum to be somewhere here, good luck doing that. Like so it's it's really hard. So so there are methods that have been developed, you know, for example, to to do that specifically. Or for example, you know, suppose you know that the the optimum has value zero. For example, you you're minimizing some some kind of loss function which is defined mathematically, and you know that you know the optimum is at zero. So the value of the optimum is zero. So say so so that that should be giving giving you some information that you can ⁓ exploit. Now, again, putting that into a Gaussian process, very hard. In fact, you know, there are specific papers that solve this specific problem, right? You know, okay, so how do I tell how my GP that you know my optimum is at zero? Or how do I modify my Bayesian optimization in such a way that, you know, ⁓ the optimum I can't you know sample from this region because I believe the optimum is around this region. So once you see it probabilistically, yeah it's super easy. Like you know, yeah, you have a prior on the location of the optimum. Yeah, okay. You can just put a prior on location of the optimum. Or I can condition on my knowledge that the value of the optimum is zero. Then you know so in A's what you do you condition on ⁓ you know y star equals zero, you're conditioning on that and you're predicting and you know. So anyway, so this we do as a kind of like a a a minimum like a side quest hidden In the appendix, pretty much. And so I say that should have been a separate paper, obviously. Well, let's say, well, ⁓ that's not how we we were working at the time. So anyway, so that's why I bought I wrote a blog post because I want to kind of kind of just, you know, ⁓ so everything is about prediction. And ⁓ so we can just of course it's the the the title is a bit of a provocation in the sense that it's not that easy. So there are some caveats, but let's say, but ⁓ it's an interesting way of thinking about it. Let's say so think essentially kind of think of your problem as a prediction problem and you know. How far can you go just thinking of, okay, I really want to predict this. Can I do it? ⁓ that's not the it's not the full story, but it's you know, you can you can do a lot just just with that.

speaker-0: Hm, okay. Yeah. Yeah, and I encourage listener to to check this blog post out. It's it's really good. ⁓ you can you can skim it and get the idea pretty fast. The links will be in the show notes for this ace paper, this blog post, and also the Aline paper. ⁓ I do wanna make sure you can talk about what you're doing currently, so ⁓ maybe we'll we'll be able to talk about Aline afterwards, but first You told me while preparing for the show that your group has ⁓ several paper had several papers at ⁓ ICLR this year. ⁓ and you had a bunch of really practical things ⁓ to talk about here and you're doing very interesting things. So yeah, can you walk us through that and and tell us what ⁓ you've been up to lately and where where your mind is at?

speaker-1: Okay, yeah, thanks. Yeah, so I'll ⁓ I'll try to be to be concise. So we have time to touch about the these different points. Yeah, maybe there are a couple of papers that you know are kind of more practical ⁓ from our kind of iClear batch. One one is a this ⁓ it's called a prior guide. And the idea is suppose that you okay, you results pre-training that you can do, let's in amortized inferences, you know, you train your your neural network model. ⁓ to to do inference. So you spend it doing this kind of relatively lengthy ⁓ pre-training effectively. And then you know you deploy it and then you know at runtime, okay you give it a data set and then you know you want it to give it this data set, for example, giving you give you back the posterior distribution of the parameters that might have generated this data set. Now it's a posterior distribution and as the audience will know, obviously you say that depends on the prior and the in your data. So how do I specify the prior? ⁓ so okay, so of course I say you can specify it a pre-training, but suppose that you know you're kind of pre-training your model and you know you get some additional information later, which you know modifies your prior in the sense that, you know, or you know, let's say another situation you can think of, especially in kind of as we go towards foundation models for inference, let's say someone else pre-trained the model, okay, with a specific prior, maybe a broad prior, you know, which kind of accounts for many phenomena. And now you I want to use that model, say for something more specific. So I know that you my my prior is is is more specific. So if I use that board prior, essentially I'm kind of you know throwing away information I could use. So how can I inject this information, you know, when I use the model? Now that's that's a big question. ⁓ we try to do that also with ACE ⁓ to a degree, but say we we kind of tackle you know more aggressively in this kind of more recent paper, ⁓ prior guide. And the idea essentially we use the machinery of ⁓ diffusion. I'm not entering in the in the details, but say you had also an episode with ⁓ Jonas Aruda. So it's it's probably it's very related. So the idea is that so we use the machine of diffusion to kind of so you can use diffusion again as a method to give your distribution. So let's say so pip you know, we can use do simulator based inference ⁓ to condition on some data, and then you know you do your kind of diffusion thing, and then let's say you get a dis samples from your posterior. Good. Now the nice thing about diffusion is that you can use ⁓ guidance. ⁓ which is kind of this technique which ⁓ essentially you kind of steer the the diffusion which you know the sampling in one direction or another. And if you kind of choose your guidance in a some you know ⁓ ad hoc way based on your problem, you can effectively you know have your model do different things at one time. And the idea is in fact you just in a in a nutshell that we find a way we find a way to encode the guidance which kind of moves the the sampling in such a way that you can use a different prior at runtime. So say so you train you or someone else trains the big model ⁓ you know on a broad prior, like you know, because you the imagine got you kind of foundation like so it's very broad, many different parameters. Then you know at runtime, ⁓ yeah, I have my experiment and you know I know actually that you know the parameters tend to live in this regional space. I can put that prior ⁓ at runtime using you know our technique, for example. And ⁓ again, and it's part of a broader kind of area of research that we are really interested in, which is again this idea of yeah, test time compute. I mean, which of course is very relevant also to LMS, but say, you know, so essentially how can we add information or you know additional computation at runtime? Before I was saying, you know, we want to amortize everything. That was you know few years ago. Now I kind of, you know, there is a caveat to that. Let's say there's an addendum, let's say that yeah, so maybe we cannot amortize everything because that would mean essentially kind of pre-compress the entire universe in a single model. Maybe we cannot do that. There will always be some situation that you know you have not quite encountered before and you need to kind of spend computation at runtime to figure out what's going on. And ⁓ so essentially so yeah so we're kinda you know following the trade you know in a way the the the the steps of you know LLMs and agents in the sense that ⁓ yeah so you do some pre-training, but then you probably also want to spend some time at runtime because you know you have some information that you get only at runtime and now you You want to use it. So prior guide is just one specific instance of that. There might be, you know, well, there are surely many others that we are exploring. ⁓ you know, that's what kind of so it's part of a broader agenda. So how do we do, you know, amortized inference? But again, not everything is amortized, something might be changed at runtime, et cetera. So it's it's very exciting because there's other stuff that can be done. And ⁓ the other paper that I want to mention very briefly is this efficient auto-aggressive ⁓ you know ⁓ inference for a transformer probistic models. And actually Kind of the the the the tagline is is is kind of is very short. The idea is that okay, so I I discussed before how it's important that ⁓ all these ⁓ models ⁓ kind of preserve ⁓ this set property. ⁓ you know, so let's say so you can kind of move around your data points and you know, but let's say your the your prediction doesn't depend on the ordering of your data because it's set. Now it turns out that that's all very good and nice, but it makes really nasty effectively kind of expanding this set. So if you have got more data. ⁓ since it's a set and it's using kind of this ⁓ self-attention, which is you know kind of between all the points, every point that you add, you need to recompute everything from scratch. So there is no concept of for example, you know, KV caching, which is you know kind of a staple of modern you know transformers and modern LMs. The idea that you know you kind of store some computation you've done so far and you only need to compute the additional tokens that you learn. So the idea is that of course we could do that, but we would possibly learn ⁓ to possibly lose something. So the idea is that okay, so what if We we use this idea of a buffer in the sense that ⁓ so we keep the context you know fully you know set-based, fully self-attention. Then you know we can add kind of something which is doesn't do full attention with the entire set, which would require us to recompute everything, but effectively only does what's called causal attention, which means that each tokens kind of only watches the previous tokens. So now this is not permutation invariant anymore. So we're kind of breaking this permutation invariance. But we are doing it only for kind of these additional points that you add. So say so the idea is that you do kind of add new points very quickly and you can make predictions very quickly for new points. And then you know once you're done with that, you can kind of okay, you know, now you have time, you can put them back into your set. And now you can do you can use low update in which you you update everything. So kind of you know, so and this actually has a lot of usages because it's a because it's kind of so you can keep your context, you know, permutation invariant. Then you you can do a lot of kind of even in parallel, you literally sample, you know, thousands of points, you know, using this technique. ⁓ and then, you know, let's say you can kind of merge it back, all the information back into your context. Anyways, so it's it's maybe in in a way it's kind of more yeah, kind of architecture, more engineering in a way. Let's say there was a lot of problems that need to be solved, like ⁓ engineering wise. ⁓ but let's say, yeah, it's it's something that we are we're kind of building on because say it kind of lets us again, it's an unlock because we can do a lot of things with this now.

speaker-0: Damn, yeah. Yeah, it's we definitely need the the links to these papers in the show notes. in so what what are the the cool things you we can do now with that? Like I what are you working on right now based on these papers? I'm guessing you're trying to add some of these ⁓ possibilities to ⁓ to some packages. So yeah, what ⁓ what are you planning around that to help people use it?

speaker-1: Well, so the the the grand vision here is that we've been discussing say my group, let's say, you know, I I have the you know, I can I can show the the logs. I've been talking about you know foundation model for inference for for many, many, many years. And and ⁓ so that's something you know we've been kind of discussing for a long time and that's always been kind of the you know the north star for us, ⁓ for my group. Especially you know, building exactly this this foundation model for inference. And ⁓ in order to get there, yeah, we kind of you know need to solve ⁓ a lot of problems. ⁓ shout out of course goes to what have become foundation models for inference recently. I mean recently, past few years. all the tabular foundation models have been kind of have shown us the way how you can build ⁓ a foundation model for inference. ⁓ of course they say they apply to tabular data, but I mean a lot of things. can be put in a table. So it's a so from from my perspective, let's say they're kind of there are general purpose ⁓ foundation models for inference. Let's say even though people cannot call them tablet data for historical reasons, I guess. But let's say so you can apply them to a lot of things and you know and they've been kind of doing ⁓ you know amazing. ⁓ so we are very interested essentially in a in a in that direction, ⁓ to expand in that direction. And there are a lot of things you can do. Let's say but you say so of course it's a ⁓ there is a lot of New ideas that can be tried in the space. And luckily these models are still fairly small, let's say, compared to LLMs. So they're still nicely within the reach of what we can do again, you know, within academia. So we don't need to kind of spend millions or even hundreds of millions or billions to get ⁓ training done. So let's say so so that's quite exciting because we can still kind of contribute. And ⁓ for example, yeah, just you know, to mention the buffer that I just mentioned, yeah, can be applied to Transformer, neural processes, etc., but also to tablet foundation models. So let's say so ⁓ one particular thing that you can do with this kind of fast prediction that doesn't require you to recompute everything, ⁓ is unrolls, which are very useful for if you're doing reinforcement learning. So if you kind of want to do you know fast predictions to see what happens, you know, as you add more data points and you want to do you know RL on this, ⁓ the what we have essentially would be built ⁓ allows us to make it like hundred times faster or something like that. So it's ⁓ no, we know with with much, you know, with a hundred times here fewer memories. Like it's it's like yeah it's it's super ⁓ useful. So let's say so that's something we're going to use definitely. I mean also others can can use. And ⁓ just you know just mention a direction. So a lot of the things we are doing are part of a kind of a grand plan. And you we're kind of building one step at a time. But let's say there is there is a there is an arc where we're we're going towards

speaker-0: Yeah, actually you wanna talk about that long arc or do you wanna talk a bit more about Allen, whatever you want, because I I do wanna be mindful of your time and and start playing us out here.

speaker-1: Yeah, yeah, thanks. So I'll ⁓ I'll give you maybe ⁓ I think ⁓ a line ⁓ Aline doesn't doesn't take a lot of time and actually it's a good segue to kind of discuss more on the ⁓ foundation model direction. So the idea of Alin ⁓ it was to ⁓ take what we did with A's. So say you can predict things, ⁓ you know conditional things, you can predict things, and now we add the fact okay, we also want a model that does active data acquisition, which means you know Because ⁓ ACE is passive, it's just doing Bayesian inference. You just feed it the data, whatever data you have, you feed it whatever information you have, and then you know that's predictions. Now we also want to collect data, ⁓ you know, do active learning or experimental design, which means essentially you need to select which points in the space. And again, your Bayesian optimization is a form of active learning, right? You're you're sampling points in the space so that you know you want to find the minimum. So That's the that's what we want to do. ⁓ we had some work before ⁓ with Dao Lang, and this is done of course work done ⁓ with ⁓ particularly with ⁓ Sami Kaski at the University of ⁓ Alto University. And ⁓ so Dao Lang is a Ph student shared between Sami and me. And ⁓ the idea is okay, so let's use effective reinforcement learning to train ⁓ another head on top of let's say an ACE like model that Instead of predicting just the value, you know, say p data predictions, it works as a policy head and tells us which points to to collect. And ⁓ what's the what's the goal here? Well, ⁓ the goal here, for example, can be our ability to predict other data points. And and this is kind of ⁓ kind of self-reinforcing in the sense that you have one head that you know makes predictions, then the other head that you know kind of compute the policy that lets you predict better. So let's say so these two lost functions in a way kind of work hand in hand. And ⁓ so we have a single network that does both. And of course it's con it's both convenient in the sense that okay so now you have a single network that you know essentially you can kind of use it for data collection and then you can also make predictions with it. Nice. But I say I think it goes kind of deeper than that because now we have a single network which is learning to do this task. And these kind of tasks help each other in the sense that the prediction head is actually used as a target for the policy head instead that you know use the policy head to want to improve the prediction, to make you know, for example, you more precise, improve the ⁓ you know, the proof the log density, et cetera. So, you know, again, it fits ⁓ yeah, there's this nice ⁓ synergy between the heads. That that's the idea. Then you know, this trained via, you know, reinforcement learning and so on so forth. So then you there are also kind of technical aspects, but it's you know, kind of the the the high level idea is is this one. And this brings me to the kind of foundation model idea. Because if you look at many say the modern foundation models, like you know, again, like in this and again in in this field, ⁓ you can think of tabular foundation models because those are kind of you know ⁓ what occupied space. Of course, you know, there are there are others. the point is you know that they do prediction. So you just you know fill in more points, et cetera. But I say so yeah, the division I have for a foundation model is is more than that. It's ⁓ so of course it's ideas a core of prediction and inference. Of course, ⁓ you know, as we said, it's very important. It's you know as the base and the the basis, well the base of everything. And ⁓ sorry, I had to do that. But the the point is we we want to do tasks with it, right? So, you know, say ⁓ you want to be able to do for example, you again based optimization is an example, ⁓ experimental design is an example. Again, the goal is not similar. So it's not the same, but you it's similar. ⁓ so you want to collect data so that you know. ⁓ you want to improve your predictions for you know for certain certain parts of your space, for the parameters. ⁓ you know, so you might have a task. So say so and the point to me, let's say that that's these these goals are not kind of, you know, an appendix to the model. But say to to in the I see that as part of your foundation model. So their foundation model say ⁓ you know has ⁓ kind of an array of tasks that is trained to do. And the point is, I mean, again, ⁓ the and this is where I think let's say the we we will benefit from the magic of of meta learning in the sense that the the the model, you know, again, ⁓ if it's big enough, is it's learning all these kind of representations to do all of these tasks. And the point you know is kind of using them, right? You know, so so it's learning to to generalize between different or at least interpolate between different tasks. And then you know, of course it's yeah, I I kind of spilled ⁓ the beans in the sense that yeah so of course you you hope that you know then at some point you start seeing some degree of generalization. And you know so essentially as you do more and more tasks, ⁓ not just prediction but other tasks to say, you know ⁓ Yeah, the y y y you get something more ⁓ and again, you we we have glimpses of that with ⁓ you know, with again with agents and Rams. Let's say clearly, of course, yes, you know, they again it's very jaggedy, as we know, et cetera. But still, let's say, yeah, I mean, they're definitely not just literally doing what they're training on, but say there's a bit of ⁓ you know, jagged generalization and maybe limited, but still, you know. I mean I'm I'm pretty sure especially a lot of the problems that I use them for, you know, they've not been trained exactly on that. So anyway, so let's say so. ⁓ again, you know, can we can we do something similar? Would that even you know, what would that even look like? Like in other things, it's not obvious, right? ⁓ so I say there are many, many, many open questions here and I think let's say, but that's anyways, that's what I see as the big direction. So eventually again, the dream is this kind of big oracle foundation model, which you know, you you know, you can't and you know, and LLMs might be a part of it. ⁓ I know that you asked, you know, I discussed it with with this Stefan and ⁓ I think maybe we have a slightly different ⁓ version of that. Of course they say yes, ⁓ I I totally agree with you know ⁓ agents that you know, like modern agents that just use kind of Bayesian tools as, you know, let's say as tools. And that's that's definitely let's say I I see that totally happening. I also say I I think we can also push on the concept of big models for inference that just you know that just are focused on doing inference and learning to do inference and you know, and they're good. are doing inference. And then well, maybe these models can be then called by agents. So the the the two things are not ⁓ kind of opposed.

speaker-0: Yeah, yeah, exactly. I think we we we spoke about that at length with ⁓ Stefan during two episodes, so definitely recommend listeners. ⁓ listen to these ones. The one with Jonas Aruda also was a bit more focused on diffusion models and he also ⁓ did a live ⁓ a live demo also. ⁓ definitely recommend that one too. And yeah, in general really agree with with what you just said here and I think it's a very important and interesting part of your research, ⁓ not only yours but like your your whole group where ⁓ you're that's really interesting to see how the NLMs and Asian part or merging with the amortized inference part and the and the more Bayesian models side of things. And actually of course you've been working a bit about that right now. You're also doing some work on LLMs and agents. So yeah, what I'm I'm curious what you're mainly doing right now about that and and what was the question, you know, that pulled you in and how do you see it connecting back to amortization and probabilistic conditioning?

speaker-1: Yeah, so the the so the work strictly on LMs and and agents, let's say still in stealth, I would say, in the sense that we haven't published ⁓ so to say. So we are yeah, we are we're working and that's let's say that's that's ongoing. And ⁓ although let's say so we it's an idea that you know, some some of these ideas have been kind of going back and forth, ⁓ but let's say so the angle that we were exploring at the moment ⁓ is related to there was recently a a position paper. ⁓ this idea say the idea I've been working on, say we've been working on it say come earlier, but of it's say, you it's in the the zeitgeist. And ⁓ so I say so there was this ⁓ kind of large position paper recently that says, you know, we should think of agents as, you know, say in the context of ⁓ Bayesian decision theory. And ⁓ and more than ever effectively we need Bayesian ⁓ decision theory. because you know essentially that's what's happening, it's not they're making kind of you know small decisions ⁓ all the time, constantly. And so I say so that that's kind of was my my The hook for me was kind of start to play around with ⁓ s local LMs, ⁓ small agents, ⁓ because they say I'm interested in ⁓ resource constraints. We haven't talked about that at all, but it's a you know, one line of research in computational neuroscience has to do with ⁓ such an inference under resource constraints and we have some work on that in more on the neuroscience camp, ⁓ by anyways. But it's a so I think that's very interesting because say that there is this idea that intelligence kind of you know arises under ⁓ boundedness of resources. So let's say so you need to, you know, the idea that so the fact that we need to compress memories actually you know makes us kind of generalize because we need to kind of compress things to then generalise. Anyways, so I I think the the whole idea of yeah constrained ⁓ resource constraint system is very interesting in itself for for a whole lot of reasons. And so let's say so for these reasons the local LLMs or you know small LMs or tiny LMs even are very interesting. So you want to them to be able to do things. But of course, you know, they're very limited. So I say, so how do you make them more usable, more reliable? And well, it turns out that you know I have a lot of work on sample efficiency, etc. So essentially, so I've been kind of combining these these ideas from sample efficiency in the kind of small, I mean L LM world, but I say, of course, the application mostly would be for the small LMs, also because they They're local, you can play around with a lot of stuff with ⁓ the way in which you know you sample tokens from from this. ⁓ and you know, to to be concrete, for example, you know, if you think about ⁓ what small LMS do or small agents do, for example, you know, when a small LM in general, also agents in general, you know, decide to call a tool effectively and kind of needs to emit a token, you know, ⁓ okay, I'm I'm calling a tool, and you know, there's a specific token for that that changes de depending on the on the ⁓ on the family. And know, and then you know say so okay, now I'm calling a tool or I'm not calling a tool. And you know, if I'm calling a tool, okay, you know, then you if you you for example, tool name, ⁓ you know, it's it's another kind of token, or maybe ⁓ a few tokens. And you c you can think of that or you know as kind of small choices or small decisions that are made. And they're probabilistic in the sense that, well, first of all, they they they're well, they're stochastic because LM tends to be stochastic, you know, unless you p put the temperature to zero. Also they're probabilistic in the sense that you they they based they're based on limited information. So you know, you can think of you know, effectively you can think you can build a posterior over this and you know and work with that and do kind of the old Bayesian ⁓ stuff. And so anyway, so it's ⁓ that's kind of the the things that ⁓ we've been ⁓ exploring.

speaker-0: Okay. Very, very interesting, very cool. ⁓ definitely sounds like ⁓ worthy of another episode when you guys have that out. ⁓ yeah, so as listeners probably know I've been also working myself on some on some ⁓ Asian stuff. ⁓ I'll refer you to at least episode one fifty seven and one fifty eight with Stefan. ⁓ but also you can check out the blog post section of the website because I've also worked on a Bayesian workflow skill and a causal inference scale. The one with Stefan was really about the the amortized workflow skill. And yeah, ⁓ I mean that's also why I I love talking about you guys do on the show because it's also something I'm I work ⁓ on myself and I think has a lot of potential to ⁓ help people in the future and make sure the good practices that come from latest research actually ⁓ gets distilled into the general public. ⁓ so so yeah, thanks a lot for for doing that, Luigi. And actually I'm also curious, you know, to finish on a practical note for an LBS listener who currently lives in Stan or PineC or an Empiro, where should they actually start with the tools we've mentioned today? So mainly Pipe V BMC, Pipe ads, base flow, what's the smallest interesting problem to try first, you would say?

speaker-1: Yeah, that's that's interesting. So yeah, these ⁓ essentially yeah the tools we we we discussed and then ⁓ yeah base flow essentially which you know which kind of like a general purpose amortized inference ⁓ tool kind of apply it to ⁓ mildly different situations and we've been trying to kind of combine them in the sense that you know w with our ⁓ amortized Bayesian workflow paper, ⁓ again with ⁓ with Stefan ⁓ and others and it was ⁓ led one yeah by by my now graduated student ⁓ Chen Kun Li. And so ideally you want to start actually with amortized methods because say they're especially if you have many data sets to to analyze, you know, that that you can have a a very strong, you know, win there. ⁓ because say, you know, if you have many data set to analyze for the same model, then you would need to r run you know MCMC or you know any method from scratch and that's that can be very expensive. Let's say so amortization would pay off. So let's say so if you in that case, let's say you should consider ⁓ using amortization. And then you know, for example, I mean, of course, it's a bit of a plug-in, but ⁓ we have these ⁓ these these ideas for how essentially you know you can then check whether your amortized methods ⁓ you know, say your amortization has worked, and or you know, you can trust it. And the point is, of course, if you don't trust it, then you know, for those specific data set that you don't trust, you those posteriors that you don't trust, then you know you can do kind of MCMC. And you know, in a way which by the way is supported by amortization. So you can use amortization in in some way. So that's that's essentially kind of ⁓ a natural way for, for example, someone who does MTMC and does not trust amortized methods. And fair enough, you know, let's say I I I I understand, you know, where this kind of ⁓ mistrust can come from, in the sense that of course MCMC has built a lot of diagnostics over the years. And you know, it took many years to come in to the modern diagnostics, right? So let's say so that's ⁓ so you know that's something you know we we we in the community ⁓ are are are building also for amortized inference, right? So say so because again, ⁓ understandably, in order to be use a method, you need to know when the method failed. It's probably one of the most important things ⁓ to to know ⁓ with a method ⁓ as a practitioner. And so I say so yeah, the ⁓ so yeah, many data sets, then you know you should definitely consider amortization, ⁓ because that might really kind of speed up ⁓ your workflow and you know let's say yeah if you look at our our tricks our suggestions then you know you can still you know kind of trust the the whole process because say you in worst case scenario you kind of go back to MCMC for those specific problematic data sets. That's one thing. Then you know there is the other angle which is if your model is as we discussed say expensive in the sense that you know each evaluation of your ⁓ log density of your model ⁓ takes time or we know has a cost of some in some sense you it typically the cost is time in computation then you know then is you're in the case in which you might want to look at ⁓ for example ⁓ Py BMC and to be honest I I'd love to connect ⁓ Py BMC more to the PyMC and ⁓ base flow ⁓ environments so that you know they could kind of speak more to each other and that's something that you know we've been discussed discussing but say yeah we haven't managed to do yet. But now you know with the agents, you know Maybe we can kind of speed that up. And ⁓ so anyway, so if you have something expensive, etc., then you know definitely let's say ⁓ you might want to consider methods that kind of do efficient Bayesian inference in the sense that they can get you to an answer ⁓ with a relatively small number of evaluations of the log density, like in again of the order of you know, maybe up to you know a few hundreds. And that that can be you know, of course, ⁓ kind of very useful. And finally I would say, know. But of course, you know, that's not something necessarily I would say for the in a in a Bayesian statistic podcast, but let's say, you know, so there are people who don't care about no say for the number of reasons, let's say they might be okay with a point estimate. And of course, there are reasons to find a point estimate because, for example, you know, ⁓ even if you're doing ⁓ you know, if you want the full posterior, ⁓ you know, doing like a a a rough optimization, maybe you know, you don't want to start you know exactly from the map for a number of reasons, especially if you're in you know moderately high dimension. But if you're in a relatively low dimension, you know, there's nothing particularly bad in ⁓ in starting you know near the map, like the the maximum a posteriori or or mode of your posterior. ⁓ and in typical, you know, actually, you know, it's it's a decent starting point. ⁓ again in low dimension, in high dimension, things are very different, so you shouldn't do it. But let's say in ⁓ in low dimension, let's say so again, you know, how do you start from the mode? I mean, you can run optimization. And in fact, let's say that's for example, how many practitioners use. kind of pie bars in combination with PyBMC. So first they run optimization such that they would kind of find you know a place near the optimum, which would be the mode, and then start PyBMC from the mode. And that actually this works well because as we discussed before, you know, typically the posterior is kind of nice near the mode and might be quite nasty elsewhere. So I say so if you were to start PyGBMC or even MCMC. ⁓ from you know from farther away in details of your posterior, things you know can be quite nasty and bad, you know, and you'll say have kind of numerical all sorts of numerical issues. But you know, if you kind of start ⁓ near the mode, things are nice. So anyway, so say that's how kind of you you can kind of this kind of this nice synergy again within PyBuds PyUMC.

speaker-0: Hmm. Yes, definitely. ⁓ and I know also some work has been ⁓ done lately, especially by ⁓ Alex Fengler, who'll be soon on the show. I'm working on that. making a bridge between Pipe C and Baseflow. And so yeah, like ⁓ definitely that resonates with ⁓ what you've you've just mentioned. ⁓ fantastic. Luigi, ⁓ any any other topics you you would like to discuss?

speaker-1: One thing I say, but you know, I want to kind of state again is that ⁓ the the motivation here is always practical in the sense that we want our tools to be used, right? So we are not doing I mean as much as you the trade aspects is very interesting. And but let's say so we want to build tools. So say, yeah, so if you you know if you kind of need to categorize what we do in our lab, I think you we are we are tool builders. ⁓ Anyway, we build even you complicated tools, but let's say but the idea is that eventually we want people to to use what we're doing. In the case of for example, you know, some of our methods, say it's kind of taking a bit longer because say we are kind of building ⁓ towards the the the releases in a way of you know things that we can actually recommend. ⁓ but yeah, we we will you know I I'm very confident you we we'll get there. ⁓ but say I so the feedback from you know users, et cetera, it's actually super important for us. Like, you know, what are the interesting problems? And you know, it's all you know, also. You know, if if ⁓ if a method fails, let's say we are actually very happy to hear ⁓ and yeah, been in touch, you know, with many users, like, you know, to know ⁓ what works and what what doesn't. And you know, and it's it's kind of keeps research very healthy because it's a okay, these are the problems, you know, ⁓ that you know people care about.

speaker-0: Yeah. Yeah, ⁓ definitely resonate with that as as everybody knows, ⁓ and that's also why why this show is here. So thank you so much, Luigi. That was that was really great. ⁓ and same, I could I could do a a six hour ⁓ ep stop with you. But I feel that would be much better to do it in person. So let's ⁓ let's postpone that ⁓ until ⁓ I visit House in Key. Yeah. in in the meantime, I'm gonna ask you the last two questions. I asked every guest at the end of the show.

speaker-1: I look forward to that.

speaker-0: So first one, if you had unlimited time and resources, which problem Sorry, ⁓ problem with my with my mic? I'm telling you to the keys a bit ⁓ okay, so ⁓ let's do that again. ⁓ In the meantime, I'm gonna ask you the last two questions I ask every guest at the end of the show. First one, if you had unlimited time and resources, which problem would you try to solve?

speaker-1: Yeah, I had to ⁓ Well, I mean, okay, so I can give you the kind of gut answer, which is if you look at the the title of of you know well the name of my of my group is called you know the the Machine and Human Intelligence. And ⁓ it's not just a a fancy name in the sense that, you know, the underlying kind of you know, the Feel Rouge, the connection ⁓ is that I'm interested overall in yeah how intelligence works and ⁓ and not necessarily the human intelligence that's the point right you it's machine and human or you know say so and even an animal so let's say so for it's kind of all the same in the sense that it's some sort of ⁓ general ⁓ concept underlying you know say how how do intelligent agents work, how does that arise? And of course let's say the the the work I've been doing let's say it's kind of very takes a a specific angle which is intelligence has to do with inference. It's ⁓ Of course, say it's kind of it's a very opinionated take on what intelligence is. And of course it doesn't end there, but it's a you know there is a you can you can say it covers a lot. And some people say it's actually everything there is. ⁓ there are you know ⁓ active inference camp, etc. But anyway, that's all another discussion. And ⁓ so anyway, so yeah, so big question is okay, what what's intelligence? How does intelligence arise? So what makes you know a brain, you know, work intelligently and know, an like an agent, you know, and then people yes, people are you know asking, are you know, modern as of twenty twenty six agents, ⁓ intelligent. Some people will, you know, yeah, scream and say, no, that's not intelligence. ⁓ that's, you know, whatever. And that's, you know, just talking predictions, which, you know and ⁓ other people will say, Yeah, of course, you I I see some elements of intelligence, et cetera. So anyways, so understanding what's what's the secret of of intelligent, I they said that's you know, that's one of the most interesting questions.

speaker-0: Hm. Yeah, yeah, definitely agree. ⁓ that would and that would ⁓ solve a lot of problem of problems also, you know, as a as a trickle down effect. So ⁓ so definitely agree this is ⁓ this is a good one, ⁓ because it's not it's not a contained ⁓ solution. So ⁓ very interesting. and second question if you could have dinner with any great scientific mind, dead, alive or fictional, who'd it be?

speaker-1: Yeah, that was a that's a hard one, of course. And so again, I I didn't give it too much thought because otherwise kind of cheating. And ⁓ I would say ⁓ Johnny von Neumann and as ⁓ mostly because I'll be interested kind of, know ⁓ Piccoli's brain, which is of course ⁓ famously very ⁓ very big and ⁓ a smart person and like you know what what the things about yeah the modern you know developments, you know. in artificial intelligence, etc., I think he'd be very fascinated by it. So I yeah, I don't claim I would be able to candle much at any in any way this inter but I would definitely kind of benefit from, you know, kinda okay, ⁓ you know, given the insight that we have just by being born in in this area and having seen this explosion of artificial intelligence. ⁓ okay, so what what what does you know, what would he make of it? And that I would be very interested in that.

speaker-0: Yeah. Yeah. ⁓ for sure and and for sure a dinner where there is a lot of things I would not understand. Let me know, ⁓ let me know and I'll I'll come. I'll I will for sure learn a lot. awesome. So please make sure to add the the latest papers you've mentioned ⁓ in the show notes because I have the ones from last year, but not the ones you've been working on this year, so ⁓ add that to the show notes I'm sure.

speaker-1: So

speaker-0: People will ⁓ will look into that. So yes listeners, make sure to look at the show if you wanna dig deeper. And Buigi, thank you again for taking the time and being on this show.

speaker-1: Thank you, it's been a pleasure.

speaker-0: This has been another episode of Learning Basion Statistics. Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbastats.com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's LearnBaseTats.com. Our theme music is GoodBayan by Beba Brinkman. Fit MCLas and Megaran. Check out his awesome work at bebabrinkman.com. I'm your host. Alexandora. You can follow me on Twitter at Alex underscore andora like the country. You can support the show and unlock exclusive benefits by visiting patreon.com slash learnbase tax. Thank you so much for listening and for your support. You're truly a good baby and change your predictions after taking information. And if you're thinking I'll be less than amazing, less adjust those expectations. Let me show you how to be a good daisy. Change calculations after taking fresh data. Those predictions that your brain is making. Let's get them on a solid foundation.