speaker-0: We are working on this little tool that ⁓ we call it Bayesify. ⁓ it's a tool where you can go and upload a research paper, also your own, obviously, and get a structured report on strength and weaknesses stepwise on how you applied the ⁓ Bayesian workflow, right? W including then hints at how to how to improve how to improve the paper. Right. If it's your own, this can serve as a review engine. ⁓ to drive you to a good point. If it's ⁓ other historical papers, it serves as like a educational resource and later downstream possibly as a as a well of ⁓ data through which we can do some historical forensic analysis also on let's say the quality right of ⁓ Bayesian analysis over time. ⁓ so I could go here, right, load a paper. It will take like ⁓ a minute or two to to run this.

speaker-1: you

speaker-0: So I can just upload the paper here, right? But I also have it ⁓ preloaded on this on this tab. So this is like my own paper now, right? ⁓ Like Yude Approximation Networks for Fast Inference of Simulation Models and Cognitive Neuroscience, right? So what we'll get is this little report here. So what is the report about, right? ⁓ first it has a relevance gate, whether the the the paper is even you know in the realm of what makes sense to analyze through Bayesian workflows. Then we have a rubric here. So we have ⁓ we allow anchoring the analysis on very specific workflow papers. There's for example a famous ⁓ paper from Gellmann at all, a Bayesian workflow, right? And we have a few options and then we also have our we call it now the gold standard ⁓ synthesis, which is a synthesis of ⁓ a few such candidate papers, right, that is then projected into ⁓ a synthesis version of the stepwise approach and we'll always link to ⁓ any particular step to the sources that were used to define the step. Right. We we'll see in one second. And it ⁓ it classifies the paper type. So in my case it's method development. So this is not just an empirical data analysis. So it will then check which steps of the workflow are useful. Nine remain of an original eleven and it will give you a score on how many of them sorry, in this case it's ⁓ seven out of nine. Sorry. ⁓ and it will give you an overarching Bayesify score, right? So far it's been harsh on any of us who tried it. ⁓ have not seen a very high score. ⁓ did you try it on the actual Bayesian workflow paper by Gellman? It's a well calibration will be defined. That paper will be defined as method development. So we should you're right, we should do we should do that. ⁓ but since it's defining the framework, you couldn't necessarily ask it to follow itself the framework. That's ⁓ that's a recursive approach. but yeah, we are we are still working in generally speaking. ⁓ the goal here is to have ⁓ a what we'll call a gold set of like human rated papers and ⁓ test the engine for how well it's calibrated to real expert judgment of ⁓ papers. So we're currently in the process of collecting such a gold set so that we can check the scoring mechanism here against what experts in the field would have actually said. So you can get you'll get yourself this ⁓ report, right? Many s many different steps here, how well they are applied, ⁓ whether or not you were adequate. You can see I was adequate only on a few of them. ⁓ I have a couple of them missing in my paper. ⁓ in hindsight, lucky that I got it published. And then you'll You'll get suggested fixes. ⁓ it gives you the sources of evidence that were used to judge the step, right? And it will give you the the standards that were let me pull this in. The standards that were applied. Sorry, where is it? Here. ⁓ so this particular step, right, posterior predictive, ⁓ retrodictive checks, ⁓ is based around the scoring mechanism is based around these two papers the visualization invasion workflow ⁓ and the Gelman Bayesian workflow paper. So that's a tool we're we're working on. ⁓ it's very close to it's actually already reachable now. We have Basify.org. ⁓ but we're still making refinements to this to make sure that ⁓ eventually we can ⁓ properly claim that it's ⁓ solid and it also tracks ⁓ in a calibration sense real expert judgments.