Jarie Bolander: Hey everyone, welcome to the OT crowd. My name's Jarie. That's David, and that's Andrew. And today we are actually gonna do a demo, a live demo, which I think I'm a little nervous about considering I don't know, live demos always are a bad idea. you can probably hear in the background some sort of like fan going, right? And that's actually our demo system, which I'm about to put on the screen in a second, but What we wanted to do and w why we wanted to do this is because this demo is actually based on the OPAS standard for high availability, which I'm gonna have both Andrew and David talk about since I don't know much about it other than high availability seems like a good thing. So what I'm gonna do is I'm gonna share my screen. For those of you that are just listening, you'll just have to kind of bear with us. So we can walk you through what you're seeing on the screen. So, Andrew: Theatre of the mind. Jarie Bolander: Theater of the mind, exactly. So David DeBari: Ha ha ha. Jarie Bolander: yeah, why don't you Andrew, quickly Andrew: Yeah. Jarie Bolander: why don't you talk a little bit about H A, what it is, and then I'll kinda go through what people are seeing right now. Andrew: Sure. Yeah, so H A means high availability. it's a little bit different to redundancy, where redundancy is just means you have more than one device trying to keep uptime, but high availability is specifically about the uptime that you're trying to achieve. so it can be could be one, could be two, could be n number of control run times trying to keep the system running in case of failure. there's a lot of different ways to do redundancy and high availability through different kinds of network architectures, different control architectures. you can even go all the way down to redundant sensors. so when you talk about high availability and redundancy, you have to be sort of specific about what you want. but in this case, we are showing high. availability of the control runtime. Now OPAS specifies recently added to the spec some information that we felt was needed for the OPCUA redundancy capabilities high availability capabilities because that while opcua does specify it they did not It was not quite complete enough for us for in the forum to make the final call to say, yeah, that's that's gonna work or not. But there's too many open gaps that different implementations from different vendors could have ended up incompatible. So we decided to specify it a little bit further. those changes to the spec are currently informative, part of part four. I think they're released as part of part four right now. And if they're not, they'll be out soon. And the difference between informative and normative is that an informative is just informational. There's no tests for certification for those parts of the spec yet. later when there is testing and certification to those specific profiles, that would then make them normative. then you can get your little opass sticker for the high availability. I I'm not sure how the profile's called right now, if it's high availability or redundancy. Jarie Bolander: Cool, cool, cool, cool. And so David, like why is this important? David DeBari: Well in the process industry, some of the processes you you can't tolerate downtime from failed equipment, whether that be the processor, the control logic itself, or or any of the other equipment. So redundancy has been a part of automation systems for quite a while, but through the work with the OPA forum and with end users, what we realized is we can make it even better. You know, redundancy as Andrew said is usually based on some very specific hardware software capabilities that are vendor specific a lot of times. we tend to call it symmetrical in that if you have a device you have to have the same exact device as its Andrew: Mm. David DeBari: peer to be redundant. when you start getting into high availability, you mix in some software capabilities that modern IT folks have been working on for many, many years and proving. We see it in business systems all the time. and we have we have made it very capable in the OT world. And so this this idea of having high availability is something better than redundancy. It's redundancy plus, redundancy on s on software steroids, I like to say. And you start seeing what is possible, this is this is better than the redundancy model. And what we have to show today actually is very interesting, given that it is capable of getting beyond the one to one and to the one to N, as Andrew had said as well. so if you're in a process industry where the you know, the the chemistry of the energy, the the you know the dangerous materials that you're using require it. This is a mandatory piece. If you're gonna make an automation system, it has to have that redundancy or what you really wish for is a high availability capability. Jarie Bolander: Totally, yeah, totally. You know, I mean as we when we talk to people about this, it's actually one of the things that gets them excited because of what you David DeBari: Yeah. Jarie Bolander: just mentioned about being able to have less downtime and less ability to like do these sort of soft D C Ns and like yeah. David DeBari: It it seems to be a ticket to entry these days as well. Even for people who haven't traditionally used redundancy, there are some things and and we've realized that there are life cycle events that aren't failure, but also could benefit from high availability, things like patching and replacement of devices. And and so the OPA you know, sort of preaches this idea you can change devices and that's a good thing. well if you're gonna change the device you have the potential for downtime and nobody likes downtime. unplanned downtime Jarie Bolander: No, no one does. No one does it all. David DeBari: and a failure is really is bad and and and unpl and planned downtime is hard as well, especially given the you know, economics of the world today. Being down and not making profits a bad place to be. Jarie Bolander: Yeah, yeah. So I mean clearly the business case is there and and I do think as you mentioned before, which I think is very powerful and obviously the reason why we chose open for our architecture or, you know, our product is because you can kind of mix and match this hardware and software in David DeBari: Right, right. Jarie Bolander: some cases. And so j just so everyone knows kind of what they're looking at, so this thing to the left here is what we call our Copa five hundred system. This thing right here is a simulation of a process. You see that little ping pong ball? We're trying to keep that ping pong ball kind of like in the mid, you know, at the set point, which if you can see the set points, you know, right over here David DeBari: Fifty. Jarie Bolander: at 50. you'll notice the little on this HMI, the ball's jumping around and the ball levels at around 50. Hey, how cool is that? the thing that's interesting about this whole system, right, is that this is all. running on different hardware. So this is the ACP right here running on a super micro. These two things right here are DCN one and DCN two and you can kind of see that from the HA status, right? The leader and the follower, DCN1, DCN two. These are both ASROC devices. Down here, this is the Phoenix IO and that's actually controlling the little ball thing with the fan and all that sort of stuff. over here we have a a Cisco switch. And then we also have some net gear switches down here. And the reason we needed that is because we had ran out of ports here. So we're gonna definitely need to add some more ports there. And then this thing right here is the power supply, Phoenix power supply. So very kind of simple, obviously, system, you know, with some really so yeah, simple IO and and this thing. Yep. David DeBari: basic automation system. We got control, we got I.O. We've got some you know the power supply and the and the network and we've got the server based systems like your your your HMI and your logic engineering capabilities Jarie Bolander: Yeah, and and just so what what and what you guys are actually looking at is I'm logged into what we call our engineering workstation. So literally on this thing right here, and I've got a live stream. So there's a camera that's pointing at this and and in a second you'll see me go in inside and mess around with this. And so everything is now I'm logged in running, you know, securely on this, so that we can do this demo. And again, like what's really cool about this is It is running all sorts of different other companies software and hardware. Of course, you know, we've got Codasys running the ring this this loop, the control David DeBari: Control logic, yeah. Jarie Bolander: logic. We've got ignition from inductive automation that's the HMI. And so this really does show the power of everything is working together and and interoperable when applied, when a part of the OPAS standard. So What I'm gonna do is I'm gonna go to the other room. It's literally right over there. Actually, I had to shut the door because it's pretty noisy. And what I'm gonna do is I'm gonna mess around with this little valve right here. You'll see the ball go up and down. I'll come back in to make it stabilize. And then what I'm gonna do is simulate, and whether or not this is 100% real or not, this is like the best way for us to do it. I'm gonna simulate this DCN1 going down. So I'm literally gonna pull those network cables, like pull it out, and What you should see and and look over here at this HA status is that you'll see the leader and the follower switch. So I'm gonna go on mute because it's gonna get a little bit noisy and I'll have my other colleagues talk about kind of what's going on, but I'm gonna go do that right now, so it'll be right. Andrew: Yeah, like what Jarie said, he's gonna go turn that little valve on top and that's gonna introduce a disturbance to the system. and the David DeBari: Yeah, airflow change. Andrew: yeah. So then the P ID should correct for it. depending on how that P ID is tuned, maybe faster or slower, but David DeBari: It'll start trying. Andrew: Yeah. David DeBari: I think he usually opens it back up before it's gotten too carried away. Jarie Bolander: Did it work? Did it do it came back? Andrew: I didn't see you move it. David DeBari: What did you do? You didn't do anything. Jarie Bolander: No, I haven't moved it yet. No, I just I just did I just did the little thing back and forth. The ball went up and down, right? I hope. Recovered. All right. David DeBari: No, we didn't see it. We didn't Andrew: I didn't David DeBari: see it. Andrew: s I didn't see you touch the valve, no. Jarie Bolander: Really? David DeBari: We didn't see you in the picture. Andrew: Yeah. Jarie Bolander: no. okay. Something's wrong. there it goes back. Okay, let me Alright. This okay, let me go back again. Okay, I think what happens is it okay, David DeBari: Yeah, the the ball hasn't been moving around much, so maybe it was Okay. Jarie Bolander: as with any good demo, it it locked up. David DeBari: Camera froze. Jarie Bolander: looks like can you see it? Yeah, okay. I'm gonna go do it again. Hold on. I'm gonna Andrew: All right. David DeBari: Okay. Jarie Bolander: go on go on mute. Hold on. Andrew: Alright, I still see the ball jiggling up and down so I don't think it's frozen. There it goes. There it goes. Actually that's tuned much better. David DeBari: I think when he goes on mute he's freezing. No there he is. There's his hand. So we turned the valve closed. The ball dropped. Yeah, yeah. They've been working on it. So Andrew: Yeah. David DeBari: they Andrew: That's a much better team. David DeBari: Yeah, so we closed the valve, the valve ball dropped because the airflow was restricted. Opened it back up, it it popped it did, and we saw you too. Yeah. Jarie Bolander: It worked? Did it work? Cool. Andrew: Yeah. We saw it we saw it that time. Jarie Bolander: All right, awesome. So this is what happens when you do a live demo streaming on a podcast. David DeBari: That's all right. That's okay. Andrew: Yeah. Jarie Bolander: yeah, it looks like the the video camera is a little bit thing of demos. Right. David DeBari: Murphy's Law of live demos, yeah. Jarie Bolander: So now I'm gonna pull DCN one, which is these this right here. So you'll you'll be that I'll be right back. David DeBari: All right. Andrew: Alright, and if you watch on the right hand side screen where it says HA status leader and follower, leader will become DCN two, at least I think. Yep, so he's disconnecting one, so David DeBari: Yep, and the follower will be blank. Andrew: yep. David DeBari: None. 'Cause it's off. There is no follower. Jarie Bolander: Did it work? A leader. Andrew: Yes, sir. It's fine. David DeBari: Did. DCN two is now the leader 'cause it's the only one connected. Jarie Bolander: Awesome. So now what I'm gonna do is I'm gonna unplug this D C N two, just so you know there's no shenanigans going on. David DeBari: Yeah, there will be no control at that point. We'll see what happens. Andrew: Yep, no hidden control coming from the Phoenix IO David DeBari: No. Andrew: or inside the box. David DeBari: The I.O.'s not sell set to foul last either. That's it. Andrew: And it's not set to hold last. Yeah. David DeBari: Fan stopped, ball dropped. Have a nice day. Jarie Bolander: All right. Andrew: Looks good, Jarie. Jarie Bolander: Cool. No one's in control. David DeBari: No one's in control, as we thought. Jarie Bolander: Now I'm gonna plug in D C N one again and you'll see it come back. Andrew: cameras froze again. I think you're having Hey Jarie. I think you're having issues with your Yeah, with your RDP. There it looks like your RDP came back. Jarie Bolander: Yeah, no. there we go. Yeah, I thought maybe I'm ebbing. Andrew: Maybe because you're not clicked into it for a longer time. It stops streaming or something. Jarie Bolander: Maybe. Here, let me make sure I click into it. Yeah. Andrew: 'Cause I could hear it I could hear it running Jarie Bolander: Yeah. Andrew: from the other room. Jarie Bolander: Yeah, it's just so here, let me it's it's like a bit clunky that way, but hold on. Let me Yeah, it's my it's probably because my bandwidth. But you'll notice it came back. Ha ha it look it worked. And then I'll go let me go plug in this one again. And you will see see it come back fully. Andrew: Yep, so now both leader and follower will be back. And what's really cool about this system is it's able to do more than just two. So you could have three, four, five DCNs in a high availability configuration. and start doing really cool stuff like migrations, David DeBari: The Andrew: have a couple of your DCNs being older hardware. start installing some newer hardware and then migrate over to the newer hardware by then once they're all running slowly disabling the older hardware step by step to maintain running David DeBari: Yeah. And Andrew: the whole time. Jarie Bolander: Yeah. David DeBari: if you want to do a patch on one you can restore with a third unit, you could actually have a two, so you're no longer in simplex. And then as you restore the the one you take down with patching or you know, whatever you might do to it, you can bring it back. And the idea is that they're not there's this is not a symmetrical requirement. In other words, the two ASROCs don't have to be ASROX. They don't have to be the same model. They could be one of them could be virtual. More a couple of them could be virtual, which is a real interesting capability with high availability. It starts Jarie Bolander: Right. David DeBari: to remove the the tightly coupled software hardware model that has been used. in this the Jarie Bolander: Totally. David DeBari: high availability does this for you. Jarie Bolander: Yeah, I mean that's the whole and also that's the whole point of OPA where you can mix and match hardware, have David DeBari: Right. Jarie Bolander: different software. and just just so everyone knows, like these two this you know, D C N one and D C N two are only connected through to the network. There's no stuff behind them, right? Which I think is another important thing. David DeBari: Connected to the network. Yeah. There's no there's no bus, there's no private cable connecting them. They literally are just part of the the environment via the network cables. Okay. If you go disconnect Jarie Bolander: Yeah. Yeah. So so there you go. David DeBari: DCN one and Jarie Bolander: Okay. David DeBari: and Give it a couple of seconds and then plug it back in. We'll see D C N two move to be the leader and then DCN one will come back in as the follower. Jarie Bolander: Yeah, that should work. Let me go do that right now. Hold on. Andrew: This can also be forced through various David DeBari: Yes. Andrew: software interfaces. Fusion offers a way to force, but it's also possible to force within the Codasys application. So for some reason within the control runtime you need to force who is leader, then you can do that. David DeBari: Now he's going to plug back in DCN1. And in this situation, it it doesn't force DCN1 to be the leader, but that is also settable. They can be prioritized so that if DCN1 returns, it can become the leader and the Jarie Bolander: Yeah. David DeBari: DCN2 would become the follower. And that's really the beauty of the high availability. It's based on the software settings. So as you can tell, you know, we have. when we took all the control off, the IO went to fail low, you know, went to zero, but you could set it to fail last. There's a there's a lot of other things that could be done in a system which are typical of automation systems today. But the way we've done it here is just whichever one is running it stays the leader and then anybody that joins in becomes a follower. obviously we've we've all run around with with people who wanna have the left hand one should always be the leader if it's available or the right hand one should be the leader if it's available. You know, it it it is definitely settable and as Andrew said, you know, if you wanna ch if you wanna make it swap, you can do that too, 'cause the tool allows you to force the swap if you want. Jarie Bolander: Totally, totally. And I do think, you know, and you mentioned the virtual DCNs, which at some point we'll do another demo on because we're still kind of getting through through David DeBari: Right. Jarie Bolander: all that. But it is amazing and and I think you mentioned this in your CODISIS talk the other day about how much compute power, like compute power is growing faster than like say IO capability is just a broad David DeBari: Right. Jarie Bolander: example. So more powerful industrial PCs, more powerful compute you can do a lot more. So as an example, these these two DCNs could drive way more obviously than this I.O. and have David DeBari: lots. Jarie Bolander: multiple, you know, replace multiple PLCs David DeBari: Right. Jarie Bolander: because they are so powerful. And because you have that capability, I think you guys mentioned the virtual you know DCN upgrades, swapping without losing any downtime, which is pretty amazing. And The reason that I think an OPA system is so powerful and why obviously we're proponents of it, not just our solution, obviously anything that that can be OPA is going to be very powerful for the end customer. David DeBari: Yeah, a few interesting facts about our HA is that it doesn't it doesn't have a resynchronization scan cycle. literally the device takes over the leadership in the next scan cycle. So there's no where were we check, if you will. it is literally one scan cycle. It was one device, the next scan cycle, it is the the follower that's been promoted to leader. It is in charge. So there's no skipping a scan cycle. And and that's a very interesting capability as well that it that it doesn't seem to miss. many systems can tolerate a scan cycle skip, but this HA doesn't have it. Jarie Bolander: Yeah, totally. And again. All based on OPA, which I think is even better, which David DeBari: Yeah. Right, right, right, right. Jarie Bolander: which is awesome. So so thanks again everyone for taking the time to listen to the OT crowd. if you listen to this on Apple, Spotify, or whatever, please leave a rating in review. If you want to send us a message or whatever, you can go to the OTcrowd.com. And That's it for this time. Thanks so much for joining, and we will see you next time.