00:00
This is the Art of Network Engineering podcast.

00:11
In this podcast, we'll explore tools, technologies, and talented people. We aim to bring you information that will expand your skill sets and toolbox and share the stories of fellow network engineers. Do you want to know what it is the networks are everywhere, all around us, even in this room? You can see them when you look in your data center or even up at the ceiling.

00:36
And don't even get me started about what's up there in the clouds. But what if I told you there was a whole other kind of network. A network that not everyone can see. One with a whole different set of devices and needs. Unfortunately, no one can be told what this other network is. You have to see it for yourself. In one hand, I have the IT network. It's comfortable, warm, and fuzzy. In the other, I hold the OT network.

01:05
It is largely unknown, looks different to you, and maybe even a little scary. Which would you choose? Find out the exciting conclusion of this story on this episode of the Art of Network Engineering. Oh my God. Welcome to the Art of Network Engineering. My name is Andy Lapteff and I am joined tonight by... That was Morpheus, right? Because I thought you were going with Neo, but then you sounded like...

01:31
Morpheus. Yeah, I need to start off with an apology to Lawrence Fishburne. I can only imagine he's a fan of the show. So in fact, I should extend that apology to the entire Matrix franchise. But hey, Andy, dude, that was awesome. That was so good. That's that's the greatest thing I've seen in a minute. So here we are, man. How how are you, Tim? Let's let's start with how Tim is not too shabby. I've been I've been running my

02:00
I've been trying to mimic you. However, I've been running my easy bake oven outside. Um, I'm not at the skill level that you are with your, uh, your briquettes and your wood. I use the little rabbit turd pellets. Um, Oh, you got a, um, a pellet, uh, smoker. Yeah. Oh dude, they're, I mean, that's, that's great. They're set in, right? Yeah. That's exactly what it is because I don't know what I'm doing. So if I can just put something in there and come back a little bit later, to an extent.

02:27
pretend I know what I'm doing, Tim, but I have charcoal, but then I bought a system. It's a, it's a four probe thermometer system that has a pit probe in it with a fan that feeds the pit air. So I basically have what you have, but I pretend that I know what I'm doing because I'm using charcoal. You're just feeds the pellets and regulates temp. It's all good. It's all about controlling the fire. What are you, what are you cooking? What are you barbecuing? Um, all kinds of stuff.

02:56
You know, anything from burgers to, I would tell you, we did a meatloaf on there a while back. And I will never have or never want just a meatloaf out of the regular oven again. It's just the consistency, the taste. It's different. I do got something going with the, somewhat the temp right now. Sometimes it'll think it's overheating and shut off. So I got to figure that out. But other than that, it's good. What's new with you? Delicious.

03:22
I'm happy to hear it, man. I started grilling as well. I did some ribs the other night. They came out pretty well. The pool was open today. This is our first full summer having the pool. We built a pool last year and it didn't open until the season was almost over. Yeah, so they opened the pool up and that is exciting. The water's a little chilly. Work is going great. Really enjoying what I'm doing at Juniper. And I started lifting weights this week.

03:51
And I'm starting to think it was a mistake because everything hurts, man. My wife is teasing me like, are you okay? I'm like, oh, those are those good first week back feels. Yeah, man. It's awful. That's awful. Isn't that how you know it's working though? If it hurts, it's working. Isn't, isn't that the thing? I hope so. The bad part is when that, that first, first or second day, when you go to put your hand down on your chair to sit down and you're

04:19
Your whole arm gives out. You end up on the floor. Yeah. So the voice we're hearing from the ether is our guest tonight. Um, Mr. O.T. himself, Josh Varghese. How you doing, Josh? Hey guys. How are you? Good, man. Who, who are you? What do you do? So my name is Josh Varghese. Uh, I am the owner of Trace Route. I started Trace Route, uh, about five and a half years ago. Uh, we are just a small little consultancy.

04:48
And we focus all our efforts on design, configuration, product recommendation, procurement, pre-config, testing, training, auditing, troubleshooting. And the unique part is that it's all for industrial networks or OT networks, which is maybe a big question mark for everybody right now. But that's what the whole point of this episode is, is to find out what the heck is an OT network and what's unique about that.

05:17
not OT, right? What is OT? What's the acronym? What does it mean? Yes. So the acronym is to differentiate away from IT, the information technology. The OT is operational technology. The general concept where these phrases branched out from is if IT is supporting like data systems and business systems that OT is supporting. Basically, OT is hardware and software that interacts with the physical world. So operational technology, the giant umbrella term.

05:47
The way I like to think about it is to think about where it is and what verticals it's in. So your local municipality that delivers your water, that's OT. That infrastructure is operational technology. The pumps and motors and drives that work, the valves, the pipelines to move that water, their wastewater side. If we talk about manufacturing, food and beverage, consumer packaged goods, oil and gas, electrical

06:17
All of these verticals have this side of the house that interacts with the physical world to make the product they're making or move around the thing that they're moving around, that they're all interacting with the real world in some way. And these OT systems, there's a bunch of phrases that I would say for the purposes of this discussion, we can treat them as interchangeable. You might hear the term industrial control systems. You might hear the term SCADA.

06:45
SCADE is an acronym for supervisory control and data acquisition. You may just hear people talk about controls or industrial automation. All of these are kind of in the same category. There are these systems out there that are operating equipment, operating physical stuff in the real world to make things happen. As we're saying that, I've actually got a few pictures that I thought we could share to kind of put some images behind all this yapping of mine.

07:14
those images up. I kind of wanted to ask if, so I get the idea behind what the OT devices are and what they're supporting. But what I'll ask is, we talk about OT networking as well. Why do these OT devices, why do they have to be or why should they be on separate networks? And is that just ethernet? Is it other types of networks? Why do we have separate networks from IT and OT?

07:42
All good questions. So I think to answer that, I think the first good thing to look at is just what are the elements of an industrial control system or an OT system. And so at a very high level, these are just some examples of the large major primary assets in a control system. And so to your point, Tim, like on this sample screen, maybe these blue lines represent an ethernet network. And then the major components sitting on this ethernet

08:11
are, I would say that the primary asset that we're going to talk about is the PLC. That stands for Programmable Logic Controller. And if we look at one of these PLCs, and here's a picture of one like in the real world, like installed in a control panel, controlling equipment. But if I go back to this, what I want to show you is, so this PLC has multiple elements to it. There's a power supply, there's a CPU, there are network communication cards. Maybe those are

08:40
Ethernet networks, there could be a bunch of proprietary field bus networks that are vendor-specific or certain instrumentation type specific. So you've got your communication cards, and then you've got input cards and output cards. And these are the ones that you have hardwired connections to equipment you're controlling in the field. So you're going to have a wire running to a drive that's controlling the speed of a motor or that's literally telling that motor that pump to turn on or off or that valve to open or close

09:10
50% open or you're reading instrumentation from the field. We got to monitor this temperature, this pressure, this flow, this pump speed. So ultimately these PLCs, their job is to read those inputs. These PLCs are going to be programmed by PLC programmers, industrial automation professionals, controls engineers. These are the professions you'll hear as the jobs that are the people who program these PLCs.

09:35
they're gonna write logic in these PLCs that might look like any of these things here. There's a bunch of different programming languages for these PLCs. Ladder logic is still a very popular one. It reads a lot like an electrical schematic. You've got some called function block where you've got blocks performing functions, basic functions like and, or, or, if, then. You've got structured text. You've got these different programming languages running in these PLCs so that you can basically process inputs, execute logic, process outputs.

10:04
And then ultimately, you've got operators of these systems. Sometimes these systems are highly automated. Sometimes they require an operator to look at what's happening in the system and make decisions in real time. So those operators got to look at some screens. So you have what's called the, the, the HMIs, the human machine interface. Now everybody, whether they've heard of OT or not has probably interacted with an HMI. Basically anytime you've interacted with some sort of kiosk, whether it's at the gas station pump or buying tickets at a movie theater.

10:33
That's an HMI, right? Like it's a simplified user interface where you hit some buttons to make something happen. So the HMI is in these OT systems. The only difference between those, like here's some examples of some HMI screens. So again, that same system integrator, those controls engineers, industrial automation professionals, another part of their job is not just programming those PLCs, but to build these graphical user interfaces that the operators...

11:00
are going to interact with. So you could picture that maybe there's a control room for a facility where there's one or more operators sitting and looking at the statuses of the equipment within the facility and making decisions based on a recipe they want to run or based on the production targets for that day and literally potentially manually starting and stopping equipment or starting and stopping sequences. And if I go back to this screen, so we've got the programmable logic controller, we've got HMIs.

11:27
We've got control room engineering stations where maybe the development software sits, that if you need to make changes to these things, where that work's done. You've got these servers facilitating, collecting data from these PLCs and maybe historically recording some of this data. Maybe we're not just interested in real-time data. We want to know what was that temperature over the last 24 hours? What did it look like 10 minutes ago? What did it look like 30 minutes ago? We've got to store that data and be able to run trends and let an operator quickly see

11:56
What has that process value been doing for the last hour, day, or month, or whatever? These pieces combined, the PLCs, the HMIs, the servers, the engineering stations, it's very common for these, I'd say in the last 20 years, for these to sit on an Ethernet network. This is where things start to get interesting. When I say Ethernet network to most IT professionals, I think they're probably picturing some Cisco

12:26
because what else would it be? That's where the switches are, other than maybe occasionally in above some ceiling somewhere, you might tell me, depending on the size of the organization and the business, but this is where things start to diverge quite a bit. So maybe what's best to talk about is like, what is the difference between an IT network and an OT network? And I can hit kind of a list for you there. So I just figured the OT stuff lived on the IT network, but that's not.

12:54
I guess. So what I'll say to you is that that happens on occasion. Now I would clarify when you say it sits on the IT network, do you mean sits on the same hardware or literally sits in the same VLANs as IT equipment? You know, I don't know. It just sounded like something smart to say. So are these separate systems? So the blue lines in your drawing here represent the network. And OT or a bunch of these systems that have to communicate together. So you're going to plug them into a network.

13:22
Yeah. If it is ethernet, I would assume you're using some type of iron that has ethernet ports and layer two capability to do all that stuff. But, yeah, I mean, I have so many questions like, are they separate people? For instance, like, is there a network person that sets all this up and then you plug your PLCs into it? Like, so, so we'll get there. So let me say for starters, to address your direct question that like, uh, what I'm missing in this list of pictures, that's one that I missed. But what I would say is in most organizations.

13:49
Where they are or where they want to be is that their OT network is separate from their IT network, either completely air gapped, which is less and less realistic these days with where they want to pull some of that data into higher level business systems. But from a production and robustness and reliability standpoint, historically, these have always been their own things for a number of reasons. And you hit on a couple of them. The hardware is different. So we'll talk about that a little bit.

14:18
The hosts that they're supporting are different. The protocols that are running on these networks are probably different than the top 20, 30% of protocols that are running on your IT networks. How this equipment is managed is often different. The topologies and the protocols to support the topologies are different. So deployment, go ahead. Can these, I have so many questions. This is awful. It's gonna be like a four hour show. So can these places- Let me say one more thing, which is.

14:46
I said they want to be air gap sometimes, but it's not realistic because like, let's say there's data in these servers that they want to feed to upstream business systems. So what ends up typically happening? Like I'd say, roughly speaking, 80%, 90% of organizations, there's going to end up being some firewall between the OT system and the IT system. Maybe there's some DMZ between the two, something like that. But generally speaking, there is, if there's connectivity and there's a need to push or pull data,

15:15
from OT into IT systems, there's gonna be at least one firewall ideally between the two. Now to that point, or to my point about that I'm gonna get to about these systems sometimes are a disaster, we do audits all the time in smaller facilities where IT and OT is all sitting on the same hardware and I'll talk about why maybe that's not great. In the same VLAN. So- Yes. My question was is,

15:40
Can the OT systems communicate with each other without the IT network? Because I'm understanding them to be two separate things. Yeah. I would say like for this system to operate, like what the communication that needs to happen is between these PLCs and these SCADA servers, these clients, and these SCADA servers, these SCADA servers, and it's all kind of self-contained within that control system. But for the PLC to talk to the SCADA server, it doesn't have to go through the IT network. Yeah. So these SCADA servers typically sit on the OT network.

16:10
The OT network, right? Yeah. I guess I'm trying to, I'm trying to draw the demarcation in my head of it sounds like the OT system has its own network. The IT system has its own network. And for some reason they need to talk to each other. So that's what I'm trying to break apart in my head. Yeah. And the perfect example, like when that happens is for example, um, there are, um, there's a couple of different software package types, like, uh, manufacturing execution systems.

16:39
and a couple of other ones that tend to sit on the IT side of the house. And so they're interested in some of these production. I talked about recording like historical data. They want that data. They're running reports off that data. They're making business decisions based off that data. So when that needs to happen, what's typically happening is there's a few different ways to do it, but you're going to end up dumping information. Like you're going to do like SQL database replication or something like that from a database that resides in OT into like...

17:08
a mirror copy of the database sitting in IT that they just access that one. Right. Like there's reasons. Which might be running a different telemetry collection application that's going to do your analysis. And okay. Yeah. I think. And there's, there's a couple of reasons why security is high on the list that like OT and IT is increasingly understanding this and these organizations are working together, but like they don't want IT reaching into the OT network to get data. They are nervous. Like they can, like the OT network considers the IT network as untrusted.

17:38
as the IT network considers the internet. Yeah, yeah. Are they different teams? I know it's gonna depend. Yeah, so when we say teams, which bodies are we talking about? Like responsible for the network? So that contention between IT and the OT folks, it sounds like there's two teams and they're very leery of each other because of security concerns, right? Yeah, so this is on my list as far as differences between like IT and OT networks, like.

18:06
late in that list I also talk about. Historically, there's a pretty negative high friction relationship between the OT side of the house and the IT side of the house. In my opinion, I've been in this for, I guess, 19 years total, 15 specifically OT networking. In my opinion, that friction is just, it's basically ignorance on all sides. IT has language and syntax that OT doesn't understand.

18:36
There's different priorities. One of the things that comes up a lot in IT OT conversations is that the CIA triad of confidentiality, integrity, and availability, that IT puts that into certain priority and that OT values that a different way. If you're the OT guy, they would say their complaint is like, IT doesn't understand our production needs and how sensitive the system is.

19:03
might roll out patches to equipment in the middle of a production run and take equipment down. We're sitting here targeting 100% uptime. There's just these, I would say, this list of the differences between the networks ends up creating this giant gap of understanding between both sides. I don't think it's single-sided to blame here. It's just the fact that they traditionally have very different responsibilities and priorities.

19:29
Yeah. So I was thinking competing priorities and if you, you just said a hundred percent uptime. So I'm thinking the design has to be pretty important. Like if you're going to have to make upgrades to systems, but you have to keep them up all the time, I guess is the design critical so that you can take parts offline, bleed, bleed traffic off, do the upgrade, bring it back up. So I'm going to come back to that point because I'm going to paint a picture for you. That's going to, that's going to seem very mismatched between what seems like should be the case.

19:56
And what I'm saying is the case in a lot of systems out there. We jumped right in real quick because this is again, amazing. It could be like a three day long show. How, how did you wind up in OT? How did this happen? Like, how does someone get in? Cause it seems like a very specialized area, right? So I did, uh, I did power systems, double E in school. Uh, and I ended up working for a water and wastewater, like engineering consulting firm.

20:25
Electrical engineering? Is that what that is? Yep. But I did not do anything with power systems. I did some interviews and I knew some people at these engineering firms. And basically I ended up becoming one of these controls engineers, automation engineers. My first four years out of college, I was doing like designs for water, wastewater. And I say designs loosely because designs sometimes means copy and paste the last design and do small tweaks.

20:53
That's familiar. But that group happened to not only have design engineers, but they had a programming team as well. So I got to take this one project where we were building new pump stations for a large water utility. And I got to work on the design of that system. I got to work on the construction services side, making sure that the system integrator and the contractor was providing the equipment that's in the plans and specs.

21:19
And then I got to actually work on the PLC and HMI programming for that same system. So I kind of got to see this project across a couple of years from literally like birth to construction and startup fully completed and operational. So I was basically a controls engineer for a couple of years. I was doing this programming and stuff like that. I think I decided pretty quickly that like, I don't think programming is my skill set. Like I think I'm fine at it, but I'm definitely not great at it.

21:49
and I couldn't see myself doing it for a long time. And what happened was we used to have a company that would call on us at this engineering firm because we're creating these plans and specs for these water utilities. So we'd have different vendors and distributors calling on us to do lunch and learns and talk about what equipment we should be specifying, what's the newest, latest technology, all this stuff. And one of them was a networking vendor. They were specialized in industrial networking. So they didn't ever sell into the carpeted environment.

22:18
Their entire line card was gear that went into these control systems. So it's ruggedized, hardened equipment for switch route, wireless, all that stuff. And, uh, I basically ended up jumping ship over there when I found out like, Oh, there's a little bit of a, they're kind of a technical distributor. So they had a need for like pre-sales and post sales, like not only technical support, but also doing some kind of projects with customers, that sort of thing.

22:43
And after almost a decade of that, I realized that like, man, the need for consulting and subject matter expertise in this space is so huge that like it's not going away anytime soon. And I decided to hang my own shingle like focusing on not tied to a line card. You see this lab wall behind me. There's got to be at least 15 different vendors on this wall behind me. And part of that is like, I am so passionate about finding.

23:11
new, better, different, figuring out the bells and whistles on everybody's platform and being able to find the best solution for any given project or opportunity. So that's why we do what we do is so I could offer that consultation on this niche area and be free of any specific vendor locked in relationship situation. Yeah, it is, it really is. I don't know where to begin. I know Tim's got a list of good questions. Tim, I'm at a loss. I could just dive into.

23:41
It seems like such a cool world, like even the way you got into it, you know, your electrical engineer, you, this, that's who got you the job. Like, I didn't even know this stuff existed and it's just so amazing. And our whole infrastructure is running off this stuff. Like you said, the power plants, the water department. Like I'm so glad you phrased it that way. We don't even know it's there, but man, if it goes away, we're in big trouble. Every, everybody I work with, you know, friends, colleagues, customers, everybody in industrial automation and control, we all talk about the fact that like.

24:08
It's so weird how everybody landed in industrial automation itself and that if you don't know somebody in it or happen to take some factory tour on some weird occasion, nobody would know that. Nobody thinks about it. You don't set foot into any of these facilities and you just don't think about it. But it's crazy what's happening in all these manufacturing facilities or on the pipelines or...

24:35
at these substations. It's like, oh man, there's a lot going on there. And the reason I wanted to talk to you guys and talk to y'all's audience is to say like, oh, networking in this space specifically, it's remarkably underserved. I can talk about why that is, but it's just a huge gap. And so I think knowing what you guys, you guys talk a lot about career development and how people got into IT.

25:01
what I want to flag to everybody is man, there's this whole other area. Like there's a different area in this country and all these different verticals with networks that aren't served by IT traditionally that need a lot of help. And I bet there's people that will be listening that have an OT side of the house in their organization where today like this is a problem. Like they are struggling with their networks on the OT side of the house. And for a number of different reasons, like those two parts of the organization aren't talking or whatever. But

25:30
these people could make themselves like there's an opportunity to make yourself really valuable on that side of the house because this is an increasingly recognized large gap for a lot of these organizations. So that's the major thing I want to highlight today is like opportunity abound. And you talk about network management. So I want to get into that a little bit and how we know we have a traditional IT network and now we know we have these separate often air gapped OT networks. So

25:59
What makes OT networking different from a management perspective in that to me, and if we're talking ethernet, specifically ethernet, to me isn't a switch a switch? Are we talking specifically like you mentioned like ruggedized gear and that kind of thing, but from a management perspective, from a design perspective, what makes it different is that we don't, maybe we don't need as many features.

26:29
on the OT side from a networking perspective that we have on the IT side. What, what, let's, let's compare and contrast for a minute. So let me, let me hit this list about what I kind of jotted down as like unique about the OT networks. So the first one on my list is the host that they're supporting. If we pop back to this slide, like, okay, uh, NUX or servers, like this looks familiar from the IT side. Anything PC based I'd say looks familiar. But when we start talking about PLCs and these panel mount HMIs, like

26:59
this guy or these right here are called VFDs, variable frequency drives. These basically control the speed of a motor. They're a very prevalent item on OT networks. And you'll notice there's ethernet cables coming out of them, right? So there's gonna be VFDs like this. There's gonna be servos, these PLCs. There's all these unique host types, assets types that you'll never see on an IT network, right? And one other thing that's unique about them,

27:28
is unlike maybe a Windows box, I would say a lot of these OT devices and these OT device types, they run on like real-time operating systems that are like purpose-built for their platform on maybe resource-constrained hardware. I shouldn't say resource-constrained. I mean, relative to like a large Windows box, I would say they're resource-constrained. It's a smaller CPU. It's less RAM. It's purpose-built, right? That's right. It's purpose-built. And the problem

27:57
with that is when you start deploying these devices at scale, they don't handle extraneous traffic as well as a Windows box might. Where it might take a lot of extraneous or unwanted traffic to lock up a Windows box, it's pretty easy to tip over some OT devices. If a broadcast domain gets too chatty, you can start tipping over some of these OT devices. We might touch on multicast later.

28:26
makers used and there wasn't enough education. And all of a sudden there's a bunch of multicasts on these networks with unmanaged switches. So there's nothing to mitigate it. So these multicasts turn into broadcast. Now all these OTAssets that can't handle extra traffic are getting drilled with a thousand or 3000 or 5000 packets per second. A traffic dot meant for them. That's locking them up. So there's all these situations. Hang on a second. You said unmanaged switches. I did. So we have some of these. So we may have a big facility. You just said it that...

28:56
You could have a large broadcast domain and essentially have a broadcast storm take down some of these devices. So how do we, back to Andy's point about design, I mean, how much does design play into that? If we have a bunch of unmanaged switches potentially, I mean, it's all one VLAN, right? Unless you further physically segment it. So how does that even, how does that design, I mean, who gets involved to figure all that out?

29:25
So this is where I'm going to start to break Andy's brain on like, I feel like I know what this system should look like. And I said I was going to mortify you. So let me start with that. Let me start with the why and then I'll get into the what I'm going to kind of back into it. So the why is these systems like everything you see on this page, the PLC, the HMI, the skater servers, these clients, the who like who is developing these systems, who is

29:51
procuring them, who is configuring them. In some cases, if it's a large enough end user, maybe they've got controls engineers in house. In a lot of cases, they're dependent on outside parties. That group is typically systems integrators. That is the company type. These systems integrators have like instrumentation or field techs, and then they have automation engineers. And these are the guys that are programming these PLCs and these HMIs and these SCADA servers. And ultimately that system integrator

30:20
Like the project that they're bidding or they're scoping for a customer is a control system upgrade or maybe automating a process that hasn't been automated yet. And so historically, this has been the job of, generically, the controls engineer. But my main point is often it's been the job of an outside resource to the organization, a contracted system integrator to design, configure, install, startup, and maybe to some degree maintain these systems. And for 20 years...

30:49
That typically meant this food and beverage facility needs a new production line. So the integrator would light up one line. Maybe there's one of these control panels, uh, like this with some equipment and a PLC. So maybe he's got an unmanaged switch in here to connect in that PLC and the local HMI. My main point here is for a long time, these were a lot of smaller islanded systems, even within OT, they didn't even talk to each other. It was just, you can imagine a factory floor with.

31:18
20 little islands that are just doing their local process and that's it. But over time, obviously, they want some overarching ability to see the entire system. So now they're like, oh, we need a plant network. Literally, sometimes our phone call is like, we need to build our first plant network, like the facility-wide OT network, because it hasn't existed yet. We just had a bunch of these islands. And so comment number one is, if you're a system integrator, and your choice is to purchase...

31:47
an industrial managed switch. Let's just say it's an eight port gigabit copper switch. They tend to look like this, an industrial switch. So what do you, I'm curious, what do you notice when I show this to you? What is the same and what is the, what is different about this and what you guys are typically dealing with? I mean, the power connectors stand out to me. It's also a lot smaller form factor. Yeah.

32:16
So, industrial managed switches, typically you're going to have DIN rail, which in the picture we're looking at right now, you'll notice all this equipment is sitting on these rails. Everything behind my head is sitting on similar rails. Those are DIN rails that sit in these control panels. So a lot of industrial equipment is designed to sit on DIN rail. So just like this PLC sitting there, they want this switch to go on there. So yes, the mounting is different. Andy, you said the power.

32:46
So on the IT equipment, we're typically hitting it with 120 volts AC. Yeah, you don't typically want 120 AC in these control panels because they want the control panels touch safe. They want people able to get in there and rework wiring and stuff without risk of electrocution very easily. So for that reason being touch safe. And there's a couple of things about heat and size that gets into that like YS24 used in industrial in general that's in the mix.

33:15
Ultimately, the main point here is your industrial equipment is often powered by 24 to maybe 48 volts DC coming in on these hardwired terminal block connections instead of like an IEC connector on a typical like catalyst or something like that. The temperature rating on something like this could be, I'd say standard industrial temperature range is like zero to 60C. Extended industrial temperature range is like minus 40 to plus 85C. These things are going out.

33:44
in outdoor panels in West Texas in the peak of summer and supposed to be running the whole time. They're fanless. If somebody sticks a fan on an industrial switch, they're basically going to get left out of the room. There should be no moving parts because that's just a maintenance nightmare for the environments these are going in. So, I would say extended temperature, shock and vibration specs sometimes based on the environment they're going into, power, mounting. Sometimes there are...

34:12
specifications or certifications that are specific to different verticals. So maybe in electrical utilities, there's a very specific shock and vibe spec that has to be met. So there's a... Or in oil and gas, there's a certain thing about electrical hazards in certain situations. So there's almost a ceiling requirement called class 1, div 2 that basically a spark can't generate an explosion in that area because of the environment in that area.

34:38
So all these things make the hardware unique. And this is why the vendors in industrial networking, there's some crossover. Some of the IT vendors like crossover into OT, but there's also a lot of players that are uniquely OT, exclusively build gear for OT networks. So we've talked about the types of hosts they're supporting and that those hosts are kind of sensitive. We've talked about who's designing these systems and what they're deploying. I didn't finish the thought. The thought was,

35:07
If your choice is to spend $1,200 on something like this, or I can buy an unmanaged industrial switch, which might look the same, but just literally has no ability to be managed. You can't assign an IP to it. The unmanaged version might be these days, a hundred bucks. So 9.9 times out of 10, the integrator is going to buy the integrator or the end user or the machine builder. They're all going to pick the cheap thing. If the cheap thing's going to work, we'll pick the

35:37
Because you have people who don't understand networking, building networks. That's right. Yeah. And I think they would say they, I think some of them would say, we're not even building networks, like we're just facilitating this little island. You're not until you connect the islands. That's right. And, and this is, and this is the crux of the whole, like, where is this problem? This tornado that I'm going to, this tornado picture I'm going to paint. This is where this starts from is that historically the

36:03
People responsible were responsible for small islanded pieces. And then in the last increasingly in the last decade, uh, these end users are saying, well, we don't want islands anymore. We, we want all the data all the time. So please connect them. I want to dig into that point. You might've just answered it, but it sounds like a big cultural shift in the OT industry from islands to this unified plant system, what big cultural changes in, in verticals are rare, right? So what, what.

36:33
What facilitated that? What precipitated? We don't want islands anymore. We want, which then created a whole networking problem, but Oh, I think it's just there. I think it's the recognition of, uh, data being their most valuable asset that a lot of these people like would create processes and crank things out and try and improve processes and then slowly they learned that like, Hmm, if we knew this about what we did before, like we could do better. So this general mantra

36:59
That's right. Make better decisions. That's right. Do that if everything's an island because you can't get it off it. Right. Correct. Yeah. That is absolutely what drove this like explosion is this recognition of like, we need more data. We need more data. We need more data. And these used to be like siloed pieces of data and people are like literally putting spreadsheets on the flash drives and sneaker net walking it around. And like, this is terrible. These things, they sit on ethernet networks, right? We can get all this data. Just plug it all together.

37:29
And did they do that? Yeah. So what would tend to happen is the end user themselves or the system integrator would be like, yeah, I guess we can plug line one and line two together. Maybe they'd get away with that. Right. But we know where this story is going to end. They're connecting all the unmanaged switches to each other. Right. Is that? Absolutely. Or, you know, they're installing some 24 port rack mount switch maybe in a closet somewhere or maybe on a column in the middle of the factory.

37:56
And then they're home running all these unmanaged switches back to this unmanaged 24 port switch. And they're like, boom, we got it all. And that's where you start knocking devices over from too many broadcasts. The broadcast domain is the first issue, right? Literally when like we do this two day industrial ethernet class, I've done one a month for the last six months. It's been really, there's been people asking for it left and right this past year. When we teach this class, like my number one topic across that two days is.

38:25
Minimizing the size of broadcast domains because in this environment specifically, it is often the killer. Like they have organically grown to this, like, um, 200 assets, 500 assets, a thousand asset size on a big flat, unmanaged network. And it is the root of all their network problems and downtime. So if I can guess you're tearing out all the unmanaged switches, replacing them with managed and partitioning.

38:54
layer two, right? To constrain the broadcast. So literally our conversations are this like kind of what one of my comments about OT versus IT is I've been, like I think I said, I've been at this for like 14 years and somehow I'm considered like a subject matter expert in this space. And what's funny to me about that is when I compare it to IT and

39:19
what's available in IT in terms of technology and protocols and what's happening in IT. I feel like for my entire career, I've been doing less than CCNA level work. And that's not a knock. I'm not even trying to be self-deprecating. What I'm saying is that the basic blocking and tackling of network fundamentals, that's what these customers haven't reached that yet. When I talk about these maturity assessments or maturity scales, I'm like,

39:48
Yeah. So if we're talking like zero, one, two, three, four, like a lot of these customers are at zero or a half. And I'm going into all these organizations and we're taking them to step one, which is exactly what you enumerated. Like let's rip out the unmanaged. Let's get some managed switches in there. Let's actually manage them. Let's do some VLAN segmentation and we'll figure out, we'll, we'll discuss like how it makes sense to break up your network based on your system. And you're going to have, you're going to be in a substantially better position today than you have been for the last decade.

40:18
Basically, I mean, that playbook, as simple as that playbook sounds, you've just described 95% of my engagements for the last 14 years. And they have to. That's the only way to do that. If they want to follow that cultural shift and get access to all their data, it's the only way they can do it. If the end hosts are going to fall over with, you know, too many broadcasts. The other thing I'll tell you that's funny is like, there are still system integrators in this space. Like I get into these arguments on LinkedIn sometimes. Like I've got an integrator. It tells me all the time.

40:47
You don't need managed switches. They're just, they make things worse. And I was like, I can appreciate where your sentiment is coming from. And I understand you probably had some frustration. Like I get it. It's not that like, I don't understand how you're even pulling this off unless all you're working on is small isolated systems. Like you will have the same problems everybody else has if the system gets large enough. And you're also just kind of shooting yourself in the foot from a troubleshooting standpoint. Another thing we talk about in the class is how

41:16
70% of these problems on these networks are physical layer problems. And I love to tell them how like, guess what? These physical layer problems cannot hide from you if you have managed switches. You are going to stack up CRC errors because you've got poorly terminated cables, or you've got unshielded cabling running across 480 volt VFDs, or you've got duplex issues that are showing up as collisions. They cannot hide from you. It's my favorite thing about networking, is that...

41:42
At least when it comes to wired networking, I legitimately feel like there's not a problem I can't solve, at least in OT networks. There's a systematic approach. I can troubleshoot any problem and identify either where something has been misconfigured, miscabled, or eventually, sometimes I do end at a like, oh, this piece of equipment is not doing what it should be doing. We've got a bug in this stack, which that's not uncommon either. Some of these OT, like the TCP stacks into these OT devices.

42:12
Uh, they got some, they got some issues. It's, I'd say the closest thing in IT networks is probably printers as far as like just being nightmare sometimes. So I'm just, I just had a little light bulb moment, which is embarrassing. Even say this wasn't in my bones before, but yeah, non-managed switch. You can't even see a CRC error on an interface. So you can't see anything. So you have no visibility. It just, it's like it ended and walk away. Yeah. Which is what they loved about it. Well, I've done it at home.

42:41
Right. But like in an environment like that, yeah, once you have problems, I mean, you can't even troubleshoot. That's correct. Effectively. Right. So, so I would say for a decade, these, these Island networks, or as they, they scaled poorly, these networks on unmanaged switches, they were, uh, we, we call it plug and pray, right? Like you just kind of plug it all up and you cross your fingers. And when stuff doesn't go well, they just like unplug, try different port. They cut off the end of the cable. They re-terminate it. They replace the switch. Like.

43:10
That's about the amount of troubleshooting they can do because it's all that's at their disposal. Well, was this the gap in networking that you alluded to earlier? So you said there's a huge gap in this space and that networkers could like, you know, how this is relevant to our audience and maybe somebody trying to break in. I mean, it sounds like the level of networking knowledge in these places is very minimal. And I guess that's what you're doing with your company is coming in and trying to help them with that. But so there's so.

43:36
Yeah, like one of the things on this list about the differences and you touched on it earlier is like, well, who's responsible? So I mentioned like these integrators are responsible for maybe like the design and the initial like installation and configuration and everything. My general comment there is like, so these networks are owned typically by controls, automation, OT, not by IT, right? Like these OT networks are not maintained by the, there's a few exceptions here.

44:03
In some organizations, they do have IT take over the OT networks. But in my entire career, 95% of them, IT is not touching these networks for a number of different reasons. So ultimately, that means that that same controls engineer, the local plant engineer, he's responsible for this network. And automation engineers are some of the smartest people I've ever met. The good ones have multidisciplinary engineering skill, ranging from this primary core skill of automation to mechanical.

44:32
electrical process, chemical engineering, it is crazy the range of skills that these guys have. So there's not a capability problem here, but there is this very abrupt all of a sudden in the last couple of years, like, Oh, now these networks are tipping over. Now I got to be an IT guy too. Like that's the natural response is like my plate is overflowing with what I have to know and understand and I'm responsible for. So

44:59
This is the gap I'm talking about too. Like who is responsible for the network today, but what their level, like they're so resource constrained. Like these organizations are tremendously resource constrained. It is the very tiny exception. And this is just starting where I have large customers that have large controls engineering staffs. Instead of having one, they might have five, 10, 15, 20 people and they're recognizing like.

45:25
Okay, well, if we have 10 or 20 automation engineers, we probably should have like a controls networking person. It's like one of the first times in my career that I, one of them came to me for help on like writing a job description because like they're using us for like outside consultation, but like we can't be there 24 seven people. So they're like, what would it look like for us to hire someone into this role? But that, that, the idea of that even existing as a job is very, very fresh and new. Is

45:53
I had a question about security, but that's probably a whole other rabbit hole. I do want to lean into that. I didn't want to have a conversation about OT without talking about security. Josh, you had mentioned earlier that these, these networks, these OT devices going down can be very detrimental from a, from a revenue standpoint of companies. Now it's been around for a while, but I would say ever present.

46:18
over the last few years, especially with attacks like ransomware and things like that, that you keep hearing the phrase, especially from government agencies, critical infrastructure. We're talking about things like power companies, water treatment centers. You mentioned earlier about how there are use cases for OT and IT networks connecting for things like data transmission for data analysis, potentially support.

46:48
One of the big things that has come up over the last few years is remote work. We want people to be able to remote in and help troubleshoot things. You gotta have a link to the outside. So what are your thoughts on security and segmentation for OT networks? So I was wrong earlier. I do have this slide that I hoped to include in here. So Andy, so Tim, this goes to your question and Andy, this may help with what you are trying to put.

47:16
tie together in your brain at the beginning of this conversation. If down here is what I'm calling the OT network, maybe we... So do you guys recognize this architecture, by the way? Have you seen some version of this architecture at any point in your career? The term that gets... This gets presented a lot as the Purdue model or the Purdue Enterprise reference architecture. And it's kind of like this guideline. I don't want to call it a standard because one, like I've rarely seen it.

47:45
directly implemented this clean. But it does provide kind of a guideline framework architecture for the assets in your OT system and how to architect them into an IT and enterprise network in a secure fashion. So what you'll notice down here is in this example case, we've got one firewall between like this level three where we've got...

48:09
things like the application servers and database servers. And then maybe up here at the full level three, we've got this demilitarized zone, this DMZ that is, yeah, this is where assets sit, where there's maybe shared data between OT and IT. So we can imagine that this firewall is pretty important. This firewall is controlling like ingress from the IT network into this DMZ and likely is not allowing any traffic to traverse directly from IT into the OT network.

48:38
So if there's any need for like shared resources, they would typically ideally be designed into a DMZ like this. You'll notice there's a remote access jump server in this DMZ to go to that question. Again, everything in this picture is ideal. Is this what customers have deployed? You know, what percentage of the time? Like, it's good. These are great aspirations. It is rare that I see this type of deployment. And I should mention too, like we work on

49:06
tiny little systems for small companies to we've been on billion dollar capexes for Fortune Five. And you would think, I keep waiting for the project where I'm like, I'm going to meet the company that just crushes it. We went into one a year and a half ago with a Fortune, to not give it away, I'll call it a Fortune Three. And this was a...

49:32
$1.4 billion capex project, this company, through this one facility, it's an air hub at an international airport domestically, where they have a hundred of their planes come in and like they can facilitate a hundred planes at a time and they can do a million packages in a day. And I was thinking going into this, and we got hired as a third party contractor to help like the lead system integrator with the OT side of the network for this project. And I was like, oh man, dealing with this end user, they're gonna be...

50:00
so locked down, like this is going to be amazing. And there were some things that were better than typical, but in general, what was funny is even their IT department, they basically told the integrator, Mr. Integrator, we're going to deploy these ISA 3Ks at every single one of your panels. And there's going to be a cable up. The ISA 3K is basically the ASA in an industrial box. So we're going to drop these little cables to these ISAs and we're going to facilitate the data we need through that ISA. And then below this box,

50:29
You do your thing. Like we don't wanna know.

50:34
And so yes, there is like a global control specification that this company has written that these integrators are meant to follow. But as I worked on this project, I kept running into things and I'd ask my, like our lead on that project. I'm like, I just found this, like, I don't see this anywhere in the spec. They're like, oh yeah, that was an exception. And then over the course of a year and a half, I found that the system was like 98% exception and 2% spec. And that's familiar. That's even in this.

51:03
Even in this billion dollar capex where I thought I would find like the perfect situ I'm like, Oh no, this, if this is a disaster here, like no wonder we're so busy, like this is a disaster everywhere. So this architecture again, like is this good framework to address what you're talking about from a cybersecurity standpoint? It is a, it, it takes, I mean, just looking at this, you can picture what it takes to deploy this, right? Like that requires a mature organization with resources to do this.

51:33
And I would say a lot of that's it and OT expertise, having both of those and having them work together, which I think has to be difficult. Yeah. So like the top half of this drawing, I'd say like, well, yeah, that probably isn't a lot of organizations. The bottom half is where this starts to get sketchy because like this firewall, for example, like who's who's responsible for this firewall? I already said earlier that it's going to be hard enough for an automation engineer to tack on.

52:01
networking expertise on top of the 9,000 other things they have to do. And now I'm saying, not only do I need to teach you about managed switches and VLANs and redundancy, you're going to have to learn how to maintain a firewall. That's a lot to ask of that automation engineer. So they have a huge gap in networking skills at these places. Is it because they either don't want to or don't have the resources to pay for a knowledgeable, dedicated...

52:27
network person, because I'm thinking of the place you just talked about, and I don't know specifically, but considering the size, I mean, was it, was everything a one-off because that was just technical debt over time? Like where I'm really getting with this is like new Greenfield stuff with a big budget, would it look like this drawing? Would they have a dedicated network person to help or they're, they're still like, they're just building weirdly, right? Oh, so here's the part I left out of that story. Okay. So I said that the end users, IT department said like.

52:55
we're just gonna drop this ISA in at every panel, y'all do whatever you want. And there's a hundred of these panels on this job, right? So there's a hundred of these ISAs under one roof in one facility because the way IT wants to manage the connection to their systems. But then below this, the question is, okay, well, there's gonna be an OT network below these ISAs. What does that OT network look like and who's creating it? So again, in this case, there was a system integrator hired to design, procure, configure this system. That system integrator.

53:24
You can imagine they're a large company if they're facilitating an end user this large. Hundreds, I'm sure it's thousands of employees on this project. There was like, I'm pretty sure triple digit controls engineers working on this project for a year and a half. They have an IT department. And so you would think like, okay, they're going to tap their IT department to help them with building the network for this facility. Their IT department said, we don't want to touch OT networks. We don't know it. We don't like it. You all always yell at us. We don't know what it is. Go find somebody.

53:54
And that's how we got pulled in is like the integration manager on that project, like found me on LinkedIn is like, can we have a phone call? And he described to me what was happening. I was like, oh yeah, this is exactly what we do. Like we get pulled into these situations to fill that exact gap. And if again, that's another value there. That, and that's another example though. Like if on a project of that size with an end user of that size, with that integrator responsible for that job, if they're also told within their organization, like by their IT department.

54:23
don't want, don't know, it's different, go do it. And they're having to hire outside help. You can imagine what it's like for these, forget the fortune too, what about everybody else in the country? This is a complete nightmare for everybody else. Your local municipality for water, forget about it. Even the people that are, the private companies, the manufacturing companies, this is a major struggle for all. One of our customers is recently venture funded with...

54:51
$750 million to solve battery recycling. It's the ex-CTO and co-founder of Tesla is their CEO. And this company, I mean, they're moving so fast building out these facilities. And they're an example of one of these organizations that I'm talking about that like, well, they've got a lot of controls, they got a lot of bodies. And they're one of the first ones kind of navigating this thing with me of like, okay, help us get started. And then what would a resource look like?

55:20
Like we have some IT help and we have some IT support, but they get the similar thing, right? Like IT doesn't understand the unique things about the OT network. They're like, I think we need to hire a body of our own that's an OT networking person. And the challenge there is, because this is sort of still fresh, that companies are recognizing this as its own discipline and a need for this, because it's relatively fresh, this is why it's drastically underserved. It hasn't historically been a role that organizations have had.

55:50
So nobody has gone to school. And I mean that even just professionally, like no one has worked towards this role because the role hasn't been available. So I'd say kind of the space is still, even though I'm 14 years into it, I would say it's just starting. And to be fair, you're not a networking professional by trade, right? You're an electric engineer, you got into this stuff. I mean, not that you can't manage a network, but you seem to be the OT guru for like, how to fix this stuff. And networking seems to be

56:20
Oh, that's a great point. One story I tell, like when I used to hire at our old company, I would do interviews and I would tell these people like, because a lot of the people I did, like one of the questions is do we interview, excuse me, do we interview a controls engineer and teach them about networking? Is that who we want to hire? Or do we hire an IT person and teach them about OT?

56:42
And there's, there's not a clear answer to that question. Like it's always automation and networking. It's the same. It's the same thing they've been, we've been doing for a decade or two. How do we, so how are you guys doing it? Well, what I would tell these guys is, uh, look, even if you've never done networking, like within about two months of us just doing some, like on the job training, you're going to know more about this than 99% of people who ever call in here asking for help.

57:11
and you're going to feel like a wizard doing what you feel like are very basic things. And my favorite thing is when one of those new hires would come in in about a month or two months and they'd stroll into the office and they'd be like, I just get off this phone call and they asked this question and I told them this and they were so excited. They were so happy and they said like, I was a wizard. I was like, what did I tell you? And remember my comment earlier about like, I feel like what we do is less than CCNA level work. Like,

57:37
I mean that like we're doing like basic blocking and tackling, but these systems are so immature from a networking standpoint that that basic amount moves mountains for them. Wow. Yeah, that's telling. The world is really held together by duct tape and superglue. Absolutely. Huh. I got one more question in the security vein before we wrap this up. Sure. And it's about potentially tying IT and OT networks together. So yeah.

58:05
I'm interested to get your take on, do you think software-defined networking potentially changes the ITOT game at all? What I mean by that is we have these vendors that are coming out with these SDN products that you can plug a device into the network, it can hit a AAA server, it can be profiled and it can even go into separate VLANs or even separate virtual routing and forwarding instances, separate VRFs from anything else.

58:33
Do you see that potentially changing the game at all and that we could have OT devices more on IT networks because while they're not physically separate, they can still be logically separate? Okay. I'm so tickled that you asked about SDN. So for me personally, as an individual, SDN is the most exciting thing I've seen in my time in this space. I mentioned earlier that I'm just a...

59:02
I'm a gear head. I'm constantly wanting to play with new platforms and new tech and figure out where it fits. What I'll say is I don't see any time soon SDN solving the convergence issue, but I do see the potential for OTSDN. So it's still maybe an isolated network, but if it's SDN instead of traditional networking, that completely changing the game on this diagram we're looking at.

59:30
OT cybersecurity has also gotten a lot of increasing attention and funding and new startups in the last several years. The challenge is, I'm telling you guys that these customers are so resource constrained that they can't even stand up a managed network. We're trying to also get them to stand up a managed and highly secured network where a lot of these OT security startups are struggling.

59:55
They've taken all this funding. They've got these platforms to do things like visibility and monitoring and detection in the OT space. And then they're going into all these meetings and finding out that their customers have flat networks. They don't have managed switches that they can span. They can run span ports off of to feed these sensors. So all these vendors are like, uh-oh, like we can't deploy our system because they don't even have a network that's mature enough to deploy our system into yet.

01:00:20
So SDN is one of the only things I've seen where just from a high level technology standpoint, I could picture how the model of how these systems are maintained today, meaning by a controls engineer. There's two vendors today developing solutions that are specific for OTSDN. And one of them, for example, I shared a demo on my LinkedIn a couple of months ago. It has this GUI that you're in the controller.

01:00:46
and you see the assets, right? Like I see the PLCs, I see the workstations. And in this demo, he is just point and clicking two assets, picking out of a list, the specific industrial protocol they need to communicate and hitting allow. And this is in a redundantly path system. And we see the thing go from not communicating to communicating. And prior to SDN, anything like that level of like lateral control in a system, like it's practically unachievable. Can you deploy ACLs on switches? Sure.

01:01:15
based on everything I've told you, is that ever gonna, in what universe is that practical or scalable? It's not at all. So if someone can deliver the right solution that combines ease of use with what SDN is capable of, I absolutely think it could be a complete game changer. What's funny about this, the reason I laugh when you brought it up is, so I've been geeked about this for four years now. That's when I first got introduced to one of these vendors working on it in OT.

01:01:45
It's not here at scale yet. Like none of these vendors are have a bunch of production installs. They've, they've got some, but it's so early for everybody that like for the entire rest of the space, they're not even ready to hear this yet. Well, one of the things on my list is, uh, as far as differences is deployment life cycles, what would you say is the typical IT life cycle for equipment? Three to five on a good day. Yes. On paper.

01:02:14
Yeah. On paper, yes. On paper and depending on the majority of the organization. Exactly. Yeah. In OT, I would say like traditionally that number has been 20 plus, you know? Like that number is getting smaller. It's been shrinking, but there are absolutely like Hirschman industrial switches that have been installed in plants for 20 years. And that switch hasn't been touched in 20 years. And the same can be said for their controllers, like their controllers are running for 10 years, 15 years.

01:02:42
So these deployment life cycles are very different in these environments. And so when we start talking about new tech, as excited as I get about it, like I have to be realistic about traditionally, historically, what has been the adoption rate of new technology into OT. Like OT is the laggard. Like I think I said earlier that like I'm teaching people about VLANs in 2023. And we go into systems where customers have recognized we need to do, we need three different networks here. And so...

01:03:11
I'm looking at pictures or I'm doing a physical audit and I'm like, why are there three switches in this panel? And they're like, oh, you're going to see three switches in every panel. Once for this network, once for this network, once for this network. And I was like, did anyone ever talk about VLANs? And sometimes the answer is no. And sometimes the answer is yeah, but nobody here knows how and it was just easier to do this. And that is not an uncommon thing too. Some general recognition that like...

01:03:38
Well, this went bad when we combined all these things. So let's separate it. So let's buy 3X the hardware and deploy three separate physical networks in these same panels. And when you show them like VLANs, it's like, oh, well that enables us, we could, you can just watch their gear start turning. And I love it. It's- What the hell did anybody tell us this before? So that's what your whole, I mean, is that the whole mission of your company, basically? Is this what you guys are doing? Absolutely. And I mean, even that trait, like, I don't-

01:04:07
Um, like training isn't like a big, I don't promote our training a ton, but what does happen is anytime we have a new customer engagement, I advocate that we do that training as soon as possible because it completely changes what our conversations are like over the next six months, we're able to have a more mature, meaningful conversation. Once we're using that same language about managed switches and V lands and building out.

01:04:34
redundant rings in their facilities. And we talk about network management and integrating that data into their SCADA system, like there's, there's so much raising the bar to do to just get on a common, you know, kind of common platform to talk about how to improve their system that we often encourage, like, can we kick this off with the training? I know we're, I know we're wrapping up here, but I'm curious, how, are you, are you local or your regional? How far will you guys go to help a company?

01:05:01
Yeah, so we're based out of just out of Dallas, Texas. And today, like we bubbled up to on large projects with subcontractors and stuff to like six, seven people. But today, like it's me and one other full-time engineer, but our projects have gone, yeah, have definitely been all around the country. Another interesting point is a lot of our work has been remote. That huge project I talked about, we did 3000 hours and we went to the site maybe three or four times for a few days each. But the vast majority of those 3000 hours

01:05:30
working from our houses. We were configuring switches remotely and then working with people remotely to check on things or make small little changes. And once we completed that project, it was the perfect example for me, for any other customer, it's like, man, if we could do that project at that scale, mostly remotely, there's very little we can't get done remotely. I'm impressed that you saw a need and filled it and had the gumption to...

01:06:00
to go out and do it. Why'd you choose the name Traceroute? Oh man, when I was, first of all, it makes me so happy to hear you say Traceroute. You'll notice our logo that I even did the two tone to separate the trace and the route. Despite this, I regularly get, including from people who work at like vendors and networking, I'll get the question, oh, when did you start Traceroute? I'm like, I'm sorry, what? Sorry, what?

01:06:29
I can appreciate the root one. I got plenty of people, I got plenty of people in Europe and vendors and stuff like that. The root one's fine, but the tracer route one kills me. That's good. I never heard that one. When I was thinking of ideas, like, you know, my wife and I were kind of bouncing names back and forth. And I was thinking about when I work with new customers or when I work with a new engineer, this like general troubleshooting process for a problem and how often I would find myself in, you know, we mentioned that often.

01:06:57
what we're going in and doing is doing like kind of the first managed segmented network, which means maybe the first time, not maybe the first time this customer has had a layer three switch in the system that is routing between these different VLANs. And so often we would have these systems go up and either the end user or a new engineer is calling saying like, okay, we think we lit it all up, but something's not working here. And so inevitably in this list on my list of questions for them is like, we'll have you run a traceroute.

01:07:24
And so I was just thinking about like just the analog to like, yeah, just taking a problem and having some diagnostic to figure out where the failure is and how to go fix it. And I thought it was a good fit for a consulting company for, for networking. It's a great name. Yeah. I don't know if you have any competition, but Tim, if you want to start a company with me as a competitor, we'll call it Ping. We'll be Ping. They'll be Tracer Alden. It sounds like there's plenty of work to go around.

01:07:54
Oh man, there is a, do y'all know Thousand Eyes? They're like a Cisco partner. Yeah. They, at one of the Cisco, at Cisco Live a year ago or two years ago, they were handing out stickers. And this is, this was like one of their taglines for a while, which I don't want to cuss on here, but it was basically F Trace route. Oh, good one. And so some people at Cisco Live brought this sticker back to me. They're like, what did you do to them? I was like, no, this isn't about me.

01:08:24
I got some of those stickers somewhere. You know, you're doing it right when a thousand eyes is cussing you out. Yeah. That's awesome. That's really funny. Well, Andy, you were not kidding. This could have been a three day show. I think, Oh, Josh, I got to tell you, man, you know, I don't know if we can have you back someday or what, but I feel like we could just keep, keep going and digging. And this is really, this is the most fun I've had in a while, just because it's also new, you know, a lot of times we talk about

01:08:53
Not the same thing here, but man, OT is just, I mean, there's networking in there, but wow, what a different world to, to, to be in. Yeah. And it's the reason it's one of the reasons I'm so passionate about it is because not just underserved, but literally like, yeah, there's just so much opportunity to help people. And so, you know, like I said, talking to you guys, talking to your audience, like what I want to flag is, Oh, there's a big unknown untapped area to make a big difference where there's not a lot of people doing it.

01:09:23
You know, and like I said, even within the organizations you're in there, you may not even realize it yet. And they, your organization may not realize it yet, but like befriend that controls engineer within your organization. Take him to lunch, take her to lunch, ask them some questions. You may find out when you guys start talking like, oh, there's an opportunity to maybe improve their own station, improve their situation. Because maybe that organization hasn't even realized yet that like, you know what, we're big enough. We're having all these problems all the time. This person could, we could create this new job role.

01:09:52
for this person, it would help us tremendously. Like I think this is going to increase just we are on the very beginning of this hockey stick of these organizations recognizing how big of a problem this is and how big of a need this is. And to me, that's the perfect time to just be watching it and checking it out within the company you're at, the companies around you pay attention to the job listings, befriend some of these people. And you know, you may blink and find all of a sudden, you know, you're an OT.

01:10:21
So cool. Take us home, Tim. You taking us home? Who's taking us home? Yeah, sure thing. Josh, thank you for having this conversation with us. Where can people connect with you online? Where can they find your work? I'm on one social media platform and it is LinkedIn and I am noisy on there. So for sure, absolutely connect with me on LinkedIn. Just look me up, Josh Varghese or Traceroute and you'll find me. I'm pretty active on there regularly, kind of posting content about both products and projects.

01:10:49
troubleshooting activities. So if, if you're interested in kind of keeping tabs on like what's happening in OT networking, uh, happy to be your connection. I was going to ask you because we threw a tweet out there advertising this episode and I'm like, does he have some obscure just Twitter handle? I cannot find this dude, but that, that answers my question. Okay. Yeah. The industrial automation community somehow, um, is incredibly active.

01:11:19
on LinkedIn. They are there. They are present. And it's been a huge boon for us. In five and a half years, I would say 95% of our projects and customers have literally just been people finding us on LinkedIn and growing that community on LinkedIn and over time becoming the guy through one or two degrees of separation. Everyone's like, oh, OT networking thing?

01:11:43
This is where to go. So that's where I live because LinkedIn has been good to us and I'm good to LinkedIn. Well, thanks again, Josh. Thank you for having me for, uh, for listening to this show. You can find us at art of network engineering.com on Twitter, other social media platforms, just search art of net eng, be sure to check out the cables to clouds podcast. We're really, really excited with what that team's doing.

01:12:11
dot cables to clouds.com. Find them on other social media platforms as well. Thank you all for listening to this episode of the art of network engineering. We'll see you later.

01:12:25
Hey everyone, this is Andy. If you like what you heard today, then please subscribe to our podcast and your favorite podcatcher. Click that bell icon to get notified of all of our future episodes. Also follow us on Twitter and Instagram. We are at Art of Net Eng. That's Art of N-E-T-E-N-G. You can also find us on the web at artofnetworkengineering.com where we post all of our show notes, blog articles, and general networking nerdery. You can also see our pretty faces on our YouTube channel named The Art of Network Engineering.

01:12:55
Thanks for listening.