Introduction

SPEAKER_00 0:00

This is the Art of Network Engineering. Where technology meets the human side of IT. Whether you're scaling networks, solving problems, or shaping your career, we've got the insights, stories, and tips to keep you ahead in the ever-evolving world of networking.

SPEAKER_01 0:13

Welcome to the Art of Network Engineering Podcast. My name is Andy Lapteff, and I am joined by the man, the myth, the legend, the person we all know because he's always at all the events, including where we are today at Networking Field Day 41 SCOROBOM. Hello, Andy Laptoff. Good to see you, man. How was that intro? Pretty good, right? I I better than I deserve. Better than I deserve. So right before we start a recording, like, what are we going to talk about, man? And I think we have to talk about AI. Yep. Sorry. Because I like what are we going to talk about? PGP? Hey, let's uh you know, we've I feel like our industry and all the content creators and the podcast, like we've we've talked about all the things. If you're a route switch person or an automation person, like you can find content talking about the things. What is the big thing happening, obviously, is AI. And it's so hard to figure all of this out. So for me, if I like I put myself in that cable guy truck studying back in the day, how do I navigate all this? Right. Because you've been in the industry a long time. You've had all the jobs, and like people like you are still trying to figure out, right? Yeah. What does this all mean and where do we all go? So I guess let's frame the conversation. Like, what how is this a different conversation than every other AI conversation you've heard?

Building trust in new systems

SPEAKER_01 1:19

What are we going to cover here?

Scott Robohn 1:20

Let's talk about building trust in new systems. Right. And how do I do it explicitly? And that's not just an AI thing, but it's important in the AI moment, especially the agentic moment.

SPEAKER_01 1:30

Trust is like an interesting. I always compare it to my own experience. Like, I didn't understand a lot of the programmatic stuff. So I don't necessarily think my constraint was trust. Right. But people that understand development deeper than I do, there seems to be a trust component. Sure. Like I can break all the things fast. How do I trust it? Right. Right. There's been some cool technologies that come along, like digital twin and pre-validation and like Git. So I guess that's the answer to some of that. But trust is an interesting, you know, like in software dev, I remember my old jobs, you had like a development environment. Right. And they could, you know, spin up a branch and test all the things and see if it worked and then push to production where in networking, traditionally, it's like it's physical infrastructure.

Scott Robohn 2:12

Yeah. It's not the same.

SPEAKER_01 2:13

And you're like labbing in production, kind of. I mean, I know that's not ideal. Right. And there's there's other solutions. So trust is like, I think that's why networking is so different. Not that we're saying anything unique here. Right. But you blow up an app, okay. You blow up the roads that the whole company uses, right? The blast radiuses, everything. The bridges out, right? Yeah. Quite literally and figurative. So grounding trust in the network, right? Right. Like the infrastructure, you have to trust it. It has to work. Sure. If it's working well, it's not even um, it's like transparent,

Ethernet interoperability and vendor bake-offs

SPEAKER_01 2:42

right? It's just yeah, yeah.

Scott Robohn 2:44

So so there's some really interesting analogies here, right? If I take a take a walk back with me, Andy. Let's go to the 90s. Right. And um, so Ethernet wasn't always Ethernet and defaults weren't the same, right? And whether they were auto-negotiating for full duplex or set for half duplex, you know, different vendors had different defaults. And so what you would do was test things and make sure at least you knew vendor A behavior and vendor B behavior, so that when you had a mix in your network, you knew what the things were going in and you knew how to set things. So it worked. Is that interop that's and and we we stand on the shoulders of interop giants, right? We did a lot of work with network infrastructure for decades to now make sure Ethernet pretty much just works. OSPF pretty much just works, right? There aren't a lot of flaky variants between vendors and implementations. The now that I've said that, you will have um viewers come in and say, Well, I know of a situation where I have a a keystone example that I think about on this, right? One of my big network operations architecture engineering jobs was at Bell Atlantic. And we went from old school Bay networks, well fleet routers, and I think Cabletron hubs. Like those were that was our campus infrastructure. And we said it was time to go with somebody else. So we brought in four vendors. You know, we didn't just say, yep, Cisco is going to be fine. Yep, IBM is gonna be fine. You know, we we basically wrote test plans, figured out we had an architecture that we were um looking at for the network, and we ran every vendor through a very explicit series of tests to know that we could trust them when we put it in the network.

SPEAKER_01 4:31

Would that be called a POC, like a proof of concept with the vendors?

Scott Robohn 4:33

I would say this is even um, we would call it a bake-off. Okay. Right. You know, again, you know, four vendors head to head against each other.

SPEAKER_01 4:40

Do they give you infrastructure? Like how do you test their you get loaner, loaner equipment, right? So you bring it in, you stand it up, you're running.

Scott Robohn 4:46

We brought it into we brought it into our lab, uh the data network control center in downtown Washington, DC, which is really cool place. We had all sorts of vendor kit there for all the different services that Belt Atlantic supported. But you know, we didn't pick a vendor ahead of time. We brought in four and kicked the tires really, really, really hard. To establish trust.

SPEAKER_01 5:07

To establish trust, right? In the new thing you might want to do. Was it a new thing? Like you, I guess. So what I wanted to ask is part of like end of life, right? Like we're replacing Adran with whatever.

Scott Robohn 5:15

The big new thing we were implementing at the time was ATM as a backbone technology. Yeah, don't laugh. There was a time I worked on that stuff. Um and so we basically had you know closet switches that had Ethernet termination, but ATM um backbone connections to a big core network.

SPEAKER_01 5:33

Now, that was the new stuff or the old stuff? That was the new stuff. Okay, so the new stuff was ATM coming in. That's right. We need a bake off because we don't want to go to this new equipment and these new systems and have everything fall over, especially in telco. Like it has to work, right? Yeah, exactly. Commission critical way.

Scott Robohn 5:45

And even though this was the IT network for a telco, you're you're a telco. You know how to provide network connectivity. Yeah. Right. You're not going to you know just willy-nilly, you know, pick with some pick some other vendor, right?

SPEAKER_01 5:56

I know I know this is a trust conversation, but just to dig in a little bit there. So when you have the bake off, I don't know how to test applications necessarily. Like what are you measuring to see if uh you can trust the solution, right? So that's a really funny element of this conversation.

Scott Robohn 6:11

You can understand your application environment to a certain extent, right? You know, ports and protocols, right? So I know that I have this training group in this building that uses X Windows for lots of remote um training delivery. That's another story for another time. But like, so I know that okay, I need to support, you know, X Windows over this, you know, type of environment with servers in this location and clients in another location.

SPEAKER_01 6:34

Question. So are you do you have validators that are testing applications, or is it basically like network connectivity? Like, oh, I see the route. I can ping the thing.

Scott Robohn 6:42

Um probably one level beyond that, right? Like you know, having some test traffic.

SPEAKER_01 6:46

But you're not for all four bake-offs? Because where I'm getting is like you have to keep pulling these people into these windows to test, right?

Scott Robohn 6:52

So so your test plan is gonna be applied to each of the new vendors, right? And here are the things that I need to see, this minimum set of requirements that I need to see working across your kit and your kit and your kit and your kit before I'm gonna say, okay, let's go to a limited production trial.

SPEAKER_01 7:08

Okay. And that's all to establish trust in the infrastructure. Correct. So trust is huge in infrastructure. There's a process you go through. And that sounds I'm going somewhere with this because AI, everything goes fast, and we're talking about I think trust.

Scott Robohn 7:21

Faster than anything we've ever seen.

SPEAKER_01 7:23

Right. So uh the speed seems to be the big variable that I keep hearing in these different conversations. I I I try to ground in in history to sure, you know, as as people, right? Like look at history and what have we seen, like the industrial revolution, um atomic energy, things that were disruptive, potentially uh whatever, dangerous, and and we got through. I I keep going back and forth. You know, you have I forget the guy's the gentleman's name, but um, he was like the grandfather, godfather of AI. He worked at Google, he left because he was so concerned about and you know he this guy who I think created a lot of this, right? He's freaked out, right? Like without safety, like like end time stuff, right? So you have and and Tim, but this is the guy who created this stuff. Now, maybe he's just older and losing his marbles again. Like, I don't know. He seems like a really smart guy, but then there's the other side of the table that's like how I've experienced this in my life, it's really been an enabler. So it's so hard, which is why I like these conversations. It's so hard to try to figure out sure is this the best thing since sliced bread for all of us? Or is this gonna like end all the things and the speed? So getting back to like so when you were going through the um bake offs, how long would that take? I guess it could be weeks, months, or four different vendors. Like, this is a long process, right?

Scott Robohn 8:38

I think it was a probably a couple weeks per vendor. So it was not an instantaneous, you know, click your heels, there's no place like home.

SPEAKER_01 8:45

So maybe a month per vendor, four vendors, yeah, maybe six months by the time you start all the calls, plan this stuff.

Scott Robohn 8:51

Like, yeah, that if if my memory is fuzzy, but I I think it took less than six months.

Adoption pressure versus production responsibility

Scott Robohn 8:57

Okay. But to your point, with things moving so fast today, I think a lesson learned here is whoa, whoa, whoa, whoa, whoa, whoa, whoa. Right. And like, you know, the the media environment, the social media environment, LinkedIn, and so forth, right? We've got so many voices telling us this technology should be used for everything that you're doing right now. And of course, those voices don't have any production responsibility in networks.

SPEAKER_01 10:28

And those voices, I'm assuming, are also um like follow the money. They're motivated by the adoption of this, right?

Scott Robohn 10:36

And to give them the most slack possible, they're doing their job, right? They're trying to get their message out there in a very crowded environment, noisy environment, noisier than it was in 1996. When we say they, we're talking about like the frontier model folks who are trying to push this out or and networking services companies that are saying, yes, you should use these things for that part of the equation.

SPEAKER_01 10:55

I hate to cut you off. Are they monetizing AI? Because it from what I can tell, it's just an add-on to all the existing things. Like you don't pay a license for the AI feature, right? Sure.

Scott Robohn 11:05

Well, I think that's an it depends. Like, how do they get the return? First of all, network engineering, Andy. It depends.

SPEAKER_01 11:09

And that's a whole different probably conversation. Like, I don't understand how we this year alone, how do we get a return on $625 trillion in infrastructure investment? I don't think you do. Yeah.

Scott Robohn 11:17

Well, it doesn't happen immediately.

SPEAKER_01 11:19

Unless all the humans are replaced, and let's not go there. It's a different thing.

Scott Robohn 11:21

Well, it is it is any longer conversation. Like, what does the ROI look like? Right? I'll come back and we'll talk about that another time. Maybe we'll get Bouchon and chat through that.

SPEAKER_01 11:29

But why are we so there are people who are motivated to get us all excited about this and how it's going to help us? I guess they're motivated by uh, I don't want to say money, but like well, that's part of it. That's that's part of it.

Scott Robohn 11:42

But there's also potential. Like, and you and I have done enough stuff to know that there is real potential for many use cases, not all of them, and it's not fairy dust. It doesn't make everything better or easier. And we're in this really, really interesting time of okay, what can it really help me with? And what does it make sense for given the state of the technology, my increasing comfort level with it, my ability to write the rules for the, you know, so the lanes are well defined and the guardrails are put in place. I think the tension is really important. People pushing features and capabilities and us saying, I need to know how to implement it responsibly, responsibly. I think that's a real, natural thing. And it's not the first time we've seen this. It's not just an AI thing.

SPEAKER_01 12:23

But this speed, and and again, I'm I try to stay as aware of my biases as I can, which is tricky because you're they're usually subconscious, you're unaware of them. So I but I always hear Bouchang in my head, not to make this the Bouchang show, but you know, like we're all driven by subconscious stuff, system one thinking, condiment, and all that. So we are being pushed. The the tension between the speed, this is amazing, you gotta get it, you gotta do the thing, and how to establish trust. And if it took Ma Bell five months to gain enough trust in just a network solution, right in in you know, one area location business unit, I don't see that level of care and deliberate, you know, thinking through of things with this new chase to thing. And it reminds me, again, I'm always looking for historical. It's so funny. I hated history as a kid because the way it was taught it was so boring. And now you found it's very useful. Well, the history channel came out. I'm like, whoa, this is like a movie, but it happened, you know, but it's just not how it was taught back in the Catholic schools of this is the date, here's the person, memorize it. But it reminds me, at least it it it it rings of like atomic energy. Like there was a race. Sure. And we had to beat who was it? Was it the Russians?

Scott Robohn 13:38

I like whoever we were we were concerned about the Germans and the Soviets.

SPEAKER_01 13:43

Yeah, so we we had to race to um you know figure out atomic energy. And I forget the gentleman's name. I don't think it was Oppenheimer, but he's like the famous guy from the movie. But there was one of the smartest people working on the thing, and they weren't sure the first time they tested it if the reaction would stop.

Scott Robohn 14:01

Right. Are we going to burn the uh atmosphere? Right.

SPEAKER_01 14:04

There was a non-zero chance that testing that first atomic bomb could end civilization. Right. It would literally react into because the chain reaction. Again, I don't understand this stuff, but the atmosphere would be gone. Good luck without an atmosphere. And the reason I bring that up is now we're in this race because we have to beat uh another the Chinese.

Scott Robohn 14:28

We we we the Chinese are often cited as this is the other global entity that we're up against. Yeah, yeah.

SPEAKER_01 14:33

So so there's this us versus them thing happening. Sure. And right now it's China. And I I guess when I look back at history, like uh again, this is the trust conversation. When I look at history and I look at atomic energy, we raced until we won, quote unquote. A lot of people were harmed slash killed when we used it twice. And I don't know if it was the horror of that that then created this slow down safety, like, okay, maybe we should have protocols for missiles. Like, I I don't know.

Scott Robohn 15:08

Do you think is there a question, Mr. Laptop?

SPEAKER_01 15:12

No, we're going so fast in AI. We went so fast in atomic energy. We went so fast until we killed 200,000 people in 45 seconds. And in my mind, if I were there, maybe I'm some overly empathetic, I would go, oh no, wow, this is like awful. You know, maybe we should slow down. Um, so in AI, like in my mind, we're just gonna go fast until something happens. I don't know if the cyber attack, I don't know if the robots take over at Star Wars, but in my mind, looking at history, right, a bunch of people may be harmed and then we'll finally go, oh. Now I know there's a lot of talk right now and trying to slow down and get some safety in place. I see the Frontier folks on the news talking about this. Yeah. I don't the cynic in me is like, I don't know if they're just it's a good thing.

Scott Robohn 15:57

Yeah, my cynic, my cynic, my cynic numbers are are high there. Right. Well, let me let me bring it to more contemporary, recent example. Certainly to table stakes regarding the destructive power of atomic energy. You you could argue whether or not that's a good comparison for what we're gonna do.

SPEAKER_01 16:11

What would be your historical comparison? How do you think about it?

Scott Robohn 16:12

So the event of social media, like so. If you think about the last 15 years, about you know, social media

Social media, trust, and agent permissions

Scott Robohn 16:20

just kind of happened, right? And we have seen over you know time really harmful effects on groups of kids. And the promise was it was going to connect us all. Right. And the opposite happened. Well, both can be true. I don't I don't think just the opposite happened. But there was harm. There was harm. And you know, whether we think about it consciously or not, we don't want that to happen again with this. There is already a little bit of the cats out of the bag here. Um, but that's a very recent and I think more analogous issue here with, you know, what's going to happen when my 13-year-old just wants to have a relationship with a chatbot versus re meeting real people in meat space, right? So those are those are real concerns that we have that are connected to this. If I were to pull this back and like all these extraneous issues, extraneous sounds dismissive, all these other issues outside of the outside of the scope of network engineering, architecture, and operations are important. I feel like the space where we can get our hands around things are, you know, this technology isn't going to force its way into my network. How do I set the right guidelines? How do I control permissions for agents? Right. We've got some good starting paradigms there, right? You know, I've got, you know, your user account at your company has certain permissions associated with it. That's probably a really good starting point for agents and much more reduced permissions and really locked down.

SPEAKER_01 17:47

Was there a race in social media? I guess they were all kind of competing for remember MySpace. Absolutely. And I I watched that uh it's a fascinating documentary, Social Dilemma. Right. And that's the first application I've seen of like AI being directed at us, trying to like hijack our you know, our brains.

Scott Robohn 18:04

Or even even just algorithms. Like you don't even have to talk about AI, right? A computational approach that knows how to exploit Andy's preferences. You know, click get a reward.

SPEAKER_01 18:14

Click get a watch what Andy clicks. Yeah. Right. And and I guess so it's a good, I like that metaphor. It's a little less scary than than than atomic energy. It's still really important. Well, it is. Did they ever? I'm not convinced today that social media, the the companies that run these things, I mean, all you got to do is watch like the Capitol Hill stuff, and like there's still a lot of harm, and there's still a lot of founders that go up on Capitol Hill and say they're gonna do the thing, and then yeah, you know, the people behind the scenes are like, yeah, the guy said the thing. So I I don't know.

Scott Robohn 18:44

I I I look at look at what it's even done with actors on Capitol Hill. And I I use that term intentionally, right? There are more and more performative um discussions in Congress just so I can get the 30-second, you know, segment out there on Instagram, on TikTok, and show how I'm, you know, I'm senator, I'm congressman, and I'm I'm doing good things to protect you, Andy. Um, and really not lift a finger other than publish the video.

SPEAKER_01 19:13

Well, that's kind of what's interesting. Like there's there's a whole uh social kind of um there's a social contract that I think we all operate under, and we believe fundamentally, I think, that leaders have the the the great the good of you know the common good above all else. Uh I I don't see a ton of evidence that that that that's true. And and and to bring it back to trust, you just made me think with social media, you know, we just give away like it's really amazing when I I heard this privacy paradox thing in this NPR podcast. It was really fascinating. They did this whole thing on privacy and the the history of you know surveillance and and blah blah blah, and the wholesale handing over of our entire lives, our pictures, our locations, our experiences to social media, if that's not a trust signal, right? Like, oh, here's everything. I I don't put pictures of my kids on social media just for reasons that seem you know relevant to me. Now my wife will, and I'm not gonna tell her not to. Right. But I trust this stuff, I guess, less than than she does. So I guess where I'm going with all this is I wonder with AI, just like with social media, if you can get it ingrained in the everyday, you know, the every Joe's experience, do you need to lead with safety or to have safety as a top priority when everybody's using it anyway? They handed over all of our data, they've all given up. There was a time that you would try to like protect your privacy. Right. And now we live in a world where everybody just hands it over for free.

Scott Robohn 20:44

It's a new information environment.

SPEAKER_01 20:46

Yeah, because when what is it, when the product is free, you are the product kind of thing. That's right. And this is based on these founder people who are like, hey, even so social dilemma, circling back to that. Um, the guy Tristan Harris, he's the guy who came out, and all the founders of these things, like the person who created the the like button on Facebook, like the person who created the little bubbles for texting, these people who created features that kept you coming, they all freaked out and like, oh my God. Yeah. Like social media is bad. We have to tell everybody. And they made this movie. So now he has this. I forget the name of the show, but he has this organization that's he's taken that and going to AI. So there's equal concern that it's moving fast, it's harmful, there's no safety. I know we're talking about trust here. So in a networking context, like I trust, I'm using this all the time. Sure. I used it to help us with this outline, which I have gleefully not looked at.

Scott Robohn 21:36

But you, you, you generated it, you QA'd it, you know the pieces you want to use and not use. It's not here as

AI as a thought partner

Scott Robohn 21:42

a third participant with us in the conversation. Right. So you have you have guardrails that you've established. That's yeah, that's a good thing. You're using it in a way that's that's useful and helpful for a workflow you're trying to accomplish. The human in the loop.

SPEAKER_01 21:53

I can parse what's important, what isn't, what doesn't feel right.

Scott Robohn 21:56

Thought partner.

SPEAKER_01 21:57

You've used it as a thought partner. In using the tooling, I've realized what it's good and not so good at. Yeah. And you and I had a semi-long text thread yesterday on our flights, and I'm tired and traveling a lot as you are. So we're both kind of loopy. And I just threw the text in there and I'm like, hey, help me pull out you know, a thread here for a show. And that's what it did. And it did pretty darn good. Yeah. Now that's what it's good at. Now, when I give it a general prompt, like, hey, write me a thought leadership blog on the importance of safety in AI, it's kind of garbage. Like, right? Yeah, I I for me, maybe because I've been a writer for so long, but I can just see. You know, it's different hallmarks with different models at different times.

Scott Robohn 22:34

Yeah, it's not this, but it's that maybe the one-liner thing. It's just so like yeah, and that'll change over time. There's a whole discussion

Critical thinking and reviewing AI output

Scott Robohn 23:13

here on content creation used to be hard. Writing code used to be hard. Those used to be the expensive tasks. Now they're relatively inexpensive tasks, especially if you're willing to accept a lot of slop. Right. Um, and now critical thinking and review is the expensive task and is the more important gating factor. Right. So critical thinking is back, baby. Uh And like I I view that as this is an opportunity for us to be really human, right?

SPEAKER_01 23:40

Can you define critical thinking? Because my uh my my understanding of AGI, which they may or may not get to, when you look at all the money being spent, more money than humans have ever spent on anything, as far as I know in modern m modernity. Oh, I never use that modernity. Break that down. Yeah, right. Um when I look for the return, the only thing I can come up with is they're gonna replace what you just said, critical thinking. Like if you could get rid of all these pesky humans that are super expensive and need time off and complain, because it's not gonna happen if if you take the population of the planet and charge them all twenty dollars a month for a subscription, I don't think you could get anywhere near a percentage of this year's investment. Right. And and maybe it's just like, hey, this is pushing humans forward, it's just an investment in the overall thing. Like, great, but again, the cynical mind in me is like, oh man. I think what what did you call cognitive what? Like the critical thinking. Yeah. I am afraid trust. We're talking about trust. Yeah. I distrust the possibility that critical thinking is back and that's what's important in the loop. Because the other I don't have any answers, right? But what I fear is we are using the thing because we are important in the loop, because we have to be critical thought people, and we're training these models to be better and better at what we do so that they can eventually replace us. I hope that I'm wrong. I'm probably a crazy tinfoil hat guy, but why would you spend this much money on something you can never return the investment on? You know what I mean? Like it's a business thing.

Scott Robohn 25:10

Yeah, and and there's a lot packed into that. Um typical Andy quest. Well, so and just for sake of trying to not have the conversation go everywhere it could, you know, in a limited amount of time, right? I take the position of I don't care what anybody in tech or Silicon Valley tells me, critical thinking is critical. Period. Like you need to exercise that as a human.

SPEAKER_01 25:34

Because that's what differentiates us from the apes. That that that's right. And you see how we treat the apes. What if something got smarter than us? Again, tin foil happens. There's there's lots of movies that address that. Um there's lots of founders that created this technology that are like, this is where we're headed. Yeah. Like this isn't my opinion, which is what scares me.

Scott Robohn 25:50

I would also take what these founders are saying with a grain of salt. I, for you know, at the risk of going down a rat hole, I don't understand why some of these leaders are allowed to talk to the public with some of the things they say. Yeah, yeah. I do think fundamentally we're talking about computational machinery. The fact that we started talking, we called it some kind of intelligence from the beginning.

SPEAKER_01 26:12

I've heard people say that was a bad that that's like that's a marketing term.

Scott Robohn 26:15

Artificial intelligence is really a marketing term. And it's a super intelligence and whatever else people decide to call it, right? It's a really interesting set of computational techniques. It's software development and software processes. But isn't it different? Like, I don't understand neural networks. Nothing is nothing exactly the same.

SPEAKER_01 26:29

But do you understand like the neural network stuff? Like, I really want to get like brilliant people on the show to talk about this stuff because like my limited understanding when they talk about this stuff is it's not traditional compute and software like a Stavon Neumann conversation, absolutely. Yeah, like neural networks, isn't it like creating kind of a digital brain?

Scott Robohn 26:46

And that's probably an overreach, but no, no, that's a good analogy. Yeah, yeah. Right. But you know, how well do how do how well does today's science really understand how the human brain actually works? It's a good point.

SPEAKER_01 26:57

We're going into uncharted territory, right? Yeah, yeah. And we don't know the similar to the other metaphors we've used. That's right. We're on the forefront, which is kind of exciting. Yeah. This is awesome. There are exciting things about it. Absolutely. Yeah, it's exciting, it's scary. So I I guess because it's a network engineering show. I have a way to pull this back. Well, yeah. So getting back to trust, and you are the guy to talk to, which is why thank you for coming on, because I have been speaking to you for years about network automation, and I think there's a I never know if it's an analogy or

Bringing the trust conversation back to networking

SPEAKER_01 27:25

metaphor, but I think there's a relationship between network automation and the establishment of trust in an organization to trust the tools to not melt the world fast. Yep. And maybe in my mind, AI is kind of like automation on steroids because I couldn't automate prior prior to the like Cloud Code as an example got so good. Yeah, I can create software to do things. Right. Some of it works, some of it doesn't. We talked about this, but it's more than I could have ever done previously. And it's very helpful. And I have kind of got some trust in there. So how do we trust how does this look for like networking? Because as far as I can tell, it's just gonna be an add-on in tools. You're not gonna pay extra for it. AI ops is gonna be there. Ta-da.

Scott Robohn 28:01

It could even be just how certain features are delivered, and you don't even need to care that it's AI under the surface. Right? So, so have an example of that? But so Python. Yeah, right. I can use Python to write scripts to automatically log in to a device, right? And I remember how happy you were when you got your first Python script to log into a device. That's right. But I can also use Python to plow through tons of data and do data analysis. Python is a tool, I can use it for different things. AI is not one thing. AI as a set of tools, and some of those tools can be used for network automation, but I can also use it to write um proposed scripts for a webinar or a podcast, right? So the question, how do I establish trust? First of all, what is the problem you're trying to solve?

SPEAKER_01 28:46

Do we need to establish trust or do we just like we accepted everything else, including social media? I mean, I think it's a dumb, you know, it's a provocative question.

Scott Robohn 28:52

We need to establish if you're if you're responsible for delivering services in an enterprise or a carrier or a cloud environment or a neo-cloud environment, yeah, you got to prove that this is gonna do what the vendor says it will do and what I'm gonna attest to it, right? Somebody's gonna be hold held responsible for purchasing that product that's AI-based, letting it loose in production, um, and measuring what am I getting out of what I decided to do with this.

SPEAKER_01 29:18

Is this different than network automation that has been a laggard? It's similar. It's similar. I'm not gonna say it's exactly the same. It seems more accepted though, like quickly, right? Because of the It's being pushed, it's being pushed much more quickly by people that benefit from pushing whoever that is.

Scott Robohn 29:33

Correct. Well, the the the vendors and the technology stuff. Weren't they doing the same with automation?

SPEAKER_01 29:36

And and didn't the industry push back because I can melt all the things too fast, I don't trust it. I think I think we're circling around trust still, right? Yeah.

Scott Robohn 29:43

So so this is a little um the best example I can give you is a study that I did with Lumen last year where they had a process, a multi-year

Testing automation workflows and handling exceptions

Scott Robohn 29:57

process for defining over a thousand workflows, testing them, automating and orchestrating these workflows one at a time, and not pushing them into production until they met certain criteria for each workflow. Automation workflows, I guess. Correct, correct pre-AI. This is even before the AI conversation. Right. And they took a deliberate database approach, right? Greg Freeman, he's uh shout out to Greg.

SPEAKER_01 30:21

So like we have to test this X number of times before we say it's good to push and all that.

Scott Robohn 30:26

And I need to have good error handling around that automation because I I wanted to do the same, I want to do the same thing all the time. But if I if I get the same result 99% of the time, and then I have good error handling for that 1% of the time that it does something that I don't expect and kicks it up to the knock operator to say, hey, this is not right. You know, we need human approval um before this goes at forward, right? You have a structure, you've tested every every workflow according to a test plan that's appropriate for that workflow.

SPEAKER_01 30:58

Do you want to hear something funny? I'm thinking of the software that I've tried to build to run my operations. Right. Uh, I don't think I tested it. And you get a little too vibe cody there, Andy. I was drowning, bro. Yeah,

Andy's software and silent failures

SPEAKER_01 31:12

sure. You know what I mean? Yeah. I can't afford to bring on help and you can only work 20 hour days so long until you're like, what am I gonna do? Yeah, yeah. So half of it worked and it helped, but out of desperation, I'm just laughing as you're talking because I didn't do the things you're saying you should do, which is test, make sure it's good, validate it. I just kind of threw it out there. But even error handling, I realize that I don't have any error handling. Yeah. So when something's failing, it's silently failing. Yeah. And then three weeks later, when I go to generate a media kit and all the numbers are wrong, I'm yelling at the AI. And it's like, well, dude, we've been telling you, and I'm like, oh, so so there's so much to know about software development, automation, and all that.

Scott Robohn 31:48

The concept of error handling is like that's fundamental and like it just makes so much sense once you get your head around it.

SPEAKER_01 31:53

Yeah.

Scott Robohn 31:53

Now having the time and the discipline to actually implement it, you have to know what a what an error looks like, I guess.

SPEAKER_01 31:58

Correct. Like what would failure look like to push it up to I mean, these software.

Scott Robohn 32:04

Take the take the mirror image of that. Okay. So I know what every here's every case that I expect. And if something happens that doesn't match an expected case, if anything else, then then I need to uh take a closer look at it.

SPEAKER_01 32:15

So you don't have to know the specific failure cases. That's true if it doesn't meet this list of things I need. Yeah, okay. Yeah. That makes sense. How do you know all this? You are years of You did take C, right? You I did.

Scott Robohn 32:25

I was I made sure I got a C. I was a Fortran programmer, baby. Um, and I made sure I got C because I kind of had a sense that that was going to be important. And it was for a time. So it still is.

SPEAKER_01 32:37

Performant real-time system. You teach me C? I failed at a C twice in college. And you've you've got LLMs you can ask, and they're actually really good places to start.

Scott Robohn 32:45

Don't we even need C anymore? The patient tutor, dude. So, yes, anything that has to be close to the hardware, the memory, the I.O. I I understand that many folks are moving to Rust for that because it has better memory protection mechanisms and so forth. But uh the idea of being close to the hardware for performance is still really important, especially for systems like routers and switches and firewalls.

SPEAKER_01 33:05

So I told you I was gonna make notes and not look at them. So I'm just we gotta have you out of here. Well, no, because I want to get you out of here. Um, in in full transparency, Scott was very patient with me. He was my first recording here, and he sat here for 40 minutes while I set up. So thank you for being such

Permissions, read-only access, and deliberate testing

SPEAKER_01 33:18

a patient friend. I'm just wondering, you know, as we wrap up, if I missed anything here, I like what you said about the permissions and RBAC and how we're gonna maybe tie in some uh trust systems that we already have in networking with maybe agents and AI or agents are kind of in my mind being anthropomorphized into these like digital employees that can do things for us. I guess permissions would be really important to make sure they can't do all the things they shouldn't.

Scott Robohn 33:43

I don't like the digital employee nomenclature. Um, but just because you know I've got IAM set up for Andy and Scott, or I've got our back set up, that is a good place to start. It might not be where we land. I would I would kind of summarize it all like this. Just slow down and pause, right? Think through what is the problem you're trying to solve, and then write out the steps where you like if I want to use agents for this particular, you know, switch troubleshooting scenario or this BGP troubleshooting scenario, what would it look like for a human to do it? You know, what are the things I'm not gonna let the human do? You know, start out by not letting the agent do that. Be be deliberate, write it down, write test cases.

SPEAKER_01 34:26

Read only, too, right? I remember you saying that.

Scott Robohn 34:28

Starting read-only is a really is a really good place to start. It doesn't mean you need to stay there, right? You know, but you put the right guardrails in place, you can you can make use of right operations.

SPEAKER_01 34:37

I think we've been here before with uh automation. Ding, ding, ding, ding, ding. Yep. But man, you would know better than anyone. What are those numbers for the the survey data on how many networks are operated?

Scott Robohn 34:48

And it's oh I well, so Chris Gruneman drives all that, and he is he is doing a refresh of the state of network automation survey.

AI-assisted troubleshooting and log analysis

Scott Robohn 34:56

Let's get a link on that for the show notes, and he'll be talking about it at Autocon 6 in Tucson.

SPEAKER_01 35:01

But I remember last year's numbers were you know underwhelming as as they usually are. So we have so many methods and networking that we do to make the stuff safe and the permissions and testing and you know, read-only. I hope. Well, I don't know if hope is the right word, but automation has been lagging behind every other tech stack for so long and networking. That is AI going to be different? I think so. I think we have some really good techniques. Like I'll I'll I'll wrap it up with this. You know, when I when I was a WAN guy in data centers, I would get paged all the time in the middle of the night if something wasn't reachable. And I would get on the bridge and nobody did anything. Right. And, you know, whatever. I mean, they're level one, maybe they're not comfortable. I get it. But what I eventually before I I moved on to to other things, I'm like, guys, here's here's just a it's just a word document. I couldn't code anything, but you know, here's like five commands. If you could just run when the ticket comes in, yeah, is the route there? Right. Like, can you reach things? Were there any changes that carl aid in time? Like and you know, you could probably run a script for that. And then I guess the next iteration in the AI uh conversation might be like an agent. Like, okay, a ticket comes in, an agent spins up, it does some work, which is validation, so that when the tier three, whoever the heck gets on the bridge, there's a document. Like, here are the things we checked. Because that's what I've seen is compelling is the troubleshooting. Like, I for me to look in seven different devices in my transit gear looking at logs to all the logs associated with that. Like what happened, when did it happen, and what's the commonality? I have seen the LLM tolling just parse all that data and give me signal from the noise like really quickly.

Scott Robohn 36:36

I'll rip off John Cabo Bianco. It's enough, not finding a needle in a needle stack or needle in a haystack, it's finding a needle in a needle stack. And LLMs are really, really good at parsing through piles of log data on that.

SPEAKER_01 36:47

Yeah. So finding what it's good at and then leaning into that, much like we talked about the outline for for this show. Like it's good for that, maybe not so much for blind blog posts in your network, if it can go through logs and try to help you. And you know, maybe there's a read-only agent you can figure out to help your people troubleshoot because we're all short on people, things are breaking, uh, outages are expensive, and they usually take a really they take longer than they probably could to try to figure out what's happening. Yeah. And we have this tooling that can stuff like that. It's just all that. Right. Yeah. And it's a starting point. Yep. Scott. Thank you so much, man. How did I do on top? Oh, good. All right. You're we're getting you out on the um thank you so much for being here. Uh, thank you for joining us on the Art of Network Engineering podcast. For all things, uh, Art of NetEnd, you can check out our website, uh Artofnetworkengineering.com.

Wrap-up and community resources

SPEAKER_01 37:33

All the things are there. Uh, we're adding monthly to the resources section. Um, so check it out. A lot of free, uh, awesome things. If you're trying to break into networking, if you're in and you're trying to skill up, if you're trying to pivot to this AI stuff, um, I so I created software with an LLM that crawls the web and finds new free resources that we can share with the community. So, like another cool use of, you know, um, it's been really good.

Scott Robohn 37:56

Networking needs networkers.

SPEAKER_01 37:57

It does. As always, thank you so much for watching and listening, and we will catch you next time on the Art of Network Engineering podcast.

SPEAKER_00 38:04

Hey folks, if you like what you heard today, please subscribe to our podcast and your favorite podcatcher. You can find us on socials at Art ofNetEnge, and you can visit Linktree forward slash Art of NetEng for links to all of our content, including the A1 merch store and our virtual community on Discord called It's All About the Journey. You can see our pretty faces on our YouTube channel named the Art of Network Engineering. That's YouTube.com forward slash Art of NetEng. Thanks for listening.