Intro

andy lapteff 0:00

This is the Art of Network Engineering. Where technology meets the human side of IT. Whether you're scaling networks, solving problems, or shaping your career, we've got the insights, stories, and tips to keep you ahead in the ever-evolving world of networking. Welcome to the Art of Network Engineering Podcast. My name is Andy Lapteff, and in this sponsored episode, we're bringing back an old friend of the show, Nox Hutchinson. How are you doing, Nox?

Knox Hutchinson 0:22

What's up, Andy? It's good to be here, man.

andy lapteff 0:24

I'm so happy to be back. I know it's been way too long. Um, you've been very busy building something very cool, which we'll get into shortly. For newer fans of the show who may not know you, who you are, what your background is, give us a quick like who is this Nox

Meet Knox Hutchinson

andy lapteff 0:40

guy? Sure.

Knox Hutchinson 0:40

Um, so you, if you have heard of me, you've probably heard of me through either my YouTube channel, Data Nox, or through CBT Nuggets. I have been a trainer at CBT for almost eight years now. It'll be like eight years next month. So that's where my primary branding and following comes from. But me personally, professionally, I've been a software developer the majority of my career. I've worked from small businesses to government entities to large fintech firms, uh, which is kind of interesting because it is different from how I am branded online as a networking and network automation guy. But there's a whole story behind how that came to be.

andy lapteff 1:21

I'm so glad you

How Knox Found the Show

andy lapteff 1:22

mentioned that, Knox, because I know you from CBT Nuggets. I know you from your awesome YouTube videos, and I know you specifically through the Art of Network Engineering on you, you were on episodes 14 and 15, very early. I just looked it up October of 2020. So we started in July of 2020, and shortly thereafter, you know, over five years ago, if my math is right, time is hard for me, but I think it's been five years. You were very early, and the way you appeared on our radar, I believe, was one day AJ Murray just pings me and is like, yo guys, holy crap, like Nox Hutchinson, who has a huge following and is this awesome CBT Nuggets trainer, is like telling the world how much he loves our show. Now, I kind of forget how it went down. If you mentioned it on your YouTube channel first, and then you invited us, and then we came on your show, which was awesome, and it was really just you basically introducing us to the world. And you again, I forget, like, you know, we had three people following us at the time. And for someone as known in the industry as you so early on to just show up and appear and be like, yo, this is a great show. Y'all need to watch this, and then we had you on your show. So it was this really great moment for us. So thank you. Because like you showed up, and then Keith Barker showed up, and we're like, what is happening? Which I think is why it took off so fast in the beginning, is people like you, who the industry knew, kind of called on and were like, yo, you guys got to check this out. So it's a long way of saying you have a great history with our show. Thank you for your kindness early on. I love that you love the show enough to share us with your audience. And yes, I know you from CBT Nuggets and your YouTube channel and being on our show. And when we talked not too long ago, and you told me that you built this amazing like software for network engineers, I assumed, and I'm being a little provocative here, but I assume you're doing what I'm doing and vibe coding software, right? And the first cynical, like you know, network engineers, the cynical person's gonna be like, all right, you vibe coded an app good for you. And and it's almost like I don't want to say that people vibe coding apps get ignored by the industry because I am doing that and I hope I don't get ignored. But just like you see an AI written piece of, you know, like writing and you pass over it because you're like, all right, M-dash isn't actually, well, I'm not gonna waste my time. When you told me you were a software developer first in your career, I'm like, oh, like so for me that helps. So let's lean into, I guess, I would like to get into your history really quick before we see the cool thing you built, if that's okay. Like, yes, I didn't know you were a software developer. So, how the heck did you go from like software developer to the CBT nuggets trainer

Software Dev to Networking

andy lapteff 4:02

that we all know?

Knox Hutchinson 4:02

Not an uncommon story, but it is uncommon to do both things, to be both network engineering and in software development. And a lot of people think that like you you would leap to like, oh, he must have just gone straight into like network automation or something. And that that wasn't even true, too. I began my career in back-end data work. That's why my handles are all data knocks. And if you actually look at my CBT Nuggets content, like the first five courses I ever produced for them were Tableau Data Analytics, Power BI Data Analytics, and ETL, then SQL Server Integration Studio, Analysis Studio, and Reporting Studio. It was all certification exams back then. And I would say, if I had to guess, probably over 75% of my content with CBT Nuggets is software development, uh, DevOps engineering, cloud engineering, platform as a service. It's actually not much networking. Um, I had to, I had to like really scrap to get in on the networking stuff. But CBT Nuggets is known as a place to go to get network training. And it's to me, I mean, that's where I got my training. So that's where the story actually begins is I was working for a very large fintech firm, and my job was to fly on site to all of these customers that we have. They I would go to banks, I would go to businesses and banks that are regional banks in whatever state you can think of. I was there. And my job was to integrate their core system with our platform. Our platform did a very specific thing. So it was building APIs and building interfaces that transmit the data back and forth. Well, it's a bank, so you can't do this over the public internet, right? So next thing we know, I'm interacting with the networking team. And they're like, okay, well, we got to build IPsec tunnels, their address space overlaps this customer's address space, so there's gonna have to be VRFs and route policies and whatever. And I'm sitting there like, uh, you know, I was speaking, and I'm the software developer, I'm speaking software to them. So it's like I'm speaking Japanese and they're speaking Greek. Like it's just never gonna happen.

andy lapteff 6:03

Sounds very familiar.

Knox Hutchinson 6:04

Yeah, right. And this is true, it's not just software and networking people. This is like everybody versus networking people. And if you're a networking person, you feel that that is a pressure that you feel. So, um, anyways, like after a year of just like not getting along well with our networking team, I was like, I've I've got to learn how to speak their language, I've got to translate. So literally, when I'm sitting there thinking about it, I get this email from corporate that says, We've just bought 10 seats to CBT Nuggets. Let me know if you want one. So I literally hit reply and I'm like, yes, put me in. I want it. And next thing I know, I'm getting my network plus, and then I'm getting my C CNA. And then at some point along that journey, I'm like, wait a minute. I really like networking. Like, I love networking, like I my brain just gets it. So I transitioned out of that role into an MSP where I did most of the networking work. Um, and that was really like, okay, I'm a networking guy now. And then around that time, we still had CBT Nuggets subscription at the MSP. I was still learning, I was still getting certifications and everything. And at one point, I started making content. CBT Nuggets found my content and invited me to come interview with them, which was really cool. And you know, the rest is kind of history. The thing that really kicked off the branding is around 2018, Cisco announced the DevNet certification track. And I was like, oh my god, like I'm doing that. I do software and networking, like I can teach that. And that's you know, that's how my brand kind of became a networking-centric brand. But ever since then, I've still always done software. It's always kind of been my thing. Yeah, and to your point about vibe coding, so you know, if I want to talk about this, like this is kind of a thing that I'm fighting against now is my own brand. Uh, is that people think like, oh, he probably just sat down and like slapped something together without knowing that, like, no, I've like delivered enterprise software for large fintech firms, government, whatever, for for a long

Fintech IPsec Tunnels Story

Knox Hutchinson 8:10

time.

andy lapteff 8:10

And having worked in fintech, I'm sorry to interrupt you, but having worked in fintech and the PCI compliance and like, yeah, it's like healthcare, man. It is so regulated, it is so important to our countries, if not global, economy, that this isn't like stuff you could just mess around with. No, you being a software developer in fintech for people who don't know, means like you're legit skilled, know what you're doing, no crap, like this is real. There's no room for error.

Knox Hutchinson 8:37

I mean, that's that's the thing, is you've got with either of these like highly regulated things, like there's you could have loss of property or loss of health, loss of very private information. So it's very, very important to get it right. And that's why, you know, I I'm not just slapping something together and vibe coding something and shipping it in like two or three weeks. Like, this is something that I've been building for a pretty long time. So there's some things that you should probably be aware of as a network engineer. Think about this just about every single application that's installed on your computer right now by whatever entity you purchased it from or downloaded it from, was probably rewritten or re-architected by AI in the last few years. Every single one of them. So using AI as a coding assistant does not equal vibe coding. Unless you think that like all of the other applications that you're using today that are installed on your computer are vibe coded, then yeah, then if that's your definition, then yes. In my opinion, vibe coding is something like, hey, I've got this idea, I want to build a prototype, and let's just ship it out there. It might solve this very specific niche problem for me, but I don't ever intend on delivering it to another enterprise. I think that that's kind of the idea. Is the moment you're like, no, wait, I really want to put this out there for the world to use, you then have to ask, well, what would happen if a hospital uses it? What would happen if JP Morgan uses it? You then are like exposing a lot more risk into the environment. So to deliver an enterprise application, if you have like a pie chart, because I'm thinking about data again, if you have like a pie chart, like code writing is 10 to 20% of the work. And that's why today most enterprises, most software enterprises are leveraging AI to get through that 10 to 20 percent to move on to code review, security audit, DevOps pipeline, unit testing, user acceptance testing, integration testing, back to security review one more time. Uh and then compliance is built into all of that, pen testing is built into all of that too. Uh that's kind of my spiel is if you're just like slap together some code and ship it, then that probably is vibe coding. If you're actually giving like 80% of your effort comes after the code is written, then you're probably moving more towards something that looks more like enterprise delivery. There's a real problem that I think this application solves. And what I didn't want to have happen is the majority of network engineers can't use it because it's not pen tested and it's not SOC2 audited, and it hasn't gone through all of these things. So that's why I'm like, no, we're gonna put forth real effort to do this right up front, deliver it like I've delivered enterprise software in the past, and not just slap together something because it's a cool idea. That's where we're at today.

andy lapteff 11:22

I think that's the perfect framing. And before we jump in and take a look at your um software, what's it called?

Knox Hutchinson 11:27

What's the name of it is transit AI because data in transit.

andy lapteff 11:32

Transit AI. So I'd like to just hang in the cultural bit around networking and AI for a few more moments and then hop into the software.

Knox Hutchinson 11:43

This is this is the this is the talking point.

andy lapteff 11:46

It really is. I mean, we're all going through a huge transition because of large language models and machine learning and AI. I don't even know what to call it, right? Like, is that all the same thing? I don't know. But here we all are trying to figure it out, and we get tremendous efficiencies with it. I am someone that, you know, I guess I'm a vibe coder, but I also have some smart people teaching me about like spectrum and development and like security stuff. And so I'm what AI has done for me, like AI isn't writing my podcasts, writing blogs, writing scripts for me for work. It's allowing me to create solutions I couldn't before. Some automations for the show, some things at my house. Now, I don't have the skill set you have, so I don't necessarily think that you should take the software that I've created to solve my own problems and then deploy them to finance or to so there is a huge difference between Knox Hutchinson and Andy Laptef's software. And I think it's important for network engineers who are about to see your software hear that because network engineers historically have been resistant and afraid of network automation. And I think that these tie in together, right?

Knox Hutchinson 12:55

For me, AI big reasons why.

andy lapteff 12:57

Yes, in a sense, AI is so I always see AI as like network automation on steroids or like gasoline on that fire, meaning that that's not a bad thing inherently, but because network people don't have a development environment that they can test changes in, like software people, because network engineers don't have a complete replica or lab or digital twin of their entire network with state, so that when you make a change, it's not just testing protocol behavior, but it's testing what happens to the flows when we do the thing. And because the networks, I'm going somewhere with this, I'm almost time, but because the networks are the roads that everything travels on for network people to deploy automation, which makes things faster, not better inherently, for them to lean into anything AI when there were hallucinations early on than the models, when am I going to break, you know, network people are afraid to break all the things. So I'm really glad that we're kind of touching on the cultural aspect, how automation, and not that it's the same thing as AI, but now AI, how networking people are afraid of it. And really where I'm going with all this is let's jump into transit AI. And if you could frame it in a way, at least in the beginning, of I am a network engineer, I'm afraid of automation, I'm afraid of AI. What have you done when you built this software as a real developer who has proven success for decades in things like FinTech and HIPAA? What is this software? How can I use it? And how is it not going to do the bad thing I'm afraid of, which is break all my things? So I think you're spot on.

Knox Hutchinson 14:32

And I want to say, like, I've also been a certified network engineer too. Like I've been you and I know the risk. And that's why I think I'm in a unique position to have created this application. As a software developer, like you just said, you can blow up your entire code base right now on your laptop, on your desktop, and no one will ever know. It's just git restore or git checkout, and you're out a bad afternoon, but that's it. You or just delete the branch, go back to main, branch off main again and try again. Like that's it. You like you, there's no there's no real risk when you're doing software development, especially if you've got a really good testing pipeline that comes after the code is written, which is really the entire point. That's why you have enterprise apps today. With network engineering, it is a lot harder to get away with that. And this, you we started to see this with network automation when it started becoming a real thing in 2018. Network automation has been around since 2003, when Juniper created JunoScript. Let's throw that out there right now. JunoScript ended up becoming Netconf in 2006, and then Yang came after that. So network automation has always been around, but it wasn't a buzzword until 2018 when Cisco dropped the DevNet Associate exams. And now everybody is starting to be automation curious. Some people have like fully adopted it and gone all in, but some people are starting to get software curious. And this is where we go back to what I was just talking about, that the writing of the code, even for network automation, should only be 10 to 20% of the work. You, when you're writing code for as a first-time code writer, you're like, oh, this is so exciting. I can't wait to run it and see everything work and it does the configuration. But the distrust comes in with like, well, what if it doesn't work? Or what if it does something even worse? What if it breaks things? That's why the 80% after the code is written is all about testing. And you do need a digital twin. You need a digital twin that mimics your environment right now to run tests against. And however many tests you think you need, triple it. If you're like, I want to run 100 tests, make it 300 tests. And in the age of A high, you can be like, I need 300 tests, and at least it gives you something to start with before you just yellow code out there in production and see if it worked or not. So, first, like I want to say, like, I know because I've sat in that chair when you confidently type one wrong command and the building goes black, and everybody is now standing, go, hey, is the internet down? Yes, it's down. I know it's 2:30 in the morning. Ah, like, you know, like you're you're screaming. So, like, I know what that pressure is like. I know what it means to do that. There's also another fundamental thing that's pretty interesting that you should talk about. Networking is hard, and there's not many people to do it. There's a lot of people who go to school and get a computer science degree and become a software developer. There's a lot of people, and I don't mean any of this with any disrespect, but there's a lot of people who come in off the street to become a junior sysadmin. So if we're just highlighting just fundamentally what is different about networking and why networking engineers, as someone who's been one, is looked at as the weird one in the IT group. If you take someone off the street, I could probably teach them how to reset a password and Active Directory and change group membership and maybe interact with GPOs on day zero. I can teach someone to do that today. I cannot teach someone how to create a VLAN on day zero, the most fundamental building block of networking. Understanding the OSI model and VLANs and segmentation and 802.1Q is hard. So by default, network engineers get weeded out really, really early. Most people, if you go into an IT shop today, if there's 50 people working in IT, how many of them can touch a network or allowed to touch a network? Three, maybe, four, I don't know. Um, it's not many because it is fundamentally hard on day zero. So by default, you are back against the wall on day zero. The job that you do is hard. There's not enough people to do it. So you are on call. And

Introducing Transit AI

Knox Hutchinson 18:42

if one thing goes wrong, that's it. Like it's over. So and it always goes wrong in the middle of the night, Knox. Yeah, yeah. So like if I'm bringing up PTSD, I want you to know it's because I have PTSD. And and when I set out to build this application, transit, it was with that in mind.

andy lapteff 18:59

And my understanding of transit AI is you are going to be enormously helpful to the network engineer. Let's say at two in the morning when there's an outage and all heck's breaking loose. I I told you when we were talking a week or two ago that when I would get called for an outage in a very big place in the middle of the night, and I would get on the bridge and there'd be 15 people on there, I would almost never have any intelligence that was collected when I got on. So I'm exhausted. I worked in maintenance the night before and probably weeks. Like, I'm just I'm wiped out and they call me on because I'm the tier three escalation. And I'm like, okay, what what do you guys know? Oh, well, you know, we we we haven't checked anything. I'm like, could could you at least, you know, check the route tables? Is the path there? Like, can you prove out trans? Like, can you do anything? Like, we have net flow, like, look, right? Yeah. But to me, it's a troubleshooting tool. So I'm gonna shut up. But like, I I can see this tool as I get on at two in the morning and I have a paragraph or two of data that transit AI collected that tells me, here's what we checked, here's what we saw, here's what we did, and and here, like I think we're in a good spot to jump in and take a look at it.

Knox Hutchinson 20:04

So when I set out to build transit, I want to tell you here's what my motivation was and has been. Fundamentally, software developers had the debate about AI in 2023, like three years ago at the time we're recording this. They don't debate if they should use AI anymore. That that debate has been had and it was solved two years ago in 2024. Now, software developers are debating what is the right AI architecture? What AI writes code, what AI reviews code, what AI So it's not about whether or not to use AI, it's about how much AI to use. I think network engineers are still in this like weird environment where we're still debating whether or not to even use it at all. And then importantly, some people are gatekeeping it to be like, no, you're not allowed to because you're not good enough yet. And I think that's a really big problem. It's a really big cultural problem, and it's a really big problem in general, but it's mostly a cultural problem. So, where this motivation came from, as someone who is wearing both hats in 2025, early 2025, I would open up Cursor and I would have this incredible software development experience. It looked and felt like a tool that I used via code, but it also had a chat agent where I could say, like, what is this line of code doing? Or how do I fix this part? Or I want to add this new feature, but give me a design idea. And it would just do it. It would just rattle off and it would become an iterative collaborative process. Then I would go to log in to a switch and I would open up an application from 2013, and it looks like it's from 2013, and they haven't shipped an update since 2017. And I'm sitting here going, like, hang on, like that user experience in the software world is incredible. And if you look at the market, the the all knowing market, the economy of things, it all seems to be rewarding that user experience. Meanwhile, over here on this side of things. Where I'm using SSH, it's like completely stalled for a decade. And I'm sitting here thinking, like, oh, I think we could fix this. I think we could fix this. But importantly, do it in a way that solves the problem that network engineers are terrified of. I don't want to let AI have access to my box. It's going to take down the whole network and I'm going to, you know, I'm not going to know what it does. So what I built was something called transit. And what I'll do now is I'll share my screen and we can kind of talk through it so it makes more sense. Here is transit. Transit is brought up on the screen here. This should look and feel a little familiar. This should look and feel like SSH apps that you have used in the past. And that's exactly what it is. It is an SSH app that you have used in the past. I can launch eight sessions, tab sessions across the top, right here. And you know, it's just typing, it's an SSH tab, just like you've used in the past. But what makes it different is that there is an AI copilot built into it. And the AI co-pilot is interesting because it is investigation only. It's not allowed to change the configuration and it's not allowed to change the system state. It's not allowed to reboot your machines. It's only allowed to help investigate. So if I were to give a demo, there this is a lab environment. There are eight Cisco routers. Some of them are running DMVPN. And of those that are running DMVPN, the spokes, two of them are having issues, and they're two different issues. So if I were paged at 2.30 in the morning and it's like, hey, DMVPN is acting really funny. I can't tell what's going on. I can say DMVPN, I'll just type it with typos, is running on some of these machines. Find out which. Then two of them are having problems.

Why Networks Are Risky

Knox Hutchinson 23:52

Find the root cause for each. So I'll fire this off. And this is where it starts to get interesting. The first thing that, in this case, I'm using Opus 5 as my model. We'll talk more about models in a second, because that's also a very important talking point. The first thing Opus V is going to do is it's going to find out which ones are running DMVPN. And it's going to do this by running show commands. Show commands are inherently investigation only. So what it is allowed to do is to propose a command one at a time on each device of what it wants to run to perform the investigation. Now, because I trust all show commands to not uh break the configuration or reboot the machine, I'm going to go ahead and set all eight of these devices to allow show commands to run automatically. So I'm still going to approve this for all eight. And this is an important thing because you might be fine with like, yeah, you can automatically run show commands on the access switches, but on the firewall or on the core switches, I want to still be in control. So that's why we do it on a per session basis.

andy lapteff 24:55

Questions? Yes. Some of us are afraid of LLMs. Uh-huh. So now you're talking about permissions and what the software is and not allowed to do. So are we trusting the LLMs to make that decision, or is that decision made in your software?

Knox Hutchinson 25:11

It is made two parts, both have to be true. The LLM is allowed to propose a command. The first thing that the software does, before you ever see it as an approval, the first thing that the software, the app itself, not the LLM and not a prompt, the first thing that the software does is it validates that this will not change configuration or state.

andy lapteff 25:32

That's what I'm digging at. So it's the software. We're not blindly trusting an LLM. Your software says, no, no, no, you can't change anything. And if you can can it can the LLM ask to?

Knox Hutchinson 25:43

It happens all the time. Okay. This is so this is a big architectural discussion. LLMs come with guardrails, meaning there are some things that you tell an LLM you're allowed to do and you're not allowed to do. But guardrails can almost always be broken. And that is a very, very important thing to understand. This is why like mythos was pulled off the rack. If you remember when mythos dropped. Oh, by the way, the root cause was found just now in about 60 seconds. We'll talk about that in a second. Um so LLM. So if you want me to fear I can demo this, like right now.

andy lapteff 26:20

Yeah.

Knox Hutchinson 26:21

I can say run the fix for me. And I'll show you.

andy lapteff 26:25

You know what I'm getting at, right? Like I'm I'm afraid, I'm a network engineer. What is this thing? AI is gonna ruin my career. The the the app won't allow that.

Knox Hutchinson 26:32

Yep.

andy lapteff 26:32

By the same time.

Knox Hutchinson 26:33

So it says I can't, rights are blocked at the policy gate. That's the that's the thing. And I'll say try anyways.

andy lapteff 26:39

So you said fix it, and it said I can't.

Knox Hutchinson 26:41

It said I can't, but then I'm gonna say try anyways. And it says no, I can't.

andy lapteff 26:46

So transit AI is controlling that, which is good. You're protecting me from or junior from doing something dumb they don't understand, right?

Knox Hutchinson 26:53

But the important thing is I didn't prompt it to. Only once it passes the policy gate do you see that approval to run the next command. Okay. And the command will not be run unless both things are true. First, it has to pass the policy gate that's built into the app, then it has to pass your approval. So that's there's two stages to make this investigation only. And that's what that's what fundamentally sets us apart is we architected the application to be this way, and we filed a patent for that. We filed a patent for the policy gate. Now there's other things that the policy gate does. It's not just that it stops the configuration from happening. So if I say, watch this, I'll do show run. When I go up to show run, uh-oh, what's here? Passwords. So if I now say read the scrollback of R1 after I've just run show run, the question becomes, did I just leak this secret to the LLM? And the policy gate stops traffic going in the other way. So now I'm still thinking about DMVPA because it's got the context of DMVPA. And I'll say, did you see the password for the user? And I'll just pause. No. It came through the redaction filter as redacted Cisco username secret number one. I never see the bytes. So the policy gate is built into the app. It's not in our cloud, it's not in the LLM, it's not a prompt. And the policy gate stops traffic going in two directions. It stops configuration changes from coming in or state changes like system reboots. And it also strips secrets and passwords, well-known secrets and passwords and uh strings and arrays from ever exiting and making its way out to the LLM. Now, aside from the obvious things like we don't train, we don't use this data for training, we don't even store this data. Anything that's happening on your box or anything that's happening in your chat is not stored on transit AI infrastructure. Outside of those things that are clearly defined in the privacy policy, I'm taking it a step further and saying we're still gonna strip the secrets and redact them, and we're still gonna stop configurations from coming in.

andy lapteff 29:35

So you have a big focus on safety and guardrails and making sure because you know the the challenge with some of these tools, at least for me, is I'm not clear what damage I could do if not constrained. And you have really built in the exact safety and guardrails that someone like me would need to not misunderstand. I've seen this before in production. This isn't an automation tool, but in in the in the scope of automation, I've seen someone not understand the tool and push to change and have unexpected results because they didn't understand the logic of the tool, right?

Knox Hutchinson 30:11

That's a real thing that happens out there today. People are trying to build AI runs your network tools, and I'm personally not comfortable with that yet. I'm not there. So I think, and I don't I it's my also belief that large enterprises, especially ones with hospitals and banks, are also not there yet. And a lot of the feedback that I've been given so far is that. So what you gotta think about now is now that we're like, okay, I trust that it's not gonna change the configuration nor release my secrets. So what did I actually just get for this? Well, what you actually just got for this was a complicated troubleshooting scenario solved in about 45 to 60 seconds.

andy lapteff 30:52

Can we go back to the original um symptoms? So I know that I know that we're troubleshooting DMVPN,

Troubleshooting at 2 AM

andy lapteff 30:59

but let's get in the seat of the operator. What happened and why are they looking at DMVPN? Did an alarm come in? Did something happen?

Knox Hutchinson 31:05

Exactly. A ticket, an anomaly was detected, you were paged at 2 30 in the morning, whatever the case is, a problem has occurred.

andy lapteff 31:12

And what did you ask it? Like, hey, we have DMVPN, something's weird, I forget the problem. That's it.

Knox Hutchinson 31:17

It was it was it was like two really vague sentences. I didn't really direct it at the specific device or even the symptoms. Perfect. I just said DMVPN is running on some of these machines, find out which. Then two of them are having problems. I didn't say which machines are having problems, nor did I say what the problem was. So this would happen.

andy lapteff 31:36

You there'd be a ticket, you get vague information, you plug it into this tool, because I am the tool in that maintenance window.

Knox Hutchinson 31:41

Without this, I'm this is the equivalent to the internet is down or the internet is slow. Okay.

andy lapteff 31:46

That's all you guys can know off of. Let's see what it did and figured out here.

Knox Hutchinson 31:50

Yep. So the first thing it did is it ran show em DPV DMVPN on all eight devices, and it immediately found out which was the hub and which three were spokes of the four that I have. It found that wait a minute, R5 is stuck in NHRP and R7 never came up at all. So right there, it identified which two of the four that were in DMVPN, which two had problems. Then it ran show run, show IP, NHRP, NHS detail, and then pings to make sure it can reach the loopbacks of each. So with that, it was able, with that context, it was able to find the root cause of each of these problems. R5 had a tunnel key mismatch, and R7 had the wrong NHS address attached to it, which made it look like it was flapping. And this all happened in about 45 seconds. So, what you should be thinking about what this unlocked is if you're a junior engineer or you work with junior engineers, they just became safely in power. Because this ticket would have been really, really hard for a junior engineer, someone in their first three years. This would have been really, really hard. So, how much time would your junior engineer have spent looking into this? And would they have come up with a root cause at all in that given time frame? It was probably been more than 45 seconds. And the answer is I don't know how much time they have. If it was five hours, maybe they have the answer, but how mad is the customer at that point? So this is a way that wait a minute, we can now safely empower the junior engineer to start digging into things because we know it's not going to change the configuration. They could also learn from this while this is going on and they start to understand their customers' networks a little bit better. Secondly, let's say the junior engineer now does have the root cause, but they're not allowed to make the change, especially one as significant as DM VPN configurations. What they can now do is generate a summary report of the DMVPN findings and return to me in let's say something like markdown. So what this guy can now do, guy or girl, can now say, like, okay, I've now got the root cause. I know what happened, where, when, and even have a recommendation of what to do to fix it. They can now generate this full report of exactly what to do. And they just scroll up to the little copy button right here and paste it into Slack to send to the senior engineer, put it in a ticket comment and escalate up, send it to the customer as a report of what happened. So now this isn't just about finding solutions quickly and closing tickets quickly, which is a great win. It's more about collaboration is happening because this is the real problem. The real problem is that there's not enough network engineers to do the work in the first place. High risk tickets come in. A junior engineer is either afraid to perform the research that is needed to do this because it could ruin their career, or they're not afraid enough and they cowboy in and break everything. Then, subsequently, if that problem is solved, they then start collaborating with the senior engineer who they were afraid of because the senior engineer has been hoarding all this work in the first place, and they're buried under it, and they can't focus on the design work they were told to do. Now this collaboration gets unlocked. And now the senior engineer is about to be handed a full report, which they can then log in and verify themselves. They don't have to just take it and copy and paste and see. They can then log in and verify it themselves, and they even have next steps of what commands could fix the problem. That collaboration gets fixed. And then what is really does this also solve? Think about how their manager feels every day. Every single ticket that comes in the door is giving that manager some form of stress. They're like, oh gosh, can they solve this in time? Or oh gosh, this is one more ticket that's about to get added to the senior's queue, and they're sitting on top of a mountain of tickets, anyways. And what they're really doing is they're really looking at your desk and seeing there's $600,000 of monthly recurring revenue. And if that person rage quits from stress, all of that walks out the door. They're in an absolute crisis, they have no documentation because that ACL policy lived in that guy's head, and he's been buried for so long that this junior who got this ticket, spent five hours troubleshooting it, couldn't find anything, the customer's now threatening to sue, you know. So here, senior, good luck. Call the customer when you're done. I I wouldn't blame them. This is not an uncommon thing. And for anybody who's listening, I'm not saying this is your story, but there's probably a piece of this that you can relate to. And if you had something like this, where it's like, hey, I got I gotta bounce an idea off of you. Give me your thoughts in 60 seconds, or this is a problem that I've just been given virtually no information. Can you at least point me in the right direction? The senior who's about to implement a massive MPLS change could be like, hey, validate this configuration before I commit it and tell me what do you think will change? Is there a way to harden this? Let's say you're taking you're an MSP and you're taking on a new customer. You walk into the network and you have to discover what they even have in the first place. You can do that now. You can have a tool that can just start an investigation for you. And yes, it's a it supports console cable. It supports console cable? It supports console cable. You just plug a console cable. Yay! If I plug a console cable in right now, I'm plugging it in. I just plugged it in. Look at that, right there.

andy lapteff 37:38

You don't have to look at device manager and figure out the comp port. It's right there.

Knox Hutchinson 37:42

You just plug it in, it's right there at the bottom. You can plug in the bottom. You know what I'm talking about.

andy lapteff 37:45

From that, from that old tool in 2013, it's like, what's the com port and what do I gotta assign to the thing? Remember that?

Knox Hutchinson 37:51

Yeah, yeah. I'm I'm plugged into the console port right now of a Juniper switch. So if I do show configuration, that's all in the console port. And yes, I can now use the AI to talk to the console port. So even if this device doesn't have an internet connection, and we can talk about this in a second, if I have an internet connection, I can get it. And importantly, I think this is gonna be a big one. Like your audience is gonna ask me about this. It does support local LLMs. So even if I don't have an internet connection, or if I'm not allowed to use cloud AI, if inference is not allowed to use my network, it detects it right here. So I've got Quinn38B that I'm running locally on this machine. So I can now be out in the field and get AI assistance right there on my laptop.

andy lapteff 38:36

Let me run some thoughts by you. As you were showing this, I wrote some things down and they really landed with me. So networking is hard, it's complicated. And I think that DMVPN is the perfect thing to show here because I helped co-design and run a DMVPN environment at a very big place. For people who don't know DMVPN is um dynamic tunnels that, you know, you have a hub and then they build tunnels out to places, and then depending on configs and how you do it, the spokes can talk to each other, they can talk to the hub. There's a lot of going on, and I kind of forget the whole protocol stack, but like I I know NHRP, NHS, there's a handful of subprotocols that make DMVPN do the magic, right? So there's a lot of moving parts and they all have to work together in a very particular way to make this all work. And as someone who helped co-design and run a DMVPN global environment, I forget how it all works and I wouldn't know what to check. So to your point, it's complicated, there's not enough help, and outages take longer because for me who maybe have been working on a different section of the network for a month or two, and now I get pulled into a DMVPN outage, our documentation isn't great. I haven't touched this in a while. I kind of forget how it all works. There's seven devices I have to look at. Am I looking in logs? Am I doing show and HRP status stuff? Like I just, you know, I gotta Google the commands because I forget. So this saves an enormous amount of cognitive load for the network engineer. And I always lean into that two in the morning outage. I know this can be used in non-added situations, but like this is hard. We don't have enough help. Can someone please help me? That's what I see this tool doing. The way you described it, I wrote down glue, but that it empowers juniors, level one people, knock folks, to help their senior people, their level tier three, you know, the people who can actually touch a data center, blast radius, and and make a change to fix it. The collaboration that it creates between them, because yes, when I was in a knock and all heck was breaking loose, and I needed to talk to a level three person who was asleep, it takes long, you know, it took me a long time to gather the information I needed to tell the level three person what we needed done. The level three person's asleep. It takes an hour to wake them up out of bed to approve the thing for us to do it. The fact that this troubleshot the pro complicated environment in less than 60 seconds not only gave me the potential root cause, but then a summary that I can hand to someone. Like, I did not have this in production. And I remember thinking, God, can someone please hand me something when I join a bridge, guys? I'm tired, I need help. It helps the juniors and it helps the the higher ups who who have you know um all the access and and all the right capabilities. Uh the local LLMs thing I think is really compelling because when you talk about finance or HIPAA, like there there are places that don't want you doing things out to places. I've worked in them. So can we touch on that?

Knox Hutchinson 41:26

Is it hard to plug in a local LLM? Like how would I it's not hard at all. If the easiest way to do it is download an app called Olama on your local machine and then just use Olama to download a model.

andy lapteff 41:38

And that's it. Do I need like transit automatically detects it? Do I need fancy Ulala GPUs to run a local model? Like how can I run it on my laptop? Can I use an old 1070 Ti? Like, yeah.

Knox Hutchinson 41:50

Yes and no. Uh so I am going to have a very definitive answer to that question by the end of this week. Okay. So, Andy, this is the first place that this is being revealed. Okay. You're hearing breaking news.

andy lapteff 42:04

I'm sorry if I'm asking you questions. I shouldn't. I'm just we haven't scripted this. As the questions are coming up, I'm like, oh, local is good. How do I do that?

Knox Hutchinson 42:11

And to

Safety Guardrails and LLM Limits

Knox Hutchinson 42:12

your audience, I was not prepared to talk about this today, and I'm not even sure if I should, but I'm going to. Because it's my company and I get to say what I want. No one will. So, yes, you can get started with local LLMs right now. You download Olama, you use Olama to download a model that can run on your computer, and transit will automatically pick it up. And it will even be the default model selected. You would have to explicitly choose to move it back to a cloud model. But this begs the bigger question that I am now very deep in the weeds on. Which model is right? Which model is best? And how much investment do I have to get or have to have to put in to get a worthy model? And that is what I am quite literally on another computer right now putting together a very, very rigorous, almost academic level research to give you the answer to. So on my MacBook Pro, it's an M5 MacBook Pro, not an M5 Max, it's an M5 Pro. This is the first thing. Transit is multi-vendor. It's not just Cisco. It will literally support any vendor out of the box. So the question becomes then okay, which small model is still accurate enough to be worthy to use? And the truthful answer is not very many of them at all. The best small model that I've found so far across a wide array of vendors, Unify, Juniper, Palo Alto, Fortigate, PF Sense, Cisco, and not just Cisco, iOS, iOS XE, iOS XR, NXOS, all of those, the best small ish model that I have found is only about 46% accurate. It's not great. And this is using a four or five thousand dollar laptop, which is Not a cheap laptop. You're not going to be able to do that.

andy lapteff 44:01

Networking is complicated, so I'm not super surprised that we would need a powerful model to parse all this complexity.

Knox Hutchinson 44:07

So then you're like, okay, well, let's say I go to my my business and I say, hey, if we spend a little bit more money than just a laptop, if we bought some GPUs, not even the biggest, baddest GPUs, but if we bought some GPUs, what kind of performance gains for accuracy and for speed? Because the other thing with these local models, if you get like a bigger local model to get accurate, it goes very, very, very slow. Like it will take, you know, you saw it right here. It answered this question, all of these in about one minute. On a bigger local model, if it finds the right answer at all, it'll probably be more like five to 10 minutes. Still not bad, still beats five hours, keeps all your inference local, but it's not, I'm in a pinch, I need an answer in 60 seconds. You're not going to get that from even a bigger local model. So if you start getting into buying GPUs, does that buy you significantly more accuracy? And the answer doesn't really get to yes until you're talking about like six-figure investments.

andy lapteff 45:10

And hardware for GPUs, right? Yeah.

Knox Hutchinson 45:12

So now for some enterprises, that's not like you might say, like, hey, we'll spend 100,000. Like, like this is a big business. We can absolutely spend $100,000 right now to get a model that is 85% accurate and can still get you answers in a couple minutes. They might do that. And that that is the type of thing that's out there. But you have things like the whole world right now, as of today, is talking about Kimmy K3, the open source model that dropped yesterday that competes with Mythos. And you're like, wow, I can't believe they made Mythos open source. And it's like, cool. All you need is a couple million dollars to run it, and you're off to the races. Like, you know, this is this, so it does, it is a sliding scale of like the more accuracy and speed you need, it is an exponentially more investment you need to have. So that's not to say, like, don't worry about using local small models because they'll never help you. They absolutely can. It's just that they're not very good troubleshooters. They understand the protocols really well. They understand the devices really well. But this right here, the agentic loop, where it ran a command, gathered information, then ran follow-up commands, is where small models degrade very quickly. So, what I'm telling you as transit AI, since we're offering, you know, this product out there in the world that supports local models, the immediate next follow-up question is which local models are good and what is the investment needed to get them? And that is the research that I plan to have published free out there in the world by the end of this week. So there will be a follow-up topic that comes after that. The follow-up of what should we do about it will come after that. That will take more time. But I want you to know that this is a big conversation that weirdly not many people have had. If you look into like network troubleshooting benchmark scores, which is what we're creating, we are creating a benchmark score for local models for troubleshooting specifically. There's not really anything out there. So what I am doing, like as we speak, I am renting boxes with these H100 GPUs and B300 GPUs, or even RTX 6,000 GPUs, much smaller, and I'm running all of these benchmark tests. It's like close to a thousand tests being run. Uh, it takes days to run all these tests. I'm gathering and aggregating all the data, and I'm going to publish a report very soon.

andy lapteff 47:41

And to be clear, for anyone who isn't following, like this isn't a constraint of transit AI. This is just how complex networking is and the level of model you need and the power and the size of it to be able to parse all this. I mean, when you said, you know, Cisco is an example and all those NASA, like just to try to parse all of that, right? And the different versions and the syntax changes, and like it's there, there's a really there's a lot there, but you're thinking about this in the right way, as far as I'm concerned. You're looking at all that, and what can we do locally? Because some folks will have to do that.

Knox Hutchinson 48:14

We're getting toward I need to I need to clarify. When I say locally, I I don't mean it has to be installed on your laptop, it can be local to your LAN. So you can buy a big server and serve your inference locally in transit in the configuration, it's right here under local models. You can either point it at OLAMA on your computer or you can add a custom endpoint right here where you say, like, what's the name, what's the ID, what kind of model is it, and specifically, where is it served? What's the endpoint, what's the port?

Speaker 48:45

It's easy to say.

Knox Hutchinson 48:46

So you can say, like, I'm running this on my computer across town in a data center, and you can still connect to it. And it's still that that inference never leaves your infrastructure.

andy lapteff 48:55

The fact that you're thinking about local models and you build it into your software just shows me that your head's in the right place, right? Because people are gonna there is a subsection of folks who will be interested in that. Even for me, I have a bunch of old cards laying around from things I used to do, and I have Olama on my laptop that I haven't messed around yet. But um, I had to do a bunch of transcription stuff for for the podcast, and I was gonna, you know, try to do it on my GPUs, right? Just to, oh, you know, this will be a use case for for local models. And honestly, it came down to time. Like, you know what? I can throw it at a cloud service for like 30 bucks and just be done, or I can spend a week trying to get my three GPUs in parallel mining to like do the thing, right?

Knox Hutchinson 49:35

So that's there, but uh if the application that you're using, like transit here, doesn't use anything for training data, we don't store any of the chat history. Uh so like so, and if I send something to anthropic, anthropic like legally has to store the chat history for 30 days for abuse detection.

Speaker 49:54

Okay.

Knox Hutchinson 49:54

So like that that is out there, but then it gets flushed after the abuse detection period has gone. And it's not used for training. So it's like, I'm not saying you're wrong for still being conscientious and not wanting to do it. And that's why we ship it with local models, because there are definitely enterprises out there that are like, we've already invested in this infrastructure to get around any privacy concerns. Awesome. I am with you, and I think that is a really great idea. I love local AI, I have invested heavily in it. I guess what I'm saying is like there's a lot of people out there who are like, I only want to use local AI because you're gonna use the data for training and it's too cost prohibitive to use local AI, but I don't want to do the cheaper, affordable way of cost prohibitive. So it's like you're kind of like canceling yourself out of using AI at all.

andy lapteff 50:45

Right. You paint yourself into some corner. Like I I love that transit AI has the easy button. So pay the thing, it's all

DMVPN Troubleshooting Walkthrough

andy lapteff 50:52

built in, and yay. But if you're the person that wants to do it locally, you can. Is there anything else you wanted to show before we tell people how to find it and get it?

Knox Hutchinson 51:00

There's a there's a ton of features that we could demo. They're all features that you would come to expect from an SSH client. Things like you can see the button bar, the broadcast bar, you know, launching sessions in bulk, closing sessions in bulk. One of the features that we just shipped recently is the ability to actually do a snapshot. So if I I'm gonna demo this one real quick. If I do, like, let's say I want to do capture like this show run on Cisco R1 is my golden state. So I'll capture show run right here. Then let's say somebody comes in and creates router OSPF three and sets a network one to one to one to one. You can kind of guess where I'm going with this 000 area three. End.

andy lapteff 51:46

Oh when I go ahead. Hold on. You just lost me. So you're you're doing configuration commands in transit AI, right?

Knox Hutchinson 51:53

Yeah, it's an SSH app just like any other.

andy lapteff 51:55

Ah, okay. So earlier when we said so the the LLM, the AI can't make configurations. You could so so this is helpful, I think. This is basically an SSH client, not basic. It is. It is so this is is this free? I could just download this and use it as an SSH client.

Knox Hutchinson 52:12

You can download it right now and replace your current SSH client, no cost for free. Okay, and the feature that I'm demonstrating right here, it's packed into it for free. All right. So I've got a golden state configuration. I've then made a change. Somebody else, I should say, excuse me. Why would they do that, Nox? I would never, but somebody else makes a change, and now things are acting funny. So what I can do is I can take a second snapshot and then compare the original snapshot to the new snapshot, and it shows you a doing a diff on the golden, yeah. Exactly. So you can detect state change and configuration groups directly.

andy lapteff 52:57

This is really, really useful.

Knox Hutchinson 52:58

This is built into it for it's all sorts of stuff like that. Session logging, exporting the scrollback, uh small things like you know, your credentials aren't stored in a text file. You would think that would be common sense, but nearly every SSH client stores your credentials in a text file. Um, our all your credentials, if you are typing in just a username and a password with basic authentication, it goes to your operating system's keychain. So Mac OS Keyring or Windows Credential Manager. Alternatively, we integrate with one password. So if you store passwords in one password or SSH keys in one password, you can now authenticate to all your devices with a fingerprint, a fingerprint scan to one password. And if you're in an enterprise where like one password is where you share credentials, you can point it at that and your whole enterprise can use transit.

andy lapteff 53:48

This is multi-vendor. Yes. So uh let's say I have devices that aren't in the pre-populated list. Yes. Is it easy to create a new, you might have called it a profile or something earlier? What's that called? This is just a device.

Knox Hutchinson 54:05

So that's a good point. So how does transit know that the command would change the configuration when the actual configuration statements differ from one vendor to the next? On uh Cisco, you type conf T, right? On Juniper, you might type edit to get into configuration mode. So how does transit know that? So that's why the policy gate works on a per vendor basis. So when I'm creating a Juniper connection, I'm gonna say this is a Juniper device. So it knows what to evaluate it against. And it also knows what commands it's allowed to run. It knows, like, hey, show interfaces terse is different from show IP interface brief.

andy lapteff 54:48

Is that something I do initially when I install this? I gotta tell it what the devices are.

Knox Hutchinson 54:53

So these are the operating systems that ship with it out of the box. Junos, all the Cisco ones, Arista, Palo Alto, and then Linux is the one weird one where we we set it to unrestricted because there are an infinite number of commands that you can chain together on Linux and still get it to run a configuration. So the flip side of that that we did is there is no like auto-approved. You saw me do auto-approved to show commands, like everything that begins with show, I'm fine. On Linux, that's not allowed. You have to explicitly approve every single proposed command on a Linux device. Now you're sitting here thinking, like, wait, I don't see Fortinet, I don't see Unify, I don't see uh MicroTick Router OS, I don't see whatever Adva and RAD if you're in a service provider environment. So we ship something called bring your own policy. It is very easy to create your own policy to support whatever operating system you want. You simply create a policy, it's a basic YAML file, and you can use Chat GPT to generate it. And then you just tell it the policies live in this directory. From there, whenever you create, say, a Fortinet, if I have a Fortinet policy, which I do, I then just tell it, use the Fortinet policy right there in the drop down. So now I can connect to this Fortinet device. Uh let me close the import sessions here. I've got a Fortinet session open. I can now switch this to Opus, and I can now say, I'm only interested in talking to my Fortinet device for this chat session. I can now say, tell me about this Ford iOS device, give me a lay of the land, which is a great thing to do before you make a configuration change. Just tell me what I'm up against. Tell me what's going on. I've got 300 customers and they all have different vendor equipment. Like, remind me where I am. So it's now allowed to propose Fortinet commands like get system status, finding out the route table, and so on, and it'll generate a quick summary of like what's going on on this Fortinet device. So that's how you can bring your own policy, the entire setup start to finish to support an operating system like Unify. I have my UDM Pro right here at the bottom, or Fortigate, like I just demonstrated. You can now support any operating system in about two minutes. It is just as simple as creating a YAML file and pointing to it in your configuration.

UniFi Home Lab Demo

andy lapteff 57:13

You just blew my mind. I run Unify at home. I don't really troubleshoot in it. Occasionally I'll get weird messages and I, you know, I'm kind of lazy at home. I don't go into troubleshooting, but are you telling me I can use this on my UDM Pro?

Knox Hutchinson 57:33

I'm doing it right now. So give me a lay of the land of this UDM Pro. So I actually used this about a week or two ago to discover an issue was going on with my IPv6 environment that I didn't even know.

Speaker 57:44

Wow.

Knox Hutchinson 57:44

It was like it's it told me like you only half configured IPv6. You forgot to like check this one box.

andy lapteff 57:50

See, this is fascinating. Now the engineering me is coming out in the operator, but I I believe that that platform gives you some form of troubleshooting stuff, but it I don't get the experience there that I'm getting here. What I mean is I'm in my office, I have a hard line here, it hits a switch locally and then goes down to my DMARC to my you know where my UDM switches and router is and stuff. And I occasionally get notified in the app that there's um, I don't know if it's like CRCRs or there's there's some kind of issue happening on this particular feed. And this is my office, this is my studio. Like I don't want to have that problem, but it's never really manifested as a problem. Like I don't have problems in these recordings. I'm not telling people they can't hear me on calls, but there's something happening in my local network. I don't know what it is. And I guess why I'm excited is I would like to download this, get it working on my local environment, and ask it. Like maybe it could look and figure it out because I don't really, it just tells me there's errors on a port. That the local thing. It doesn't tell me why. Like, is it CRCs? Is something loose? Like CRCs to me, it's like, okay, there's physically there's something happening. It's a bad cable, something's crimped, right? Like there's errors that the frames are getting damaged. I'm not beating up on the system I have, but I just don't get that level of granularity where I think so so show me what you have up here. What's it telling me about your Yeah?

Knox Hutchinson 59:13

So I mean I just told you to give me a lay of the LAN. It tells me, you know, the version, the controller version, what the WAN configurations are like, the LAN configurations. It's asking me follow-up questions. Do you want me to pull switch port status or connected clientless next?

andy lapteff 59:28

So could I ask you, like, look in the logs for an issue and tell me, like, start troubleshooting? Like my my errors on a port as an example. I would like transit AI to investigate it for me. Like, why do I keep having this intermittent problem, right?

Knox Hutchinson 59:41

Are all ports healthy?

andy lapteff 59:43

Yeah. Let's see and then if if not, you could dig in, right? I mean, that's Yep.

Knox Hutchinson 59:47

It's just gonna start running. Oh, there's the policy gate in action. It stopped a command from running. So see, it proposed ETH tool switch zero, and the policy gate stopped it from running that. There's a few commands that stopped there. Mostly healthy. See, look, I didn't even know this.

andy lapteff 1:00:03

Right.

Knox Hutchinson 1:00:04

So, like, even from the case. I didn't even know there's no problem. But right there, like, there's there's receive errors right here. Negligible rate, but there's air. I didn't know this was happening on my own stuff.

andy lapteff 1:00:12

Well, this is magical, right? Like proactive troubleshooting. I worked in a NOC for two years, and when things aren't on fire and you're just sitting around, that's the perfect time to dig around and try to find issues before they manifest into outages. There's that. And I could see this doing that for me. Like, hey, it's quiet. Let's dig around. Let's look. Is anything happening? And to your point, like, okay, it's not a lot of errors, but like something's going on there. Is that going to escalate into something eventually? Is that small problem going to amplify into something bigger?

Knox Hutchinson 1:00:41

If you're not proactively monitoring that now, then this is definitely something, like, if this is the tool that you're already using, you can jump in and just ask a question in plain English. And I think that's the real win. Is for me as someone like before I make a change, because that's where I've I get cautious. Like, yes, I can use this tool to troubleshoot all day long, and I do in production. I want to stress that right now that I use this in production nearly every single day. The other thing that I usually do is like before I make this change, like look at the environment and tell me what's going on. Because there might already be errors or things or misconfigurations that I missed or did not know about, like these receive errors right here. That's absolutely something that I do. Just the give me the lay of the land. That's the prompt that you saw me type right up here. Give me the lay of the land. I always start with that. It helps the LLM build context, but it also helps me build context before I'm like, I have this great idea. Let's implement ISIS instead of OSPF.

andy lapteff 1:01:44

You know, like you know, like the last thing the last thing I'll share before we get into how people find and and download this and try it out for free. You and I were talking before that I had an issue in production that I think it took me like seven or eight days to figure out. And I had like a handful of people working on it, but I had a new circuit I was bringing up. So, you know, I was I was in the data center, so we have well, I don't want to give away the thing, but I guess I will give away the thing. So we have jumbo frames everywhere, like MTU 9000, whatever it was. And I had a new circuit I was bringing up at a site connecting back. I might have even been in the data center, but the long and short of it was uh we were building tunnels and the tunnel was bouncing, and I couldn't figure out why, because I've configured thousands of these tunnels and I'm very familiar with the protocol. And like, I'm telling you, man, like seven or eight days, and I have people looking at it, and then I finally asked the senior guy who's been around 40 years, guy John, I'm like, dude, because I can't, I'm burning in the circuit, right? Meaning, is this stable before we put traffic on it? And it's bouncing. I'm like, what the hell's happening? And I couldn't figure it out. Documentation, I don't know if I opened a ticket. Long and short of it, John's like, did you check the MTU? I forgot on the new the interface on the new circuit connected to my device. I forgot to configure a jumbo frame. So there was an MTU mismatch on the two ends, and it was bringing now that felt never occurred to me. It was eight days in for me to figure that out until I finally was able to fix that, get it up, and put traffic on it. Transit AI, as an example, probably would have found that in less than a minute, I'm guessing, based on your DMV, because that that's way less complex than the DMVPN example, and that would have saved me a week and countless hours of bothering everyone that would talk to me. Like, can anybody look at this and figure it out? Because I can't. So I think just one of hundreds of examples, right? You made the point for consistent.

Knox Hutchinson 1:03:31

So, like I've I've demoed this on YouTube and LinkedIn and socials a couple times. There's OSPF MTU mismatch, BGP MTU mismatch. I've demoed transit handling that exact scenario. Yeah, stuck in Xtart.

Speaker 1:03:44

Yep.

Knox Hutchinson 1:03:44

Is a big one, and it's not obvious. No, it's not, it doesn't look like an OSPF configuration problem because it's not, it's an MTU problem.

Speaker 1:03:52

Right.

Knox Hutchinson 1:03:52

So I have demoed this on YouTube, on LinkedIn, all the places. I think it solved OSPF in like 45 seconds and BGP closer to 90 seconds.

Speaker 1:04:03

Faster than eight days.

Knox Hutchinson 1:04:04

Yeah, exactly. So I mean, but that's that highlights the real problem. Like, you know, you could have been empowered, you were maybe hesitant. I don't want to say afraid, but you were worried about escalating this up to the senior because the seniors are all grumpy, right? I'm kidding. But like there's there is the if you are a senior, you need to know that most of the people around you are afraid to ask you questions. What if they weren't? Like this tool doesn't have to be for you, the senior, who knows every command by heart, right? It's so it's for the people around you. And those people around you, they're a pressure release valve. All of a sudden, they're capable of doing things that they couldn't and ended up getting dumped on you at the last minute. What if they all of a sudden became confident when talking to you and it became a collaborative environment? This is the real decades-long problem that we have, not because network engineers are inherently grumpy people, but because the work is hard, because it weeds people out early, and because there's not enough of us to do it, and because the stakes are high. Like you just gave me the exact example. Yeah. And I've lived that exact example too. I want to be very, very clear that I have lived it many, many, many times. My the example that comes to my head when we're talking about this, I had a spanning tree issue. So I was in my first year, I was working for a government entity, they had all Cisco on a multi-building, multi-story environment, you know, IDFs and MDFs every floor, multiple of them. And we were converting from Cisco to Juniper. Well, Juniper and Cisco don't have the exact same flavor of spanning tree. I I swear I spent and we we did this, we started it on a Friday night. We didn't finish it until Sunday night. Because we ended up having to get Juniper support involved. And it's like, what if this could have just solved it in like 30 to 60 seconds? It probably would have. It's pretty darn good at spanning tree. Really, the LLMs are pretty darn good at spanning tree. That's just where my head goes. And I was terrified to go to the senior who I was working with and telling them, like, dude, I'm stuck. Like, I will because I wanted to prove myself. Like, I had been in tech, I've been writing software for a decade at that point. I'm starting over from scratch as a network engineer in 2015 or whatever it was. So I really felt like, oh, like, oh my God, like I've, you know, because I've been in tech for so long, I've got a higher than normal salary for someone in my skill set. So I really got to prove myself, but I'm too terrified to get this wrong. But like I don't know enough about Juniper's version of spanning tree to compare it to Cisco's. So I'm just gonna sit here for the entire weekend. And I did.

Speaker 1:06:59

I'm laughing because you just you just described my entire career in production.

Knox Hutchinson 1:07:03

I I think I described a lot of people. I don't I don't want to be presumptuous, but I think a lot of y'all have felt that. And that's time, man.

andy lapteff 1:07:12

I I I keep coming back to time, right? Like if you can be more efficient with your time, if you can troubleshoot something in a minute as opposed to eight days, there aren't enough of us, it's super complex. You don't want to bother the seniors. Like the problem that this solves is the right problem in networking. Um, if you can empower juniors, if you can give insights, and your boss is terrified of losing you.

Knox Hutchinson 1:07:39

Like, you need to know that your boss is terrified of losing you.

andy lapteff 1:07:43

Like everybody's stressed, everybody's I don't want to say scared, but right, like, don't break the things, keep it up. Like, we're all trying to create as much uptime as we can, and there are is just an endless array of issues and things and weird stuff, transient, intermittent, like any intelligent software that I could point at it safely, right? Which is what you've baked into all of this, makes a ton of sense to me. Um, this is a free-forever SSH client replacement. Is that correct? The app. So I can download it and use it today. Yes, the app, like just the SSH client.

Knox Hutchinson 1:08:18

You do not have to use an SSH app from 2016 anymore. Right. You can download this right now, no account, no problem. The AI is the only thing where a subscription gets involved.

andy lapteff 1:08:28

Correct. And I believe there's a 14-day trial on the AI. Does that sound correct?

Knox Hutchinson 1:08:32

There is. There is a 14-day free trial. So, you know, if you're listening, if this were me, and I'm not telling you what to do, but if it were me and I'm working for a business and I'm troubleshooting spanning tree, I would go to my employer and say, like, there's

Documentation and Trust

Knox Hutchinson 1:08:47

a 14-day free trial for me to try this. Are you okay with me trying this for free for 14 days? And if I don't like it, I'll cancel it on the 13th day. I would ask the employer about if we can use this because I think they will also see, like, hmm, this is a lot better than hiring a new employee for $65,000 a year. We could, for free, give it a little pilot run uh and see how it works. If it were me, that's how I would approach trialing this.

andy lapteff 1:09:16

I would and your documentation is dense enough that you could give it to your manager and be like, listen, look, like it's right, it's safe, it's good, it's right. It's not this vague tool that we just have to trust this Knox guy. Like they can look and see what it's doing and if it falls within their rules of engagement in their environment. That's right.

Knox Hutchinson 1:09:33

That's I I never want you to do something that would violate your own corporate policies. So don't do that. Do not go download this right now and use it in production without permission first. Yeah, uh, that is a very important thing.

andy lapteff 1:09:46

But you get a 14-day free trial with the AI. There's I think multiple tiers of AI, right? Heavy users, light users. Um, there's some local LM stuff you're looking at. Yep. Um, you can go to transitai.app slash a one to check it out. Yep. Um that's transitai.ap slash a-o-n-e. It's not a brand new tool that you have to learn. It's something that'll feel familiar, but it has a chat bar with no friction.

Knox Hutchinson 1:10:13

That's what I want you to know is you're not having to deploy new infrastructure and learn a new tool. It's the same app you've used for 20 years, but with a huge facelift, some more modern features, and then the option to add the AI to it later if that's what you want to do. A lot of people are downloading the app just to use the app, and then later saying, like, oh, I'm really stuck on this problem. Now I'll check out the 14-day free trial. And I think that's perfect, great. And if you if the AI doesn't help you, then cancel it. Like, I still think that the app has a lot of features in it that are really cool and

Free Trial and Where to Find It

Knox Hutchinson 1:10:47

are worth using.

andy lapteff 1:10:47

Where can we send people uh if they're a shorter form? So this is going to be a longer episode. Um, do you have a YouTube channel or a place people can see demos and things like that?

Knox Hutchinson 1:10:56

Sure, sure. There's the Data Knox YouTube channel. There's also a transit AI YouTube channel, which is more of the how-tos, how to set it up, how to create an auth profile, how to bring your own policy. I would say go to transit AI if you really want to learn how to use the app. But you can always, and I mean always, send me feedback on Twitter or LinkedIn. Find me. If you have something, if there's a feature that it's missing or something feels rough, you have a direct line to me and I am the one who's making it. So I need to hear your feedback.

andy lapteff 1:11:30

Nox, it was great to see you. I hope that we get to see each other more often. I missed you while you were gone building a very amazing uh app that's gonna help the network engineering community. For all things, Art of NetEng, you can check out our website at artofnetworkengineering.com or artofneteng.com. All kinds of uh amazing new resources in there in our resources section. Uh we just launched our A1 newsletter today, so go uh sign up for the newsletter. Woo! It's good, I read it. Oh, good, thanks. Yeah, we're we're distilling you know all the insights from six years of conversations with brilliant people like you and how it can help uh you know your career so that you don't have to make all the mistakes we made. As always, thank you so much for watching and listening, and we'll catch you next time on the Art of Network Engineering podcast. Hey folks, if you like what you heard today, please subscribe to our podcast and your favorite podcatcher. You can find us on socials at Art of NetEng, and you can visit Linktree forward slash Art of NetEng for links to all of our content, including the A1 merch store and our virtual community on Discord called It's All About the Journey. You can see our pretty faces on our YouTube channel named the Art of Network Engineering. That's youtube.com forward slash Art of NetEng. Thanks for listening.