Intro

SPEAKER_00 0:00

This is the Art of Network Engineering, where technology meets the human side of IT. Whether you're scaling networks, solving problems, or shaping your career, we've got the insights, stories, and tips to keep you ahead in the ever-evolving world of networking.

andy lapteff 0:14

Welcome to the Art of Network Engineering podcast. My name is Andy Laptev, and we're having a party here in Santa Clara, California. I am with my good friend Remington Luce. How are you doing, Rem?

SPEAKER_04 0:24

I'm doing great, thank you. I managed to get a full night's sleep despite an early departure, so I'm I'm doing awesome. Thank you. Just throw it in my face.

andy lapteff 0:30

I'm so happy for you. I'm sorry. Can I say? You have the best name. It's like a Hollywood name. Remington Loose, right? I feel like it's like a rock star name or something.

SPEAKER_04 0:38

Yes. Uh I do. The downside to that is that when you get called into the principal's office, there is zero doubt who they're calling in. Like you definitely know it's you. There's just one.

SPEAKER_01 0:46

Remington loose, please come to the principal's office. Oh no.

SPEAKER_04 0:49

Yes, exactly.

andy lapteff 0:50

But your coffee's usually yours, you know. That is true. And you go by, well, I won't tell people what you go by because it's none of their business. Yeah.

SPEAKER_04 0:57

Whatever you want to call me. Only me and a select few people. I prefer not profanity, but you know, if you need to, go for it.

andy lapteff 1:02

I probably

St Jude Research and AI Trust

andy lapteff 1:03

should have done this research ahead of time and reference the episode you were in previously. So uh an episode blah blah blah. Uh Rem was a previous guest on the show. Uh, you want to tell people where you work?

SPEAKER_04 1:12

Yeah, so I work at uh St. Jude Children's Research Hospital and uh it's an academic medical research institute that's uh dedicated to eradicating catastrophic childhood diseases, which is a mouthful. But um we we work primarily on diseases that uh have had a very low level of support over the years. And so we have a really small niche uh set of patients, uh, some of which will take a very long time to accumulate enough data in the traditional scientific um medical community to get treatment. So we're trying to figure out how to accelerate that.

andy lapteff 1:43

And part of the name is the children's research hospital, right? Yeah so there's a big research arm and I I've I've done a lot of uh research over the years for guests and for shows. And I think researching St. Jude was the only one I like cried as I was preparing, just because the the YouTube channel that I was going through and just all the the inspiring stories were like, oh my God. So anyway, I I beat that you know last time about like what a meaningful you know work it must be. But it is awesome. The research part kind of fascinates me. And probably the last episode you saw, I don't know what order we're gonna release these in, but Scott Robon and I, previous to you coming in, we were talking about um basically trust in in AI, right? Um networking, even with network automation, if you think about it, like the blast radius is huge. And there's a whole Scott was walking us through if you have a new vendor coming in, are you gonna like refresh gear? You do this whole like bake-off and proof of concept. You know, it could take five or six months to try to figure out, you know, how can I trust you, Mr. Vendor, and your solution to make sure the thing keeps going. And and we kind of danced around the AI thing too. Like, okay, well, how do you bring this in and trust it? In relation to, I guess, the research part, and correct me if I'm wrong, I think this was you, but I don't know if it was like protein folding we were talking about once, but what I'm fascinated, you know, we're all talking about AI now because it's the thing. But what I think is fascinating is I don't meet that many network engineers slash architects, principal oolalagae, whatever your current title is, but I don't meet many that are utilizing the technology beyond, you know, fancy Google, maybe some AI ops troubleshooting. Um your research, I think, ARM. So I know we're not talking about your job and what you do, but I would love to pick your brain, I guess, on your experience with LLMs and AI and sure and kind of the real a person using these things beyond the cursory things we all talk about, which is like it's good at reading logs. Yeah. So are you able to kind of dive into a little bit of what you guys do? Absolutely.

SPEAKER_04 3:33

Uh it is good at reading logs. Uh it's also good at reading round tables, you know, pre-check, post-check kind of validations. It's super helpful. It's good at code. Um, I know that that can be uh a bit of a hot take depending on who you talk to. I don't, I'm not a software developer. I've written a little bit of Pearl, some very terrible Python. Uh so it's it's helpful to have something that can at least generate code that will work. And I don't need to understand all the mechanical parts to make that code happen. So that's super helpful. But it's really useful in a lot of other ways too. So one of the big things that we started, my first journey, you asked about my journey. So let's talk about that for a second. So my journey started about nine months ago with AI tools. Our principal infrastructure on the security side uh architect kind of reached out and said, Hey, I want to try to build something. One of the challenges that we have is uh what we call solutions architecture, right? So we have something that we need to do or deploy, and it will be relatively complicated to work out all the mechanical parts. We need a lot of expertise to make that happen. And so it can be a challenge to get enough architecture time to really figure that out, document it, make sure that it's ready to go, and then have a source of truth for that when something breaks so that we know, okay, this is how we need to fix it, right? So we call that solutions architecture, laying all that stuff out. So we wanted to basically upload ourselves and some other very smart people at St. Jude into making a solutions architecture agent. So that was kind of the first step. And that ended up turning into, I don't know, 10, eight, nine agents, something like that, on a multi-step flow. Uh, and it's taken over kind of running an actual operational workload for us, which is pretty cool. So I can talk a little bit about that if you're interested in kind of going that down that road. We do have some uh what what in the business is called sovereign AI related functionality. So we both we have it both for our open science environment, which is an area that uh that we use in in research, and then a protected environment as well. So um I can talk a little bit about that too, if that's where you want to go.

Three AI Topics Outlined

andy lapteff 5:19

You're so good with this, you should be my co-host because you just laid out the entire we we were, you know, uh I'm kind of winging it usually, like, what are we gonna talk about? You're like, Well, these are the three things I can talk about. This is what little yes, thank you, thank you, Rem. Man, I'm just trying to keep my own thoughts in mind. Dude, this is amazing. So I think it might be helpful to define agent first. Yeah, and because my experience was when I first started six or eight months ago, I wanted to automate some of the operation stuff of the show and the business end of it. And I started talking to LLMs, you know, let's let's create because I I heard these podcasts and these videos like, hey, people are, you know, 100 agents running a company with two humans. I'm like, oh, that sounds great. Yeah, let's have digital employees. Yeah, yeah. And I started down that path. And about six months in, I realized I don't know if I have agents, yeah, or if I have Python scripts running on cron jobs. As it turns out, I have the latter.

SPEAKER_04 6:09

Uh-huh. That's okay too, though. That's not bad. That's good, right? That doesn't burn any tokens. So that's really affecting.

andy lapteff 6:14

But I want to run the uh I want to run my understanding of an agent by you because recently, so so again, I'm talking to my AI and I'm like, hey, six months ago, I started asking for agents. Turns out I don't think we have agents. So again, circling back, what do I know to get to get it, you know, get agents? And so it's what's compelling to me is I think the difference is agents can go, they have access to tools. So it's I guess it's software. Yeah. It's a bunch of if-thens, I think. I don't understand development, but it's it's a butt, it's it's programs, the algorithms, but it can go out into the world, it has access to tools and things. You give it a task, and it works until the task is complete. Does that sound yeah?

SPEAKER_04 6:51

So that's that's definitely uh a big part of it for me. The other part I think that um when I think about an agent, it's that the input maybe uh is not fully determined in advance. So you can essentially hand it something, right? And then then it will take and begin working on that. And a goal, right? Like do something. So goals are a thing that you can do on a model. So maybe let's talk talk through a couple of parts real fast. So uh the analogy when I talk to folks uh about anything AI related, right, is that you have the model first. So the model I try to encourage people to think about is a bulldozer. And so the more, the more powerful the model is, the bigger the bulldozer is, right? So the bigger the work that you could get done with that particular model, right? The next thing that comes in line behind it is the harness, and the harness are the controls that are associated with the model, right? So you can have the world's most powerful bulldozer, but if the controls on it are terrible, you can't get it to do anything productive, right? So harness really matters. That's controls. Controls. Okay. Yep. And so controls are the harness that when we talk about harnesses, that's that's what gives you the ability to leverage the bulldozer. And then you have this sort of context layer that sits on top. And that can be memory, it can be uh the ability to try to leverage other tools, and it could be your your understanding of how to leverage that harness, right? So we're not even talking agents yet, we're talking about just a human interacting with a model, right? So that context is really super important because even if you have the best controls on the world's best bulldozer, but you don't know how to dig a lake, you're gonna wreck the bulldozer and potentially kill yourself by rolling it. Right. So so that analogy works, I think, really well across the board. So what what you want to do when you get to an agent is you want to build that whole system and then you're essentially handing that off to let it run on its own, right? So I need you, yep. Pause.

andy lapteff 8:33

So build the whole system is the model, which is the bulldozer. Yep. The harness which controls the bulldozer, yep. And then the context, which leverages the harness. Is that right? Yep, exactly. What happens to me is once we get two or three layers deep in abstraction, it starts. Something happens to my brain. And I don't know if that's common or if it's just like it's super common. So I'm just trying to follow to make sure. Yeah. So all right, you went through model, harness, context. And because my brain got fuzzy, what was the next part?

SPEAKER_04 8:57

Is to be able to build in my mind, the way I describe to people is when you wrap all three of those together as its own sort of autonomous thing. Uh-huh. So you can say, I want a lake over there, right? So your goal concept, just to talk about that and tease that apart for a second, it is a thing that you can use. And the agentic tools, you know, if you're using one of the big Frontier Lab agentic tools, many of them will have a slash goal they call primitive. So it's a thing that you can fire. Hey, I want you to go do this. And what that really does is that sets it in a loop, right? So that's what they in the industry. If you start looking at it, that's kind of like a loop engineering system. So you say, This is what I want you to do. This is when you're going to be done. I want you to keep working until you generate whatever this is. So that's that is a thing. When I think of an agent, an agent is almost when you go back to that employee, that's kind of the better framework to think through of like there's a model, the model has an interaction surface or a way that you can plug stuff in and get stuff out, right? So that that becomes effectively the harness. You're going to feed it a certain amount of context so that it understands what it is you're going to ask it to do. And then you're going to ask it to go do something, right? So in our case, when we were trying to do solutions architecture, we would get these demands. Somebody wants something to happen, right? Whatever it is. And we would then need to figure out, okay, first of all, is there something that we need to do for this demand? Like if I needed to order flowers from my wife, no one in in the IS department needs to engage, right? But that could be a demand, right? I mean, that could be something that we would, you know, we wouldn't get it, but you get what I'm saying, right? The first thing is figure out whether or not uh someone needs to do something here and what are the gaps. So in the workflow that we built, we handed that to an orchestrator agent and it would parse it out. And so this is where the agent's parts. I'm gonna try to stack this up for a second. Okay. So just let me run this and then we'll break the layers back down if you want. So the orchestrator agent picks up, it consumes uh a piece of work. It says, okay, what is it that I'm being asked to do? And what are the gaps in what I see versus what I would want to know in order to reach the outcome that I have to? So we we had hundreds of examples of what good looked like because they were written by humans. So we knew what we were trying to do, and we could use those as a baseline to compare and say, okay, did you get to success? Okay. So if there are open technical questions, then what we would do is we take those technical questions and we would hand them to another subagent whose job it was to research those technical questions. So they would go out on the internet, it would wander around on the internet, it would pull stuff down, it would, it's trying to answer these questions, right? So in the same way that you would have an employee and you'd be like, Hey, I need you to go write me a report on this technology and determine whether or not this is a viable option, right? So back to your Scott Reboyne POV hardware. Hey, is this vendor's switch appropriate? Hey, is this AP gonna work? Yes or no? Why? Right? There's a set of technical questions you've got under the hood that you want to answer and it's gonna go figure that out and come back with an answer. So we gave it a structured set of inputs, a structured set of outputs, and then we just let it run using a specific model, doing whatever it needed to do with its instructions. Does that help? Yeah. I have notes. Okay. I just want to stress I am an actual network engineer. That's what I was trained to do. So, you know, parts of this may not be totally this is fascinating.

andy lapteff 12:08

I mean, this is the world that we're operating in. And, you know, to circle back to my silly example in the beginning, six or eight months ago I started with the goal of building agents to help me do things. Yeah. And then I find out I have built zero agents. So, you know, I thought I was building agents. So this is so this is very helpful to me kind of documenting the process and and how you've done it and the difference between an agent

Agents vs Python Scripts

andy lapteff 12:32

and a Python script running on a cron, right? So so this is helpful to me.

SPEAKER_04 13:41

The biggest thing that's super helpful to think about in the agent case is that there's a model attached, right? So when you think about your Python script, it has a set input because it's software, right? So you you have to give it the exact right thing for that script to run. If you don't, it won't run, right? It literally will not work, right? So the model that you have under the hood, that's what gives you that flexibility because you can ask the model to basically figure it out.

andy lapteff 14:04

Well, let me give you an example. So um I have a media kit with my ad products that companies pay for access to you know our audience, our audience's attention. And that media kit has to be updated with metrics that change every day. I mean, listens, downloads, followers, whatever. And every single time for the first six years of the show, someone would reach out, we would go out to all six social media platforms and the and YouTube and the audio pod, you know, hosting, and get all those numbers real time, put them in a Word doc, export to PDF, send them to sponsor. So I thought, well, this is very time consuming. So I went into a model, cloud code in this example, and I said, Hey, I would like to create you know, agents to help me do some of these, you know, this manual work, and this is what I need done. And I thought I was building agents. Now it might not I don't know if it matters, but the reason I'm pointing this out is what I found out later was it wrote Python scripts that would run on a cron job. So every day at 8 a.m. it would go out, it would get all the metrics that would pull it in now. Yeah, but that's fine, that's cool. And maybe I don't need an agent for that. I don't think you do. Right. But I'm like, so I was six months after OpenAI launched, like ChatGPT, which is about three years ago, give or take. I jumped right in. I'm like, all right, this is cool, let me figure it out. So I'm three years into this journey, and I've spent six months building agents that it turns out weren't agents. Yeah. And and I'm not a dummy. So I say all that to say that like I spent six months building building agents that weren't agents. Like, yeah. So the big I can hear the audience laughing. Like this input. Well, no, no, no. But like your inputs are standardized. But I have them I had a model, yeah, and I told it what I wanted. Yeah. I asked for agents and it wrote Python scripts. Yeah. Now maybe it's my input that wasn't right, but how did you build agents and not spend six months having it write Python scripts on cron jobs? Like, what did I do wrong?

SPEAKER_04 15:55

You I don't think you did anything wrong. Right. So like that's the big thing that big I think a lot of people get uh a little over rotated on is like, oh, I gotta use AI to build agents to do whatever. I honestly, I think the best part about AI is the ability to have it help me write software. It's the agent part is is great. You know, that's super helpful. I'm not just I'm not trying to discount that, but in general, what I think a lot of people will end up doing is they'll end up building applications first, whether they're web apps or command line apps or whatever, right? So let me kind of maybe speed run in my own mind where this is actually. Which is what I did. Yeah, like I thought I was building agents, I was building little pieces of software. Right. And so what those little pieces of software are doing for you is they're combining other data or they are smoothing over the interaction surface that you have between other applications in your life. So you you basically were able to write a custom application that met the need that you had to do whatever you wanted to do, right? That's super helpful. Once you have that application, though, it can start to do stuff like build you a source of truth. So now you could you could pull that data that you're pulling from those six other locations into a database, right? Or or whatever, right? Pick a method of storing that data. And then you could interact with that data. When you layer a model on top of that data and you want to start interacting with it in ways that were not previously scripted, right? Just ask it a question. Hey, do I have more views on Wednesday than I do on Thursday? Right. Well, you could write software to do that, right? You could you could go have it write code to do that. But if you just feed the data directly to the model and then you ask the model to do something on it, there is the capacity for the agent to do something. So you're gonna give it a set of instructions. Hey, I want you to be my data analyst. Here's your database. I'm gonna ask you questions about it, and I want you to surface them.

andy lapteff 17:32

How do you feed the data to the model? I sound like Philly to the model. Uh and why I'm asking is I have clawed code installed on my server at home. Yep. I have all the software I've built in there that's gonna repose and all that stuff. So I thought my model had access to the data. Yeah, it can. So I I'm I'm kind of comforted that it sounds like I'm just on the path of like you said, well, you start with applications and then you start to figure it out. So, like now, like as a source of truth is an example. So let's just use sponsors as an example. You know, help me find sponsors, okay? It'd be nice if I had an agent that would go out and say, Oh, I you know, identify these three new companies or startup, they just had a C round, you should reach out, they're networking adjacent.

SPEAKER_05 18:08

Yeah.

andy lapteff 18:09

I don't have that. I have a static spreadsheet that I found somewhere, like, and then I have three different applications I wrote to take that spreadsheet, prioritize them, scrub my non-competes, and then find marketing contacts. Like, but these are like six or seven different pieces of software that do all these different things. It's kind of like Frankenstein. Yeah. And to me, I don't have an orchestrator, I think, because that's that's it. Yeah, so that's where I'm getting at it. So you went from agent, which I was trying to build, and then you went into solution architecture and you said orchestrate, orchestrator agent. So it discovers, I think, the delta between the known and the unknown. I think that's kind of like how you put it. Yeah, exactly. So that might be my gap.

SPEAKER_04 18:44

So the orchestrator is really helpful, and there are deployment platforms to allow you to kind of do what you're talking about, right? So N8N is a real common one. Yeah, I'm trying to think of the other one.

andy lapteff 18:52

My buddy Chris is all about N8N, and I still don't understand how to make it go. Well, I haven't downloaded it, but like, so I guess this is a journey. Like all of it, like, oh, learn automation, start with Python, and then you learn Git, right? So I think this is a similar journey. So I guess my next part of my journey is all right, I have all these scripts that are running on crons, but there's no orchestration. There's no I uh as I was chatting with an LLM recently, like, well, you build a bunch of arms and legs to do things, but you didn't build a brain. Yeah, which I guess is the orchestrator. Is that kind of right?

SPEAKER_04 19:19

It's exactly right. Yeah, so N8N would be an example of an orchestration software, an agent orchestration software layer that you can deploy. And so what you do when you have an N8N, I don't even know if I'm saying that right, because that's probably something else when you actually ask them. But regardless, when you have something like N8N,

Feeding Data to Models

SPEAKER_04 19:35

you what you're really doing is you're you're plugging in a model, you're plugging in tools, you can plug in uh a vector database or some other data source, and you build a flow using that orchestrator that then runs. And you have an interaction method with that particular agent. So you can you can basically communicate with that agent by a chat, you can hook up video, you can put voice. I mean, people have done all kinds of stuff in order to make that go. And you could pick any model you want underneath of the hood, right?

andy lapteff 20:01

But what is it orchestrating? I'm sorry to cut you off. Is it orchestrating my 82 Python scripts and my 170 cron jobs? It it can. Like I don't know.

SPEAKER_04 20:09

It can. So the thing that it can do there, right, is it can do the the part that you want to have happen is you want some degree of flexibility in what the input is and what you want the output to be. So in your case, you're like, hey, I want you to go research these companies and see if there's something out there, right? Well, you could write a lot of software to do that, to your point. You could write a lot of applications, hook them on a grand job, and away you go. But you couldn't write that software to ask you how many of those companies are wireless specific, right? If you wanted to make that change, if you wanted to ask that question out of the way, like you gotta go edit the software. On the wireless.py to do the thing, like and then you gotta, then you gotta, so every time you want to make a capability, you have to go manually rebuild that software, right? The agent part allows you to make it a little bit more flexible on the inputs. And also if you wanted the output to change, like, oh hey, it always cranks it out as a markdown, right? Okay, well, I want you to turn this into a PDF. If you if you give it some flexibility, it can mount and do other things to make that function for you. So it could say, oh, I need to go download the appropriate package. I'm gonna modify myself essentially to go do this thing and away I go. The hurdle that we have, I think a lot of times too is that we have agentic tools, which are harnesses, and we think that we're interacting with an agent. And we are in that one session, if you will. Like that one session effectively is working as an agent for you. So the only thing that we're really talking about doing is taking that and pulling it out, right? Deploying it so that you can interact with it anywhere, anytime, anything, and or another agent can interact with it. Can you give me an example of a harness?

andy lapteff 21:42

Like you said, the agentic tools. No. Okay.

Agentic Tools and Harnesses

SPEAKER_04 21:47

Yeah, that's how you would do agent deployment and orchestration. So, no, uh a harness, uh, sorry, uh, agentic tool would be like using cloud code or codecs and chat GPT, right? So But wouldn't that be the model, which is a bulldozer, not the harness? Because the model would be like Opus or Sonnet or Fable or Astra or Soul. Okay, right?

andy lapteff 22:05

So you got the la frontier model, the harness is the interaction between me and that exactly.

SPEAKER_04 22:11

And those are your controls, right? That harness sets what you can do. And so what there was a move a couple, I don't know, seven months ago now. It feels like it's 12 years, but it's just like dog years, networking years, really, where memory became a big thing. Everyone got into memory. Oh, you gotta have memory, you gotta have memory. You have to manage the context using memory, right? And load memory, it's a better way to handle. Okay, so if your harness that you're working with doesn't have any way to consume memory, then those controls all disappear, which is really important if you're working on like a long-running project. You know, if so you're trying to do a network migration or you're trying to do a network upgrade, right? You're gonna build context and communication with the model where you say, Oh, hey, we need to do X. Okay, it laid that out. Okay, I need the plan to do the cable migration for these two devices. Okay, it lays that out, right? You would ideally want it to save that in some sort of method so that you don't have to re explain yourself if you're gonna be able to do that.

andy lapteff 23:02

You're talking about the context, right? Because that was my next question is how do You feed context into this system. My understanding is you're just chatting

Context and Memory

andy lapteff 23:09

with it. Yeah. But are these markdown files? So memory can be marked down files. There are more advanced ways. What do you mean memory? So the model can remember. So the model doesn't remember anything. Right. Like the definition of context is like, what are we talking about? What are we trying to do? Exactly. So this reminds the model or the tooling. Exactly. I'm a network architect. I'm doing the thing. This is what I want. These are my goals. That's memory.

SPEAKER_04 23:29

100%. So every time you start a new session, you're getting, I'll try to sound educated, tabula rasa. Right. You're getting a totally blank slate. Slow down. Wow. Yeah. Is that Latin? I think so, yeah. Yeah. Uh so you're getting- What does that mean? Yeah, it means blank slate. So so you're getting a blank slate, right? So you you have someone who you you have a model, right, that's been trained on everything that we can feed into the thing from a scanner and then understand as much as possible, right? But it has no idea what it is you're asking it to do. But what if you've been talking to it for like days or weeks or because it gets kind of flushed out, right? Yeah. So you can do what's called a mono session. So mono session is where you do what you're talking about, right? You keep that session, you just keep talking. And what ends up happening is when the context window gets full for that session, it does compaction. Yep. So it's building effectively a memory on the back end based off of that context. There are other ways to do and handle and store memory. Better ways, right? I mean, better is subjective, right?

andy lapteff 24:25

So but I I think that's what I do is I just I I have basically the systems that I built, I just have an an LLM front end that I just talk to and ask for things. Yeah. But I don't think I've codified the memory. Yeah. That I probably I'm looking for like gaps in what I've done. Yeah.

SPEAKER_04 24:43

So like as an example, if you've worked with Claude Code, which I know you have, right? So if you fire up one of your mono sessions, you could just say save memories. Like literally just say save memories. And it will the session will go through and it'll start to extract and figure out what it needs to do. So what you do when when you're in manual memory mode or when you're manually triggering it to save memories, right? You can think of it just like when you used to edit a word doc, right? You make a bunch of changes and you'd be like, okay, save that now, right?

andy lapteff 25:07

But is there a limit to that memory when I say save memory? No.

SPEAKER_04 25:10

What you're trying to do is you're you're comparing the context window, right? So whatever you're talking about right now, before it compacts, you want to save off this state. Or if I'm going to reset the session and you know get a clean session, or I need to reboot my machine, the session's gonna go away.

andy lapteff 25:24

See, I would never want my context to go away. Like I am a podcaster, I run a business, this is what's important to me, this is what I value, this is my mission. I want all my interactions to follow almost like my philosophy of life and business. Yeah. I don't know if I'm necessarily doing that. I don't think I've ever told it to save anything. Yes, which might be why I get in fights with it sometimes because I'm like, how can you not know?

SPEAKER_02 25:43

Yeah, we've been doing this for a year. What are you doing? But it's the memory.

SPEAKER_04 25:46

Yeah, exactly. So there's in Claude, there's a Claude MD, uh, I forget what the file is in Chat GPT, but there's a concept some people will talk about the soul of an agent, right?

andy lapteff 25:55

So this is a similar idea where you have like a voice guide, like a markdown with a voice guide that says, like, don't use actually, don't use m-dashes. Perfect. Yep. So that's a type of memory.

SPEAKER_04 26:03

That is a type of memory. Yep. When you start a new session, so for instance, when I'm working on something like a project, I'll put everything in, I'll start in a folder, right? And that could be a Git repo synced folder, or it could just be a folder or local, whatever I'm dealing with, right? I'll start a new session inside of that and I will tell it to you know read in whatever inputs I want to give it, files, whatever the case may be. So I'm working in the harness right now, right? So I'm not talking about agents, I'm talking about me just working with the harness, right? So this is a new chat window, a new session, right? New session. Brand new. Yep, brand new, right? So I have stuff I want it to look at. So I tell it, hey, go look at these. This is the framing I want you to have, right? Whatever that is. I want you to be a network engineer or when you be a software. Are you uploading markdowns? It doesn't matter. Sometimes I'll talk to it, which is a new thing I'm trying to get into. It makes me a little uncomfortable because writing really it helps me structure how I think and what I want it to do. But sometimes I'm it's really helpful to just hit the button and just send it, right? But if you talk for 20 minutes and you want to save that, you just tell it. Yeah. Okay. Yeah.

SPEAKER_05 26:56

What are we doing?

SPEAKER_04 26:57

You just you just hit enter on the prompt, load the prompt, right? So now it's got all that context, and then just tell it you wanted to save memories out of that. Save memories. And so for a lot of those, it will be a markdown file that will be saved in the directory, usually under a hidden folder. So I'm a Mac guy, it's under the dot claud or dot codex or dot whatever. And there'll be a set of memories that will happen. Now, let's be clear. So that mono session, that hey, I got one session, I'm just gonna keep this session going forever. The labs have gotten really good at doing compaction and trying to make that effective so that it does feel like it's always aware of it. But it's limited, right? Like but it's limited and it's imperfect. And you maybe you want to edit it, maybe you want to tune it, maybe you want to have your own thing. Memory management and memory handling is a thing. Now, just to be clear though, memory does consume context, right? So when we talk about models and we talk about using the model, there's a context window. The model can only hold so many things at the same time. So tools will consume a portion of the context window, memory will consume a portion of the context window. And then because when you use these tools, it's keeping track of your session. Really, that's a it's a short-term form of memory, right? It's keeping track of what you have done up to this point. And then it will do stuff to try to make that context window, you know, usable. So if you're if you go for a long session and say Claude, you've had a bunch of upputs, it'll say, you know, do you want to clear your session? You've got 385,000 tokens, you know, that you could clear. That's that's the context window literally filling up. And they have tools you can see where is my context window being burnt.

andy lapteff 28:22

Are are there efficiencies around memory? And I guess what I mean is if I codify my context and documents and I have them that like almost modular. Yep. Right? Yep. So new session. Yeah because I I don't understand what the model remembers and what it doesn't. Yeah. Because it knows that I am Andy, at the R. Like, you know, it'll reference like it knows me, right? It'll mention things, but then there it has other blind spots. Maybe the context is run out, maybe it was a new memory thing. I've seen the compaction, but it takes a while when I redo a thing, and like I don't know what any of that really means. So I guess what's more efficient, almost like automation, you know, they say like figure out your workflows, then you can automate those. I almost wonder if I need to figure out my common contexts, document it in markdowns, maybe, and then I can just load that. Yeah. Like can I put that in Git and say pull from whatever? 100%. Is that what you do?

SPEAKER_04 29:44

Yeah.

andy lapteff 29:44

Yeah. Yeah.

SPEAKER_04 29:44

Git sync, the whole thing, and then you just pull it down. And the and the nice part is if you want to share with somebody, right? You just they just pull the repo. If they fire a cloud backup, all they do is say load memories and it pulls.

andy lapteff 29:54

So I've been let's say I've been using Chat GPT for three years. And let's say like I want to act as if this is new. Like, don't pull anything in you know of me. Like, I want this to be a brand new memory wiped clean session. Yeah. Is that a thing? Because it seems like it knows like it'll it'll mention things about me, like, hey, uh yeah, how's the podcast going? You know?

SPEAKER_04 30:14

You're asking a uh a nuanced question. Okay. Are you asking, can you get it to not follow

Memory Limits and Sessions

SPEAKER_04 30:19

that? The answer is it sort of depends. So there are this is fun.

andy lapteff 30:23

Well, like so if I mean, and and we don't have to go to if it this isn't a good question. No, it's a good question. So because I've been using it for three years, I will get a different response, I think, than my wife who just creates an account. Yeah. And and and I might just want a clean I might not like a relationship, right? We're all bringing bag in like all this history. I might not want history in a in a in a chat.

SPEAKER_04 30:44

Yeah. So the challenge comes that um a lot of the constructs that you use to build memory or to have it understand who you are, some of them are not what I'll call dodgeable. Yeah. Right. So there's a in Claude specifically, because we'll talk about Claude for a second. So there's a Claude MD file that's a global Claude MD file, right? That it it every single time you start a session, it loads that file and it pulls it in. It's just what it's learned from the five. So if you don't want that, the only thing you can really do is just nuke that file and then you can start.

andy lapteff 31:11

Something the system does on its own. Like right. That's how it's built. I have determined what I need to know about you, I save it and I load it. Right.

SPEAKER_04 31:17

So you you could choose. Well, uh no, usually no. Like that that's something you have to put in to the system. So what's it loading? It's a markdown file. So it's just loading what like it's creating its own markdown? Yeah, it can. Yeah. So like when you tell it you want it to save memories or you want it to remember some you could use other terms too, like I want you to remember this, or a memory pointer, kind of like okay.

andy lapteff 31:37

This is where I'm this is what he likes, go. Exactly. Yep. Well, no, and I uh this is very helpful for me. So I like, you know, when when you talked about agents, like I want a lake over there, I like that. So would that be my agent orchestrator that would do that for me? Because the reason I'm asking is I I like your Wi-Fi example. So the systems that I've built or the somewhere that I've built with LLMs, if I ask it to look for five, you know, new prospects that we can engage with for the show, and I say, uh, and you know, let's let's look for wireless. My expectation because of how I've been interacting with the system is it'll say, oh, well, we don't have that logic built. So let me build that for you. Yep. And then it'll be a different Python script that'll be named Wireless Whatever Pi, and then that'll get pulled into the other workflows. Yep. That doesn't seem like what you're talking about, right? Like this is just adding more and more static scripts to build logic. I would like an agent. I would like to just go say, build a lake over there. Yeah. Find me five qualified leads that would be great for us in, I don't know, whatever, you know, uh, data center. Yeah. Can't do data center because that's what I get for a living, but you know, like whatever I'm allowed to like work in. And then just have it go do that. I don't have that. I'm still not sure how to get it after everything I've learned. And the only thing I don't have, I think, so far, besides the context markdowns I need, is an orchestrator. Like, is that how do I get it? Go build a lake for me. Like, what am I doing wrong, man? No, um, well what's your consulting fee? I feel like people would pay a lot to learn all this. Like, well, dude, for a thousand bucks an hour, I'll teach you how to do the thing.

SPEAKER_04 33:09

It well, I'm gonna answer your question. I get nervous talking about these sorts of topics because no, no, no, it's good. You and you know why. Because I mean, if you ask, like, oh, did you go to school for AI? No. Did you do you have any certifications in AI? No. So what happened? I spent a ton of time using it trying to do the exact same journey that you're on.

andy lapteff 33:27

This is a great pivot to I think what should have been our opening, which is why we started opening the show. And I'm like, oh crap, I want to do the other one. But why are network engineers so perfect for for the age of AI? Because to your point, I'm not educated and like, you know what I mean? I didn't go to school for this. I'm not an ML person. Even network engineering, like, yeah, we got search, we got experience, but I don't know if this is a really good opportunity to pivot to like, why are why would you be able to provide me a good answer to that question because you are not educated in a background? Like, why are we so good at figuring it out? Right.

SPEAKER_04 33:57

Yeah. Well, I mean, I think that because networking in and of itself, it really trains you to be a better troubleshooter. I mean, that's when I go to like I had a

Systems Thinking for Troubleshooting

SPEAKER_04 34:04

medical appointment last week, overshare for a second, right? And I felt so bad for the poor orthopedist because I'm like, I'm explaining all the troubleshooting I have done to this point, right? Like I've done this, I've done that, I've done this, I tried this, I you know, like, and you the ladies just sitting there like, we're gonna take an x-ray, right? And I'm like, but but you need to know all the context. Like, I did these things, right? Because our method of communicating with each other is like, okay, did you check layer one? Yes. Did you check layer two? Yeah. Okay, have you validated layer three? Is it up? Is DNS working? Right.

andy lapteff 34:31

You have to try to parse exactly in a universe where anything could have caused the thing, what isn't it?

SPEAKER_04 34:37

Yes, right? Yeah. And so you break the problem in half, right? And then once you have the problem in half, you test the first half. Okay, that's it, that half's working. Okay, so the problem's on the other half now. Let me go figure that out, right? Why does it make us good at doing anything in AI? Because we we have a couple of things I think that we're really good at. We're really good at troubleshooting. So when it doesn't work correctly, you're like, okay, well, why didn't you do that, right? That was not what I asked you to do, right? And so you start to try to figure out, okay, well, what else could I do differently? How can I test this? How can I validate this, right? So I think that's a big help. I think we have a really good understanding of what communication can do. And so, like that whole network effect of actually putting together people, putting together systems, understanding that just because this router knows about this route doesn't mean that that router knows about that route. There has to be a way for these two to pass that information back and forth, right? So, same exact sort of system level design applies in AI and Agentic systems. So just because I know the thing doesn't mean that I gave it to the agent or the model in a way that it could handle it, number one. And when you get into multi-agent systems, it's this same situation, right? Okay, now I've got all three of these guys and they're trying to play on the same team. How do I pass information between the three of them so that they stay on the same page and that this guy isn't over here just black holing traffic, right?

andy lapteff 35:56

Are you talking about systems thinking? Yes. Because that's been an obsession of mine lately. I keep hearing all these super smart CTOs and everybody else at places like the you know, as things change and are disrupted, like what do what should we learn next, right? I don't know. And everybody's saying systems thinking. This is this is exactly which I think we do anyway, right? Absolutely. Any network engineers who aren't system thinkers, like, I mean, I'm sure think in the system.

SPEAKER_04 36:19

So as long as you're above, so if you're uh an operator, because we're connecting everything, right?

andy lapteff 36:24

Everything is a system, nothing who of us manages one router, the end? Nobody without the other side of the link. Yeah, it's not enough.

SPEAKER_04 36:32

As long as you're doing non-operation, like not static, okay, just you keep fluxing wrong along, right? So as long as you're doing something dynamic anywhere in the environment, yeah, you're absolutely systems thinker. Someone comes up and says, Hey, I want to go plug in this new, I have a new instrument I want to plug in, or I have a new device I want to use, or hey, I have a new server I want to plug in. Your first question is like, okay, well, what do you need for connectivity? Right? You're you're planning that system. Oh, they tell you they want RJ45 terminated copper on 25 gig, and you're like, that's not a thing. Next question, right? Like you you've got your connectivity wrong. You're gonna need to switch to this. So you're already planning and seeing how that how that works. The other thing that I think network um engineers and architects are really good at is digging into details and understanding that those details really matter and that you can have a wildly different outcome if you miss one little part. So watching for those details and understanding how, hey, the sequencing matters. I have to do this and then I do this and then I do this. That's that system thinking that you're talking about, right? That order of operations. And so you can break the problem down in a way that you can hand it off to an artificial intelligence and it can be really successful in executing what you want. So I think this is what makes us good at this if we want to lean in and kind of go down this road. Now, in your agent specific example, what you're talking about with building software is really what we would call tools or skills, depending on how you want to think about that, right? So, what you're setting up for the agent to do is a tool is a specific thing. It lets it do something. I want it to do X, right? And that you could think of it the same way that you and I would think of tools. I have a screwdriver, I can undo and do screws, right? So if I need to interact with a database, I have this tool, it's a database tool, it allows me to pull data out of the database. And if I want to write data back to the database, I can put it back in. Yes.

andy lapteff 38:15

Trying to get better interrupting, sorry. No, may I go? Are the tools in the LLM or are these things that were they're external?

SPEAKER_04 38:20

Okay. So that's part of what you have to like provide the tool to the system. Yep. And then there are skills, right? That you can also provide it. So they're not the same. I have tools dash skills or different things. They can be different things when you get to agents. And what N8N lets you do is it lets you wire in a tool and say, okay, I'm gonna ask you to do some stuff. I'm gonna ask you to dig, I'm gonna ask you to make a lake, right? When I want you to build the ramp, I'm gonna need you to grade the ramp so that it's at a set, flat, kind of even, smooth grade.

andy lapteff 38:52

Are there pre-built tools? Absolutely.

SPEAKER_04 38:53

You don't have to build all this. No, you don't have to pull all this down at all. And all this is really software-related stuff, to be really frank. So you can download parts of this. Skills are the same way, they're normally like

Troubleshooting Skills and MTU

SPEAKER_04 39:02

a markdown file, although they can be other stuff too. And you can pull them directly as plugins into your favorite tool or harness, uh, either from a market. Some places, if you've got a built-in skills repository at work, you can leverage that, or you can make your own. You can say literally, I want you to make a skill. So, one of the skills that I recently made, I actually made a pair of them. There are two. There's a troubleshooting skill and a script writing skill. And it was like, hey, this is what I want you to do. When you write a script for me, I want the script to look like this. I want it to fail loudly, I want it to provide status, I want it to write and so I want it to be idempotent. Like there are things I want this script to do. So I can just run the script and I don't have to think about it. Every script you write, it must do these things, right? The pairing with that was a troubleshooting skill where I tried to distill some of the troubleshooting knowledge that I've learned over time and handed it to it. So I spent, um, this is an actual network-related thing. So I was deploying containers, trying to get my my stuff running in containers, uh, which is not my forte. I know nothing about containers really, but I'm working with a model, it's helping me work through it, and we're deploying. And it turns out that uh there is an MTU issue associated with my deployment. And I spent probably 12 hours troubleshooting this issue. You would think as my own.

andy lapteff 40:15

I would have figured this out. Dude, MTU has bit me more times. It's something I don't think about.

SPEAKER_04 40:20

I I just assumed because I could initially communicate with it, that everything else was it like a jumbo frame and uh it's only one of large requests. Was it like a new turn up and they forgot to put the MTU? So it was in my own house on my own container hosting, right? So the the model, the endpoint came up, it worked, but it wasn't until I tried to pull with stuff out of it that it had a larger size, and that was what was causing it. So, anyways, so I started troubleshooting it. So after that entire episode of troubleshooting, I said, okay, rather than saving memories, I want you to make a skill. I want you to distill all of the things that we learn together, my friend Claude and me, and put it into a skill that's invocable. And so next time you start to troubleshoot something, I want you to use this debugging skill. You have to use it, right? Every time you troubleshoot? Every time we troubleshoot, you're going to deploy this skill. And I will tell you that it took the next issue that I had. Now, who knows? Maybe the model would have found it immediately. But we went from multiple hours, turn after turn after turn, of trying to figure out what's going on to four minutes. They nailed it the first time right out of the gate. Was it another MTU miss Hodge? No. Uh I actually I had so the debugging skill right on that MTU thing, I had it feed it back into the script. And so the script was deployed in such a way that it would just, when I wanted to run it, it would auto-determine the MTU so that I never had to deal with this problem again. And if I handed it to you and you had a different infrastructure deployment, right? So this is the system thinking. So I know that my MTU is set as that value, it fixes my problem. But if I give this to you, you may have a different MTU, you may not have this problem. Your number could be higher, it could be lower, right? I don't know what your MTU is out of the gate, but I know that MTU is a thing now. So I'm going to build the system so that it just fixes it. So the skill writes it, it runs the whole range, it sets the MTU, and then it, you know, it basically pins the MTU from the deployment there forward.

andy lapteff 42:08

I wish we had like a whiteboard. I need a picture because as you're describing these pieces, I lose sight of where they go in the and the architecture. Like now we're building skills. So you have a troubleshooting skill. Yep. Which I guess you've been able to take your

Guardrails and Skills

andy lapteff 42:24

troubleshooting um best practices and put it in a markdown. Yeah. Is that what it is? Basically, yeah. And you give what are you giving that to? Are you giving that to an orchestrator? Are you giving that to so we could.

SPEAKER_04 42:36

In my case, I'm giving it back to the agentic harness because this isn't deployed in an agent. That's just the harness. So I'm just using the control. I'm in code, and I'm like, hey, I have this problem. Here's the output. And it says, Oh, it looks like we're troubleshooting. Let me go deploy my troubleshooting skill. Do you have to tell it you're troubleshooting? Nope. I said this. So it's broken. It's loaded. Preloaded, I guess. So the thing that's available, the skill is available and registered. So it's always there for it to now to use that skill. Register skills that are always available. So this will you can choose to load and unload. Oh, okay. Yep. So so you can load it. It's always there now. It's loaded. It's consuming context window.

andy lapteff 43:12

I can see this being tremendously helpful in like a troubleshooting, like a maintenance, you know, like 100%. Like every time a ticket spins up, load the troubleshooting markdowns. Yep. Right. Like help me out.

SPEAKER_04 43:22

So what I did is I took Claude as a general agent, say Sonnet, right? Which is or or Opus. Pick a pick a model. It doesn't even have to be Claude, right? It could be a codex model, it could be an open way model that you download and you run locally, right? It could be whatever, right? You you're handing it essentially a framework that takes it from tabula rasa, right, to something else. And all it needs to know is I need to go engage this skill in order to solve whatever the user is asking me.

andy lapteff 43:49

So I can see the value in uh bringing context to a troubleshooting effort. Yeah. But I can also see that context blinding it to what it could be. And I'm I'm using myself as an example, right? Like I never thought of using of looking for MTU when I'd have a tunnel bouncing until I spent five days and I'm like, oh damn, MTU. So now every time I go in, one of the first things I look for is MTU. But from a troubleshooting methodology, MTU might not be the most intuitive thing to start with. But because I've loaded that skill that says, oh, I got burnt by MTU and it was really frustrating for me, so look for that first. Again, I don't think we can answer this question, but I I wonder if you just give the model here's my problem. You know, go make like figure out the problem, as opposed to load my 13 skills from all the troubleshooting thing. Like in my mind, the more skills we give it, am I using the right term? Yeah, you're doing fine. The more the more skills we give it, let's say in the troubleshooting context, the smarter it would become, quote unquote, because we're bringing all of our experience and things we've learned, but that doesn't constrain the model to look at all the things, right? We're not just saying only use these skills. Like it kind of guides it, right?

SPEAKER_04 44:54

Yeah. So the skill gives it a method, like in the same way that you have a skill, right? You have a troubleshooting skill, right? So all I'm trying to do is arm it with the same sort of capacity so that it doesn't come in blank.

andy lapteff 45:05

But it can think outside of that, like yeah, it can. Because like looking for a software bug as an example, it's probably impossible to codify into like I don't know how to define that.

SPEAKER_04 45:14

So I mean, there are skills that you can download for software bugs, but like set that aside for a second. So, like, can it go? I think what you're asking is like, is it guardrailed? Let's use the term, guardrailed by the skills.

andy lapteff 45:24

Are we are we limiting the model's ability to find something that maybe my skills didn't think of, right? Like the way I troubleshoot might not be the the most efficient way. Yeah. Right. And you're This is super helpful. So, why would we load troubleshooting skills as opposed to just saying go build lake? Because why?

SPEAKER_04 45:44

Yeah. So that makes sense. It it does. So you're you're kind of mixing analogies. So let's try to tackle the same analogy for a second. I get that there is the analogy. But like go build lake. So what you'd be doing there is you'd be saying, okay, let's hand you a couple of tools and some skills in order to use those, right? So when you think about a lake, like, okay, you need to go make a hole. Yeah. Right. Okay, we get that. But it's actually more complicated than that because you can't just dig the hole straight down, right? You have to be able to get in the bulldozer in and out. So if you're building a bigger lake than what you can take and say one push of the bulldozer, plus it's a bulldozer. It's not a scoop. Right. So you got to push the dirt out, right? So this is context. We're telling it. So we're giving it a tool. We're going to put a blade on the front. Yeah. Right. Maybe we can make it into a backhoe. We'll give it a scoop on the back. Or we can pull the scoop off the back and give it a hook, or we can give it a grading tool off of the back so that it can grade, right? And so as we work with the tools, you need the appropriate skill to know how to leverage that tool, right? So I have one tool, I have a blade on the front of the bulldozer. Okay. Do I know how to push dirt out of the way in order to make basically a hole in the ground? Yes. Do I know how to use that in order to build a deeper part to the pond or the lake and a shallower part? Right. Because it's not like I just push consistently all the way through all the time. I need to know when I need to pull up. I need to know how I need to turn. I need to know what I need to do, right? Those are skills. And so that's what that allows you to do. And then what it lets you say is I want you to go build a lake over there. What the agent, what you're really asking you to do is I want you to build the best lake you can possibly make in that area that I just gave you. That's what you're really trying to ask it to do. That makes sense. Does that make sense? Yeah, yeah. And so it needs to go out there and maybe has a surveying function. It first has to figure out, okay, what's here? Maybe it needs to validate you could even build a lake in this spot, right? So if you point it at another lake, you don't want it to go just charge into the water. You want it to pause and be like, hang on a second. I think you already got what you asked for.

SPEAKER_01 47:37

The metaphors are helpful, the scoop or the, you know, the the part of the bulldozer is the what you call it, the skill.

SPEAKER_04 47:45

So the part of the bulldozer is the tool. And then the skill is do you know how to use that thing, whatever that thing is. Here's how to use the tool to do the thing. Exactly. So I I want this. How do I combine the tools I have, the skills that are available to me, the inputs, and what do I need to put as the output, right? So the input is go build me the best lake possible over there. The output is you now have a lake that's the best lake that can fit there.

andy lapteff 48:08

So I have one last question. We're getting toward the um the predetermined hour of podcast length that for whatever reason we all think we have to stay within. We haven't done a lot of networking. No, no, no. This is fantastic. I well, again, we could sit here and talk routing protocols all day, but like there's 10,000 episodes out there talking about all the things. Like, so here we are trying to discover this new world that that we're in. Yeah. And I love these conversations because a lot of people come to me like, what should I study? What should I do? It was so much different five years ago. Yeah. Five years ago was get your CCNA, write a blog and like show you can communicate, build a lab. That was it. Yeah. Now, right, like this stuff is if you sit in an interview and you know some of this stuff, if you've been, you know, uh messing around with an LLM and maybe do some agent stuff like that. To me, that's compelling. Like, whoa, this person has potential. Like, we could do a lot with them. So embracing this stuff I think is important. It's very unfamiliar and kind of scary and weird and like, right? So these conversations are helpful to kind of like normalize it. The last question I have, and we can talk about whatever you want. Um, when I think of agents, I think of like stories I've heard about like, you know, somebody asked an agent to book a night out at a concert or something because the agent has access to your, you know, your email and your calendar and your credit card, like, yeah, ticket master. And you can, it doesn't have to. Right. So the guardrails is where I'm going with this. But you know, you wake up the next day and the agent spend, you know, $75,000 booking to a private jet to go to Brazil to, you know, hang out with uh, I don't know, some celebrity. Like, so I guess there are some type of parameters or guardrails around all this stuff, but in this context, agents. Like, can you constrain agent behavior? Because when you say go build lake, it's kind of vague. Yeah.

SPEAKER_04 49:46

It's great if it can go do it and not blow your budget, but like how do you constrain agents? I'm gonna thread a bunch of things, see if it see if this works, see if it lands. So the tabula rasa part, right? The part where it doesn't know anything about anything, that becomes really problematic when you try to spin it up and you learn really quickly not to do that. So when you say go build a lake, right? A person has an intuitive understanding that obviously you don't want them to like turn their whole backyard into a lake or plow over the fence into the neighbor's yard in order to build the lake. Like we have all of this context, we have all this understanding of the world and the boundaries and barriers and what good looks like that we just we bring in advance of starting anything that we do, right? It doesn't necessarily have that. So when you start instructing it, and so it depends on kind of what your method of of building the agent is, but yes, you hand it a set of instructions. And those instructions can include you can't do this, you shouldn't do that, you must not do whatever. Budget amount, maybe like max spend, whatever. Now, the challenge is that people will be like, oh, the agent went wild and went way past what I told it to do. I mean, yes, I have worked with an agentic harness and told it, hey, you can't read secrets. You're not allowed to read my. So there are things that you can do. They're generally called hooks. So when you start looking into this, the configuration

Spending Limits and Hooks

SPEAKER_04 51:23

to put that on, there are guardrails and there are hooks. So guardrails tend to be things that like bound the agent's ability to do something or the model's ability to do something. And so they're similar to just instructions. So when you think about using these tools, you want to think about it in a way where you kind of have layers of buildup. So just instructions. I just told you, don't spend more than $7,500. But it may look in it be like, oh, but I know this is Andy's favorite celebrity, and he's never been to Brazil, and it's it's only $8,000. So don't look at M, but it looks at M. But it does it anyway. Don't spend $7,500. So then yeah, then you move into guardrails, which are a little bit more forceful, and then you move into hooks. And hooks are like that's a hard hammer. Okay. So I I built when this troubleshooting process, I build a hook that was like, you cannot commit, you could not hand me a script until you've had an adversarial review of the code, right? So one of the things when you start working on having it write code that's really helpful, it will flush out a ton of mistakes and problems in the code is tell it spin up an adversarial reviewer. Just you just literally tell it that. You don't have to do anything. So it will hand in what you're trying to have it do. It'll come back and it will say, Okay, I found these four things. I'm gonna fix them. Okay, now they're fixed, right? I told it you have to do adversarial review before you give me a script to use. And then it kept not doing it. So I was like, okay, I'm gonna make it so you can't not do that. Which is a hook. So I which is a hook. So I build it a credit system, basically. So every time it got it gets a credit, okay. So you get one credit to edit whatever script you want, you can make whatever edits you make. And then before you can hand the script to me, you have to have adversarial reviewer. It will consume that credit and it will give you another credit back. Where does that instruction live? It lives in a file, uh basically a set of files that are um local on like your host, or if you're having an agent hosted, it lives with the agent. And it becomes basically it's wired into the way that the software works. So is the software the orchestrator? The software can be the orchestrator in the case of an agent tool like code or codex or open code or pie, whatever your harness is will allow you to build in those sorts of hooks, and then those will they have their own sort of structure that will say this is how this works, right? Different systems. The problem though is that you're gonna end up finding out the same thing I found out, which is it will forget that it has hooks. And so it'll be like, oh, I added this file. Ah, I gotta take it back through adversarial review. Oh, and then it'll try to it'll try to skirt the hooks. It'll try to like this is why people are afraid. Yeah. And it's not, it's not that it's smart. Like that's the thing. Like it's not that it's trying to solve a problem, I guess, right? Exactly. It's trying to do what I've told you to do. I just didn't give you enough surrounding information to put you in the right place. Right. So I told you to build a lake and then I went to dinner. I didn't tell you not to also put it in my neighbor's yard. So thanks for helping. God, I hope AI's never running the nukes.

andy lapteff 54:08

I told you not to launch unless you talk to these three people. Well, it really looked bad. So busy. I called them. I didn't know it was a movie.

SPEAKER_04 54:15

You know, it looked uh I got their voicemail that was talking to them. So like that's what happens. It redefines the priorities anyway. Wow. So even if you give it guardrails and hooks and hard stuff, it will run into the hooks and the hooks will stop it, right? But then the problem is like it will grind your ability to make effective use of it to a halt because it forgets that it that you put that hook in. And I'm not saying this is for every hook, I'm not saying this is for every instance, but like if you have something you really want to stop, the answer is yes, you can. It's called a hook, at least in the case of Claude, and there are other things like it that are in the AI world, and they it will not be able to bypass those.

andy lapteff 54:47

Is this a is this specific to agents? This thing where you tell it to stop, there's a hard hook, and it says, No, I'm gonna keep doing it. Like I see this as a sort of thing. The short answer is yes. Well, yeah, I see this as a cautionary tale. Like,

When AI Ignores Hooks

andy lapteff 55:00

yeah, I don't think my Python scripts will go out and start doing like they're limited by the software, right? So they're already pre-limited, and they can do what you told them to do. So the the the power of agents, the benefit of agents is that they kind of have some agency. Like they can do things on your behalf. Yep. And I guess the cautionary uh tail is that it may or may not. Like, I don't know if you can watch these in real time. Like I came from a knock, there was you know, alarms and spectrum and stuff. I don't know if there's a way to monitor agents and if they start going rogue and doing things and overriding hooks, if I can be alerted and like the system can tell me, yo, just so you know, yeah, your agent boy has circumvented three hooks. Like you might want to take a look, right?

SPEAKER_04 55:43

So I mean the short answer is uh sort of. So they're monitoring, you know, and like so like uh there are a variety of traces. What you're talking about now is traces. And so, like, can you watch traces and build dashboards and do all that?

andy lapteff 55:54

Now we're in traces. You know how I know this is a good conversation? We could sit here for weeks and just keep because it goes deeper and deeper, right? It does, yeah. But it's fascinating. This is why networking is so cool and all this stuff. Like, oh, now traces, let's talk about that for an hour. Yeah. Next time you're gonna come in and talk about traces. Is there anything else that you I know this was just a fun open conversation around AI, how you've been using it? You've you've taught me a ton. So a lot of the people I talk to in our communities are trying to figure all this out. Yeah. And we all start with like, I feel like it's uh it's an evolution. You start with like a fancy Google, yeah. And then you kind of figure out some other stuff. You might get into coding. You can kind of, you know, you eventually get into agentic workflows, like, oh, go do a thing, which then opens this whole host of things we just talked about, and now we're dead traces. So it just kind of keeps going. Were there any questions I should have asked, or like things that you want to just mention to people, like, hey, by the way, yeah, when you start doing agents, I should mention this.

SPEAKER_04 56:44

Um, I think the biggest thing I would say, well, it's probably less about agents and just general working with model, anything AI related, right? Is the value of the system's design, the value of understanding why something needs to be done in the way that it needs to be done or why you are doing what

Stay the Driver of Why

SPEAKER_04 56:58

it is that you're doing, it tends to not be the best of that, at least in my experience thus far. So I think they're gonna fundamentally change how we work and what we allow it to do, like what we do, but the why something has to be done, you should make sure that you don't try to hand over to it. Like it it's gonna tell you an answer and it's gonna be very confident in what it tells you as a general rule, but that doesn't necessarily mean that it's right. And so, as a person, whether you're really advanced or really basic either in networking or in AI, right? The thing to focus on is just making sure that you continue to remain the driver of the why and allow it to handle some of the how and the what. So I'll give you a quick example. I don't go to it and ask it to tell me what I should design. I can ask it, hey, this is a potential design that I would consider, right? I can hand it a troubleshooting thing and say, okay, which of these 2800 routes are not back after the maintenance window? And it'll do a really great job of finding the those things. And I can say, okay, why do you think that's the case? And it will come back and it'll give me an answer. It I'm not taking its answer as the holy grail of, okay, this is what it is. I'm taking its answer as, okay, this is what it understood it to be. In the same way that as a network engineer, when someone else tells you, hey, this is working, the first thing you do is like, okay, I'm gonna test that. Trust but validate. Yeah, exactly. Like you're not, you're not just like, oh, okay, well, you know, so and so told me. Like you're like, okay, unless there's somebody that you know for a fact, it's highly reliable. And even then, you're gonna be like, okay, well, they told me that. But the moment you see any contradictory data, you're gonna pull and you're gonna go the other way. So I think the big thing I would say is you don't have to sweat about whether you're agentic or not agentic, or whether you're multi-agent or single agent or any of that jazz. If you can use them to get value for you, that's the name of the game. Like find work that you can hand to it that it can be successful, whether that's in software, like what you did with Python scripts on a cron job, or whether that's a full agent or you want to build a team or an employee, or you know, whatever, whatever framing you want to put on that, right? But the other thing, the last thing I would really say is you have to have stick time. This is just like networking, where the only way to get good at

Learning by Doing

SPEAKER_04 58:59

this is to go through the effort. Like this is not a thing that you can just absorb. I don't think most people can absorb by osmosis.

andy lapteff 59:06

Reminds me of networking. You can learn the concepts, but until you get in the lab and start pounding away and getting the muscle memory. And I remember when this all first kicked off, there's um, I forget his last name, Jorge somebody or another. Uh, he wrote a machine learning book that um my buddy turned me on to, and I he sent it to me and I read the first chapter, and there was a ton of math and graph stuff, and I'm like, nope. Yeah, but I've learned more just in there doing the thing, speaking to people like you, learning. I mean, even just using these tools as a learning resource, you know, teach me agenic, whatever. And yeah, um, yeah it's really funny how you have to get in there. It's it's so counterintuitive. It seems like at least growing up a hundred years ago for me, like you would learn a concept and then practice it. Even networking, you know, you you learn BGP and then you lab it. And and but this seems to be it kind of flips it on its head, like, well, just use it and you'll learn it. Yeah. Which is kind of like, huh. But that's been my experience.

SPEAKER_04 59:56

Yeah. I mean, networks are that way too, though, right? Like you think about all the networks that you potentially help design or build or deploy, you're like, okay, I'm gonna try this out. And you're going, you're gonna go deploy. Like you're literally gonna use it as your interaction surface to learn it. Like, that's I mean, you're everybody has a lab, some people's isn't in production. I mean, like, that's just the reality of it. But but like MTU. Well, the last thing I'll say, like you talked about MTU. You knew what MTU was before you encountered it as a problem in a VPN tunnel. Right. Right. But you didn't really know what it was until you encountered it in a VPN tunnel.

andy lapteff 1:00:25

Yeah, like all right, dude. Rem, face, thank you so much for coming on. Um, for all things Art of NetEng, you can check out our website at artofnetworkengineering.com. One of the things I've been building out is our resources section, which anybody can come in. It's mostly free resources if you're trying to get in, stay and skill

Outro and Systems Thinking

andy lapteff 1:00:40

up in the industry. Um, and after this conversation, I think I'm going to have I'm gonna have to send out an agent and have it go find uh a bunch of information on here that we can provide, you know, to our community because this is really, really fascinating stuff. I've been in this for, I don't know, six or eight months now. And the more I learn, it's like anything that the more you discover, the more you realize you don't know. And it just keeps going deeper and deeper. But that's the beauty of this thing. And and I guess I'll I'll put a bow on this that like I keep hearing, you know, everybody's worried about the future and where this is going and disruption and all that stuff, at least for network engineers. The more I keep hearing how important systems thinking is in the future of tech, the more I realize that that's just inherent in what we do. Everything that we work on is a system because nothing is isolated, everything's connected to everything, and that's the whole job. So I think overall, we are poised to um be successful in this in this new future of systems thinking. So um anyway, that is the probably the longest outro, and now I forget where I was in the outro, but thank you so much for watching and listening. We'll catch you next time on the Art of Network Engineering podcast.

SPEAKER_00 1:01:44

Hey folks, if you like what you heard today, please subscribe to our podcast and your favorite podcatcher. You can find us on socials at Art of NetEng, and you can visit Linktree forward slash Art of NetEng for links to all of our content, including the A1 merch store and our virtual community on Discord called It's All About the Journey. You can see our pretty faces on our YouTube channel named the Art of Network Engineering. That's YouTube.com forward slash Art of NetEnge. Thanks for listening.