Ep 113 - Russ White

00:47
This is the Art of Network Engineering podcast.

00:58
In this podcast, we'll explore tools, technologies, and talented people. We aim to bring you information that will expand your skill sense and toolbox and share the stories of fellow network engineer.

01:14
I'm out.

01:26
Let's cut it over to Tim for your OSI 7 layer forecast. Thank you Andy and may I say that your hair is on point as usual today. Ya punk. Anyway, yes here's your look at the 7 layer forecast. Definitely some troubling probability percentages of high ticket volumes in the coming days. Folks at home may be quick to equate this pattern to slow moving fronts coming in from layers 1 through 4. However I would encourage you to compare this forecast and high ticket volumes to your local change calendar.

01:55
You may quickly find that heavy database modifications are causing a fire NATO up in the Northern application layer. Also friendly reminder that it's broadcast storm season. So batten down those switchboards and keep that spanning tree rapid. Now back to you Andy and your amazing full head of hair for this week's episode of the art of network engineering. Classic, I love it. I love it.

02:21
Welcome to another episode of the Art of Network Engineering. I am AJ Murray at No Blinky Blinky. That was Tim Bertino at Tim Bertino on the Twitters. Tim, how you doing? Good, AJ. What's happening? I do want to take that intro a step further. So there's this local meteorologist, and he's got the nice button-up shirt and the tie and everything. And the running joke is that when stuff's about to get real weather-wise,

02:49
they'll cut to him and he'll be on screen and the sleeves will be rolled up. And when the sleeves are up, you know something's about to happen. So I was kinda trying to equate that to network engineers and I'm thinking, geez, if that was the case with us, we just have to cut the damn sleeves off because they'd be up all the time. Right, right. But no, I'm good, what's going on with you?

03:12
Oh, not too much as we as we live and breathe. This is our first recording of the new year in 2023. So happy, happy new year, everybody. Thanks for joining us. Andy at Andy Laptev. How you doing, Andy? Good to see you. Good to see you, man.

03:27
I got a cold, which is why I sound funny. Got a little roll of toilet paper. I'll be blowing my nose throughout and muting myself. You too, she's on those though, right? It was just a cold. I was doing shots of Dayquil about an hour ago because I refused to miss our guest tonight. I'm very excited to talk to this gentleman. So yeah, other than that, I'm good, AJ. Christmas came and went, it was great. Had 11 days off of work and school and just hung out with the family, went on a little road trip. It was nice to unplug and just be with the people.

03:57
Awesome. That's great. How about you? How was your holiday? It was great. We did Christmas at home and New Year's on the road, visiting a family. So it was a good time. A real good time. Awesome. Awesome. Well, as, as you alluded, we have a very exciting guest this evening. I, I am, uh, I don't even know how to introduce, uh, introduce him. So let's just jump right in. This, this is Russ White joining us this evening. Russ, thank you so much for, for giving us some of your time. Oh yeah. Cool. No, it's always great to be on as well, by the way. So as long as my,

04:27
Mike and video sound okay? We're good to go. Yeah, absolutely. You're sounding great. I don't know what microphone that is, but man, you sound great. Russ built it himself. Yeah, exactly. Danny, I was gonna ask you, school, are you going to school or do you have kids in school? Kids in school. I mean, I'm always in school. I'm in networking, right? Yeah, that's what I was about to say. I finally, when I finished my PhD, people said, oh my goodness, you finished your recreational PhD.

04:51
And I thought. Oh, that reminds me. Should we call you Doctor White? No! Hello. Oh, okay. Doctor. That's a classroom thing. No thanks. What's the PhD in, Russ? Okay, don't laugh. It's philosophy. That's very cool. You can't laugh at a doctor. Okay, so let's get the backstory. Why philosophy? Well, I was looking at doing a PhD in comp sci.

05:18
But I really kind of decided I wouldn't probably learn a lot more in a PhD program in comp sci. It felt like, you know, it just like what was I really going to get out of it other than being able to teach comp sci. But the reality is a PhD is a PhD at most schools, some schools it's not. Most colleges if you have a PhD of any sort, they'll let you teach in the master's program. So getting a PhD in philosophy is just as good as getting a PhD in IT or whatever. And so I already had a master's in theology. And I thought.

05:48
Well, if I can make it into a PhD program in philosophy, I'll go do it. I was told I wouldn't ever make it into the program, by the way, that that would never happen. But I latched onto a major professor, Dr. Bruce Little, who said, no, I can teach you philosophy, but I cannot teach you to think. And you're an awesome thinker. And therefore we will pull you through philosophy, kicking and screaming.

06:13
I don't really care what it takes, but you will learn philosophy. Was it hard, Russ? Because I took one philosophy class in college. It was really difficult. They called it Intro to Logic, but it was like a philosophy based course. And it was really difficult. So in reality, my PhD is a philosophy PhD, but it's in apologetics and culture. And so it's from a Christian seminary. And it is, um, a lot of my classes were on

06:39
A lot of my seminars didn't really have classes in a PhD program. A lot of my seminars were in metaphysics, problem of evil. I had a lot of teaching seminars. I had to take two languages. But my comps, my oral comps, and my written comps, my comprehensives were the history of philosophy. Basically, they sat me in front of a computer with a blank word document and said, start with Plato. Wow.

07:06
just give us the history of philosophy from Plato forward. And so, you know, how each philosopher interacted with one another. And I had to go pretty much all the way up to John Dewey and the modern philosophers from Plato all the way through the whole mess and hit all the highlights. So that I think was the hardest part for me was getting that piece down.

07:30
I love for those living in a cave not familiar with Russ Waite, I love that he starts out with, well, I could have went for a doctorate in comp sci, but what could they possibly teach me? Oh my God. That's just how that started. That's odd. Well, I mean, and it turns out that some colleges do care. They won't let me teach with a PhD in philosophy. They want me to have a PhD in technology. But then this spring, I'm actually teaching a course. I was teaching at NC State and they didn't much care.

07:59
They were like, hey, it's a PhD, it's a PhD. The PhD is a research degree. It's there to teach you how to research and how to teach. That's really what the degree is. The topic matter is not necessarily pertinent to the whole concept of having a good research ability and stuff like that. And now I'm teaching at University of Colorado in Boulder, remote, of course, in the spring. So I intend to start adjuncting there, starting this spring and sticking with that.

08:26
Because there are other colleges I've talked to who have said, no, you don't have a PhD in IT. We don't want to talk to you. So was that the end goal of getting a PhD to be able to instruct at the collegiate level? Yeah, that was part of it, yes. And I'll probably never do it full time, but I do enjoy teaching part time at least, adjuncting and stuff. And part of it was, seriously, just like, this is an entirely new body of knowledge. And I really want to understand this stuff.

08:55
Like this stuff interacts. So a lot of people don't think that philosophy would interact with anything in technology, and that's absolutely not true. There's lots of places where philosophy does interact with technology. In epistemology, in privacy issues, it's one of the reasons I've been on a tear about privacy for the last year or two, is because of my PhD. My dissertation was effectively about the social impact of social media, the impact of social media on individuals and culture.

09:25
In fact, it's really funny, I couldn't find readers. They couldn't find anybody who could read my dissertation because it was a blend of engineering.

09:34
And the philosophers were all like, I don't know anything about engineering, so I can't read that part. No, that's a perfect way to get a good grade. If nobody understands it, then it's just gotta be good. Yeah, exactly. So, yeah. Are you able to like briefly touch on that? Because that's so, you know, you've done it all and I've...

09:57
You're in the two and a half years we've been doing the show. You're only one of two guests I've been nervous to have on. And I over-prepared and I haven't. You're telling me you've been nervous around me. No, no, just because you've- He's not lying, he over-prepared, yeah. Well, just because you've done so much and like, it's just amazing the more I read about you and research about you, just that the breadth and the depth of which, what you know in networking, it blows me away. And then I saw on your LinkedIn profile. So I have a bunch of technical things, you know.

10:23
A lot of our show is people, you know, up and coming, trying to build a career. So I have a lot of questions around that, you know, like insights about the industry. But you touched on your unfriending dystopia book and I felt that was an interesting, I saw that in your LinkedIn, you know, bio. Can you briefly touch on that? I mean, the intersection between, I mean, is it basically like how social media is tearing apart our social fabric or is it something different? So now, of course, coming from a Christian seminary, this is a very Christian take. But

10:51
Jewish and Islamic audience people would agree with the generic, like the basic concept. They may not agree with the way I've approached the arguments, but they'll probably agree with where the argument goes. Is that effectively the biggest problem, well, one of the biggest problems to me with social media is that it does what you call flattening, what I call flattening people, it others them, it makes them into numbers, and it makes all of our relationships into measurable, quantifiable things.

11:20
So one of the examples I was reading in one of the books I was using for research, the guy said, I went to an Airbnb and I had tea with this person at Airbnb. And he said, it was really weird because I knew the whole time we were both faking it because all we were trying to do was make sure we got good ratings. And he said, you know, this quantification of relationship, this concern about giving something out of every relationship and treating other people.

11:47
not as an end, but as a means to an end, which is perfectly valid in some situations to treat other people as a means to an end. But what we've got, yeah, as a means to an end, like a doctor really sees you as a body, and there's arguments over whether that's a correct way of doing medicine, but that's another entire problem. But what my argument basically is, is that social media gets us into the habit of treating other people as means to ends, rather than as ends in themselves. And if you go back to

12:17
neuroplasticity and habit formation and the idea of anonymous environments that creeps into the real world in serious ways. And so, I mean, there are other effects as well, but that was my primary argument. I mean, there's also a- So to put some meat on the bone, just to make sure I'm following you, I originally got on Twitter to connect with people in IT because I wasn't in IT yet. And somebody had said, hey, you know, you can connect with hiring managers and people that work at places and start to like create a sphere of influence.

12:47
So that's why I got on there. Are you arguing that now I could be on there just as a means to an end that, hey, everybody listen to my podcast so more people listen so I can get more sponsors? Is that kind of where you're going with it? Well, to some degree. But more along the lines of, from an individual perspective, I treat everybody in social media as an audience. And what I'm after is likes. Therefore, I'm not treating them as a whole person. I'm treating them as an audience member.

13:14
And the people who are watching me don't treat me as a whole person. They treat me as a performer, like a TV person or something like that. Transactional, right? It's all transactional. So we've made the relationship transactional by quantifying it. And the point is, is that that's cool in some situations. But then when that creeps into real life, the things like Uber and Airbnb and Lyft, wherever we are, TaskRabbit or whatever Uber eats and door dash and whatever.

13:43
you end up transactionalizing all of your relationships, and all relationships become transactional. And that's extremely corrosive on relationships. Like, you know, people don't go to church anymore because they want to, you know, because they want to worship or do whatever, they go to church to get something. And if you look at all the modern churches, that's what they advertise, right? I'm here to get something, right? And I saw a survey the other day that said that something like 30% of the pastors in Canada are,

14:12
fall on the narcissistic scale, very high on the narcissistic scale. And see, then there it's transactional from their perspective as well. I have this congregation, right? They're not people. And I think this is there. I think that social media just tends to bring this out. Like, I think people do that already, but social media makes it easy. It's amazing that the four of us on this panel were probably alive before social media. Right. And so we've, you know, we've seen like life before.

14:40
And then life with, and it's amazing how much it's changed our worlds, our lives. There was a movie I saw on Netflix, The Social Dilemma, and they were the people who built all these platforms and the insights and the scariness of it all was just, you know, it's just amazing how much it's transformed. And I have little kids at home, you know, and I'm trying to think like, geez, how do I keep this from inundating their world and, you know, making my five-year-old daughter feel horrible about how she looks someday, right? There's just so many implications of.

15:07
I mean, magazines were doing it before. Now it's just pervasive on their phones and then their tablets. And that's the difference. I think is that, yeah, we've always done this as humans. It's that social media is a tool that makes it a lot more effective to do this. Right. Yeah. And then that's- Global scale, right? Right. The distribution is everyone. Yes. Exactly. So we might jump around because I have so many things to ask you and guys cut me off if you have something, but I was looking through your, your blog, which-

15:33
I don't know why it took me so long. Usually when I go to your blog, I'm looking at your, your hedge podcast feed, but I was just digging through and there's, there's so much helpful, wonderfully written information on there. Can you talk about, cause, cause I'm a guy that for 10 years in network engineering has been overwhelmed at the rate of change, right? Like I've been crying about automation for years because I could barely hold on to what the hell's happening now. I failed at a of programming in college and now I got to be a damn dev, right? Like I just, I hate it.

16:03
And I read rule 11 of RFC 1925 on your blog and you know, how it can help a person in tech emotionally handle the rate of change. So can you kind of touch on, because I think it's super helpful for anybody in tech, especially newer people that it's so easy to become consumed and destroyed by this industry, right? Because there's just so much coming at you. You're studying nights, weekends, holidays every time you can just to try to catch up. So what is rule 11 of RFC 1925 and how can it help?

16:30
Yeah, rule 11 just basically says that everything that's ever been invented before will be invented again just given a different name. And I mean, it's Ecclesiastes, it's, I don't know, the statement, you know, the history doesn't repeat itself, but it rhymes or it echoes. And this is a common thing in human culture. So I've taken recently, and this comes out of the Rena model. I don't know if you know much about the Rena model, John Day's Rena model, which is an

17:00
And I'm very reliant on the Reno model now. I really, I'm writing a book right now for Pearson and I'm kind of having to go back to the OSI model and I'm like trying to soft pedal the OSI model in this book, because it's kind of a requirement to talk about the OSI model. But I'm really much more focused on the Reno model. But the reason is- Reno, R-I-N-A? Is that, I'm sorry to interrupt you. That's right. R-I-N-A. Recursive Internet Architecture. But basically,

17:27
What it comes down to is, is I've come to the belief that there are really only four problems that have to be solved to make a computer network go. And for each of those four problems, there are roughly four, three to six, I would say four is a good rule of thumb, solutions for any of those problems. And if you know those problems, those four problems and that set of solutions, you can pretty much understand any protocol because you can ask the questions of how it solves those problems. It's very, very simple.

17:56
And so the idea is behind this is that rather than me learning the details of every protocol that hit and how to configure it and everything else, which by the way, some people need to be fast on configuration. I got it. I'm just not one of those people anymore. I don't, I don't touch config that often. And when I do, if I do touch config, I have plenty of time to go look in the docs. Like I'm not troubleshooting config in real time. And so I see the value from somebody being able to understand or read a config because I was in Cisco TAC, right? I mean.

18:24
I knew the configs inside and out and upside down and backwards and I was intact. So there's value in that. But even so, I would say that it's better to know the protocols and understand how they work and what questions to ask of the protocol so that you can find out how to do things. So that's kind of my theory on things. And it helps me not be so crazy about new protocols. When QUIC comes out, like when QUIC first came out.

18:52
everybody was freaking out, we're all the bits, we're all the bits, how are the bits done, and how's the spin bit work and all this other stuff? Well, it's only solving the same four problems that TCP does. And so it's just gonna use perhaps a different set of solutions or maybe the same set of solutions. It's really the same thing. So that's kind of my philosophy. Did you say there's four problems that networking has to solve? That's it. Is it easy to touch on those? Because that's news to me, right?

19:22
That seems crazy. Doesn't it? That seems where have you been my whole career? You really could have saved me a couple of nervous breakdowns, Russ. How come there's more than four questions on the STIRT exam? Yeah, that's exactly right. That Russ helped right, Daniel Russ. Okay. So I say there are four things, but some of these problems do have some problems. Okay. So let's, let's say that first. But basically you have to do four things. You have to marshal the data.

19:48
which if you think of a dictionary in a grammar in a language, that's what it is. It's marshalling. Some people, I think John Day calls this transport or something like that. I call it marshalling because it's formatting. It's TLV versus fixed length format. Those are your solutions. Basically, there's a couple of other solutions in there, but those are basically what you have. You have to be able to address things. You have to be able to share multi-access links effectively. OK.

20:17
So that's the second problem you have to solve. And that doesn't just go for links, that goes for like multi-hop links and stuff like that. And then you have to be able to handle errors and you have to be able to handle flow control. That's it, that's all there is. Those are the only four things you're doing in the world. Now you can throw encryption in, right? But encryption is basically another way of marshaling. It's another way of formatting data that provides you with privacy. Can you name anything you do in networking that doesn't solve one of those four problems? I don't know of anything.

20:46
It's one of those four things. Now, routing solves part of the addressing problem. So when you talk about addressing and multi-access, sharing multi-access links, then you get to those who are kind of intertwined. And routing is really a subset with its own set of problems underneath addressing, right? That's what it basically is, is you're trying to do shared access and addressing at a remote distance, which means you have to have some way of routing.

21:15
only has four or five problems that it has to solve. And for each of those problems, there's only four or five solutions for each of those problems. I mean, think about it, right? You have layer two addresses, you have layer three addresses. You have to map them. Well, how many ways can you think of to map layer two and layer three addresses? You can have a centralized database everybody talks to, you can have a protocol like ARP, or you can have a protocol which is reactive, or you can have a proactive protocol like Neighbor Discovery and v6. You got anything else? That's it.

21:42
And no matter what protocol you look at in the world, or you can cross map them. You can do like multicast and network and OSI, and you can cross map them like CLNS does. There's four solutions. That's it. That's all there is. I don't care what protocol you look at, mapping one protocol to address to another, those are the four solutions you got. That's it. He's right. I love the simplicity of that. But here's what I'll counter with. Why has networking become so complex?

22:10
And why is simplicity so difficult? If it's as simple as you're saying, why is it so damn hard? First of all, because we like to invent new things. Rule 11. We love our capes. We're network engineers. We wear capes. Man, we love our capes. Although, if you listen to, I don't know, Mr. Incredible will tell you as a superhero, you shouldn't wear a cape.

22:33
They get cotton jets and stuff. You know what I mean? They get it. But so first and second of all, because we don't know that, right? We don't really pay attention to it that way. And the third thing is that we think we're solving some corner case. But the fourth thing is that there's a lot of combinations. Even if I say there are only four problems with 16 possible solutions, or 20 solutions, or whatever that number comes out to be. I mean, I've done charts of it before, and I don't really know what the number is. There's still.

23:02
what's 16 factorial, right? That's how many different protocols you could have, basically, or something like that. And so there's a lot of different ways of solving things, and there's a lot of different ways of combining things together. I think the key is to be able to tear apart the protocol and say, this is the problem it's solving.

23:21
and this is what I'm, these are my options for solving that problem. Which one did they, which one did they use designing this protocol? And then if you understand that, like flow control, okay, well flow control is complex, there's lots of different things to do in flow control. But honestly there's not, there's only two or three different ways you can do flow control, right? You can do a lot of fancy queuing stuff, you can do a lot of fancy numbers and number spaces and try to gauge the depth of queues at different places in the network. That's all very hard and very statistical, but in reality it's either going to be

23:51
one packet in, one packet out, or it's going to be a windowing scheme, or it's going to be I throw stuff at you until you scream or something, right? There's only, I mean, or I throw stuff at you and you throw it away. I mean, there's only like four or five solutions to flow control. Do you think vendors contribute to complexity? And I'm thinking of like EVPN, VXLA, and where each vendor applies the standard differently, which I still can't get my head around. Okay, well. It's a standard.

24:21
Yeah, so well, that's understanding the ITF process a little bit. You know, the ITF is not as technical as, well, it's technical, it's deeply technical, but it's also very political. It's deeply political. And so, you know, it's like any other law, if you write a law and you have two different people who want to apply that law in different ways, and they're both very powerful and the law can't get passed without both of them agreeing, guess what's going to happen?

24:46
you're going to write the language ambiguous enough so that both can do it the way they want to, because that allows everybody to say they're following the standard and allows the standard organization to say, yeah, we've written a standard, but in reality, it doesn't necessarily mean you have interoperability. So this is why in the IETF, the routing working group in particular, is so persnickety about interoperability. But even there, there are ways to slip around it, right? We're interoperable with this open source implementation that our coders happen to also

25:15
and therefore we're inoperable with two implementations. So that is not to say that that's all wrong. I'm saying that's human nature, right? That's just the way organizations work in the human world is they do this kind of game. And yeah, part of it is vendors. Part of it is just implementations. There are ceaseless, endless arguments behind the scenes in the IETF about, well, our BGP implementation works this way, yours works that way. If you force us to do the standard this way, we're gonna have to rewrite our entire implementation.

25:45
And we can't do that, right? And so that turns into tech debt type of, well, not really tech debt, tech debt's not really the right word for it. We use that, but sunk cost types of problems, right? Where we've sunk, every vendor has sunk cost in hundreds of thousands of lines of code. And vendor A goes out and looks at their code and says, I can solve this problem very easily this way. And then vendor B looks at their code and says, but I can't solve it that way, right? That's a huge change in my network operating system.

26:14
And so you get into these kinds of things that you run into on a regular basis. When you were a network engineer, like doing the job at the keyboard, was this a stressful career for you? Um, I think it was to some degree, although I must admit that I think that tack was very stressful, but very, very fulfilling, probably the most stressful slash fulfilling job I've ever had was global escalation because you walk into

26:43
the worst broken networks in the world 100% of the time. But when you walk out, you've either lost a customer or you're wearing a cape. There's only two possible outcomes in the global. It's only a pass fail, huh? No pressure. Yeah. But how have you managed that level of stress? What does your self-care look like? How do you walk into that kind of environment day in and day out and not have it destroy you? So for one thing, I grew up with a lot of stress. And...

27:10
I grew up in a very stressful environment. And then another thing is when I was in the Air Force, I worked on airfield systems. And there was nothing more stressful than being called out at 2 o'clock in the morning because there's a C-141 floating around overhead with not very much fuel left, maybe not enough to divert to Philly or wherever is the closest nearest airport. And the ILS system is down or not working correctly, so they can't land. So you've got 200 people sitting on an airplane

27:40
You know, their lives don't really depend on you because they could land someplace else. But, you know, their families are waiting for them on the other side of the terminal. And it's not a pleasant situation. And other very stressful things like driving out on the airfield and running head to head, head to head with an F4.

28:00
doing like 70 miles an hour or running into a tornado that was running around the airfield randomly and stuff like that. So to some degree, it's just a matter of been there, done that. It just is what it is. To some degree, it's also, I've always had a very strong community around me and I've always had a lot of strong outside of work interests.

28:24
do gunsmithing, I do woodworking, I do philosophy, I spend a lot of time with my wife, we go hiking, you know, stuff. And so I live a very, very active life outside of work that I don't talk a lot about, but that's a lot of it as well. I mean, there's nothing more de-stressing at the end of the day than to go build a table or to go to the range and pop off a couple of hundred rounds. I used to shoot competitively, so, you know, go pop off a couple of hundred rounds and...

28:53
you know, try to get your bill drill down under 12 seconds or whatever the case might happen to be and, you know, do that kind of stuff. It's very, very, very, it's a different kind of stress. It's stressful, but it's a different kind of stress. So it's actually relaxing in the sense that it drops you out of that, that IT world to a large degree. So I want to dig into that a little bit more Russ. You've been involved in many of what I'll call extracurricular activities, such as writing books.

29:20
From writing books to developing certification programs, where does the desire to do that, those kinds of things come from? Do you see gaps in certain areas where if you think to yourself, hey, if I can write a book to fill that gap or I can help out write the certification program or does it come from somewhere else? So, so most of the time I fall into these things. And honestly, I'd almost given up on writing. But Ethan Banks talked me into it.

29:50
getting back on board. That's kind of a long story, but yeah, I mean, I was, I was actually at the end of my tether with writing after one of my books and I was like, okay, I'm kind of done. You'll see a gap. If you look at the years where I published books, you'll see a gap in there when I wasn't really generating content at all. I had pretty much given up. And you can go blame Ethan for me to...

30:09
for problems and solutions and all the video training I'm doing now and stuff. But yeah, and the blogging is actually Ethan's fault as well, as well as the podcast, that's all Ethan's fault. So anyway, yeah, so when it first started, it started as a request from Cisco Press to General Insight Cisco saying, we need a follow on book to Sam Halabi's book. They had published one book and they needed a second. And so Don and Alvaro and I looked at each other and said,

30:39
You know what, maybe we go fix so many darn networks and they're so messed up from a design perspective and the reason they fail so much is because they're so messed up from a design perspective. Maybe even though none of us have ever done a ton of design, maybe we could show people what breaks, we could do the opposite direction, we could build a network design principle out of what we've learned breaks. And out of that came Advanced Happy Network Design in 99. And so that basically started, it was just feeling like there was a need, there was a place where we could do something.

31:09
And then after that, it pretty much followed along the same path, except Pearson was pushing me a good bit. What book are you writing this year? Type of thing, which was cool for a while. And then, you know, I kind of fell off the wagon a bit, but it is true. A lot of it is just because I have the deep feeling that most people don't understand networking the way that I do. And it's kind of important for me to explain to the world.

31:37
how I think about network engineering. Because, you know, as Andy pointed out in the very beginning, you know, I have this whole rule 11 thing. I have this whole four problems, four solutions type of thing. I have this whole way that I look at things that really helps me, I think, deal with the stress of change. And so I think that in design principles and stuff like that, and I just think that, you know, if I can help the world out and be around a little bit, and it doesn't help that it pays a bit on the side, it's not tons of money, but.

32:07
It's something, right? So that's kind of the main reason. I always tell people when they say, thank you for this book, I'm like, I'm just happy that it's been useful, that you found it useful. How many books have you written? It's ridiculous. It's amazing. I actually don't know. Ha ha ha. Too high to count. Too high to count, right? I do think that's a common theme, because we've had a few different writers

32:36
on different episodes now. And that seems to be the common theme is that they get done writing it and the goal is that I hope this is useful to someone. Yeah, yeah, that's exactly right. Yeah, it's not so much about your sales. It's more about even if 10 or 15 people pick it up and say, oh, that was really useful. That was good. I'm really happy I went to that online training that you did or whatever. That is kind of- Did that original design book, did that eventually-

33:04
help lead into the CCDE? Yeah, to some degree, the CCDE is a little bit of a weird story. There were a group of us who decided we were not recertifying CCIE any longer because that's not what we do. We didn't do troubleshooting. We were just kind of like, why am I spending my time doing this? It's not that it's a wrong certification, right? It's not that we were saying it was bad. And by the way, the CCIE, I was deeply involved in rewriting it a bit as well. There's a kind of funny story there

33:34
Three or four of us who were known as the top experts in Cisco TAC at the time failed our recertification on our specialty protocols or came close to failing on our specialty protocols. So we said, okay, that's broken. So they said, fine, we agree with you. It's broken. Now go rewrite it then. And so that's how we got involved with the certification people in the first place. And then- Wait, you failed the certification and they said, well, Russ couldn't have possibly failed the cert. So-

34:00
Go rewrite it. We need to rewrite it. Yeah. Realistic, go write an exam. You think you can pass, Russ? He's not wrong, we're wrong. Yeah. I was coding EIGRP part-time and I failed the CCA research and the lowest score was EIGRP. Oh, that's ridiculous. Okay. That's ridiculous. Okay.

34:25
That's funny. Sorry, but I don't think so. And it wouldn't have been so bad except like one of the top BGP experts in the world, his lowest score was on BGP and one of the top OSPF guys we had in the tag, his lowest score was on OSPF. And we were like, okay, three for three. This is not cool. Oh my gosh. So yeah. So, I mean, this is way back in the bad old days and the certification was, was very, you know,

34:54
was very light and skinny and stuff. And so it wasn't like today where it's a big production with all these psychometricians. There were no psychometricians involved at that point or anything. And so, but we basically, the same group of people plus a couple of others just said, you know, we think the CCA is valid, but it's a troubleshooting exam. I'm not in escalation anymore. I don't dive into hard troubleshooting problems anymore. I'm much more...

35:19
going out and helping customers design their networks. And this was kind of one of the fun jobs I had at Cisco was I would go out and I would talk to customers and I would say, okay, what problem are you facing? And they would explain it to me and I would try to design a solution around it. But then I would take the problem that they had raised and bring it back and turn it into features. That was basically the whole job was to go find pain points by being a subject matter expert in network design and pulling, getting, because customers won't tell you.

35:48
when you're at a vendor, I mean, they'll tell you what features they want. But that's awesome. But that doesn't necessarily mean they're right. Honestly, right. They could be asking for something because they haven't thought through the whole problem or because somebody just some other vendor has it or whatever the case is. It doesn't really matter why they ask for it. And so it often runs into this. We often ran into the situation where they were asking for something. We were like, that's crazy. Like, why are they asking for that? So I started going to customers and saying, tell me what you're doing.

36:17
explain to me what it is you're trying to accomplish. Don't tell me what feature you want. Tell me what you're trying to solve. And then we could then come back and say, no, we don't really want to do the feature that way. We really want to do it this way over here. And the customers, that's not what they ask for. But you know, that's the right thing to do. And so that was kind of, so anyway, that's what I was doing at the time. And the CCA just didn't really apply to me. And I didn't want to spend the time studying the troubleshooting bits to do it. So I just said, I'm not going to do it.

36:46
And so I wasn't going to recertify. I was going to let it drop. And this is before Emeritus and all that stuff. And a couple of particular VP's stood up and said, no, that's not, that's not acceptable. Um, it's not acceptable that you're going to appear before my customers without a certification and tell my customers they need their certifications. Like that's not, that's not a cool way to be. And so he basically said, what would it take to get you to certify at the expert level?

37:15
And I randomly said, build a design certification. And he said, here's a project manager, make it happen.

37:23
And so this project manager spent a lot of time gathering up all the other people in my same situation and getting these people together and building a team and hiring a psychometrics firm and building the CCDE. I mean, it was, it was an incredible process. Wow. That's where it came from. So it's all your fault. That's amazing. But before we get off search for us. It wasn't just me. It was Alvaro.

37:53
John, it was John, oh my goodness, what's his last name? I just talked to him the other day, but anyway, and it was Mosadoc and it was Khalid Raza, and it was a bunch of us who were in the same position. We were all like, no, we're not doing this anymore. And this VP said, I can't have all these really senior people dropping their, this is just not cool.

38:12
Like I just have too many senior people dropping their certification. So somebody told me something about Cisco certs and I, I've wanted to ask you this for a while, I forgot. So did Cisco come up with a certification program because tack was overwhelmed with cases and they thought we need to train these customers. Is that true? That's, that's pretty much what it was. That's where it started. Not even so much just to train the customers, but also to say we had too many cases being escalated.

38:39
And so if you make people get a certification before they can escalate, you make sure that you know that the customer has asked the right questions and done the right things before they call and before they make a mess out of your time. Your time can be used on. It's a brilliant move. But that was in response to the TAC caseload, right? Yeah, that was a direct response to the TAC caseload entirely, particularly higher end cases. And I mean, I don't want to talk bad about customers, but we had all sorts of weird stuff in TAC, you know.

39:07
There was one time that I had a manager on the call and this was not an engineering problem. The engineer was actually pretty decent. It was a 7500 and his problem was that they had downloaded the wrong version of code and installed it on the 7500 and the 7500 just rolled over. It just wouldn't do anything. And so their manager was on the phone constantly berating us about get this done faster, get this done faster. And so finally, now this is way back in the days of AOL, I asked him if he had an AOL disk laying around.

39:37
And he said, yeah. And I said, why don't you go create an account. And then I went on Cisco online and I found the largest physical image file I could find. Because all he had was Modem access at this point, because the 7500 was down. And I said, why don't you download that file for us? It'll be really helpful. So he went off and did that for about a half an hour. And in that half an hour, we got the problem fixed.

39:58
Watch yourself some time You know horrible horrible tech tricks Russ I have a question for you. You you have made some Significant contributions to network engineering over the years. I want to know kind of this this two-part question What got you into network engineering and what about it?

40:26
drove you to do everything that you've done for it. Mm. OK, so what got me into network engineering was when they replaced the TACAN at McGuire Air Force Base with a digital TACAN and VOR, which was cool equipment. I mean, it was really cool stuff. And when they started talking about taking down the FPS-77 and replacing it with a next-gen radar system,

40:55
I realized that the old tack-in you went in and you troubleshot by pulling things out and playing with them and using ohmeters and you know everything else to figure out oscilloscopes and stuff to figure out what was wrong with the device with the piece of equipment. The new one you sat down at a keyboard and you said please run self diagnostics and then it would run through a menu system and you would ask you questions and then it would tell you which board to replace. And I thought okay if this is the way electronics is going.

41:24
I'm really not interested anymore. Thanks, but no thanks. Like this is just not fun. And so I jumped over, I was self, I self taught myself, I taught myself C during this timeframe when I was in the Air Force. I bought C Primer Plus, this was the name of the book. It was awesome. Had a lot of stuff in it about computer memory structure and help you understand all sorts of stuff about why you were doing things the way you were in C.

41:50
like what pointers, they didn't just talk about what a pointer did, they actually laid out the entire memory structure of our computer, you know, data, text, et cetera, et cetera, and explained what the pointer was doing and how it impacted the stack and how the processor pushed the stack off onto memory to do an interrupt and all this other stuff. It was a very, very good book. I don't know if it's still out there, but it was a really good book on computer architecture. And so, somebody figured out that I knew C, so they, and I knew a little bit about computers, so they drug me into the Support Computer Support Center,

42:19
just to help do desktop troubleshooting support and stuff. And then this problem came up of replacing the backbone. And so I started diving into networking and got just got really fascinated with it. So when I left the Air Force and then I was in tech control for a while working on telco switches and crypto gear. And so when I got out of the Air Force, I said networking is what I really wanna do. So I went and got my CC, my CISC, my certified network expert engineer, CNE.

42:48
And then I studied and worked on my CBE, my Cisco, my certified Banyan. And then I decided to move to North Carolina for various reasons. And when I moved to North Carolina, I started looking around for jobs and just fell into Cisco, literally just like they were hungry for people. And I was looking for a job. And so I ended up in the routing protocols, back line TRT taking cases at Cisco. And so that's basically all I fell into it. It was not, it was not totally intentional.

43:17
on my part. If it wouldn't have been for Cisco, I probably would have gone towards the Linux host side of things rather than the routing protocol side of things. But there that is. As far as doing stuff, I just think that my electronic engineering background being banged into my head when I was a kid, doing amateur radio when I was a kid, and doing radios and stuff when I was a kid, and just the whole engineering process and troubleshooting process has kind of built in inherently built in me this.

43:45
desire that every time I see a problem, I want to get to the root of it. I want to understand the root of the problem. I want to understand not just how to fix it, but like down at the bottom. Why does that work that way? Right. Why, why is that the way it is? Um, and.

44:01
You know, like when I was working on a radio, a tube-type radio or something, it was never enough for me to go, oh, I found the bad tube, let's replace it. I'd be like, where is that tube in the circuit? Why did that tube cause that problem? Right, what's the chain of causality to get me from problem from from component X to problem Y? I want to know what that is. And so I think that has been very instrumental in my ability, again, going way back to understanding and putting structure around things. And I think that's why.

44:31
I've been so successful at understanding things in a way that I can turn around and explain them hopefully in ways that are useful to other people. Right. Sounds like a very curious. That's great. Yeah, I ain't very curious. On that note, Russ, I'm going to have you explain something.

44:50
I did have kind of in the learning vein, I had a question looking at your, I think it was the bio page on your blog site. I apologize if I've got the timelines wrong, but I think it was about 10 years ago, you went to VeriSign Labs to research global internet security and DNS issues. Correct. So I think DNS can be- Immediately after Cisco. Immediately after Cisco, right. Immediately after Cisco.

45:18
I think DNS can be a pretty nebulous topic to IT professionals, even network engineers. In fact, I think there's some organizations where the networking team may not even have their hands in DNS. What I've found, I haven't really seen any structured certification paths or learning paths that are specifically aimed at the inner workings of DNS. Do you have any advice on how?

45:45
people can get more immersed in what DNS is, how recursion works, root servers, all that good stuff. So this is why I included in a class that I teach every six months or so at Pearson, a pair of classes called How the Internet Really Works, part one and part two. I spend like two or three hours just on DNS in that timeframe. And this is why I'm teaching at University of Colorado. I'm teaching an hour just basically on DNS, because I think DNS is scattered throughout the entire course.

46:14
because I do think it's one of the most misunderstood parts of the infrastructure and not well understood parts of the infrastructure. I don't know of any other resources, partially because I probably never went and looked because I learned DNS the hard way. I got, you know, VeriSign, you're sipping from the fire hose on DNS, right? Overnight, you're like, you're- See, Russ, that's what I'm trying to avoid. But-

46:38
I don't know of any other really good resources. It's probably a good idea for somebody to do something in that area. And maybe I should get with somebody and do it. But I don't know of a lot in that area. And in fact, it's even worse now. DNSSEC is a total mess. The high-fumers, RADB thing is really making hash out of a lot of the backend work. And then there's D-Records, which are astounding to me. I can't even imagine.

47:07
I don't know, people, I think they're crazy. I think, I just think D records are insane, but there's that, you know? Now, if anybody doesn't know what a D record is, a D record is effectively a, how would I put it? It's not a C name alias, it's a wild card type thing. It's like throwing regex in the middle of your DNS name and like, wow, that's just insanity. Like it's just total insanity.

47:36
Like a C name is so specific lookup, right? Right. A D name is like, give me, give me a service under rule11.tech that might do this. Like, whoa. Or give, give me the T, give me a website that provides this service. And I don't care what domain name it's in. Like.

47:54
What? That's crazy. But yeah, that's D name is D name is pretty incredible to me. So, yeah, so I can really answer that very well, because I honestly know the answer. There's not a lot out there. Well, maybe maybe.

48:08
When you get back into writing again, that can be your next book for me. Well, I'm actually writing right now. Pearson asked me to do a book, so it's the first book I've done since Problems and Solutions. I normally don't do books, I do video now, but. Sure. Cisco's coming out with a new certification, I don't know if I'm supposed to say any of this, but they're coming out with a new certification. And Pearson was looking for somebody to write the book for it, so I told him I would do it. In fact, this is the first certification book I've ever written, really.

48:38
I don't really write certification books. Certification targeted books, I write theory books, but this one I felt like I could get away with mostly making theory and just tacking the certification bits on, pushing them in here and there, so I felt like I could do it. So that's a- So Russ, before we run out of time, I definitely wanna get into some career type advice for up and comers, but right before we do that, you're like the smartest design guy I've ever had the pleasure to speak with in person. So-

49:07
I don't know if this is a fair question, but what did most orgs do wrong when building and managing networks? Like why, why is it always such a damn hot mess? Are there, are there themes you've seen or it's just varied and sundry? Yeah. The biggest problem is, is that everybody does really brilliant design on day one. They greenfield it and then they start hitting in corner cases and things they have to support, and then they just start throwing crap at it. They just start throwing slop at whatever they're doing.

49:35
And a really brilliant design turns into total crap if you just keep throwing slop at it. It just does. Yeah. There's no thought in most networks for lifecycle control, for managing the complexity in a disciplined way. You get into a data center, and you're like, OK, I'm going to build a pod-based data center design. Cool, awesome. And then you get into it, and you realize that every pod is effectively a snowflake. They're all different generations.

50:05
And you're like, well, why'd you do that? Like, don't do that. Cause you didn't tell us not to. That's some discipline about your generation, your life cycle stuff. This is like, don't do that. Drives me crazy. That's interesting. Like, yeah, the initial design is great, pristine and you put it in, but then after that it just seems to go to shit. Yeah. And by the way, this is- Yeah, I mean that's-

50:32
I don't, we don't have time to get into it, but that's one of my major arguments around automation is that automation is great and good, but it really depends on what kind of design you have. If you don't have a design where different parts of the network are similar and repeatable then automation in my opinion is going to be really difficult. So I think at the end of the day, it all goes back down to design.

50:58
Yeah. And everything's a snowflake, right? Yeah. Every data center I've worked in, you know, the BU has a super important thing, has to go in and we're out of time and out of budget, go, go, go, and go around all the standards that we created and you do that a hundred times and then you have a hundred snowflakes everywhere. I mean, how would you ever automate that? My variables are too variable. Yeah, exactly. And by the way, this is true of protocols as well. By the way, this is not just networks. We do this in protocols too. I mean.

51:24
We do it with BGP. We do it with ISIS. We do it with all the protocols. We say, well, there's this corner case over here and I really want to solve this corner case. We have a customer who's willing to write us a $500 million check if we solve this corner case. So we've got to find the protocol to do this. And you do that a thousand times and the protocol makes no sense anymore, right? It's like complete disaster, right? And you get all these- So the business drives technical decisions, right? And that's where it starts to go.

51:53
sideways people with checks say do dumb things and we do. Right. And that's the problem, right? Is that it's fine for business to drive the technology. That's what should be happening. But business needs to have the discipline to realize this is where we don't do good as network engineers, we don't push back and say, all right, I can do this for you, but let me tell you that five years down the road, this is going to be a problem and the business people just throw their hands up and say, we don't really care, that's five years from now.

52:23
But we've got to get better. We as network engineers need to get better at actually saying, no, that's not the right way to do this. Stop, back up, let's think it through. I know it's going to take me two extra weeks. I know it's going to cost you x more. But don't bite the sunk cost fallacy. They'll all understand the sunk cost fallacy in sales in all these other places, but they don't understand it in the network. And that's a real problem.

52:52
And it's partially our fault. We don't do a good job. It's partially the problem of all the schools that teach MBAs and don't teach MBAs technology and salespeople technology, but that's another entire problem, right? That's the other side of the coin. Yeah, my experience has been that, you know, people as brilliant as you, Russ, have been in orgs and they'll tell leadership, no, we cannot, this is awful, these terrible things are gonna happen. The places I've been, they just do not care.

53:18
Our biggest customer we have gave us a huge check. We have to do this. I understand what you're saying, Mr. Smart Engineer, but we're doing it anyway. And then all hell breaks loose later. Sometimes you're in that situation. I was just in this situation kind of not too long ago. What you can do is you can say, OK, I understand the urgency. We need to get this done. Let's do it this way now, but let's put tweaks in so we can walk out of it in the future.

53:45
Let's plan to make this current design obsolete. Replace it with the right design. But let's get it up and running to meet your business problem, fine. But let's think through how we can say, we're gonna lifecycle this, okay? This is gonna have an endpoint and we're gonna replace it.

54:02
Reminds me of that. There's nothing more permanent than a temporary solution in tech. Exactly. We're just going to do this for a little bit. We'll take it out later. We swear. Yeah. It's only testing until it works, right? That's what I mean. That's right. So Russ, we're getting close to the end here. We have a lot of folks listening that are either trying to get in, or getting in, or trying to grow their career. And you've seen, I think, more than anybody else on this panel over the years in your career.

54:31
You know, how has the job changed over the years? What can somebody be doing today, now? You know, we always parrot the same things, right? Like get a certification, build a lab, make some content, try to create a name for yourself. You know, is there anything outside of those obvious things that you would tell somebody coming up or trying to advance their career? Like, what can you do to stand out and be a great network engineer? So yeah, I think all those things are important actually. Certification, degree.

54:57
Learn to be a good communicator however it is you do it, whether it's public speaking or writing or blogging or whatever it is. Learn to be a good communicator because we don't communicate very well. Engineers in general just don't communicate very well, unfortunately. And by the way, I put myself in that same category. I mean, I still struggle to communicate a lot, a lot of times in a lot of places, in a lot of situations. But beyond that, I think the biggest thing is...

55:25
to learn how to tactically push back. Again, I'm not very good at it, but learn how to tactically push back. Learn the business enough to know when the business is pushing on something that doesn't need to be pushed on, right? Learn how to say no, that's enough. And learn how to say, yeah, you might miss this market opportunity, but you know, there's gonna be another one and we're gonna be better prepared for that one. And things like that, you know, just you need to learn the business side a little bit at least.

55:53
I stink as a business person, honestly. I really do. But my excuse is always that I'm learning philosophy.

56:01
And again, philosophy is teaching me a lot about people. And so that helps me interact with people at a different level. I'd wish I'd learned about the business earlier on. I'm learning about it now, 12 years into my career. And it's something I wish I'd done. I wish I could learn more about the business side. But I don't know. It's very difficult a lot of times to get both sides of it. Ideally, I would love to see us have an MBA that is a technology leadership MBA someplace. Mm.

56:29
And, you know, there was one and they closed it. I did. I did it. It was it was a misfit. So Champlain College in Burlington, Vermont had a managing it was a master's science and managing innovation and information technology. And it was an MBA with a tech twist. And it was one of for some reason it was one of their least popular programs. And they ended up closing it. Yeah, it was it was a great program. I learned the business side. I learned how to apply the tech to the business side.

56:58
people with an MBA background learned about the tech and how the tech could influence the business and support the business. It was a great program. I don't understand why they closed it. That's what we really need. We need more of that, right? And I don't know how to convince colleges to do that, but I've been on a tear about it for a couple of years and haven't gotten anywhere, so.

57:17
Yeah, I think the biggest thing is learn the fundamentals. Don't let yourself get trapped in learning how to configure stuff alone. It's great to know how to configure stuff. Learn how it works and why it works that way. Build mental models, build a framework around what it is you're doing, and then learn the business stuff and be a good communicator. I think those are the big things to me. Certifications are great, to a degree. I mean, you know, I have a PhD, two masters, a bachelor's, and three certifications, and I think I'm...

57:46
Pretty much that's all I want to do. Although I thought about getting another certification in. What else is there Ross? That's all. You've done it all. I don't have five CCIEs. I don't have seven JCNIEs or whatever they are. Right? There's people that are ridiculous. I don't have a CCNA. I don't have any of that stuff. You know, I started out in the CCNA. The CCNA didn't exist when I started. So it was just, you went dumped right into the CCNA and that was what you got. Oh, well. Crazy. Well, I have pages of questions I didn't get to, but.

58:14
We'll just step through that. Well, you know, there's always other times. Get you back on some other time. Yeah. That's cool. Is there anything we should have asked you? Is there anything you wanted to talk about that we didn't get to? You tried to re-take that out of your mouth. Sorry, AJ. No, no, you're great. No, no, I mean, we could go on all night if we start that path, so. Well, what do you have coming out? Do you have like any new books or new courses or I really enjoy all your contributions in the community? Anything new coming down the pike? I'm doing a course on privacy at the end of January for Pearson.

58:44
towards the end of January. Then I always have my regular training schedule with them. I do three hours a month or six hours. I'm doing a data center fabrics thing that's six hours now with them. I am working on this new book. I have some other ideas on things I wanna do and I'm teaching at UC, University of Colorado. I need to go ask them like, how do they feel about people auditing courses?

59:07
from them, it would be interesting to know what their auditing policies are. Because I've never even asked. But yeah, so I'm teaching there and I have other ideas on the table I've been working on. I've been thinking about starting up my own little training channel someplace and just doing a video a week or something, but I just never have done it. I haven't really put it together and done it.

59:29
It'd be kind of nice to have, you know, some, it'd be kind of nice to have other people than me involved in it. So one thing I never do is I almost never write alone, believe it or not. It's like I always have co-authors. Not for video training, but for everything else. Everything I've written, I've had co-authors. Well, Russ, where can people find you and your content? They can always follow me on rule11.tech. That's going to be the best place. Because basically anything I write, any place else...

59:55
or do anyplace else, I cross post there. So if you follow that, you'll see everything I do. Pretty much, well, in the technology world, I write other places that are not technology. So that's another thing. And they can get your podcast feed there as well, right? Yeah, they get the hedge there as well, yeah. And you all are all welcome to come on as guests for the hedge, any old time. We're non-commercial. I love your show, I listen to them all. They're just fantastic. I really, again, I just love your contributions and.

01:00:23
I love that show. Thanks. Yeah, it's good to hear. So yeah, yeah, we can plan another one of these. It's fine. Just just let me know when. Thanks so much for coming on, Russ. Awesome, Russ. This has been such a treat. Thank you so much for joining us this evening. And thank you to everybody that tuned in for the live feed of this. You can join us anytime we're recording. We're going to tweet out every time and live stream with anybody who's willing to join and watch us. So thank you all for joining us and we'll see you next time on another episode of the Art of Network Engineering podcast.

01:00:55
Hey everyone, this is AJ. If you like what you heard today, then make sure you subscribe to our podcast and your favorite podcatcher. Smash that bell icon to get notified of all of our future episodes. Also, follow us on Twitter and Instagram. We are at art of net eng. That's art of NETENG. You can also find us on the web at art of network engineering.com where we post all of our show notes. You can read blog articles from the co hosts and guests.

01:01:22
and also a lot more news and info from the networking world. Thanks for listening.