00:00
This is the art of network engineering,  where technology meets the human side of IT.  Whether you're scaling networks, solving problems, or shaping your career, you've got the insights, stories, and tips to keep you ahead in the ever evolving world of networking.  Welcome to the art of network engineering. My name is Andy Lapteff, and if you haven't noticed,  I am  in person  with a human, not in my home studio. We are at,  where are we? We are in Silicon Valley. I guess I should introduce this nice man who's here with

00:30
Kevin Myers. Hi, Kevin. Hey, Andy. How are you? Good to meet you. I feel like this is Monday Night real life. Right? Tonight, we're seeing you the... It does feel like football. Doesn't it? Yeah, yeah. So we are at Tech Field Day's event. Networking Field Day 39 is tomorrow. And Tom Hollingsworth and Tech Field Day folks were kind enough to lend us their room. They set everything up and the guys left and they're like, you can use the room. It's great. So thank you, Tom and Tech Field Day.

00:59
super support the community all the time. And this is really incredible. Because I got here and I looked at my room and I'm like, oh my God, I am in a closet. And it's going to be really uncomfortable. So thanks to Tom and the Tech Field Day folks for hosting this for us. How was your flight in? It was pretty good. Not too bad. How about you? I know it was great. Yeah, we're both coming from the East. yeah, five hour flight. Six hour flight. I worked the whole time. It's good, man. Everything's good.

01:23
Nobody wants to hear complaining. Time just fell back, so the best way to beat that is to fly to the Pacific time zone. Exactly. So what are we going to talk about today? I thought in the spirit of Tech Field Day, right? What is Tech Field Day for people? This isn't a Tech Field Day episode, but we're here and they're being very kind. So I want to give them a shout out. The environment we're in is networking vendors come to Tech Field Day and they present what they're working on. Their wares, so to speak. And then there's delegates like Kevin. I was a...

01:50
Tech Field Day delegate prior to being a presenter tomorrow  on the vendor side.  But vendors  show what they're working on to delegates  and you get some feedback, there's back and forth, you get to see what's coming and then the delegates create content around that. Is that kind of a fair description? You've done what, 11 of these? Yeah, it's a really, it's a deep, mean, especially  Tech Field Day has all the different genres. They've got Mobility Field Day, now they have AI Field Day and all the different field days and Networking Field Day is great because if you're a networker, like you're going to be in the nerd zone.

02:20
with  all of the, not only some of the best people in the vendor space that you get to ask all the questions. You're like, I wonder who's the right person is that knows the answer to this. They usually are bringing their A team. So you get to ask them those questions, but you're also surrounded by the delegates, which are usually uh a group of super nerds in the network engineering space. So if you like being immersed in that world, this is several days of that and it's awesome. That's the awesome part of it is it's not like marketing.

02:47
people up on their blazers so don't know anything. Like they're super technical people who show up to these They're like the engineers, the builders. Yeah, usually the technical marketing engineers. And those are typically the nerds that are extroverts, right? Like  it's the people that know the geek side, but they also like, you know,  want to talk to other people. Now there's anything wrong with that for you introverts. It's just, I'm an extrovert. So it's nice to meet other geek extroverts. You know, I didn't know I was an extrovert. You didn't?  No? Because over the years I would have kind of like,

03:18
pre-getting-together anxiety. So like before I would go out in a group of people, I'd get all worked up. And I thought, well, I can't be an extrovert. If I like people, why am I upset? Well, I come to find out a little bit of social anxiety, which is a thing, and that's okay. But my buddy was like, hey man, when you go with a group of people, are you energized by it or are you exhausted by it? I'm like, I love people. I wanna be around people all the time. I love it. He's like, you're an extrovert. I'm like, oh, I didn't know. So yay, I'm an extrovert. So with that context set, we build networks, right? Networks have to be built.

03:46
They have to be maintained, they have to be built, they have to be upgraded.  And how do we do that? Well, there's vendors who make this stuff, right? Who are the big ones that come to mind? There's Cisco, there's Juniper, there's Arista, there's Nokia that I'm newly minted in.  They're way more familiar with them than I am. HP, Aruba, there's, in my world, uh Microtik, Ubiquiti,  IP Infusion, there's a few others. There are a lot of choices, right? There are, yeah. Yeah, and...

04:15
I guess that's the topic is we have to build networks. There are a lot of vendors who provide solutions to engineers who have to build networks. How do you navigate the sea of vendors and solutions? And let's say, so I'm in the data center space, just as an example, as a starting point, terrible starting point, but it's my context. Let's say you have to build a data center. You're to do EVP, NVX, LAN overlay and all the things, right? And like,  who do you go with? And I think I'll start here just because it's, I think where we all start, Cisco, I think like,

04:44
And I'm just saying that because I wanted to get into networking and they're like, go get your CCNA. It's on all the job requirements. You should probably go start there. And it's the best training and blah, blah, blah. So I did, which is a really smart move by Cisco because then it kind of gets you acclimated in the vendor space, acclimated with their products. You're comfortable with it. mean, the Cisco CLI is my jam, right? Like- Oh yeah, that's where we all started.  So that's where we all start. So I guess that's the obvious place to start is like, does everybody start at Cisco? When you're gonna build a network, what name comes to mind?

05:13
Maybe Cisco. think, yeah, I think the question that always gets asked is what kind of a network are you going to build? So you've got your verticals, you've got enterprise, you've got enterprise, you've got data center, you've got service provider, and then you've got cloud. And so you've got to figure out what kind of network am I building? And then you mentioned data center. So even within data center, there are so many different subsets because you think about there's hyperscale data centers, right? That are building these massive data centers and they care about.

05:38
building fabrics and being able to control traffic east-west in a fabric and scaling and how do I build global data centers and network them together and that's one set of challenges. Some people when they talk about data center may just mean like an enterprise data center where it's a very, know, maybe one rack of gear, right? To me, it's always you look at who are the vendors in the space for the kind of network that you're building that you care about and then what are the problems that you're trying to solve? We so often in this space put the cart before the horse because it's okay.

06:08
I got to go shop for gear and I want this product, I want this vendor and I this and I got this much money to spend instead of starting at design and requirements and saying, what problem am I trying to solve in the  data center space?  If you're looking at building a very vanilla enterprise data center network where you're going to have compute and storage and connectivity to other things for  an enterprise data center, then you're to look at, okay, you know what?

06:33
what routing protocols do I want to use? I going to need to have overlays? Do I need  to build overlays to connect the servers? What do the server teams want? What do the application teams want? I think that's the step that gets skipped in a lot of cases because vendors are great about building designs, reference designs, validated designs. So a lot of times you just go to your vendor and say, if you've been using vendor X for a long time, say, okay, what's your latest design?  And that's working.

07:00
in the service provider space and working in the commodity space, I always like to try when possible to detach, you know, instead of going to the vendor and saying, give me the next thing you got, let's talk about requirements. And then from there, let's look at, no, is vendor A still the right fit? Is Cisco still the right fit or should it be somebody else? Should we be looking at Arista or Nokia or whoever it is? Because once you get down to requirements, then you can start scrubbing that against operating systems, equipment, budget, things like that. And to me,

07:27
That's where I start the process. So I'm going to do what I do and say something that sounds not smart, but there's a reason. So you just talked about a whole lot of choices, protocols, designs. It sounds complicated. And the silly thing I'm going to say is aren't all networks the same? What do I mean? We're sitting in a carpeted space, so we're going to need an AP. The AP's got to terminate to a PoE switch, so that's going to sit in a rack somewhere. So OK, you need switching for the LAN, including the PoE.

07:56
the land's gonna hit a router somewhere to get out to the world, because it's gotta connect to a different network. Okay, got that. And maybe this is where you're going. Depending on what we're running here, we're at a hotel. So like, do they deem the data center here? Are they doing their own processing of financial transactions here? I was on, where was I?  Oh, we did a cruise ship networking uh episode. And the guy said that every cruise ship, he was with Carnival, they all build these fairly large data centers in each cruise ship. And I was surprised, because I'm like, well, can't you just, what do you need?

08:24
They wanted all their menuing. Like, so they made choices, I guess is my point. Like, what are our requirements? What do we want to do? Well, we want all that stuff local because we're not going to depend on the, on the satellite stuff. So they build data centers in every boat. in my oversimplified view of the world, like, aren't we all building the same, like think of the old school, like three tier architecture. Well, you need a core, you need distribution, you need an access. And that makes sense. And it was when Northwest and blah, blah. And now we're CLO East West, but I don't know why.

08:51
Like Russ White has a great thing he says where like there's only, I forget if it's three or four problems that are there in networking. Like Russ is one of those guys that's so smart and he's coming on soon. I can't wait. Like he looks at networking and say, guys, this is simple. Now, I don't know if he believes that or if he's being, but I really think he believes it like, it's simple to him, but you and I could sit here for a week and go through all that minutiae. Well, what are our requirements and what are we trying to build and what do we need and what protocols? And like, I don't mean to sound like disrespectful to the protocol people, but like who cares?

09:19
what protocol you're running. Like, I guess it's important for people like, oh, well, I must run OSPF because my, you know, interior gateway protocol must do these certain things that it only does. I'm oversimplifying it, but there really is a lot of complexity, right? And depending on your requirements and depending on your design, like you even said, and I'm gonna shut up soon, but you even said like, well, they all have validated designs. So if everybody kind of has the same requirements, I guess they don't, which is the dumb thing I'm saying. And if all the vendors have validated designs, this is how you design it, everybody's putting overlays on top of overlays on top of overlays.

09:49
and  I'm taking my vendor hat off because I know our differentiators, but it seems to me that all the vendors are doing the same thing. We're now talking about treating networks as cattle, not  pets and just abstract everything. It doesn't matter, automate at all. It's all Broadcom, Silicon. How does someone make a choice? And I know you've made a lot of choices. You're a white box guy, right? Yeah, I do a lot in white space. So  I guess A, are requirements so unique? We keep building snowflakes  and if all the requirements are unique,

10:18
and we have to make all these  specific choices about design and protocol and what am I need and what's my traffic?  Are there really that many corner cases of networking? Because we ever complicated, I guess is why I'm going with this. Yeah, there are. And it depends on what space you live in. Because  if you're in a very mainstream vertical, then you're right. A lot of the problems are solved if you're looking at something very simple like,  you take the hotel here, the example that you use. You have internet connectivity, you might have some security, some switches,  some POE. uh

10:47
That model's been out there for a long time. Every carpeted space is like a template, right? Yeah. They need the same thing. often in those environments, they start looking at it and say, OK, we're spending X amount of money doing this. Do we need this? Let's say that it costs $100,000 to build the network for hotel. Well, can we do it with 20 so that we can invest that money elsewhere? And can we solve the same problem with some other vendor and use that money elsewhere? And I think that's where you start looking at choices.

11:14
in the design, but to go back to something that you were talking about with Russ, because I've talked to Russ a good bit. He and I have talked architecture on his show about modular network architecture. And I think the reason that Russ says it's simple is that when you spend enough time in architecture and ops and you've done it across a long period of a career, you generally get to see the impact of your decisions, both right and wrong, and how they played out. And you start to see the patterns in network engineering and network architecture about

11:44
what works, what doesn't, and what's hype, right? Like, what are the things that you keep coming back to that you lean on to go build networks? And what are the things that kind of phase out as a FAD? And so talking about protocol choice,  if you're building, like you talked about, OSPF, if we're going to make a choice between OSPF or ISIS, then we've to look at our available vendors. But one of things that  I know from working with ISIS is that it lowers the control plane.

12:11
CPU usage quite a bit. So if I'm going to build a really big network, I'm going to want ISIS because in most of the networks that I've worked on, because ISIS is more efficient, OSPF is very chatty in the way that it does its link state database versus ISIS. It's not quite as chatty without getting into all the mechanics of it as OSPF is. And so if you're going to build 10,000 routers into a network, then you're probably going to be one of looking at something like ISIS for an IGP coupled with something like BGP. But let's say

12:40
we're going to need to run a routing protocol at this hotel here, and we're going to have maybe one or two routers. Maybe it's a firewall, and we're going to have 10 or 20 switches. Well, maybe OSPF is just fine. And those are the kind of trade-offs, regardless of the network you're building, whether it's an ISP, whether it's a data center, whether it's the hotel network, those are the trade-offs that you  have to look at and say,  what is the advantage in using this? And what's the trade-off of the complexity? again, this is all in your requirements phase. And if you've been around for a while, if you've been on networking for a while, you should be able to articulate,

13:10
the value of those because ISIS,  as we talk about it, it's a little bit archaic. It uses a CLNS style addressing format that a lot of network engineers are not familiar with. So then I've got to look at it and say, okay, if I were to use that here, what would be the benefit and is my trade off in getting somebody to support it worth it for the engineer? And so again, that's all in your design requirements phase, which then starts leading you towards that.

13:38
vendor selection phase because if I say, okay, I need to have a firewall, I need to have OSP app, I need to have POE, I need to have layer two switching, well, let's look at who the available vendors are in that space. And then if I go back to my example of, hey, we've been paying X vendor for 10 years and they're very, proud of what they built and it's very expensive. What if I want to take that money and use it somewhere else? Is it worth it to try and select a different product to do the same job? And this is where I'll get on my soapbox a little bit because I-

14:07
I often believe  one of my big things is typically when you're dealing with a vendor, especially when you're dealing with the top vendors in the space that tend to be very, very expensive, we often will go get what the budget allows. That's the way a lot of organizations work. They're like, we're going to buy as much as the budget will allow for this project, and then we're going to make it work. And that's where you're skipping the requirements phase. So what's behind that behavior? Do they have to spend that budget or they'll lose it? Is it like the user to lose it budget kind of thing? I think that's part of it.

14:34
And I think part of it is the, when you have that vendor relationship and you don't necessarily have the time to go say, Hey, we're to go evaluate 10 different vendors. Or maybe you did choose a couple of different vendors like five years ago and it didn't work out so well. So everybody has a bad taste in their mouth and like, no, no, no, we're not going to go away from Cisco or whoever it is because we tried another vendor a few years back and it didn't work out. So we're not going to do that because that can't work. But  as we get further forward  into  the world that we're in now.

15:03
Like you said, it's very much a solved problem. Like if you want basic connectivity to the internet, basic connectivity for APs, there's not a lot of differentiation. Now you can lump on all the apps and all the AI and all the hype stuff that's on there. But if you're talking about basic connectivity for something like this, there's a lot of things that can do that job that aren't 25 sacks of gold. This is why I love these conversations because I didn't realize. So design has always been a weakness of mine. Like I got my NA, I got thrown into production.

15:32
We had, I worked in organizations so big, we had rooms of architects, right? They just designed it all and did all the things. And like, here monkey, build the thing, right? I'm like, okay. I didn't realize ISAS was less chatty than OSPF. like,  oh, that, okay. So all IGPs aren't the same. Didn't know that, right? Got it. You hit so many things. I got all these notes, but so  trade-offs, right? And then money and how to spend in other places. So I guess, how does it go? Like I, so there's requirements. What do we have to build guys?

16:00
So they're your requirements. What do we need? What are we doing? And it's business requirements. think that's the other thing that That's you start, right? Like, what are we doing? Because don't, know, network engineers will sit there and argue with each other about, this is the perfect protocol. This is the perfect thing. You can't, if you do it any other way than this, I love the arguments we have. And it can never be any way other than this. But if the business comes to you and says, you know what, we could do it that way, but the business is going to lose a million dollars a year.

16:28
and we're not gonna get the value out of that. So it doesn't matter if it is the most perfect, excellent network engineering,  use of protocols, ever. If it doesn't serve the business and it doesn't impact the bottom line, then it's worthless, even if it's perfect in the world of network engineering. This ties in, we just released, uh well tomorrow, by the time you see this, it'll already be out, but we're releasing a Learn the Business, Your Career episode with Mike Bouchon, Scott Robon, and  this exact conversation. What does the business value

16:57
how are they making money and how does the network serve that? If you don't have those conversations, like I got into the nerd stuff and I love the nerd stuff and I'm like, oh, business boo-hoo. like, to your point, what is important to them? What are they doing? How do they make money? Is the network part of the product or not? Because I have been in organizations where the network is a call center, right? Not a revenue generating thing because it's not part of the product. It's something that Mike or Scott said they were like,  oh, well that kind of makes sense. Business requirements are make money, correct? I know I'm being like facetious but-

17:27
And that's, they're coming to the network or to any- But how do you from make money to like building a network? Like what happens after business requirement? We need to make money. We created software that does blah, blah, blah. And we need to, people to get this, right? And let's just assume for the second that we're not gonna press the magic cloud button and just make it happen there. So we gotta build on-prem for reasons. Make money. I mean, there's gotta be more to business requirements to make money, right? Yeah, so let's use the hotel as an example.  If you've got a thousand-

17:54
brand thousand hotels that you've got to go build a network for. And you want it to be, you know, generally the same flavor. It won't be perfectly the same because you're to have variations in size and things like that. But they're going to come and say, you know what, we want guest Wi-Fi connectivity. We want solid, reliable meeting and event connectivity. We want our guests to have the same experience no matter what, whatever they stay at. Right. Exactly. Because I got here and I remembered because I've stayed at these brand before. Oh, I got to put my room number and my last name and the active portal to get in. Yeah. But it's having that

18:25
experience across every place that's comforting to me. Cause I'm like, oh, I know where I am. I'm at the place where you do the thing and now I'm on. And if they have, if their captive portal works well and it's smooth and you're not, know, cause I mean, everybody's been at a hotel with crappy wifi and crappy internet and you're sitting there fighting for 30 minutes, trying to make the captive portal get you on the network, whatever it is, Like it's one of those things where you don't notice it if it works smoothly and works well, it's just there, but you absolutely notice it if it sucks. business requirements equal make money, but

18:54
But they're That means happy customers, clients. then, OK, what does that look like? How do we make customers happy? Is it experience? Is it performance? Is the performance tied into the experience? Because that's what they're, they don't care about the tech. I call it the light switch analogy, which is something that one of my bosses said to me at one time in the service provider space. Because we do all these insanely complicated things to get the internet from point A to point B. And at the end of the day, nobody really cares. And I know that sounds callous. Nobody cares. It's the light switch analogy. When you turn the lights on,

19:24
and you turn the light switch on, you want the lights to come on. It doesn't matter that you've completed the circuit, that it goes back to the breaker and the But I did all this stuff, Kev, you don't know how hard it Yeah, no, it goes to the breaker and that goes to the transmission lines, that goes to the nuclear power plant. All kinds of really smart people made that light turn on, but nobody cares, right? Nobody cares. Doesn't that hurt? It does a little Nobody cares. mean, it hurts. It hurts. It hits deep. To some degree it does, but that's the thing. If you do a good job in the tech world, people never know it really, because it just works.

19:54
So business requirements, push design. I like what you said. So we went from business requirements to design to then budget and then trade-offs. business is gonna make money. How do we do that? We keep customers happy. How do we do that? We build like to your point, like EasyCat to portal. Maybe if I'm in my room, I wanna like stream Netflix easily. So like, okay, I make sure that stuff can do the whatever.  Is this where the decisions begin with like, okay, what vendor provides what solution that is affordable for me to meet these requirements? So we're running this hotel, you and I.

20:23
The chain comes to us, we're running a consultancy. We're connect the thousand hotels across the country. And these are the things we need. How do you and I, like this is really the meat and potatoes of this topic. How do we choose vendor relationships? Like, oh, okay, so we built with, I'm just gonna use Cisco, just because it's where we started. How do we decide what vendor to go with? Like they're all probably offering what we need. And I guess if we have a relationship with a vendor.

20:49
and we've had a good relationship with them, and then maybe we get discounts. I don't know how all that stuff works, but if you buy more, do you get better pricing?  So is In theory. oh But ultimately, can only, I mean, the thing is... But then you might be working with a partner who's incentivized to work with a particular vendor, right? So now, do I have a choice? Maybe is the partner trying to, or the VAR trying to push me in a direction, because they know they have a better discount on a thing because of a level. Well, this is where it comes back to value. You're going to start looking at all these different solutions to say,

21:17
You've got all these different vendors out there that could go do this. You have Cisco, you've got Juniper, you've got Aruba, you've got Fortinet, you've got whoever. There's a dozen companies that are going to sell firewall switches, APs,  and the whole complement of gear. So then you're looking at it and saying, okay, we are here. If I save X amount of money, am I really saving that money or am I going to burn that up in  now I got to retrain my people, now I may not have all the features that I want,  and I may not have X. So for some companies, the answer is, you know what?

21:47
We may save some money in the short term and we may save some money here, but ultimately for our business and our requirements and maybe we don't have any IT staff, maybe we outsource that and so we've got to make sure that the company that we're outsourcing that IT to, we're using a very well known common mainstream vendor that can easily be supported by third party company.  Those are all the things that go into those kinds of decisions. So for one company, for one hotel chain, they're going to say, you know what?

22:14
I'm going to stay with  Cisco or Aruba or whoever because that's the right vendor. And then you get into things like, I've done a lot of work overseas. You start looking at supply chains and you start looking at the cost. We're looking at this from a North American perspective, a US perspective, a Canadian perspective  generally tend to be the same. You start going into EU, you go into Latin America, you go into Africa, you go into developing markets and areas. You may have a hotel chain that's going to choose something that lands in my world like Microtik or Ubiquiti.

22:43
Because there aren't a lot of choices sometimes when you're talking about exchange rates, supply chain, longevity, and you may say, you know what, this doesn't do everything that I would like it to do, but the cost and the value of doing it with this other vendor is so high that we'll never make any money. So we're prepared to sacrifice.

23:01
these features and this functionality to get something at a price point that still allows us to turn a profit. And so those are the kinds of things that would go into, hey, I'm going to get something that's really expensive, but I know I'm getting value out of it, versus I'm going to get something that's really inexpensive, and maybe it doesn't have everything that I want, but I'm going to get the right value out of it because of where I am in my business, where I am in the world, and what it is that I care about. And those are the kinds of things that I think often get missed.

23:24
in these conversations is we tend to look at it from a North American perspective, for lack of a better word. Typical Americans. whole world is representative of what I experienced, but that's a really good point. And there's a lot of different viewpoints on how to build a network. And you can absolutely build reliable and sustainable networks with vendors that are commodity, lower cost, if you know what you're doing. But then that gets into the people problem. You've got to have people that know how to do that. And are you

23:52
either going to build an organization that has people that you skill up to support that, or are you willing to pay for somebody else to go support it. And so when you get into the vendor selection, like these are the kinds of conversations that you have to have. And then you put it back to the business. The biggest thing that I think people that haven't been through architecture life cycles, operations life cycles, they sometimes take this on themselves. Like, I got to solve this problem. No, you don't. It's not your problem to solve. Your challenge is to distill the choices and the pros and cons.

24:20
of each of these paths down to the business. Then you go pitch to the business and say, here's the different ways we can go. Here are the pros and cons. These are my opinions, but ultimately this is a business choice. And that is the single biggest thing I think anybody that's been in architecture and ops for a long time will tell you is that  in many cases it is not solely upon you as the engineer to like make this choice and take it upon you. It's often a whole group of people, technical and non-technical that you're pitching these to and getting their feedback.

24:49
So there's two things I want to pivot to. One, throw it to choke, right? Yep. First, multi-vendor. Sure. So  I believe that when the supply chain stuff happened with COVID, it kind of pushed people into multi-vendor if they didn't want to be multi-vendor because the lead times were like, you know,  super long over here where we've always gone and like, whoa, we can get these other like a lot quicker.  As we are making decisions, everything you've said so far, we're making decisions to try to solve the business requirements.

25:17
And there are a lot of choices and we could go with one place for reasons,  or we could maybe, I don't want to say cobble together because it makes multi-vendor sound not great, but where my mind goes from that is single vendor, multi-vendor, and then  if multi-vendor becomes more complex, because it kind of can be, right? Like there's different CLIs and syntaxes and how things are done. Then does that lead us to automation,  maybe abstracting the complexity underneath the hood enough that

25:46
if we can treat these three different vendor,  what is it, Broadcom, Silicon, like if it's all the same stuff really, even though it's running different operating systems like NASA's, the first question I guess is why would we go multi-vendor, right? You don't just decide one day like, oh, we should go multi-vendor. Maybe you're mad at vendor A that you've had,  right? Why else would you,  because I think multi-vendor is more complex. Is that fair to say?  It is, but I think the, so I'm gonna use service providers as a good example.

26:14
they use multi-vendor because they need to solve a specific problem that not everybody is going to solve. So a good example. Let's say I started out in a Cisco backbone when I worked for a telco. We had a Cisco MPLS backbone. It went out to all the CEOs in the telco, but  Cisco kind of got out of the last mile game of making like fiber to the home access gear, like DSL, DSLAMs and like that kind of thing. They used to, but they've gotten out of that for a while. think they may be back in it a little bit, but they don't make the radios that go on the towers. Like they make like enterprise wifi and some other stuff, but like

26:42
You go on a cell tower, like it's not a Cisco radio at the top. It's an Ericsson or it's a Motorola or it's a Nokia or it's a whatever. Or, you know, let's say optical transport. It's going to be like Fujitsu or Econops or my Metro Ethernet access is going to be AdTran or Sienna or Calix. So you have all these different vendors that solve these very specific problems and service providers use them because they solve the problem well. They solve the problem at a price point that is able to be deployed at scale because this other vendor doesn't do it.

27:12
So you're basically just picking the gear that is the best for the task and then because we have interoperable standards, then your job as the engineer is, okay, I've got five different vendors here. Or if it's an SP, it's probably more like 30 or 40 vendors. Oh my God. Yeah, I mean, I'm not kidding. Like in some cases, it's- Is that how crazy it gets? Yeah, no, it gets upwards of, I mean, I could probably rattle off. always, people talk about like all the vendors. I can probably rattle off like 30 or 40 vendors that nobody's ever heard of in the SP environment because again, you're solving very, very specific problems.

27:41
So I think in enterprise, I worked in large enterprise for a while as an architect, and there was always a hesitancy to go multi-vendor until people started kind of flirting with the idea of using firewalls that weren't Cisco. And then we started using Fortinet. We started using Palo Alto. We started using these other firewalls. Well, then it's like, okay, well, I'm going to use this vendor for my firewalls, and I'm going to use this vendor for my routing and switching. So then it always goes back to, why are you doing that? Well, they make the best firewall.

28:08
So you have to kind of extend that logic and say, if that's the way that you're going with your firewalls versus your routers and switches, why would you not extend that logic to everything else? If there's another company that makes the best APs and it's so good that it's going to provide that the connectivity is going to be so much better than what you have, and maybe it's even at a better price point, then why wouldn't you go through the work to bring that in? Because at the end of the day, it's VLANs and switching and whatever, right? The AP, there's some integration work and there's some engineering work there.

28:37
But the value you bring to the business, if you're going to make the experience 30 or 40 or whatever percent better to the business for whoever it is that they care about, then that's value that's worth it. If it's like a 1 or 2 % difference, then you come back and say, no, there's no point in going with this other vendor because it's going to be a marginal difference and it brings no value. Well, I guess your skill set of your operators too could be a constraint. If everybody's trained on vendor A and then you're like, oh, we're going to go with B. I can already hear the groans.

29:03
Like, right? there's friction there. Like, oh, God, new operating system, new syntax, new systems, new whatever, right? Like, how does the tack work? So I can hear the operators like, oh, no, not another, because it just feels like another layer of weight on top of you. Like, oh, no. It is. And this is where, if I shift into the enterprise space, and I saw this in the enterprise  and  working for enterprises,  it's very different. And service provider, like, you don't have a choice. Like, you just have to figure it out because  the nature of the stuff that you have to deploy means you're going to be multi-vendor.

29:33
your ops teams just have to figure it out.  Whereas in enterprise, it's kind of a choice. And the problem, I think, is not so much  the engineers or the skill set. It's the attitude of the Fortune 500 towards their people. Because  most of these companies are going to get these people skilled up and they get a team. And then they're going to merge or they're going to fire everybody or then they're going to bring in consultants. And you ride that curve over and over. It's like  every few years you get a team built up  that is pretty good. And then everybody gets fired or laid off or whatever.

30:01
and you go right back down there. And because the organizations, to them, if it makes the bottom line work, they don't care about the technical churn. So you've got to take that into account as well. Are you working for a company where everybody's going to be gone in three to four to five to seven years, and you're going to start all over on that knowledge slope? And then you've got to take that into account too.

30:21
That's the new normal, right? It is. mean, that's the way, you know, that's kind of the way they operate. If you're in one of those, I mean, if you're looking at this as somebody that's got to build this and you say, okay, I could go do this, but I know that all the people that are going to know about it are going to get laid off in year three. Yeah, see that's whole... that's another angle of design. That's whole other thing, right? you've got to think about that's very organizationally specific. It may work for this hotel chain because they do things a certain way.

30:45
But if I'm at this hotel chain and I know that they're gonna lay everybody off in this cycle, then I can't build this network because it's unsustainable. God, all that tribal knowledge that just disappears. Isn't that awful?  It's because documentation sucks to do. I mean, it's important. We all should do it, but it sucks. Oh my God. I just got sad.  But no, there are business cycles that are gonna happen, right? Well,  we're even planning some. I'm planning this episode on  DIY automation.

31:13
And we were having, in the Discord, we were chatting, it might've even been a happy hour, you might've been there. It was a couple automation folks and they were talking about, a DYA, the one guy's like, oh, a DYA is great and everybody should do their own and don't buy into vendor automation. And then the other guy was like, well, what happens when the one guy who built all that leaves more money? Like, oh yeah. And then everybody on the call is like, well, yeah, that continually happens. They get one person in, they build all this stuff, it's awesome. Then that person leaves, no one knows how to maintain any of it. It starts to rot and then it goes away again. It's kind of the cycle of...

31:42
So  it's almost the same thing with the cycles of economics that you're talking about. We have all this stuff when we build it all and it's great, but then it's six years when they lay everybody off.  Now what do we do? Well, and that's why the cloud got so popular because I remember working in enterprise IT right as the cloud was about a decade ago or so, 15 years ago, where the cloud was just rising to popularity. And that was one of the attractive things in the dawn of cloud. Now we're dealing with the technical debt of cloud and maybe we will go on prem for a few things.

32:10
But at the beginning of it, it was, hey, let's abstract that infrastructure that we're going to pay the best and the brightest, the AWSs, the Google Clouds, the Azures, to have the best infrastructure teams, because we know that they're going to keep somebody around that knows how this stuff works and abstract all that complexity. And it worked until everybody found out how much it costs. And then, you know, it works for some people, it doesn't work for others. But, you know, now here we are kind of slowly getting back into

32:37
more of a mix between the two because on-prem data centers aren't going away. They're not what it was 20 years ago in enterprise networking. But with the AI cluster build out stuff, it's insane how much on-prem is now being built again. It's so funny to me the cycles of, like you said, hype earlier. There's hype cycles and then there's expansion and contraction. I worked at a company, we had 200 data centers and then we wanted to get down to three. They were contracting, but then they expand again. We went into cloud and they were like, oh no.

33:06
You can't just throw workloads in the cloud. You have to refactor and oh no, it's way more expensive than we thought. And then there's the, with the repatriation, like this back and forth expansion contraction doing different things. And it's kind of interesting to me that I always want a simple answer and there never is one. No, there never are. I just want simplicity. Simplicity is so hard. why, which vendor should I go with, Kevin? There's no simple answer. Everything we just said for the past 40 minutes is like,

33:30
Well, it depends. depends. Don't say it, it depends. Yeah, no, if you go back to talk to Russ, that'll be the... In fact, one of his... I don't know if it was his saying, but I've heard Russ say it quite a bit is, if you can't find the trade-offs in network engineering, you're not looking hard enough. Russ, I can't wait to have him He's coming on soon. To end this or to wrap it up, I don't know if there's any takeaways for folks. So we started with, we're here.

33:59
We're at a place where vendors pitch to engineers  and  how do you navigate the world of all these different vendors and they specialize in different things and there's trade-offs. I mean, it's really, networking still feels as complex to me today  as it did when I got in 15 years ago.  even if you think of a, not that we're talking about that, but you know, the overlays, like we're just, keep piling on complexity. I don't know why, but we do. And it gets harder and harder. And here we are like,

34:28
Hey, how do you navigate relationships with vendors and how to build networks?  this is an amazing conversation, but I don't think I'm any closer now. We ended with it depends. So that's a sucky answer, but it's the truth, I think. I'll give you my  closing argument, if you will. Good, you wrap it up?  What I always tell people is that  build the right network instead of building the network that you have budget for.  And I'll give you the best example that I can of that. was working with a company a while back where

34:56
They were a Cisco vendor. went to, there was a Cisco data center build and they were trying to solve a specific problem. They needed more leaves, they needed more leaf switches and they didn't have enough budget for it. And so that we had to do all of these stupid network tricks because they'd only bought so many sets of leaves to be able to build this leaf spine data center. And I went back to them and I said, you know, we would be able, if we took a commodity vendor or an open networking vendor or somebody else and built the design that you wanted to build instead of being hung up on the vendor.

35:26
And we built said, if you know that you need, you know, 20 sets of leaves and then, you know, some service leaves, two pairs of service leaves to do firewalls and exit routing, whatever it is. And I'm not a data center guy, so this is a while back. But I played in that world enough that it's the same for any network, whether it's an SP network or an enterprise network. This is just a really easy example because the leaves were in pairs and they didn't have enough. So they had to do all these stupid things because they wanted to stay with that vendor. If they had found a vendor.

35:56
that they could get the right number of leaves, they, pairs of leaves that they needed, that wasn't that vendor. They would have built the right network and they probably would have had a way better data center network, even though it didn't have that shiny vendor badge on it because it was the right design for what they needed versus let's just buy what we can afford and we're gonna make it work. Why did they wanna stick to the one vendor? I guess there's like a warm and fuzzy familiarity. through, was  our engineers know it, it's what we've been using. We tried this other vendor a few years ago and it didn't work out.

36:25
at the risk of bringing up the old trope of nobody ever gets fired for buying Cisco, that's it's still a fake. Yeah. There's so many like, so I've learned in the past few years, different biases, like cognitive biases, like the mental game and like even just us as humans, it's so hard to, like you just said, like, oh, well we tried another render and it didn't work. So that means that none of the other renders will ever work again. And we do that internally without even knowing it. Like  our brains are just like, oh, we went outside of the thing and it was bad. So never do that again. And they'll just.

36:51
run the organization for the next three decades doing that, like must stick with this, it worked once. But you're coming up with when the pandemic hit, like we didn't have a choice. There was a supply chain crunch and those excuses wouldn't work because it was like, okay, well, I don't care if we tried this, we have nothing. Like right now we have vaporware. So let's try this and make this work. And that's what, you know, opened people up, I think, to looking at different vendors. think not that we're out of  the big mainstream vendor, most enterprises still use it, but it did start to get...

37:20
organizations thinking about, there is other stuff out there. We can make the business function on other vendors that we're not traditionally used to using. And maybe in some cases, that's the right choice. And to your point about multi-vendor tying this all back around,  diversification of your supply chain is every bit as important  as  having the right protocols in your network stack. Because if you can't get equipment to build the network, then your perfect design, again, doesn't matter.

37:47
So we were almost out. We talked before the recording and we're like, maybe we can do a half hour episode and we'll do two and we'll bang it out. But you said something, which is now gonna send us for the next 10 to 15 minutes, this is gonna be an hour episode. So you're almost out and we'll do We can do a two-parter. Well, no, so white box, right? Build the right network, white box. And I have like vendor freedom question mark. I think we should talk about white box and then maybe that'll be the end unless you say something else brilliant, but we're gonna have it. So why, I don't really know white box, right?

38:16
I know that you're very comfortable with that. So  what is Whitebox and is it a vendor or is it not a vendor? And  what are some of the advantages? Like you would say that, so that one leaf, that company of like, oh, well, we need more leads, but we can't afford it from vendor, blah, blah, blah. So we can't do it. And your point is like, well, we could do Whitebox. That's what got me thinking of this, right?  What the hell is Whitebox?  Where does the gear come from? What is the network operating system? Is it open source networking? oh

38:41
I know this is like a- of.  So this is where it gets a little bit murky because you've got a little bit of marketing in there and you've got several companies that have done this. But  the explanation that I always use is if you think about IBM mainframe networking in the 1980s and uh some people listening to it wouldn't have been alive back then, that's, you know,  for those of us that were, that's how we did servers back then until we had x86 servers and you bought your hardware and you bought your operating system from the same company, IBM, just like Cisco, just like a vertically integrated vendor where you buy the switch or the router.

39:11
and the operating system from the same person. The server world started to evolve when x86 came around because suddenly we could put Windows or OS 2, which I'm really going to get a dig blast from the past or whatever, right? And then Linux came along and blew it all wide open. And we have all these different operating systems that can run on  x86 or now ARM64 or all these different architectures. And so the server world got very comfortable with taking open components and putting them together in mission critical infrastructure. And it took a few decades to get there.

39:41
But that's all they do. I mean, I remember a day when enterprises said, we will never have anything other than Windows and paid support and all that. And then I got into an org where it was, you know, was hundreds of thousands of Linux servers and they were just downloading, you know, Red Hat or Ubuntu or CentOS or whatever with, you know, no support. And eventually those models got there, but they proved that you can take open components and build critical infrastructure. Whitebox is very much the same thing for network engineering where you have OEM.

40:09
switch manufacturers.  And there's a bunch of different names. Edgecore is a big name in there. EufySpace is a big name in there. And Penguin, Quanta, there's like a, you know, there's a few dozen So they're just stamping out iron switchers, routers. They're building  the rack mount unit with the power and the interface to load the operating system,  usually on top of something like Broadcom Silicon, Marvel Silicon.  I don't know if Nvidia is doing white box anymore. I they kind of came from that world of Mellanox and Cumulus Linux, but I'm not sure they're quite in the white box space anymore. But

40:38
If you buy merchant silicon that you can load your own operating system onto, that's essentially what white box is. And it's not really any different than a server. It's just because it uses an ASIC, there's a little bit of extra layer of complexity because you have to have an OS that can put it into hardware forwarding, whether it's routing, switching, whatever it is you're doing. So when you say your own operating system,

41:00
What's an example of a white box NAS? So a white box NAS is that are pretty common. IP infusion is probably one of the ones that's pretty well known. So you can download it. It's free. You throw it on the box, and you can do routing and switching? You have to pay for it if you want to get the license to run it on a switch. But they have a VM you can download that's like a Cisco CLI. You can put an EVNG or whatever, or GNS3. But it's less expensive than one of the major vendors. Yeah, generally, you'll find. And this is why service providers started deploying it, because they could take and.

41:29
This kind of dovetails into the broadband build out, you know, ever since the pandemic. We've got all these federal funds and all these initiatives for building out broadband, building fiber. And so there's all these regional ISPs that have popped up that are trenching fiber and building out connectivity. And they would never have been able to do that if they were going out and buying $50 million worth of gear from the most expensive vendor out there every time they had one of these projects. They had to have something else that solved.

41:53
the problem that they were trying to solve, because we need BGP, need MPLS, I need IPv6, I need all the things that SPs need, but I need it at a price point that is going to be able to scale with this bucket of money I have. And so that's where White Box really started to get kind of some traction in the last But who builds all those features? it open source people doing it for free? No,  there are a few, actually. There are a couple.  There's DanOS, but there's another one. The name escapes me. We'll put it in the show notes. I'll I don't understand the financial model, so it's less expensive.

42:23
There is support or there isn't? No, there is support. So like  if you go buy a switch, usually the hardware manufacturers are typically not involved with the end customer. They sell the box through a reseller. Whoever makes the operating system that you're putting on to this switch is responsible for interfacing with the customer and interfacing with the hardware manufacturer.

42:44
So that, and that's the way it works in most cases. So if I have an RMA, like a bad something or another,  I'm going to my NAS. You would go to your VAR, you'd go to your VAR and you'd buy a box, you'd buy a license. And then all of your support chain would be through whoever makes the OS. like  like IP Infusion, for example, you would go buy your software license and your hardware through them. And then if you have any issues or you need to open a ticket or whatever, and it ends up being something on the hardware side, they reach out to the hardware people on the backend.

43:12
So it ends up being a little bit like a vertically integrated support experience in that perspective. And there's a couple other  operating systems that are out there. And if you hadn't put me on the spot, I could probably rattle off five of them. Are there VARs that specialize in white boxes? Is its own kind of market, its own thing? It's not so much white box as it is like. uh They're mostly in the carrier space. like the, because carriers don't tend to, they tend to use, uh they don't really call them VARs because they don't tend to use the same services that enterprises use.

43:41
Most carriers have their own teams of people that like are going to build and design this. So they just need to buy the stuff. Whereas a VAR, you're like, hey, I need your, I need your consultants and I need all this. I need all these things from you, which is the whole value added resell. In the ISP world, they like just sell me the stuff and I know what I want to do with it. So it tends to be something like a reseller, but yes, the people that, that sell the same boxes that are going to sell the gear for the towers and the fiber optic stuff. A lot of those same resellers will sell this to carrier world, but.

44:09
That's spilled over into the data center and the enterprise world. There's other companies that are doing this stuff in the data center space because ultimately, the other problem that we haven't even touched on is the as a service licensing. So one of the things that's pretty common in the white box world is that you tend to find perpetual licensing. Oh my God, you're gonna make me sad. The old perpetual licensing. Go ahead. Whereas  in  the, most vendors have gone to as a service, like type licensing.

44:37
So this is something that- Have to watch what I say, Kev.  Yeah. Well, so  I came up at a time when perpetual was what you got. Yeah, that's what everybody did. What I loved about perpetual, which we all love about perpetual is there's one license, it lasts forever, and you never think about it again. And then later,  it was the as a service model in a different job I had, and you had to think about your licenses  every day.  Yeah. Why would I say that?

45:04
We might've had a virtual thing running that was tied to a license that we didn't realize expired and the 10 gig throughput license that we had expired and it went down to the default one and a half megs per second and it dropped all the packets on the floor. And one of the biggest financial companies in the world were very upset with us because they couldn't do their monthly thing that they do with files. And it turned out it was because, cause people tell you like, oh, well, know, licenses aren't tied to, it won't fall over and break, right? Like it'll keep working. Well.

45:28
Unless you have a bandwidth license. Well, some do. But even the ones that'll tell you, like they told, like, oh, that can't break anything. Like, well, if we're pushing 10 gigs and it goes to one and a half megs, well, you had log messages, you should have checked. Well, I'm sorry, we have a bajillion log messages a minute. Like we weren't looking, right?  Now they do.  But we, so yeah, the as a service thing just gets me going because I a couple outages because of expired as a service licenses that we were told.

45:53
can't break production and they did. Yeah, I mean, there's some legit reasons. It's not to like knock all as a service because there are some legit reasons to use it. There's some legit reasons that exists, but... Again, I'm being careful. It was just a bad experience at a place like... It's... I work at a vendor and... I like the whole perpetual idea as a network designer because it's a lot easier to deal with in the design. simple. And deal with the value of the whole network. But I think the point that I'm making is that is something that was done a long time ago and you generally don't find much anymore.

46:22
But it is still pretty much a thing in the white box space.  There's more of that. So people are drawn to that, especially the reason it blew up in the carrier space is when you're an enterprise and you're funded by revenue and you're selling widgets, right? You've got to make your quarterly numbers, you're publicly traded, whatever. Well,  you are funded in a very specific way. You're very much based on revenue.  For carriers that are being funded through grants or these other large cash infusions of private equity,  they get one bucket of money.

46:49
because we  had the pandemic, we realized there was the digital divide, everybody's got to be online. So a lot of them get this one time,  one big bucket of money, and then they got to go spend it. So they've got to say, hey, I can't be locked into just burning all this cash every month. I need a network that I can pay for once relatively, like  with minimal recurring support costs. And that's the world of the service provider. But I think  some enterprises and data centers would like to get into that too.

47:17
Unchain themselves from the as-a-service model. They're gonna be wrong for private equity and shareholders and anybody on the business side as a service is awesome Revenue  revenue. Yeah, but for the people that have to consume it. It's a challenge for budgets It's back to money. Isn't it funny?  It's always it is even in that learn the business,  know episode that we're releasing it It's I'm in there complaining about oh, I can't stand these, you know  all hands meetings or all they talk about is financials and they're like, well, that's

47:44
what those meetings are. I'm like, well, why are they talking to me? I'm not on the board. I'm not the street. But it's really all about money. And if you don't connect to the business, you're kind of not in the loop. But to tie this all back together into one of the points you made earlier about logistics and supply chain, that's the other reason that White Box has gotten really popular. And like I said, it's huge in the service provider space now. It's starting to make inroads into other verticals.

48:12
But the supply chain is way more diversified because think about it you're building a server. Go back to the server analogy. Well, I can go  download Linux or go pay for a commercial version of Linux from whoever and then I can go to whoever to go get my box, right? It doesn't matter who I get my box from as long as it meets my specs. The same is true in the networking world. If you've got the same series of Broadcom chip or Marvel chip or whoever in your box, you can go source that from whoever. And when you have some logistical supply chain issues,

48:40
that becomes really attractive to insulate the business from, oh my God, vendor A has been the most amazing vendor for 30 years and now all of a sudden, they're not shipping anything for two years and everything is cut off.  was helpful. Two more hours?  Thank you, man. uh This was incredible. I learned a lot  and that's kind of the point, right? We pull people along. I like this in-person thing. I'm going to come to... Where do you live?  I'm way down south,  I'm in Mississippi. I'm way down south. Am I allowed down there? You are?

49:09
You get a passport to come south of the Mason-Dixon. We'll get you a visa. I know Kevin. For all things Art of Network Engineering, you can check us out on our Linktree, Linktree forward slash Art of Net Eng. Check out the Discord server. It's all about the journey. There is new merch up. I am adding new merch by the week. Now we have mugs and hoodies and shirts and all kinds of stuff that nobody's buying, but I'm updating the merch so it's there. ah But as always, thanks so much for watching. We'll catch you next time on the Art of Network Engineering podcast.  Hey, folks.

49:36
If you like what you heard today, please subscribe to our podcast and your favorite pod catcher.  You can find us on socials at Art of NetEng, and you can visit linktree.com slash art of net eng  for links to all of our content, including the A1 merch store and our virtual community on Discord called It's All About the Journey.  You can see our pretty faces on our YouTube channel named the Art of Network Engineering.  That's youtube.com forward slash art of net eng.  Thanks for listening.