this is the art of network engineering podcast in this podcast we'll explore tools technologies and talented people we aim to bring new information that will expand your skill sets and toolbox and share the stories of fellow network engineers welcome to the art of network engineering i am aj murray at no blinky blinky and joining me tonight lexi at track it pacer lexi how are you doing i'm doing fantastic aj i've had a great day some stimulating conversation um i've been feasting on the knowledge base of twitter all day long i thought that's a question about ethernet and oh my god it's been amazing isn't that cool you can like put something out like that and get like not only responses period but you get good responses yes that's exactly what i was gonna say i can't it feels almost like sacrilegious that i can just like scream into the void about like the ethernet standard and like the people who wrote the ethernet standard will respond to my fucking tweets it's bonkers it feels like i shouldn't be able to like summon all the ccies in the world who are like on twitter just by like sending out a stupid tweet which by the way i didn't even word properly um but i sure did that and it's it's been great like i'm learning so much i can't even um i can't even believe so i i've had a great day that's awesome that's awesome i had a similar experience in the discord this past week so i am doing the uh data center design course on cbt nuggets and jeff kish teaches that uh and i watch a nugget i had a question and i can go into the discord and i can like talk directly to jeff kish about like something that he was talking about and i'm like but wait jeff what does this mean that's awesome and then like you know we get into it a little bit just like a discussion on the topic and i'm just like you know it's not lost on me that i can watch you in a video have a question come here and talk to you like i feel you know like this this is really cool so that was fun awesome and then there's andy at andy laptop andy how you doing i'm awesome i spent way too much time fighting in linux today i want to go on record as saying i don't like linux i don't know if anybody likes linux people like linux people who are like software people like i love linux like how can you like linux anyway uh but i learned some cool stuff i'm trying to install some software on a server and there's no virtualization on the server so i have to install it in the shell but i've never done that before yeah and i'd be curious what your thoughts are maybe some other time but like i'm fighting for like we need some virtualization here this is silly why we're just installing it right on the like i i don't i don't know why we're doing that i i guess it depends on what you're going to do with it it there there's certainly reasons not to do virtualization um really oh yeah there's just some things that need direct direct access so maybe sometime in the future when we talk about data center design maybe we could talk about you know instances where you would not want to virtualize in the data center that would be we'll have to write that one down i'm going to write that one down all right but yeah i i i had to edit what is it called vi i don't even know i'm like linux yeah yeah they were they were showing me vi and you go in and you edit the stuff and extra delete something and i to insert and i i edited my first is it a linux file i don't even know i don't know well that's a long answer i'm sorry i spent a lot of time in linux and i'm still recovering from it i'm curious what are you doing like what what you doing what you up to in linux world i am trying to install nms open source nms software so that i can play around with some telemetry okay and i went in the discord i got some recommendations and you know again every every time in prague or at home i've messed with um installing software on a server there's a type one hypervisor running on bare metal i get you know whatever ova or i forget what it's called but you get the image you throw it in there and you you know you sign your course in your ram and you're done i'm i'm just in linux hell of like oh you know etsy this and go in there you got to do that and enable this service and then you know you need your sql database like oh my god i could just but i'm learning which is great yeah you know um but i'm really just trying to get some nms software running on a linux box which flavor playing with uh centos sent us what you mean well that was a follow-up question yes so so you're doing centos and what are you doing for the nms uh zabbix okay yeah yeah yeah zx yeah yeah but i've heard that it's tough to set up compared to other ones like network pistol arrow um just telling me that like oh this other one's a lot easier but you know go ahead man good luck i'm curious as to why you guys aren't virtualizing that yeah so we're having good discussions right like i'm on this team my buddy chris my boss were you know they're in there they're helping me we're banging it out and then we're having those discussions because i want to virtualize it because i'm more comfortable in that environment oh sure but sometimes it's good to be uncomfortable and i'm being forced in the linux in linux shell learning linux that people been telling me for years i need to learn right so even if even if you were virtualizing it like i think that zabex has an appliance like a already pre-formatted virtual machine that you could put in there but you could also just spin up a raw virtual machine and install linux and then install zx on top and do all this it just seems like so much lifting to list install software that's all you know like it just seems like so much and maybe if i was a linux guru it wouldn't be a big deal but i'm learning and and that's fine right like all these starts we take and like when you're learning it's difficult it stinks you complain about it but then you learn something and you're like hey look now i know this thing so yeah very long that might be my longest answer to how you do it andy on the show so i apologize to the listeners that's what i've been doing yeah but it was a good day i learned a bunch of stuff and uh i really love the people i'm working with so that's nice good yeah that's awesome yeah i'm good i'm real good um work's going well a little bit of a lull this week which uh has allowed me to do a little bit more studying than usual which is fine um i i'm just waiting on other people right like a new project started up we're waiting for the customer to send us over some configs i am also waiting on our staging team so uh at a my previous var or partner that i worked at i staged the gear i bootstrapped the switches and installed the os uh but now here at the new company that i work for um if you order something with us unless you're adamant about having it shipped directly to you we will have it shipped to a warehouse we have several warehouses around the country and we have a team of staging engineers that will rack and stack the gear per the cable diagrams that i send it they will upgrade to whatever flavor of os that you know myself and the customer decide we're going to deploy and then i can provision the gear remotely using an out-of-band management system and do all sorts of testing and make sure that it is completely good to go before we ship it out to the customer which i just think is the coolest thing in the world that is really cool so they'll install the optics for you right and you can get the code upgraded and yeah that's nice i mean that like that's all stuff that i used to have to do and now i don't have to do it so like literally now i can just worry about being an engineer you used to do like the physical stuff oh yeah like i i would have to like plan you know if i was if if i was gonna go on site i had to like give myself an extra day because i had to do the rack and stack and i had to do the software upgrades and all that stuff but now i don't have to do that now we how do you like it not having to do that anymore um it's different i won't say that i like to do like software upgrades because some of that stuff is boring as heck but um i mean like right now we're working on an aci fabric project and i want to do the rack and stack like i want to play with the gear but you know i'm not going to go all the way down to the warehouse and and do that it's not like it's right next door that's for sure dang interesting switcheroo there yeah yeah i mean some gear like i if it's gear that i haven't touched or play with before like yeah yeah i want to play with it i want to rack it i want to put the line cards in i want to do all that stuff but you know if it's just like whatever vanilla switches don't really care you just you just reminded me when i used to work in the city in an office with a bunch of people in there uh asr 1002 x's were our bread and butter in the way end so i i used to have them on my desk and man when you plug those babies in it's like a jet engine taken oh yeah the whole office would stand up and look over the dividers like what is happening like sorry i gotta you know upgrade the code and get some base config before i ship it but it was so loud like you do not turn them on in an office environment with people trying to work no no no i mean i didn't maybe that's why they sent me home to be remote they have all the routers shipped to your house whatever works right right and now it's time for some wins winning in our discord channel this week is adam pawlowski starting a new role as a network engineer congratulations yeah e-l-i-g-r-p passed the comptia project plus hmm i like that here's your name yeah yeah right here's a good one john capiobanco is starting a new position at cisco as a developer advocate congratulations john and they're yeah daniel dibb passed his aws advanced networking specialty congratulations nice steel network pass to cisco devnet associate nick fournette passed the comptia a plus nice i like this username king das so this this is an interesting story he caught covid had to go on a two-week quarantine and use that time to study for the jncia junos and passed it i just had his username in my head you remember run dmc i am the king of das there is none success or maybe that's i think it was anyway sorry that's great i love it uh luke bibby certified terraform associate jonathan switch employee of the year at their business uh or at their their work congratulations jfizzle passed the jncia and uh they posted it in october in the general channel and it turns out his wife was listening to the podcast to hear his win get announced uh and so he posted it this week and so we just want to say congratulations jfizzle and uh we hope your wife hears this four shizzle dalton b pass the ccna congratulations dalton uh hemo 200 past the fortinet nse one and two congratulations uh error disabled past the aviatrix professional certification man he has just been knocking those aviatrix ones out like left and right i feel like it wasn't that long ago i was just announcing he like the associate level one and now he's already doing the professional killing it tim mcconaughey starts a new position at aviatrix this week congratulations and this is going to be my favorite win maybe of all time uh manny accepted a new position as a network engineer and he'll be working at insights global with mike v2 so very cool congratulations it's so cool to see that come out of the community so new patreons uh joining us this week chandler bassett and peter evans thank you so much for your support of what we do here at the art of network engineering uh there are so many ways that you can show support of the show you can like rate review our podcast wherever you find it subscribe to us on youtube smash that bell icon you get notified of all our future content uh like the videos you're watching and comment all that stuff is really good mojo for uh the the social media platforms there so also make sure you're following us on twitter you can find links to all of our socials at the bottom of the show notes or the show description uh make sure you check out our merch store at artofnetenge.com forward slash store every bit of profit we get from that store we put back into our community uh and of course the patreon program um cisco it blog awards voting is still open for another few days you can vote for us and your favorite network engineer content creators until the 18th of february so make sure you hit the link in the description in the show notes and go vote for your favorite ones and i'm going to vote for you and then don't vote for me for the podcast yeah vote for the podcast and i'll vote for you what's great is that you can you can vote for more than one you can vote for like five and then you can rate like one two three four five like what do you think yeah all right so i'll vote for the podcast i'll vote for you and i'll vote for me cool there we go i thought it was a one thing all right no no not this time uh and and uh this is the last time i'm gonna ask on the listener survey uh we're still looking for responses in the listener survey we've gotten almost 100 we'd love a few more so if you have some time please take a few minutes and complete our survey all that information we share with potential future sponsors and it really helps us guide where the show is going to go in the future and i am so excited to announce that we are going to do our very first art of network engineering live event coming up in april it will be april 9th from 4 to 7 at high wire brewing in asheville north carolina we're gonna put a link in the show notes you can check out go ahead andy screamer we'll put a link in the show notes and in the description if you're watching this on youtube and you can get some general information on the event um and in early march we will have a registration page we'll be able to um you know provide tickets it's absolutely free not a paid event but it is going to be a ticketed event because there is a limited number of people we can have i think it's like 50 or 60 something like that but uh and all thanks to our sponsor opengear for their support and helping us put together that event thank you so much what are we doing there it's going to be a show yeah it's gonna be a live show we're gonna meet and greet with some of our favorite fans uh we're gonna record a show so everybody there can be a part of that a little q a interactions uh it's it's gonna be an awful lot of fun yeah maybe we'll give something away maybe it'll be some merch absolutely i know yeah i'm really excited it became really yesterday when i booked my flight and we're on the same planet i'm really excited about that we're seat buddies back and forth to and from so we hope you can join us at our first live event coming up here in april now back to the show well on that note uh tonight we are going to talk about data center design we are continuing the series the design series we've already talked a little bit about enterprise design in a previous episode and i was hoping to talk about you know data center design and then we can contrast a little bit between like you know what are we doing in enterprise versus the data center um and i don't think we did this on purpose because when we planned the the topic schedule i hadn't yet decided to embark on my data center design journey but i have been recently uh studying for it so i i feel particularly primed to discuss it but not not like super tonight well you're like our resident design guy right like you have you've studied design yeah i've started to i've studied network design for a while like i i have worked in some pretty nasty networks and and they are nasty because there was no design to it there was no planning it was just oh we need more ports let's put a switch over here or you know whatever and it was just it was dirty like there was so much latency in the building that you could be on vpn and have a better connection to the servers like that's how bad it was like i guess if you don't know any better right like you could just keep daisy chaining switches forever and that's what i've seen people do yeah but then there's a i forget from ccna but there's like a circumference you know there's a hop of like you shouldn't go past a certain amount because then like what doesn't work does it mess up spanning tree i forget that there's like a diameter diameter that's what i was trying to think of like you you because then you'll run into problems i guess right the diameter should be equal on both sides like if you're at the core and you're gonna daisy chain switches if you have to they should be the same it should be equidistant but if it's lopsided it's just the whole thing's gonna be weird so what should you do home run each switch back to back where were you running them to where where's back to either like if you're doing a collapse core model you're probably going to home run all the switches back to a core or like an mdf that has a distribution block in it right so so before we get too deep should we like yeah yeah should we should we get so we're in the weeds already right right this is like so this is data center design and i was even thinking today walking around the house thinking about tonight like we should probably define a data center right because i don't even know i've managed data centers i've worked in data centers and i'm not even sure i have a great definition when when family asks me like what i do or like where i work they'll be like oh it's you know it's a big room full of fancy computers well what do the computers do well you know when you go on facebook or instagram that those things live on these fancy like you know you try to dumb it down i ask you those questions andy no one ever cares enough to ask my dad who is listening he's listened to every episode he's been trying to figure out what i do for a living yeah he's been trying to figure out what i do for a living all this time because like you know if you don't work in tech it's magic right like it's just so nice that he wants to find out because most people i talk to unless they're in tech in which they don't ask anyway because they already know right like what it is most people don't no one really asks me yeah my dad's the man that ended up really sad sounding i did not have to be like that and the show took a turn no one loves me no one cares well honestly i mean you're right most people don't ask me my my wife doesn't want to know doesn't care just make the chromecast work you know what i mean and you're right like most people don't yeah most people don't want to know like chromecast yeah but so so how would you decide i mean let's define a data center right like why do data centers exist they say everything in tech solved a problem so like have data centers always been around since network started i don't think so i don't know if they've been around oh well i mean mainframe days the mainframes were probably yeah they probably have been right because you had to have a place when computers were first start starting out like to be massive in general yeah they were so big you had to have like a dedicated friend if you wanted like a four megabyte disc you had to like move it in on a pallet so so the i believe i could be wrong i believe the first computer was at the university of penn in philly was called the eniac i think and it was like a couple city blocks long so i almost wonder if that was the first official data center like it's a building that houses computers stuff there's cooling there's power i don't know have we just have we defined data center yet i think it's i think it's in the name right like there's data there right and you store all of your applications and data for for business or for an organization now a lot of that particularly with cloud the data centers are becoming decentralized right like the data centers used to be the main focal point everything would connect through the data center and then out to the internet you know from there or the internet connection would come into the data center and then you would break it out to all your locations well with sd-wan with cloud like it's it's becoming very decentralized but if your organization still runs applications on-premises or has their own homegrown applications or you know they're definitely going to have data like customer data and stuff like that not to say that you can't run it in the cloud but data centers still exist there's still a thing and and that's what happens in the data center that's where all your applications are that's where all your data lives all your compute to operate all of those things so that's loosely when you can sit at home on your computer and go somewhere on the internet or if you're at a business or an office or a bank or whatever like when you're on a computing device going somewhere that and for you know that request is going to wind up probably a server in a data center somewhere even the cloud it's just somebody else's data center yeah somebody else that's where pictures live in a data center and i have a i have a cute little data center story when i got my first job like when i got my fintech job the guys walking me through the data center down in the city in philly and i'd never been in a data center and my jaw was on the floor and i'm like oh my god this is amazing and i turned to the guy giving me the tour and i'm like who's responsible for all this stuff man and he he looks at me with disgust and he takes a beat he's like i think it's you pal that was my first that was my first day on the job i'm like oh okay because i just never been in one and i was amazed like you know you have a home lab with a couple routers and switches or you know gns or whatever man the first time you walk into data center it's pretty intense i feel like i'm in the future it's very loud it's just like really loud that's the thing that i when my first time like physically in a data center i wasn't surprised by really anything except the noise because like tv and movies and stuff like you eventually see a data center i feel like and if you're like i don't know i was looking at getting into tech prior to that obviously so i like had been exposed to it so i just you know you walk in it's like all right this is what it is it's very loud it's all those fans right trying to call these things big boys well and so like that's one of the things that i'm enjoying about the data center design because we're talking more more than just networking compute storage power cooling all that stuff oh like um facilities design is that part of it that's really cool because i know literally nothing about that other than like boxes need to be cooler all the time right that's it that's all i know so much can go wrong right power draw and like hot cold aisles and things like that so i can remember being on projects where there wasn't enough power in the data center to install the stuff we wanted to so like you would think you know wow it's big fancy data center and you know the people who manage that they're like we so you know then there was a whole other project of like well we need to add capacity for power like how could you run out of power you know what i mean and then somebody on our right and then somebody that we know um i think they had like a cooling issue not too long ago and and that can be if you have an hvac problem oh my god i think things would go south pretty quick i mean i guess the worst thing is if your power goes out right well then you're dead and hopefully you have a backup like a diesel backup but i guess it's a slower interesting problem when everything starts to overheat at a previous job we had a couple data centers that like had some weird design that was not okay with the hot cold yeah i don't really know how it exactly is supposed to go but you're supposed to be what like pushing hot air out a certain direction that's consistent and then yeah like pulling the cold air in from the other when you order the gear lex you have to like i think configure that right like air direction ajax yeah like you have to have it blowing the wrong way physically you can be sucking hot air into your gear i guess if you don't configure your fans right right yeah and there's a lot of data center gear that has swappable fans like so you can swap the fans out and put the appropriate fans in that will allow it to push or pull the air into the gear so if you're going to have it mounted at the back of the rack you want it to pull in from the rear of the device and it exhausts out the front so modularity huh you can redundancy modularity let's get into it i this just reminds me because i again i have like zero data center like experience i did most of my stuff remotely i haven't worked with hardware much but like you know i just built my first like pc not that long ago and like then i had to replace the fans after that it's like i had to actually you know i added a few fans and had to like sort of design it in such a way that it like brings in as much air as possible and we're yeah so um optimizing that with just building my pc sort of give gave me a little bit of like insight into i hope you have positive pressure to keep dust out right exactly that's what i was looking for that's the phrase that i was trying to remember yeah i tried to like i looked something up online to do it i wasn't like figuring this out on my own but yeah there's like an ideal way to do it so you're not bringing dust in and uh yeah on like a massive scale you are designing like at least how fans push and pull air yeah right so what what are you studying now like you are is this a ccde like what are you exactly studying design um is it a certain this is so we we had talked in a previous episode i i have to renew my ccnp in the next 18 months it's it'll expire in in july of next year so there are many different ways that a person can renew their ccnp i am choosing to do that by getting a second ccnp and so i'm going to go for my ccmp data center so i'll have enterprises i'm going to we'll see if i can get close to don't hate on it that's a new thing right like you didn't used to be able to do that yeah you could you suffered three more examples ce credits to renew your ccie previously but now you can even do it to renew your ccna right okay or your np of course so so your np is in what my current np is an enterprise and i did that by doing the encore exam the enterprise core and then the enterprise design exam and salad right yeah the end salad and and so uh in keeping with that theme because the ccde is a goal of mine eventually um which is the ccie of design right yeah um so in keeping with that theme i decided you know when i go do the ccmp data center i'm going to do the design specialist so i have you know that consistency that design background because i appreciate your design yeah and i'll be honest right trust tree here a moment like i've i've been in organizations that like architects design things and then i either built them and or supported them so design has always been a weakness of mine in my relatively short you know networking career and even recently i've been learning a lot more about design because i need to in my new role but i i i feel like i've kind of done myself a disservice because i just like well the architects do that and i don't need to and but there's i mean here i am right like nine years into my tech career and i'm still on the show like i don't understand design aj tell me all about it so you know i i'm happy now that i'm starting to fill in my gaps a little bit so so where where can we start i mean did we go old school like the different designs like the old what was it called like the three-tier model the three-turn model yep there there's the three tier in the two-tier like so so you'll have your core at the top of the tier you'll have your distribution in the middle of the tier and then you'll have your access at the bottom of the tier um now what did each of those do i've heard the core is the glue right it connects everything to everything uh the core connects you know one building to another if you're in a campus right and then you'll have a distribution block probably within each building and then access layer switches you know throughout so all the access layer switches are wired to the distribution and that's just aggregating all that access right right and then all the distribution distribution and the the core's sole role is to just move traffic as quickly as possible and so with that in mind between what and what between building the building like you can have a core in just one building right right so the the data center can hang off the core buildings can hang off of the core your wan will hang off of the core in its own block you'll have a bunch of random stuff going on but it connects to the core so it can get that traffic to the rest of your your greater networks right so real quick so all right so customer client comes into the way in from however they do it mpls internet whatever but they they come in they hit a router they they hit the data center outright so like i don't know if access like so i'm thinking when i was in data center the access was basically like all the wan routers that provided services like and people oh yeah sure like it's not enterprise access switches and closets for like pcs the access is actually edge routers right yeah yeah but what i'm thinking is the customer comes in they hit that edge router which we're calling the access it'll hit the distribution because there's probably routing protocols to tell it how to get to that destination ip it's getting to it hits the distribution distribution sends it there's like firewalls in there somewhere right i could never see the firewalls in my job so they're like these magical like i don't you'll have to tell me where the firewall is on all this but it comes through distribution hits the core goes to a compartment where the application lives grabs the kitty cat picture and then the traffic goes back out with the kitty cat picture right and then back out to the car so that's like the that's it over simplified flow right here comes in hits all those layers and then leaves yeah okay and i've been a service provider that had basically that structure going on it's just that it was at a much bigger scale so you know customers servers would be in you know for example one data center in like california and they would want to grab something from you know access their you know servers in like india right and so you know traffic would start at that access layer go up to the distribution that's still in that same data center where they start the traffic starts out and then hits the core and the course switches it across the world right and that's as fast as possible mpls or whatever you want to do and then gets that data center distribution layer in india and then hits the access layer again um that was really easy for me to learn it that way simply because i don't know on a bigger scale it's easier whenever i was studying for ccna like they do go through that three-tiered model and the collapse core too as well and they talk about it in terms like you guys did like with building to building um and that was a little nebulous for me for some reason but once i started working at an enterprise service provider that was like global scale it was easier i guess because geographical regions just kind of make more sense or something i don't know yeah yeah so that three-tier model aj does that was that built for when eighty percent of traffic was north south like what what's that whole like north south east west thing because we'll get into that with design right right traditionally before the clients came in northbound like we just talked about came in you know and then went back out so the traffic went north south meaning you know from outside came in and then went back out that way 80 go ahead yeah in an enterprise network your clients typically aren't talking to each other so there's not a lot of east-west traffic right like you're you're usually the client wants to go out to the data center or out to the internet to get something there's generally speaking not a lot of like communication amongst pcs you know that sure there could be some peer-to-peer stuff but by and large and like you say any you know 80 of the traffic is going to be northbound out of the network and 20 would be east-west and i just learned this and i've been doing this a while so that's what i wanted to to point that out so east-west would be things in the data center talking to each other right different services different servers different servers application servers talking to database servers things of that nature there's a whole lot more east-west traffic in the data center but it wasn't always that way right like now i remember reading like 80 was north south and 20 was east west and i think that's when we did the three correct me if i'm wrong because you're the data you know you're the design expert but as i've been studying lately it shifted from 80 percent north south 20 east-west and now it's inverted now 80 of traffic is east-west and data centers 20 is north-south which i thought with because all the micro services or whatever they call it in the data center like you said servers talking to each other doing things and i thought that that drove the whole spine leaf architecture right because now you need faster east-west traffic low latency you know you can't have servers waiting to talk to other servers behind spine leaf maybe yeah maybe we are jumping ahead i just i always thought that the idea behind spine leaf was like redundancy redundancy redundancy something goes down it's immediately reachable another link redundancy and scale and and speed and low latency right yeah like in spine leaf can't you get from everywhere to everywhere and like what is it like one or two one one hop to anywhere which server to server east west you need low latency fast right like it's got to be fast communication you don't want the traffic spinning around the data center trying to find where to go no then applications and databases databases do not like latency okay like what happens if a database hits latency it takes a dump uh i mean there's just so much like reading and writing and stuff going on like you you wouldn't want lazy when it comes to databases and all that being unpleasant experience for sure um so boy we've hopped around an awful lot here so where do i want to reset back to well we were a three-tier three-tier well i think it's all relevant it's all relevant like oh absolutely existed because of north south being most of the traffic i thought right but there's been a shift to that i mean we can go back to three tier if we want and break that down a little bit yeah so with the with the three tier the core was get data to its destination as fast as possible so as a result you're not going to be doing any sort of like acls or filtering or anything like that because that's just going to slow the traffic down like that's that's processing power that the you know router has to use to check and validate is the traffic going you know should it be going that way uh rather than just getting it where it needs to go towards its destination you'll tend to see more acls in the distribution layer because that's where your layer 3 typically lives not to say that you can't have routed access layer but traditionally speaking your layer 3 is at the distribution block so all of the gateways for all of your vlans will typically live in the distribution block and your svis for vlans are sitting in the distribution layer right their gateway to get in and out yeah most of the services are at distribution right like quality service acls isn't that where all that stuff lives well quality service has to exist all the way down to the access layer oh everywhere yeah okay yeah it has to be consistent access layer you do have some security things too and the access layer depending on but it's it's usually layer two security so like you know um ice um what's the word i'm trying to think of here uh yeah okay yep so so when ice is going to categorize the device as you know this thing versus that thing or whatever it it'll use some sort of layer to identification i got a firewall question can they be anywhere because they were transparent to me in my role i couldn't see them i didn't know where they were and if traffic wasn't getting somewhere we had to get them on because i mean i couldn't see it i think typically you'll see them at the edge but if you have something within your network that you need to secure there's no reason not to have a firewall it like more inside the network but there's like logical places like like we're talking about design and where your core distribution and access would be and what services live and what like i guess there's similar direction you would take for like where to put your firewalls and what to filter where and right is that okay right yeah usually you would have it like on an edge or something like that like um you know that's your usually like one of your egress points into the network like if you're bgp peering you might have a bgp router on the edge edge and then after your router you'll have some firewalls most of the organizations that i've worked for have been very small so we didn't do any bgp pairings so our firewall lived right on the edge and we just got like a static address from the isp so your firewall always your edge device yeah okay yeah and i i guess that's one of the reasons why i appreciate design because i work for such small organizations where i didn't have an architect so i had to be the architect and the engineer gives you some i don't know experience there yeah so much good experience i don't know when you want to talk about it but it might be helpful to talk about some poor design some you know like stuff you've seen and the problems it's caused like what happens when you don't follow design best practice right and there's there's certainly reasons not to i mean one of the big things with design is like knowing all the tools that are available to you and then being able to match the technology for the business case right like what what are we going to do that's going to support the business and what the business is trying to most intimidating part of it to me when thinking about like i have to design you know this architect like what there's so many technologies and so many products out there i don't know how anyone even starts it seems like an architect would already sort of have to start by using things that they already have experience with otherwise you know if you don't know it and you're just like let's try this then you're gambling quite a bit is that is that in any way accurate to assume oh yeah i mean you can do all the research you want and listen to all the pitches in the world but there's every chance that a product's not going to do what they say it's going to do right so that's why i'm amazed at design like you said you have someone come to you with the problem like the business right like we have to support the business right it's like what's your business what are you doing and like and i guess there's a litany of questions that go along with that right like well how many people are coming in how much bandwidth do we need how much like you know do we need redundancy how many servers like can you can you give an example of like when you say we have to support the business and build something that'll work like what what's the business i mean are these all software companies right no so i work for a manufacturing company it was a global manufacturing company of pumps and motors and these motors were like the size of my pickup they weighed tons and they could move enough water um in an olympic sized swimming pool and empty it in like three seconds i mean you're talking about like 600 000 gallons moved in a matter of seconds so are these for like power plants or something like why are you doing all that industrial um you know like these things would be at sea world or uh coca-cola or okay you know plenty of different other applications for them um oil rigs okay you know either like pumping oil up from the bottom of the ocean or managing the ballast water on an oil rig so this is a manufacturing place that makes those pumps makes them and sells them yep so what do they need with technology aj what are their data center needs so when i first started there this is a great example when i first started there they had a 15 meg internet connection now granted like at the time that was probably more than enough for what they were doing they had less than 100 employees they had all of their applications were in-house and if they were going out to the internet it was probably to screw off or to do some market research and that was absolutely their applications are we talking like outlook like is this you know email email servers was hosted on servers inside the building they were using windows small business server if you've ever heard of that it was like active directory sharepoint and exchange all in the same box so this isn't like proprietary software they built for their customers to order pumps this is just like running an office right email internet and then they had like a file server they had application servers everything was bare metal they didn't they weren't doing virtualization um like the old school way right you had a server for everything yeah exactly exactly yeah but then you know over time you know all that hard work got really old really fast none of it was supported anymore it was dying application server hard drives were filling up the internet connection was taking a hit as people were starting to um shadow i.t they were doing stuff not telling the it department like they were like setting up dropbox accounts and streaming music and all this other stuff right so the user experience started getting clue g right like right you can't connect to exchange when everybody gets there in the morning it's overwhelmed or the internet's so slow because everybody's streaming and then the network itself was glued together by different manufacturers of like off-brand uh retail switches right like netgear unmanaged switches everywhere yeah yeah that we're talking about and then like they there was no thought put into a network design like it was a small building and there was no reason for that stuff not to be home run back to the core back to the server room and it was just you know you have like the server room and then another switch and then that would hang off another switch would hang off of there and another switch would hang off of that one and there was no spanning tree there was one big flat network so did they have a core switch when you say the core or you're just saying where are they where the servers were the servers were that was another yeah so how do you how do you go from like that you by you i mean like you the person responsible for the network how do you take that and then like move it in a beneficial direction right like how do you you have a good design strategy but the strategy that's been employed so far has been terrible and you don't want to take the entire network down it exists already how do you go how do you like reconcile all of that you're gonna have to take that network down let's take it down you're gonna have to swap some hardware well because they don't have a good design so there's no redundancy right so you can't take services down on the a side yeah i guess what i'm asking you really is like is it hack and slash like just just burn it all down and start over more or less or or is it better to like sort of slowly integrate like better practices until you've retrofitted the entire thing it was a lot of it was a lot of retrofit so okay we we started by moving the server room we selected a question why did you get pulled in initially did they call and say oh my god nothing works this is awful help us oh i got hired there to be a network administrator and then i walked in and i kind of did like i took a consultative approach right like i i interviewed a lot of the managers and i asked them like what are your pain points with it and i looked through everything and was like okay this is a fucking mess so how are we gonna fix this were there and so there were no network people there they said we're gonna hire a network there were some people there that did some very i.t like things but they were not i.t people okay and certainly certainly with an infrastructure background they were more like application folks than infrastructure so i was the first infrastructure hire i guess because of their growth and because the user experience was degradating degrading oh my god i just make up a word so they brought you in to say please help us we need to do things here yeah okay yeah did that scare you like walking into a nightmare of a network i was excited i was very excited because because like how often it wasn't a startup right like this this is a very well established company they've been around for a couple hundred years now granted this was like the u.s branch of it but that even then that had been around for 30 40 years a couple hundred years how did they make the pumps of that electricity 200 years ago well they're original the original thing was in pumps it was actually toilets right different business okay yeah we've always needed toilets yeah so you're excited on in what ways exactly for what reasons like this is this is was a unique opportunity for you i take it it was a unique opportunity and and one of the things that i love to do is empower people through the use of technology right like if you're doing something the old-fashioned way and it's just not as efficient i want to show you how you can use technology to you know do it do it better earn some cycles back get more done uh and i just knew i could see the possibilities because that company was so far behind the times it's just like let me show you what we can do here yeah these people are going to be amazed when i weighed my magic wand where were you in your career at this point you had enough experience to know what to do i had been working in it for about seven or eight years okay and you know today i've been in 19 now for 17 years right but you understood enough about design and how things work i mean i guess anybody with a modicum of network experience could be like oh my god you have unmatted switches daisy chained everywhere this is not good right it's fairly obvious right yeah i certainly have a whole lot more networking knowledge than i like today than i did back then but i knew enough back then to know that what was there was wrong right so the original question is what does the bad network design create bad user experience things not working right can't get the stuff everything's slow which i guess slows down your whole workforce right people are trying to do their job check email check on orders and they just sure either can't get the things or they've got to wait 10 minutes for the exchange server to give them their mail or something right as an example right okay because at a certain point they started to hire a lot more people and it was more people than that server was rated for then the server was licensed for so you have more people firing up their computers all at the same time when they come into work and they're all trying to check their mail on spinning disks not like flash or anything like that over an already impeded network for various reasons you know some switches are there was like one switch that was a gig that all of these other switches connected to and that one switch then connected back to the server room over a single link so the entire company's trying to access everything in the server room over a single link so this is a really obvious example like an answer to that question of like what's the business what are you talking about like a business trying to run everyday operations and their i.t infrastructure holding them back right slowing them down like that's a that's a problem sure so if i'm there and i hire you i know you're coming at me with a price tag like well listen we're gonna need 400 thousand dollars to fix all this like was that a hard sell did you have to go to them with a with a ticket item of like we need to buy stuff so so certainly yeah i mean there was a lot that needed to be addressed and so it was trying to prioritize in this dumpster fire what what's like the the biggest fire that i can take care of first that's going to have the most impact on everyday working lives here what do we have to get done and it really yeah because i feel like nobody wants to spend money on infrastructure i mean i mean i got them in that predicament kind of right like well it just works it's fine it was neglected for so long right right and then they just kind of like dealt with stuff that they're it's like their application server had to be rebooted every day like otherwise it would lock up and then they would have to reboot it anyway so they started preemptively rebooting it like a scheduled reboot and everybody was just okay with that just like that's that's not okay we don't need to do that so i'm trying to guess what you're like you said what can i come in here so like design conversation right like look at this awful design i don't know what your budget was i'm i'm guessing that you so what would you have to do like i mean you don't need a core distribution and access i guess right like i'm guessing you're replacing those unmanaged switches with like management yeah this is enterprise design right well we did have to design an uh a data center of sorts right like it was like one or two server racks and a telecom and that was the first thing that we had to do was identify a better location right like the server room that ultimately like was there uh when i walked in the building first it was right next to a cube farm and the the the room that it was in was not like soundproof in any way shape or form and so it was it was loud there was a wall-mounted ac unit that kept it decently cool in there but the servers were old they were super loud they were like closest to the door in that room uh and so all of that noise you could very easily hear out you know in the in the uh bullpen area or whatever where everybody sat and tried to work and it's like okay there's a there's a better room in this facility that's more centralized and it's surrounded in concrete because it's part of the building that houses the elevator shaft so let's let's throw it in there and it's going to be a lot quieter and more centralized so now we're going to pull some fiber back to this room and that's that's what we're going to do for the networking now all of the axis layer stuff was pulled into was already pulled like out to the wall right and it would just be way too expensive and cost prohibitive to like re-run every single drop in the building to a new place so we had to go with the access layer switch locations that we already had patch panels were already there so we we couldn't rerun that so but we did do is run fiber from those locations back to the new server room because that's where we would put the new core switches so you put in you move the servers yep you put in core switches proper cooling access abundant power and you ran fiber from that core to each access switch yeah is that what you did yeah okay is it a pair of quarter round paracore switches or just one it was a pair of switches yep you really redid like everything yeah huh okay so how long did that guess you could do it in one night i guess right like the cut over i'm thinking like how long were they down so what the the server room move what we did was we bought new servers and we migrated applications into the new environment so we went with virtualization so we got three new servers for compute we got a data store and we ended up doing a p2v which is a physical to virtual on some of the applications not all of them and then we built net new virtual machines and just migrated the applications over to those you know we did a backup of the database one day did the restore and then this doesn't sound cheap um i mean on this on the scale that we were operating at it wasn't it was probably it was pretty expensive for the business at the time but good for them that they had i guess i don't know the foresight's the right word but it was it was just education right like every time they wanted a new application they had to buy a new server because they didn't have anybody that did virtualization and so i showed them the power of virtualization we're gonna buy these three servers and it'll be the last three servers you'll need at least for a few years right yeah right yeah that's wild and i guess that's tough right trying to convince the business to spend money on infrastructure i mean that's got to be an art to that i guess like you have to make that case for it right sure because this guy just wants to come in here and buy all this stuff well hold on you know these are the benefits you get right these are the pain points that we're going to alleviate from this and that's that's what a lot of the design discussion was was look this is a pain point for us the business wants to scale and wants to move we want to try new applications we want to do all these things but every time we do that we have to buy a new server you know like the erp system was incredibly old we needed an upgrade we needed a new server for that what's erp uh enterprise relationship program i forget exactly what it stands for the sales tracking stuff yeah pretty much so yeah yeah it ultimately helps your financials you know yes sales uses it and manufacturing uses it but when it comes right down to it that's your financial reporting right okay you know like you you're punching how many hours it took for somebody to build a pump you're tracking all the parts and raw materials that people are moving through the shop you're you know collecting all the sales from the sales people like into that system and ultimately it spits out your ledger hey a1 fans aj here for an ally you ever heard of net ally sure you have they came from the same group of engineers that brought us network tools from fluke networks netscout and now their net ally they know networking i'm a network engineer for a partner and when i go to customers and see they use net ally i know it's going to be so much easier to troubleshoot issues we might run into the name may have changed in that ally but the way they build tools hasn't changed a bit they ask what would a network engineer want to help make their job faster and easier and then they go build it just like this etherscope nxg netally is here to help netaly simplicity visibility collaboration visit netaly.com today now back to the show so exit asked earlier how long did it take sorry i stopped on your lex it was a it was a very iterative process so you know the first part was probably done within the first year so we got all of the old systems moved over into virtualization and then the next request came down was the ceo of the company wanted everybody to have the same email address and because like the the yeah the hq the company was in the uk and they were using one email domain and we were over here in the us and we were using a different email domain okay and it was it was kind of a convoluted thing and so i had to collaborate with the it directors in other countries to get us all on the same email and domain wow like microsoft this place sounds like we did a lot of good stuff i was there for six years okay so it took you six years to fix them up before you decided to move on um yeah like towards the end like the the servers and stuff that i hadn't you know designed and installed you know shortly after getting there we were retiring those literally retiring those the day that i walked out wow yeah wow so i was going to ask like what next after you finished all of that like was your job super boring after that i mean right yeah there was always there was always some improvement to be done it was you know like like i said it's very iterative right like we started on three physical servers and then a sand for storage but by the time that i was leaving we were moving to hyper converged so so they continually were investing in keeping things up to date because they saw the power in that right because when i started i started there they like the most they had made was like 18 million in a single year which was great for them they thought but by the time i left they were making 60 million a year okay so they could afford it and they were all right that's good that's great and i think that you don't hear the stories after the first iteration after getting onto the new servers and realizing oh i don't have to you know reboot my application server every day and oh hey it's responding a lot faster and uh you know we got all new high-end uh desktops for the the guys that were doing the autocad the drafting and stuff like that and they could actually work you know they didn't have to sit there and wait a super long time to open up a drawing like it just opened and they could start working those employees must have loved you yeah i mean you improve the quality of their you know employment there's so much wow so so there is a data center design discussion in here so so yeah when we're even though that was a small room right we still had to do the same data center design that we would for any scale of a data center we had to have cooling we knew that we're going to have servers and networking devices in there they need to be kept cool when your cooling dies you're going to have that depending on the room you're going to have maybe a matter of time before that stuff starts to overheat and causes you know critical damage to those devices so cooling is of the utmost importance in a proper data center you know some one of you guys mentioned earlier you're going to have a hot aisle and a cold aisle cold aisle is where cold air is either dumped in from above or comes up from below some data centers are on raised floors and you can take these suction cup things and pop the tiles up out of the floor and that's where your cold air comes out of you can place vented tiles in front of server rack doors and the cold air will come up from the bottom and the servers will all the servers and devices in the racks will pull the cold air into the front and exhaust the hot air out the back and then the hot aisle is where the hot air is exhausted into the hot air gets taken away by the hvac system in either exhausted out of the building or chilled and dumped back into the cold aisle interesting i i can't tell you i've had to work on like a number of like stupid issues in the past because you know it the stuff was in a data center that had something wrong with the hot and cold aisles like i don't know exactly how they were set up um but things were too hot and like optics were getting messed up and failing way too fast we had like i remember just like an order of um i wish i could remember what they're called they're like a single piece where you have the you know the connectors at the end the sfps at the end and the you know oh oh direct attached um so we had we had to i think i did like 80 of them in a month just in like the same data center because it was so hot and they couldn't figure out how to fix it without taking things down and moving things around so there's like real implications real bad stuff happens when that stuff is not designed properly absolutely yeah so even in that cooling discussion you know when we're looking at that room it was a small room and you know we we knew we could get an ac unit mounted on the wall that was an issue was going to dump cold air right directly like within feet of the rack right there was just enough room to open up the door on the server rack and then we would dump cold air into the front of it they i discussed with them that we needed to have something to pull the hot air out of the room and they're just like oh it'll be fine don't worry about it i'm like no it's not going to be fine i am going to worry about it so they didn't do anything to pull the hot air out of the room and sure enough like it got ridiculously hot in that room um yeah so what do you use to pull when you're a smaller you know organization like that and you're not in like a big industrial like room right what what do you what do you use for that so the top of the ac unit the wall mounted ac unit had a vented area that would pull air into it but the problem is is that hot air wasn't making its way back up to the ac evening so to provide a temporary fix until we could figure out a better long term yeah we had a fan right behind the server rack that blew the hot air up and it forced it back towards the air conditioning unit but that helped right it helped it helped a lot but it was temporary you said yeah it was temporary eventually they addressed the hvac system to be able to pull the hot air out yeah i guess you got to put some you know you got to run some vents and fans and suck it out somehow right yep and you have returns in hvac that sucks air out so i guess they just had to pay for that and right and there there was no proper like divider right like there wasn't a separation of the hot and cold area it was just one single room so eventually that hot air would just get hotter and hotter and hotter and the cold air just wouldn't even happen yeah that's wild so something just jumped in my head look i had lex i said physics that's all so i don't know if it's not part of a data center design discussion you can tell me it's not but i i never understand why people pick certain routing protocols over other certain routing protocols right like you can choose so like when you're designing a data center we're talking about physical infrastructure now right but there's logical stuff we talked a little bit about logical or earlier quality service and you know and all that stuff maybe spanning tree around so i mean like how do you choose a routing protocol because again architects design stuff and like oh it's a cisco shop they're running an ergrp or oh they want to be you know multi-vendor so it's ospf i mean is it that simple like it is it really comes down to what vendors are using and what is the familiarity of the staff that needs to run it if you're if if all of your engineers that are maintaining the equipment are classically cisco trained and have used cisco for years not to say that it's possible but suddenly pivoting and going to juniper or or any other vendor that does not have cli like like cisco there's gonna be some turmoil there right like some friction right like operationally speaking you're going to need to make some investments to get them trained which is going to add to the cost which may or not be you know more or less than just sticking with the vendor you know this doesn't even have to be a cisco discussion right like it's just it comes down to what's the comfort level of the engineers that are supporting it right now today and the other logical stuff go ahead and that could even it doesn't even have to be physical gear it could be the the technologies that you're you're running on there so it's you know are they more comfortable with ospf are they more comfortable with isis are they more comfortable with the igrp right so either vendor or lock-in or your staff you just don't ask what people are comfortable with and you do whatever you want and then they have to learn it real fast so is it you know i i'd lean to the cynical side because i think it's more realistic like do you think the routing like i feel like places buy into vendors right so if you've bought into a particular vendor solution so like i worked at a cisco shop and they ran the grp everywhere and then bgp at the edge and they redistribute you know one into the other right but then i worked at a multi-vendor shop and it was ospf right so i mean i feel like you know i'm wondering how much decisions like logically are based on like oh i you know our staff is more comfortable with this and this is a better protocol or this is the vendor we locked into and this is you know like we're either multi-vendor or one shop and we have to use this one particular protocol like do do you think vendor decisions the iron you buy and the ecosystems that come with them do you think that that drives design decisions at all i absolutely i think really there's a lot of different like philosophies right like i when i was managing the environment i was of the opinion i wanted one throat to choke right like i didn't i didn't want to have to call one vendor when my servers went down i didn't want to have to call a different vendor when my data store went down i didn't want to have to call a different vendor when my network went down i didn't want to have to call a different vendor when my mailboxes went down or whatever you're like i wanted one vendor one support line it just made it quicker i didn't have to stop and think about like oh that system went down like which 800 number do i have to call now be nice right right and so we went with a cisco hyper-converged platform hyperflex we had a cisco network because we bought hyperflex even though it was running vmware we still got our vmware support through cisco um so cisco was our single throat so you're just dealing with okay so you're just dealing with them and then they're reselling you all the things yeah is hyperflex like their server hyperflex is their hyper converged offering so hci yeah so ah is really cool because you used to buy you know pizza box servers one you two use servers that had some storage on board but you know that was probably just for the applications the operating system and then you know if you needed a lot of data you would get a data store san or something like that right yeah well hci is all of that crammed into your pizza box servers so you would already have a distributed data across however many you know devices you had in your your hci boxes you had in your cluster and you could scale with hci just by adding another unit what's really cool about hyperflex is that you could have compute heavy nodes and you can have data heavy nodes and you can mix and match most h other hci hyper-converged vendors don't have really one or the other um if you needed more data then you know you couldn't like mix and match like you could hyperflex that might be different today i don't study hci a whole lot anymore this this was years ago when i first was evaluating hci vendors so is there a ucs platform considered hci or is that a different thing hyperflex is built on ucs yeah so those are remember managing some ucl yeah yeah yeah so the ucs because i remember they had they seem to have way more stuff yeah in the in the one box right instead of like you said external sand somewhere it was all built in right into that one box and you usually with hyper converged you're seeing a whole lot more processors a whole lot more memory a whole lot more storage all built onto the same box so there's an advantage to one throat to choke right because it's simpler and you have one place you got one person you can say hey fix my stuff there any downside to one throat to choke absolutely software bugs baby right if you get a single software bug that applies to a lot of devices in your network then that one thing that one bug could cripple your entire network i've run into that and that's a lot of great everything tomorrow that's right that's i think one of the bigger drivers for people to you know spread out the love when it comes to purchasing and stuff like that and then there's also like a a piece of it like keeping the vendors honest right like if you go with the one throat to choke model if it it's possible that the vendor could you know charge you more because they know you're going to buy that vendor right so by keeping keeping the vendors honest you know like oh well i'm going to buy vendor a over here but vendor b over here and people feel like they're constantly getting a more competitive price i'm sorry no go ahead sorry um i've been in multi-vendor environments before where um we we would find a bug that was kind of far-reaching on a lot of devices we were using with one vendor so we would and by by we i mean like the organization not like the network team or something necessarily but we'd basically use that to threaten them like oh well we're gonna go with this other vendor if you don't fix this bug like pretty quickly and give us like a way to upgrade and fix it or a solution fast um which was an interesting way to handle it but i guess you got options when you have a multi-vendor environment versus just one right right just like you said right i've seen vendors put it against each other too oh sure kind of like you're saying age like if you're with one that's it but you can go to shop a and be like hey chubby over here you know even if you're intending to stay with your incumbent right provider like well i'm shopping around and they did this pack and there's this price and it looks pretty good and i've heard people kind of do that to to try to get a better price have you seen or do you think that we're heading in a direction where you think vendor lock-in is slowly going away and i say that like so we just talked to attention you know recently and you know they have this automated solution that like they don't care what stickers on your iron and juniper like where i'm at has it like i'm seeing more and more of these solutions of like vendor agnostic like you can design deploy and maintain your infrastructure no matter who stickers on the box in this you know don't say it's single pane of glass right but like so you know i i wonder if we're getting software defined networking i guess right like whatever the whatever the thing you know du jour is but it seems to be that the brains are coming out of the iron going somewhere else controller base whatever and i just wonder if having being locked into one vendor will be a thing of the past someday because it's not i don't know if it's going to matter as much if we're taking the control out of the control plane and putting it in some type of controller or brains or management's just automation plot right like right is it considered like more old-fashioned to stay with one vendor these days or does everybody sort of have a different opinion it's it's i think it's differing opinions all over the place i mean there's certain there's some someone could perceive value in staying with a single vendor because then you get the vendors total story right like if you have their firewalls if you have their data center switches if you have you know if you have all of their stuff from a to b or a to z then you you get the total story the total picture right like if you have just meraki switches you can you can see a lot about the clients that are on the network but if you have meraki switches you have meraki wireless and you have meraki firewalls you're going to learn a whole lot more about your network you're going to have a whole lot more visibility and and that's intentional by vendors very intentional right exactly so that's why with these automation platforms like you're getting a whole lot of analytics and knowledge about your network because it's that you know vendor end to end whereas with solutions that are vendor agnostic they kind of help you bypass that disadvantage maybe of using multiple vendors stuff i think a lot of the the vendor agnostic stuff out there now like yes you can have a central management device or operation or application that can manage all of those devices but i still think it's not i don't think you're getting as much telemetry and application data and and knowing what's on your network if you're multi-vendor i i could be completely wrong there's a lot of cool solutions out there that i you know don't see i've heard that said right i've definitely heard that said and i think there is something about like well i got this one vendor i got all their stuff and like it almost reminds me of apple not that that has anything to do with like data center design but it's you know it's a closed ecosystem it's tight the software the hardware they're tied to each other and it gives you certain advantages and features that you might not get and you know in a different distributed you know different software running on their hardware like it's it's why the experience with their hardware i think is so good because it's just so tightly together so like for a networking equipment they're probably doing a similar thing like you're saying you have more insight you can see more things you know you buy into this whole thing and but damn if you know if you hit that bug or if the relationship goes sour or i'm i'm i'm really i'm i'm kind of excited about like a lot of the automation platforms and you know i i hear that we're going to be able to do this stuff without so much of that vendor lock in and i guess we have some it seems like we have some ground to cover to get all that insight like you're saying in the telemetry application like because you know vendor a might not want vendor b to see this special under the hood you know hidden thing right like oh if you stay with us you can see this stuff and they can't see that which which thinks right because we're running open protocols like if we can run ospf everywhere it kind of stinks that the vendor is like yeah but there's special things that that that that automation platform can't see that ours can you know so well i don't get to a point where it's just one thing like i don't know i don't know how important the stickers on the boxes are is all i'm saying for operators like right i've never cared what i'm running when i when i worked at comcast they had everybody's box so like why would i be allegiant to any one of them right like okay we're running everything great whatever i feel like the bigger places are so we need solutions that are vendor agnostic that can help operators like that manage that infrastructure easier instead of having to log box to box to box when something goes wrong like which one's messed up i don't know let me check the logs it's so nice to just have those solutions are targeted towards larger enterprises right because you have larger teams of people that can better manage those those types of things right those different vendors whereas are there a lot of large enterprises with multi-vendor environments or are there more i mean is that more common than having like a huge enterprise with the same vendor did that question make sense do we know i guess like is there any way to know i'm just i i don't really have like my i still don't really know the trends in networking to be honest aj's probably seen more environments than us like i've i've seen two environments right you've worked out that i will caveat that by saying i've seen a lot of environments but the partners that i work for typically feed one beast yeah right so most like not to say that they don't sell others but it just the work that i do it's typically 90 of the time it's one vendor why do you think that is maybe this is a different show maybe like vendor lock in and i mean that i work for it could be a i think it could be a number of different factors i i think in the end that i haven't seen enough to to give a definitive response i think par part of my view is skewed either because of where i work or just whatever any number of reasons so you're saying you have more cisco certifications than any other vendor i'm saying that the most most of my projects that i'm installing cisco gear right but that's not that's not right there's other there's other engineers that work where i work that install other devices you know we have plenty of other engineers that install plenty of arista i don't see that right so i don't i don't know how much of them are doing that but i know that they're out there you know we have teams of people that do aruba and arista and you know plenty of other manufacturers i happen to be cisco certified i have more experience with cisco than other vendors hey a1 fans aj here to remind you about nordvpn.com nordvpn will help secure you wherever you go i use nordvpn on all my personal devices whenever i'm out about i just go into the nordvpn app hit quick connect and away i go nice and secure don't have to worry about prying eyes anybody looking at my connection if i choose to go work from a coffee shop locally or you know even while i'm traveling if i bring my personal device devices i will use new vpn to help keep things safe and secure i'm using nordvpn right now and there's no degradation in my signal everything looks good when you guys watch us on the live streams so i i can't say enough good things about nordvpn they have some great additional services included uh with their vpn product uh they'll scour the dark web for your credentials and see if they've been involved in any sort of hacks or anything and then if they have they'll let you know and you can go change your passwords and do whatever you need to do to help keep yourself safe and secure they also have a mode that will block any websites or ads known to possess malware and they just have general ad blocking anyway because you know who wants ads uh as i record an ad anyway if you want nordvpn and you do go to nordvpn.com t-a-o-n-e for the art of network engineering and you can get a really great deal 73 off two years plus four months free so again that's 73 off two years plus four months free of nordvpn if you forget that url just go to nordvpn.com at checkout you can use the promo code t-a-o-n-e for the art of network engineering and we appreciate your support as well as nordvpn's support of the art of network engineering podcast now back to the show chris denney is asking are there any physical concessions you've had to make because of one vendor or another physical concessions no not that i can think of i will tell you in my experience one of the reasons when we were looking at hyper-converged infrastructure the servers one of the reasons why we went with cisco's platform over another vendor was because another vendor wanted you to have almost required you to have four nodes in your cluster whereas we could get away with three nodes in our cluster and they kept saying oh well it's so you can put one node in maintenance and then you still have three other nodes and then so you could sustain a failure but the problem is is i i was already licensed for three nodes i already had enough vmware licensing i already had enough microsoft licensing for three nodes even though the initial cost comparison even though i was buying four nodes it was still a little bit cheaper than the three nodes that cisco had in my quote but they were all the cisco nodes were also beefier so it wasn't an apples to apples comparison but then if i went and added on the additional vmware and microsoft licensing that i needed that was going to put me over than the cisco quote so can you quickly define nodes i mean are you talking about a a rack with servers in the top rx switch sure each server is a node so when i'm talking about like my my h my hyperconverged um cluster it's a bunch of servers that are sharing the same resources okay right so whatever you were trying to build hci and the vendor said you need four yeah for the design minimum four nodes like end of discussion sorry so was that their design requirement or is that a way to sell more gear and licenses or both i think there was something in their secret sauce that they needed a minimum of four notes yeah interesting however the technology worked maybe or the things were designed yep you needed an even number maybe or whatever right so no power temperature limitations you've run into um power has certainly been a problem um the the server room that we built was power limited because of you know issues in the building and we definitely had limitations with ups um i have a hell of a ups story for you um tell it so uh we we had a uh ups system in the server rack and um when we initially got it we wanted to put it on the network so we could network manage it but the switches that we had completely full couldn't add another network drop in there we upgraded our core switches and the switches throughout the building yay we have more ports now let's connect our ups system so we can manage it remotely so i was the it manager slash infrastructure architect for this company i had hired a sysadmin the sysadmin goes up to the server room to console in to this uh ups completely unbeknownst to us apparently there are two different styles of console cables for this ups vendor and when you use the wrong console cable it shuts down the ups like completely shut it down and because the the pdus the power distribution units were plugged into the ups when the ups shut down like hard down that shut down our entire server rack plugging in the wrong console cable took down a whole server is that like a weird security thing that's a bad design this is a someone's trying to hack into the mainframe we can't let them use their own cable the differences between the two console cables were very very minut like just looking at it you would never know was it a pin out difference like i i can't even remember like what the difference is it was like a generational thing and yeah that's goofy yeah it's one thing to not be able to connect to the console port with the wrong cable right to take the thing to its knees oh man and that is that by design like that's supposed to do that i don't think it was intentional it just that's the way it ended up so they just you know they put out a bullet in a bag so you guys probably discovered it and they're like oh god um so they wrote the bug for you i was at my desk and like the internet goes down and like we couldn't get to any like even internal services i'm like well that's weird so i i go up to the server room and uh my sis admin is like on the floor like this terrified look on his face i'm like what happened he's like i don't know i just plugged into the console cable and everything shut down poor dude what did you do like everything was off like i'd never heard that room so quiet i'm like oh this isn't good he's probably gonna tell that story for the rest of his life the day he almost had a heart attack i thought he took it all down just plugging something in yeah first so aj what would you tell people that are like just starting out and want to get in like you know i i didn't learn design right other than the minimal three-tier stuff that was in the old school ccna and you know lex and i are both you know aj teach us about design we don't we never you know i've always relied on architects right and lex is like young like earlier in her career so i don't know anything but i mean i feel like i've done myself a disservice that i haven't learned more about design because i mean if you're a network person maybe you want to be an architect someday and i know it depends on the role in the company but my experience has been the architects do the design and if you don't know design you never be an architect so like i would have probably never become an architect in my career right in operations until i learned enough design to be able to design a network i mean it sounds silly to say but i like i said i wanted to be an architect on early shows and every every architect we had i'm like ooh tell me more but i never spend any time learning design right so yeah i mean you need to know design right period it's just don't do what andy did is what i'm saying and i think that's that's probably more common right like you work at a large organ you worked at a large organization that had architects and like i said anywhere that i have ever worked never had architects so it was up to us to understand the technology understand best practices design for a given technology most vendors will do um like a verified design right um you know most most vendors have a verified design for whatever platform they're selling whether it's like servers or storage or networking so those are great documents to go through and understand you know what are the tested approved validated and when they say validated it means they took this equipment they set it up this way and it works and these are the speeds and feeds they got out of it so what did you call it i thought that was like a design guide usually it's verified designs verified designs vendor verified cisco verify designs dell verifies designs so your boss says hey we want to deploy this thing go check it out you can go get a yeah whatever that document's called verified designs validated designs verified this this is what you need this is the gear this is the yeah and they'll give you like some various scenarios like oh well we were deploying this to support a thousand different locations or whatever and here's how we did it because you know what's interesting i never needed to know design because i never built networks well no yeah you built the networks with someone else's design it's just like building a building right like there's an architect that builds the building and then there's engineers that you know build like physically build a building right yeah so the moral of the story kind of like if you really have your uh your site set on being an architect one day you should try to find a position where you're kind of going to be forced into that role in the first place but maybe not with such as high such high stakes i don't know like i that's how i would think of it if i want to be an architect i wouldn't want to be just like thrown to the wolves but you also do want to learn right and that's how you said you learned like by having to do it having to know that technology and designing it and going with it was it out of necessity aj or were you interested in design because i guess i've never been interested in it because i've i i go after things i'm interested in and other people were doing it for me like was it necessity for you because you had to or were you like oh i want to know how things are built correctly let me go learn it or maybe both it's it's been a little bit of both right like i i've worked places where it was bad and it was bad because there was no design nobody sat here and thought like here's how we're gonna do it so it can scale so it can we can grow bigger you know it was just there's a need let's fulfill the need they didn't really think about how they were fulfilling the need they just checked a box right and and it's short-sighted right very short-sighted very short-term uh and so through dealing with the bad stuff i just grew an appreciation for doing it the right way right because there's like the in the design there's like modularity built in right like if you follow this design and you grow you can add modules on to this existing design and it'll work not plug in another unmanaged netgear switch in another modularity so like your troubleshooting domains are like a certain area like stuff can only break within certain confines right like then you have redundant troubleshooting you have redundancy like you know something can break a certain number of things can break and you'll still be operational you know and that's all right important pieces to keeping any business going right like you need to be able to sustain certain failures not like redundancy resiliency something else availability maybe yep i haven't considered that it'll help you isolate the fault domain if it's designed correctly if it's just the wild west god knows anything could be broken anywhere right right yeah it's probably harder to fix yeah huh do you feel like you made any mistakes when you were trying to implement you know design that worked better like did you do you feel like you made any like silly mistakes that you learned from or were you pretty much there without it like did you did you try silly stuff at first that you didn't realize was silly till you did it i feel like that's what i would do i just fuck it all up and have to start over a couple times so sorry guys i don't know um it was it was a lack of networking knowledge and it was a lack perhaps of of networking knowledge on the partner side as well so when we got new switches for that manufacturing company i was working for um i i had to settle for hp because they didn't want to shell out for cisco i was more experienced with cisco they didn't want to shell out for it so we went with hp instead um the two core switches were not stackable so that they had to operate independently and i took that at face value we didn't build any redundancy in it but my partner my trusted partner didn't push me in that direction either you know now knowing what i know today we could have had a first top redundancy protocol running on both of those switches and protected the gateway i didn't right i was running all my places to run a cable between them and run hsrp right right i had yeah yeah i had one switch that was running all of my layer three that was my spanning tree root and all of the servers connected to that switch and then i had another switch that was basically the distribution to the building all of the switches throughout the building up linked to that switch and like thinking about that and knowing what i know today i'm almost embarrassed to say that that's how i deployed that so many years ago but i didn't i didn't have the experience then that i do now right like that i could have that could have been designed so much differently and better than it was no it's not embarrassing that's just that's what i was looking for honestly it was like something i'm embarrassed for you aj i i feel like i would do that if i were just kind of trying feeling around in the dark trying to figure things out so absolutely well i do know that it like even though we didn't do like uh first top redundancy protocol i do know that what i wanted was additional bandwidth to be able to go out like the the redundant links and the higher bandwidth going out to all the switches so you know having um port channels out to all the switches i got my redundant links i satisfied that requirement i got the higher bandwidth if i had to use spanning tree across both of those switches one of those links would have been shut down by spanning tree and i only would have gotten a gig out of it so to a degree standing tree i had to do what i had to do to satisfy the the requirement but i did it also because that's that's what i knew how to do right like i could have done it a different way but it wouldn't have satisfied the requirements i know we're gonna have to rap soon we're getting way toward the end here but you said spanning true root and i just want to tell one more quick story because it it gave me i just had ptsd when you said it so like logic logical design right spanning tree we had an nmz zone which is say two aggregation switches two distribution and then they went to a third pair maybe it was ag and then distribution then there were firewalls down but anyway it was for not manage zone for clients they could come in we'd give them a cable and they could get the services so it was all into life we had to replace it so we put all the good new stuff in there i was responsible for that the ship you know the gear went out whatever but i forgot to set a spanning tree root because i have so little experience in layer two and like i didn't even think of it thought never even occurred to me so anyway the root shouldn't be on an access switch right oh no no right so because i and and again now we're like ccna level questions right so if you don't set a route it's all that 32 whatever you're saying so then it goes to like the mac address yeah so what does it go to the mac address burned into the brain so the long and short of this whole ridiculous story is one of my access switches was the root because i didn't design spanning tree at all i didn't even think of spanning tree and then i don't know if it was like root guard was then on the ag switches but because that was the root these ports shut down and like or maybe it was root card whatever it was i forget because i'm so layer to challenge but i had this whole huge mess of like what is happening like i ports are shutting down i can't get them up i'm getting weird log messages like why can't i get these ports up just because i didn't design spanning tree not only properly but at all i forgot just put in switches and it'll work so you know like you said you just didn't know any better i mean i should have known better but it was my first time dealing with the spanning tree design and i never even you know after that now i've never made that mistake again like oh yeah i better decide what the route is and do that command to make it the route you know you don't know what you don't know right i had learned about spanning tree and ccna but i had never built and designed a spanning tree you know uh design that so so anyway i i seem to learn best when i do things wrong and break them and then i'm like oh right that i i won't do that again every day laugh every day lab and prod it's still labbing yeah there you go yeah this is the same thing i like our design discussion i'll piggyback off of your your spanning tree story so um well first uh chris denney asks do you think your job at avar has helped push you along this path absolutely so i am a senior network engineer i do post sales deployment i get handed designs pre-sales designs from architects they're they're design architects they work with our customers solution architects so they'll come up with a solution designed at a very high level and then i get to take that design validate it against the customer's environment and then do a lower level design and really pick out the details so i and i love that part of my job that's like one of my favorite things tonight it's pretty fun yeah um so to piggyback off of your spanning tree story i visited a customer last year they have a very large campus and i get in and start looking at their switches i was not allowed to remote in because of the security of the environment and so i get on campus and i start looking at all their switches and every single switch had the default spanning trade priority and somehow the core switch was elected root but i feared that installing the switches i was there to install would have changed that so i told them that before we do anything we're going to set some spanning tree priority in this bitch exactly how you say it yeah because you know and then on the reverse end this was a a very large sprawling campus it was for like a park right like uh like um you go visit a park huge campus they have facilities where people live they have facilities where people uh work and store stuff and do do various things so um it was a perfect use case for these small work group switches these small eight port poe because they had voice over ip uh and and so even those had the default spanning tree priority and even though all these protocols have elections and things like that if i don't want something to ever be the spanning tree root i'm not going to leave it the default value i'm going to set it as high as i possibly can to prevent it from ever being elected the root switch so when andy went in and installed that right and andy went in he's like switches just do switching it'll be fine right right yeah i i i took all of those i touched every single one of those eight port switches and i cranked the spanning tree priority as up as high as it would let me go it's like 61 000 whatever uh so that way so by some miracle their core switches had the i forget if it's a higher so the lowest mac but whatever the hell the thing is so by some miracle their core switches had the lowest mac because everything was at default right wow yep what if that person is still out there doing that dumb stuff yeah it was it was crazy to me all right i was just like oh my gosh this is like a house of cards right like it could blow over at any second and then yeah how many they would have no idea how many devices were were participating in stp that could have it was pretty big could have had like a root switch all the way like a mile away down the green in a tiny little closet somewhere exactly last question what's the diameter of spanning tree uh i forget what you can set the maximum diameter to but the diameter is from your core out to your distribution layer out to your axis layer and and you can continue to daisy chain or hang switches off of switches it's not desirable or preferred or recommended right you know but the diameter is it should be an equal number of hops from edge to core but is there an upper limit where you got to stop hanging switches off of switches is that what the diameter is is like things will stop working once you've daisy chained to 20 seconds after like a certain distance you're going to just introduce latency in all sorts of nastiness yeah and doesn't spanning trees stop working i swore something weird happened yeah i just googled it i'm not going to pretend i'm smart but there's a max it's 17 switches it was the message okay good job lex yeah i googled something a very important skill for a network engineer i have mastered it good work awesome well this has been a fun discussion i didn't think it was gonna go this way but i i certainly had a lot of fun rehashing some fun thank you for teaching us aj this is really really helpful well i hope everybody got something out of that my rambling we only lost one patreon so it was pretty good we've been going a while and only one drop well uh we did get some great questions from our patreon family this evening if you're interested in joining our patreon program you can go to slash patreon.com of netenge and we have a couple of different tiers of patreon membership and you can join the live streams some of our patreons get early releases of our podcast episodes that's a new feature um that was andy's idea that i can see he's raising his hand no i i'm raising my hand because how dare you i'm raising my hand because i want to make sure that we mention our our new merch for uh until we have some merch thank you andy yes i am super excited you tell us about it bucks we now have women's sizes women's cut teas um in the merch store and i have my eye on a particular crop hoodie i'm very excited about so you can be stylish it's for ladies men whoever now we have all the sizes i i will say our our merch store so we do our merch store through print on demand so we don't bulk order merch store at my house and then i ship it to you we use a platform called teespring i think they're now called spring and they have a catalog of merch that we can pick what we want to put our logo on and then you go to our store buy those items and then they will print it every so many days they'll run a bunch of orders from our store print them and ship them for us they handle all the overhead we don't pay for this service we just get a cut from whatever you purchase we set the prices the prices are set i think at a fair price uh so that way they're not too expensive it there's like the recommended price and then there's like the a1 price right like we we knock it down for you um so every once in a while i go through the catalog of the merch that's available and i notice a whole bunch of ladies teas and other uh things i'm like oh yeah that's happening click click click click click click click so i'm very excited to be clear it wasn't a conscious choice of us saying no lady sizes yeah spring didn't offer them for the initial one we put up when we first put up the store there was not they did not have a lot of offerings and now they have a ton of cool stuff in there now so that's awesome thank you spring thank you for adding that absolutely dj absolutely we we need to up our merch game uh we have so many cool ideas for merch uh including uh tim ross head on a t-shirt or a coffee mug or something like that uh we've got the word shit bird on just everything i've seen the discord they want it the people wanted right right has the get off my lawn t-shirt idea come and gone is that dead now like no it's next to a design somewhere yeah i still have 80s get off my lawn acl back when i wasn't zandy and i used to yell a lot on the show i was getting off my lawn fill me in on some of these no it's just unless i don't want to be filled up just let me know offline awesome well this has been another fun discussion uh we might have to come up with a new title for the episode and do data center design uh a deeper data center because there's so much we didn't touch right like you know spine leaf there's just so much that we actually did not get into for data center design so i would feel guilty calling this a data center design episode so we'll think of another catchy title and definitely do a data center design episode in the future i blame the host you that's fine you can not not the two of us it's tommy and andy who don't know it's actually probably my fault because i think i took you down so many rabbit holes that no no i i i enjoyed it i enjoyed it a whole lot yeah me too these are always great discussions i can't talk to anybody else about this stuff so thank you for being here and having these talks with me awesome all right well we will see you next time on another episode of the art of network engineering podcast hey there friends we hope you enjoyed listening to that episode just as much as we did recording it if you want to hear more make sure you subscribe to the show and your favorite podcatcher you can also give that little bell rascal a little ringy dingy so you know when we release new episodes if you're social like we are you can follow us on twitter and instagram we are at art of net inch that's art of n-e-t you can also find us on that weaving web that is the internet at art of network engineering dot com there you'll find our show notes and some blog articles from the host guests and other friends who just like getting their thoughts down on that virtual paper until next time friends thanks for listening