this is the art of network engineering podcast in this podcast we'll explore tools technologies and talented people we aim to bring new information that will expand your skill sets and toolbox and share the stories of fellow network engineers welcome to the art of network engineering i'm tim burtino and this is the podcast where we ask the smart questions lead the study session live streams create hilarious profanity-laced educational posts and yes we work on rockets as well actually that's not we at all that is all accomplished by my excellent co-host for this evening lexi how you doing that was a hell of an introduction make me sound like i'm up there on the bottom just tinkering you know i appreciate that tim um i'm doing all right it's been it's been a hectic day i learned a lot today i can't talk about any of it you know but but it's been a great a great day of learning how are you doing i'm i'm doing well um not uh not to that level on my side by any means but yeah learning learning stuff every day still trying to figure out this uh this new role at work and and trying to kind of balance the still kind of the day-to-day operations as well as getting more involved in business level meetings and trying to come up with solutions and and high-level stuff like that so it's been exciting and still going through the um enterprise design cisco stuff so i i feel like i'm half ish way through studies that i'll be ready to try that exam so yeah i've been exciting you act like you're not learning anything you're learning probably more than me actually that's what it sounds like no it's been it's been fun it's been a crazy 2022 already i've probably said i should keep track i probably said that like 17 times already this year but i feel like that's a common theme like 2022 has been a little wild and it seems like like in general and for a lot of us in our lives individually i've heard a lot of crazy stories already i actually just brought that up to my wife earlier today i just kind of sat back and i was thinking about everything that's already happened here in 2022 i'm like was 2021 like even a tenth of the craziness that's already happened this year i think so that's yeah so i think this is going to be a wild year if the first two months are gone are any indication yeah i agree hopefully a good wild year there's good wild and there's bad wild so do you have uh anything in uh in the new town in the new place uh exciting that you wanna what's your favorite thing so far oh my god yes okay you can see mount rainier just like straight up from my nose my co-workers were asking me like hey how do you like it in the area like how's it going i was like it's pretty nice like i could see mount rainier when i go outside and they were all like okay well that's a flex wow all right fancy pants you know i didn't realize that i'm not used to like mountains because i'm from houston the land of flat industrial crap and so it's like you know i you don't never see anything in like the distance except for more like flat you know roads and stuff right so there's no like mountains um so it's just like amazing to me in general that there's mountains that exist in this place and i can look out the window and see them and then there's this big one so i don't know how used to mount rainier people are but i just assumed everyone could see it you can't see it everywhere oh it's exciting yeah it's awesome so i'm really enjoying i'm really enjoying the area for sure not so much the time zone of pst as i've mentioned before that's gotta take some time no pun intended to get used to yeah um i like wake up super early right for most people because i'm used to like being two hours ahead of this but that's like sort of where that like advantage stops right because then i have to stay later at work yeah and and then i you know i just have to rush to get to things after work basically because everyone's already started on their evenings but it's all right it's worth it yeah we're uh we're making it difficult for you i'm dan and i are at least in central time those those eastern guys those are the ones making their life i realize y'all were both in central it took me forever to figure that out because dan um is in tennessee and i just assumed he was eastern where are you tim i'm in nebraska you don't mind me oh i definitely thought you were on the east coast i'm sorry no i thought y'all were all over there except well i guess dan i knew dan was in tennessee yeah i always thought dan was was eastern time just because you know down in the south over there so but no he's he's central it's it's uh andy and aj that that bring up the eastern time god damn it those guys always ruining everything i would love to hang around and complain about andy and aj some more but it is actually time for the wins winning this week in our it's all about the journey discord channel first up john capo bianco had his last day after eight years with the parliament of canada we know john is very excited about his developer advocate role at cisco but he put so much time sweat i'm sure some tears the last year's bringing up uh the parmelec parliament of canada to uh to what it was when when he left it so congratulations john i'm sir sure was a sad day but uh looking forward to your future yeah congrats john that's fantastic i i should have led with i really uh this week's wins should win an award for the for the names that we're getting uh this week starting with john capo bianco and now this one i i need to talk to this person because i'm sure there's a backstory behind this name and uh avatar image but i'm a triscuit yes the cracker the electricity cracker i'm a trisket knocked out the devcor exam i i can't imagine how difficult that was to take on so congratulations there uh andrew andrew rodero secured another sans coin during the sec 503 class capture the flag contest he's really gotten some some cool challenge coins through those capture the flags those are really neat and michael hilton yes that michael hilton secured a job as a network engineer with apple starting very soon congrats michael awesome wow and another fun name to say net stackology passed the az900 exam congrats i love that username yeah that was awesome there's always at least one that i love every i had to make sure i set up my head a few times to make sure i was actually saying it correctly that's stackology and i'm a that's triscuit and to round it out one of our good friends from across the pond ethan passed the jncda congrats ethan that's fantastic and just one thing to add if you want to join our patreon program go check it out at patreon.com slash art of netenge all right let's get into it we have a very exciting interview this this uh this evening for this episode we have the one and only christopher hart with us christopher how are you hey hey good evening morning afternoon wherever you are i'm doing well about chill doing good we're happy to have you on this is exciting i've been wanting to talk to you in public on a recording for a long time well honored to be here as usual or not as usual honored to be here i i think my intro to you christopher has been uh twitter uh just being able to scroll and see a uh a multi-step post of of just tips tricks things you've seen on the job it's been really cool it's always it's always really interesting and really educational so that's that's my intro to you you have you have written about so many bugs that i've seen before it's so cool like it's it's it's so cool so you know i think a lot of people know you as like you know working in in tech uh with cisco right so and you you work directly with like it was the nexus platform right yes so nexus specifically running nx os um uh i'm glad to hear that the twitter threads tend to have that a lot of people seem to see a lot of value in them um i'll be quite honest i'm really just trying to see how long i can make them uh before twitter tells me to stop i'm trying to see if there's like a max limit you haven't reached it yet no i have not um so we'll we'll see though they each get progressively longer eventually i'm expecting it'll be like maybe around 64. it'll be a power of two for sure so we'll see what happens good luck please keep making them yeah i happen to be filling them with useful information i suppose so i'm glad everybody tunes in for that another awesome you wrote one about like a config save um bug and i'm trying to remember but i remember thinking like i've been gone for my last job for like maybe three months and i'm already forgetting like everything but i remember thinking like i have seen this and i can't believe you just broke it down because sometimes you know like we're i was working a break fix at a knock and we were like you know at a cert you have to find that sweet spot between like okay troubleshoot the issue figure out why but then you got to fix it and then you kind of have to move on because you're closing out tickets right so uh when i worked on that or when i you know saw it um and just working on tickets having to do with that bug like i i never fully got like the answers that i wanted behind the why just because we didn't i think at the time like have time to ask why and like learn the very minute details and you really broke it down in that twitter thread it just felt like this great sense of completion for me yeah it's um one of my inspiration first of all i thought that was a really gnarly bug it was one of those where when i when i saw it i didn't i can't really fully claim credit for it because i didn't hit it naturally it was a customer that hit it unfortunately right um i just was able to tinker around within the lab and figure out what the eventually what the root cause was and what the triggers were and such um but i remember when i was you know sitting you know in my queue back when we were in the office and just kind of talking out loud about it too i mean we're all sharing a tight-knit cube space right so we're just talking out loud about the various issues we see over like i'm like is it this is it i'm just talking out loud no way and i keep going over to my you know a teammate of mine and saying like okay it's not this but it might be this and i just keep you know going back and forth with him if i'm like oh i hit it it's this and it was like you're never going to believe what this is right um and so for for those who haven't seen that that threat or blog post um just a quick summary basically if you have an motd banner with a delimiting character that is the letter q very odd delimiting character and because it could very well be in the banner itself and you also have a cli alias configured where q aliases to the exit command then when you copy run save oh and your configuration is long enough you have enough configuration within your um within the running config and by enough i mean like literal lines within the configuration uh then when you copy run start it will uh pause and eventually timeout after 10 minutes it's such a a unique permutation or a unique combination of things where it's like wow um and my inspiration behind like talking about that isn't just like oh cool here's this cool bug i found right because i found plenty of bugs um it's more of a here imagine you're a software developer right and you're creating these the software and these features for your customers um you can test a lot of stuff um yourself with unit tests integration tests you could have a qa department dedicated to trying to reproduce these things before they come in the field but until you create that feature and you release it to a customer and you watch them use it you will never really fully understand it the the very unique ways that they use it right and that's just your feature production yeah yeah yeah and it is the wild west out there right and when you combine that feature with a bunch of other features because keep in mind if i'm the the motd banner dev i'm probably have nothing to do with the cli alias dev you would never even think to test the um interoperability for lack of a better word between those two right um so so my point is out there with putting that out there is um software is hard um it really is like as a software developer i'm not i wouldn't i'm not saying i would not say i'm qualified enough to be a software developer um but it seems like a really hard job to be able to create code that doesn't uh that is bug free so to speak i agree so yeah i really like those posts because i i subscribe to the uh cisco notification service to get a bunch of bugs for for stuff that's relevant and i read some of those and i'm like there's no way somebody hit all of these caveats in a row and i'm like well wait yeah there has to be because they created this bug so somebody had to yeah i mean some of those are just wild yeah yep and it's worth noting noting that um those are the bugs that you as a customer right right right there are tons of i mean like we have obviously have a dedicated qa department i've seen some wild stuff that they've found in between and i would say that i wouldn't you know put a hard statistic on it but i would say that the overwhelming majority of bugs that we find happen to be internal meaning they're just within okay you know builds of the code that the customer would never see um right so um yeah it's a um a a crazy difficult job and i am glad it is not mine i've got the easy job or your job i'll put it that way so let's step right into that what uh what do you do today christopher yeah so i recently took on um as a beginning of february um a new role in the united states public sector department of cisco doing solution validation services which is a very fancy word for saying we test a customer's or a copy or a lab of a customer's network and make sure that all of their technologies that they've deployed the software they've deployed the configuration maybe point and stuff works as expected and works is a very vague term they can we can interpret that however we want sometimes the customer says yeah just you know make sure things don't explode right um or they have very specific uh requirements that they outline like you know make sure latency doesn't exceed a certain amount or make sure that during this failover there's not a you know a ton of packet loss and ton is obviously quantified um and and that sort of thing which is a um it's a very interesting job as i'm learning um very different from my previous role in in the tac so um in i had joined tac beginning of 2018 i think that sounds right yeah um so i spent about with you subtract training from it probably about two and a half years or so closer to three years on the job i'm taking cases and doing some other side projects and that kind of thing yeah and like i said before that's supporting the cisco nexus data center product portfolio running in xos so if you are running aci out there uh fortunately i guess for you because i know nothing about aci outside of some very basic stuff and so how are you settling into the new role how is it very very different than tax very different um it is um a different kind of busy i'll put it that way i i i'm gonna use the word slower paced but that's not quite the right word because it's not like that there's less work to do anything along those lines it's just that the urgency of that works is a little bit different this is very much a longer project based type of role where you've got you know several weeks or sometimes even months to to put together a solution for it for a customer and give them what they're looking for um whereas in tech um the projects are much much shorter sometimes they're hours long oh wow yeah now that makes sense interesting but yeah i'm settling in it's very different but i'm learning a ton um both how to do the job as well as how to step away from the persona of attack engineer there are lots of scenarios where you see it's like oh man i know what's wrong with that or i want to instantly hop into the troubleshooting mode and kind of dig deep into a problem when it's like dude it's a docker container just just enjoy it save yourself the two hours please are you are you finding yours are you finding it difficult to step out of the tech engineer mind um yes and no there are aspects of it where that i want to keep that mindset right um or at least the mindset that i carry through tech where i'm on the lookout for things that a customer wouldn't enjoy about a product whether it's like hey this documentation isn't very good or it's wrong which is obviously pretty bad or just um you know hey i hit this snag it would be nice if this was documented or you know obviously straight up you know defects and that kind of thing in the product that aspect of it i like to keep just i think that you know as somebody on the inside so to speak i have a kind of a duty to make sure that you know if i see something broken i say something about it right um moving away from like the troubleshooting mindset is more of a personal time saver if anything um just because sure i can spend two three hours trying to dive deep into something and trying to figure out and revert it um and literally this happened today well no i saved myself today it did happen yesterday where i spent three or four hours trying to get this you know a product to deploy and because i had messed up the configuration it wasn't deploying properly and i was trying to get it to like revert halfway back and it wasn't until i had finally fixed it where i realized wait a minute i was doing all of this within a docker container i could have just like literally just exited the dock container and restarted it and everything would have been fine right um i was like okay maybe i need to to pivot a little bit sooner maybe with it and so my friend of mine who has very similar problems we both agreed five minutes five minutes is the rule before i start saying okay let's just let's step away from it and not waste a bunch of time trying to figure out what went wrong wow five minutes huh okay yeah okay which sounds kind of aggressive but it's like i mean if i'm gonna waste three or four hours on something you know that's uh my i like to think my time is valuable everybody should think their time is valuable yes absolutely that ties back to our something else we're releasing soon i think no something we've released just just recently one of our episodes on salary negotiation for sure yeah i'm thinking through that as as i know i probably can't count the amount of times that i stick with something a lot longer than i should just because i'm freaking stubborn sometimes but but yeah you you got to know when to when to cut the law i think you're almost there yeah and you like you you um you you play whack-a-mole with problems sometimes right where that's like you're almost you think you're right at the end it's like if i solve this one it'll it'll definitely work after that and then you hit the next problem and you repeat that about 20 times or however many and then you're like you know what maybe i should uh back out of this entirely wow so i want to i want to dig into the tac experience more often i'd like to to talk about your new role but i know it's it's new so we might not have a whole lot yet but definitely want to dig into the tack roll but before we do that i do you mind kind of throwing it back for us to what what kind of got you into technology what got you where you are today so i i like to say that i got into technology when i was like three or four years old um my both of my parents met online and for reference i was born in 1995. so that was a very primitive online back then they met in a very ancient version of a chat room i think i'm thinking it was like even before irc um they both had commodore 64s that's how they met and got together and then they um i'm sorry i've got the i've got the dial-up sound stuck in my head right now too that they had to hear so they could talk to each other yeah so when they when i started up when when i started in tech when i was you know three or four um the main thing that got me going was uh they would they'd throw me and have me playing some video games so like um i remember being scared of doom because you know three um so i preferred to play like everquest they were both big everquest fans back then um there was an old game called star siege tribes what i have put thousands of hours into at this point probably stemming starting at age three all the way up to probably 1450. um just you know just video games right and and even playing online with people i didn't really understand the concept of talking to people over you know that i couldn't physically see in front of me um but in order to play online you got to have dial up and so one of the first things they taught me was how to get online with zao so despite uh despite being born in 95 towards the uh end of that dial-up era i know that dial-up sound very very well my parents were doing the opposite they didn't want me to know how um so yeah that's that's where i like to say i started right just getting experience with computers and that kind of thing um so starting with like i t um in like middle school and high school i was very much the your you know typical nerdy kid right so um i was the guy who was you know always kind of tweaking his computer and fixing my friends computers for 20 bucks and you know that kind of thing um accidentally installing you know viruses on my computer and then figuring out how to get rid of them and without my parents ever knowing because that would be terrible um it was a shared family computer so it made it even worse right at least if i have my own computer like you know i can mess it up to my own content but a shared family computer they uh get will get pretty mad if i you know utterly trash it understandable um so that experience really helped like get me started right with um learning how do computers really work how does the operating system work that kind of thing just you know figuring out how to remove malware on a computer right so so um and also just the chat was the bulk of this on your own or were there any programs in middle school and high school that that helped you middle school no in high school yes um but a lot of it was just just just google man you just figured it out right you say you've got the the worst possible time they were like all right well you know my parents are home in three hours from work i got to figure out how to fix this thing right um and then the other the other aspect of it is like trying to you know push the limits of the computer and be like oh i joked i made a tweet the other day joking about you know disabling unnecessary services on windows uh from starting up with windows so you can you know squeeze out an extra bit of processor performance or something like that um and this is not way back in the days when that stuff mattered this is this is like we're talking like 2005 right when it's like no you're not gonna get like sorry that's not gonna let you play world of warcraft without a video card or something like that right it's not gonna help um but just like getting familiar with that kind of thing and be like what is a service in windows why is that important what are all these processes what's task manager just that general like investigating that sort of thing at a relatively young age um got me started right um and especially once you know in middle school and high school i had started um fixing other people's computers for money i was thinking like oh like this is a i maybe i can make money doing this i like money so that's how we all think now yeah yeah um there were programs in high school that that absolutely helped me out though because they um a lot of the stuff that i learned on my own was very um just little bits and pieces of knowledge here um but having like a formal course that was like okay this is like actually what a processor is this is what a hertz is and what's going on with math behind the scenes and this is you know memory having that that program that connected those bits of logic together um helped out quite a bit um um you might be you might be too young but i have to ask did you did you ever did you ever like have land parties yes a few times um not very often because it's the nerdy kid in middle school in high school didn't have that many friends but really every once in a while yeah um i was in band okay this is where this question's coming from we were all nerdy and yeah and we were all nerdy and we we had massive land parties at this one guy's house anyway yeah we had there were a few um just a few like okay we'll get together and do three or four when i think lan party i'm thinking like you know 10 20 people or so and there was only like one or two that i could think of where we did that um which were you know really really fun um but no it was a lot of it was just like oh yeah i've got a handful of friends that happen to play the same video game so maybe we'll meet up and play and every once in a while yeah it's funny you mentioned i was a i was a chorus kid actually in high school so yeah i was gonna say all right now the tell all secrets are coming out i knew there had to be something wrong with you i was going to say every actually everybody in band was weird to me except for um oh what is it called the percussion section yeah it's like trump yeah or whatever they're called um those kids were always really cool they were always the cool friends yeah drumline's percussion section is always the the chill ones yeah sometimes but yeah saxophones as well all right we're done talking about band i'm so sorry yeah i really just wanted to know if you like did the land thing because that was the okay yeah right don't don't yeah land parties are the one thing that you know maybe high school kids just do without realizing they're learning i don't know yeah don't cut out the band bit dan we need to keep that in the show hey y'all it's lexi aka track it pacer or as my most recent burner account hater described me quote she can't configure a switch for shit i'm really sorry to interrupt you in the middle of the podcast but i have a cool story to tell you and it'll only take a minute and then we'll get back to the show okay so i come from a knock background i started out in a knock and the most annoying thing that would happen back then is when a user would come in complaining about a problem blaming the network but our network monitoring system was showing zero signs that anything was wrong and suspend your disbelief for just a moment and we're assuming that the network actually is at fault okay sadly most network monitoring software really only pings devices and shows you utilization on maybe a few interfaces but overall this doesn't really help us in the end what if you knew everything that your network equipment knows like everything path solutions actually has a product that does this and it's called total view it provides automated root cause troubleshooting of the entire network i'm talking monitoring every interface on every switch router gateway and firewall it also goes deep collecting performance data 19 different error counters qos cdp ldp and poe on every interface the whole kit and caboodle totalview is also smart it has a built-in heuristics engine that analyzes the information to provide you the user with plain english root cause resolutions to problems that other nms systems really wouldn't even be aware of you're responsible for the entire network right so shouldn't you have visibility that matches your responsibility visit passsolutions.com today to learn more about total view and get total network visibility on your network now back to the show so what did uh what did post high school look like for you so post high school um so my family was relatively poor growing up um we were stuck in the awkward like poverty range where it's like you are uh in america where you're too poor for benefits but you don't make enough to really like actually kind of sort of make ends meet um so a lot through high school i worked as a i don't really know what the title would be an assistant something like that in a local nursery a plant nursery um so i was a lot of my days after school were out working with you know landscaping and not landscaping but like you know watering plants helping people carry stuff to their car and load stuff and that kind of thing um that was all fine and good until i graduated high school and i went to a local community college for probably a semester or two um and i realized money doesn't come from nowhere having that extra money when you're in high school is really nice but then you start kind of getting saddle responsibilities and you've got student debts that you need to start paying off and you know textbooks stuff and it's like okay well i'm hungry at school and i don't have any lunch like i need to get some income to start paying for this stuff right um so so and i i'll preface all of this by saying i'm incredibly lucky like you know like really really lucky almost under un undeservingly lucky where we lived there was a um like a local computer store probably about like half a mile maybe a mile away from where we lived um think like computer repair and that sort of thing although they sold like individual parts and that kind of thing um and so i on a whim it was just some winter day in like december something like that i walked over filled on an application and you know walked away and um they called me back for an interview a few days later or something like that and um they had said like hey you did like really well they had like a test or something that they give you to kind of filter out like you know people who think they know it but don't really and that sort of thing that sounds a little gatekeepy but it's kind of like it sounds like a shop it sounds like your first tech interview to be honest yes in a way it was and it was just like hey you know fill out this questionnaire with the best answers you got and you know we'll come back to you with your and your information and we'll come back to you if you um if we think you qualify um and they had said yeah you basically like knocked out of the park with this test when they called me back i was like oh okay i'm like does that mean i have a job and they're like yep nice um so i started there and this was before i i didn't i didn't have a car or anything like that so for the first probably like six or nine months or so i i just walked to the mile which was either actually kind of nice um and you know put in my time there and that was where um a lot of the experience that i had gotten from middle school in high school like really started clicking because i got to see a wide variety of problems um and i got to deal with uh customers in i.t and i got to learn that you get to learn the delicate balance of like uh dealing with customers in from an i.t perspective as well as like the retail perspective right so i got a little bit of the best of both worlds there um the absolute best and um that was i think i did that for about a year and a half or so and that was a looking back like a really fun experience because you know i got to fix computers all day i'm 18 years old 19 years old people bring me their computers i you know i fix them i figure out what's wrong with them you sometimes it's some you know really basic stuff sometimes it's you know kind of gnarly and you just can't figure it out right um that was a really crucial like entry level step i think for me um as with all mentor level jobs pay wasn't great i mean it was at the end of the day it was still like a retail job um but just having the opportunity to talk tech with sometimes you know the ordinary joe who walks in with a computer problem sometimes it's you know a quote-unquote fellow nerd who comes in wants to buy a new processor or something like that and being able to talk like oh yeah here's the you know this one's pretty good like for your budget range you probably want this like i don't i was living the dream back then uh that was it was a lot of fun awesome okay so going into did you go you said you took a few courses at community college did you go to university what did you end up doing after you know after that job my journey in academia is a sad story oh i'm sorry um beautiful so i no not it's not not sad like i'm emotional sad more of a um you you learn a lot when you're young right so when i went to community college initially i wanted to do i thought i wanted to do computer science um so i thought that i wanted to go to north carolina state university i'm based in north carolina which has a really good engineering and computer science program i wanted to do that out of high school but because i decided to prioritize video games after school instead of homework and doing what i wanted instead of what was good for my academic career my grades weren't up to snuff for that um so well you know what's a kid to do community college right it's cheap it'll it'll it'll get you going right um so i did that for for two years um i got my associates of science because i thought that i wanted to transfer into nc state um and they did not accept me because surprise surprise i didn't learn my lesson from high school it turns out that even in college you need to prioritize studies in order to do well in life so nc state wouldn't take me i said okay well what else can i do with this right i still need to move you know up still want to do something related to technology i.t computer science something along those lines um and i was a little bit turned off from programming from previous experiences where i tried and didn't really learn i was like eh i really like the whole like fixing computers aspect of this i want to see if i can do something with that instead of computer science which is much obviously much more programming heavy so um i looked at another university nearby east carolina university and they had a really good it program um something my high school teacher had told one of my high school teachers that told me was that ecu in north carolina for those who don't know there's a lot of rivalry between colleges so we'll all riff on each other right um i think that teacher who is from unc said oh ecu they take anybody with a beating heart um i have a beating heart actually so they did accept my application but for the i.t the specific it program that they that i wanted to go into i needed an associates of applied science i had an associates of science what what is the difference i don't know to this day i'm glad that's the answer because i was sweating over here going i'm not gonna say i don't know what it is but there's the associates of applied science is um uh has a it has a little bit more focus on like i.t specific stuff there was a specific pipeline to get into this bachelor's program that i wanted to get into for it and i did not follow that pipeline i just carried on with my normal generic two-year associate's degree and thought that would do it for me the sad part is well the kind of emotional sad part is um east carolina accepted my application and i thought i was good to go until my first meeting with my advisor and they said oh oh this isn't going to work you don't have any of the prerequisites for this bachelors and i was like what do you mean i'm like you you have the wrong degree like we shouldn't even have even accepted you i was like oh so um to make a long story short i got punted back to community college for another two years to get my associates of applied science no way and you stuck with that that's when you have to say no takes and actually with it you can't send me away you already accepted me it's over god that sucks i'm sorry but you you did it you did it i i did it and it was actually while the path sounds like it sucks and it kind of did for a while i don't think i would change it and here's why while i was doing my associates of applied science um a really weird sounding company by the name of cisco happened to do a recruiting event and literally on a whim i didn't see it ahead of time and planned for it i think i saw it like the the day of or maybe the night before and i said um yeah sure i'll i'll go do that um in but this was after i had uh left my job at the computer repair place i did a little bit of consulting for a very very small managed services provider in the meantime doing like micro business consult i.t consulting for like dentist offices and um we had a warehouse customer and just a bunch of different types of businesses so just on a whim i went to to to this event um and you know gave them my resume and talked to them and this was for a co-op position in their um in their lab they at the time they called it kayla i think that's customer advocacy lab operations they've rebranded now um for a co-op position where i would basically just be helping out tech engineers and consulting engineers and that kind of thing with creating uh smaller live replicas of customers networks and i was like yeah sure that sounds neat i mean it's a it's cisco right you know if you want to do it you've heard of cisco so sure i'll show up and talk to them and see what's up um and they signed me up for an interview that day oh wow i guess they saw my resume and was like oh you've had experience for like two and a half years at this point like yeah let's get you on board right um so i want to say that was 20 i was probably mid or late 2016 and then i started as a co-op in 2017 um doing lab stuff for um cisco how was that that sounds like it'd be fun it was um it was it was a lot of fun um like looking back on it it was a lot of fun doing the work itself um i'll just i'll never forget like the first day that i walked into the into the lab right which is um half of the uh half of a floor's worth of just racks and racks and racks of equipment most of it's cisco basically every single product that they've manufactured that does routing switching i mean everything honestly they just didn't have like collaboration stuff in that specific lab everything that they've made for the past like 20 years wow it's all in there yeah and it's just like and you you walk in and just i mean it's a data center essentially and the noise just overwhelms you and you just you know the screaming of the fans and it's like a little bit high and you're like oh okay and you're just walking through like row after row after row of racks full of equipment and me being an i.t student who hadn't really done um again this sounds kind of gatekeepy but like real i.t because i was doing consulting for very small businesses like 10 20 employee businesses where you know we buy netgear switches right and we we throw them in there right it's not it's very much a get the job done kind of business um to walk into this this was i wouldn't even call it a data center now because it's not really a data center i thought this was a data center and i was like this is cool this is what i want to do i want to touch this stuff this is cool um so i was i was basically infatuated with it as soon as i walked in i was like yeah i mean you get to play with all the stuff i get to cable it up for people i mean yeah sign me up absolutely um once i got into the job it was um i don't want to say monotonous i don't want to say like that to discredit it um but it it comes down to like okay load this version of go find this router this modern router and this model switch and connect them in this way and load the software on there and then you know make sure your connections are up and there you go we only only handled the physical layer aspect of it making sure that those links were up we didn't do very much in the way of configuring those devices or actually mimicking the customer's network and that makes total logical sense um because you don't want the college co-op to be bringing up a mimicked topology of an mpo sl3 vpn right because these are two very different things the skill set's not quite there if you can do it sure but chances are pretty good that you know you don't have the skill set there right well so okay that's pretty neat so you so how did you progress then from that position onward is was it from there you moved into tech yes um so we cisco has a very good pipeline um for college graduates well for for people in college you can get a co-op position and do a lot of cool stuff there's lots of intern um opportunities there there's also a very good pipeline to get college graduates into either tech or our professional services organization just kind of depends on your skill set and your personal preference as a co-op i was working very closely with a lot of tac engineers from a lot of different product lines and you start to get an idea of like okay i like the nexus stuff that stuff's really cool you know of course i walk into this in this lab and i'm like this is a data center so i'm like of course i'm i'm inclined to go for the data center stuff right because i think that's cool but you're like oh i like the nexus stuff like that stuff let me you know get closer with the data center team and let me in a way cherry pick a lot of the cases that they're opening with us to ask them to to ask us to cable up certain things with them right um so after getting a little bit closer with them and working with the team a little bit more um i said okay this is the team that i want to go to right and a number of the engineers on that team um liked me so much i guess or or thought my work of sufficient was of sufficient quality or whatever that they would kind of clue me in and be like oh this is what we're trying to recreate for this or they would you know help me work with the topology a little bit more than was expected of us right man those people are the best people whenever you encounter them they really are yeah yes and i as i move up in my career i'm trying to always look for opportunities to reach back down the ladder and pull people up with me right um because that's how i got to where i am um you you ask any of them they'll say um oh you you know with your skill set you probably would have been fine or whatever but that's just it's just not the case right um you you need to find people you don't need to but it makes a lot easier if you find those people and they are able to help bring you up with them um but yeah i got close with them and then they um heavily encouraged me to not only go through that new grad uh new college grad pipeline but also hand-picked me and said i want this person on my team so they they said you're coming to this team you're already familiar with the product line we already know who you are we know you're you know a decent human being come join us specifically right um okay so that happened and um oh yeah i mean it always feels great when you are hand-picked to uh when you're basically groomed and happy to come to a specific team right yeah um so yeah fast forward to to now i suppose after after my time wow yeah you did you've just like moved on up that's a really cool story and it's i really look you know i i went to college but i you know didn't get a degree in anything stem and so i i you know i feel like i can relate um when it's sort of like a i don't know do we still consider it like non-traditional for people to like you know be working in in networking but not have gotten a degree in it or something like i don't really know if that's real i feel like that's more the norm these days than anything else to be honest but i i don't know um i almost want to say that it is becoming more of an expectation that that people have college degrees really um i i think so i think if we were to go back like 10 or so years or so it would be like you would see more people without degrees than people with um and this is the do you get the degree or not to get the degree it's just a it's a what a topic right like yeah we don't need to we don't have to have that conversation here that's a huge my take on that whole thing is it helps out a lot to have a college degree so something something i'll mention is that when i eventually got my associates of applied science because there's a difference apparently and once you went to ecu i did all my classes virtually so my entire the entire rest of my bachelors was all done virtually because i was already at that point i was already a co-op um well no actually when i applied to ecu i was a tech engineer so i was doing college virtually in the evenings and then somehow learning what a nexus is during the day wow um still don't know by the way and so technically i didn't have a bachelor's degree when i was in tech and one of the things that i was wondering because because doing college when you already have a full-time job and tech is a demanding job you ask any any support engineer and they will tell you that um that was a lot that was quite a bit um and i'll be honest i probably don't deserve that bachelor's degree because i did not put in a ton of work in order to get that because because i by the time the evenings rolled around or the weekends rolled around i was so tired like you know i did not have the energy for it um so i somehow i got it but i probably did not put in as much energy as i deserve as it deserves um one thing that i had contemplated and had talked with a few people with intact about it including managers and you know upper leadership was do i keep going for my bachelor's is there actually going to be any value if i keep doing it and a lot of people said we know you you're a really good person hard work or whatever you'll probably do fine without it you should probably get it because because i do they all said i don't want you to not be able to get a position higher up in this company because you don't have it that would just suck if you suddenly hit a ceiling where it's like no sorry you can't go any higher than this because you just don't have a four-year degree and um especially in a larger corporation that is that could very well be a reality right um so i said fine and i kept going for it um sorry i'm just curious how long it takes you to once you were working in uh working at cisco and taking the courses how long did it take you to complete it i did it in the in the two years i think it was a two year spin yeah um i have been as a result of my as you can probably imagine as a result of my entire academic career i am burnt out on academia maybe that old chat attitude will change in like five or ten years because i know some people have said you know oh you should consider a masters or something like that there's a number of very of people in this industry that i respect quite a bit who have a doctorate um and dr hart really has a nice feeling to it but yeah but um for now i just um i i can't do it it's just i don't have it in me just yet there's so much else that i can do that i want to do before i make that a goal um but i would say you know to anybody up and coming i would say if you're considering like oh do i get a certification or do i go for my college degree or something like that i would say both i try to do both but i would say go for the bachelor's degree because from what i've seen that's becoming more and more of a hard requirement where they just they just won't let you in if you don't have one it doesn't matter if you're god's gift to it you don't get in you're saying so what uh kinds of bachelor what flavors of degrees should people be potentially looking at do you think i've that's a good question um like you know mine is in english literature so like obviously not that yeah well so you say that but i i actually disagree i think that if you have a bachelor's of any kind then you could like that that qualifies you so i've seen people who have transitioned careers similarly to yourself that just because they if they have a bachelor's of anything that is good enough good enough right well that's that makes me feel better for sure um yeah but that's that's i would say bachelors of i.t i think that the academic world has a little bit of work to do to figure out what that means before that has any real weight to it compared to a bachelor's of computer science or something along those lines now granted that was my attitude when i was going through college things have changed the past few years there's a number of schools out there um the one that really comes to mind is uh what is it called wgu western governor's university it's like a total i think it's all online and um i've known a few people who have gone through that really really good school i think rit up in new york is another one i don't know if they're all online but i've seen some really good people come out of that school and i think that they're an i.t focused so or one of their programs is it focused and i mean they they train you and you know your stuff heard really good things about them yeah but i've also heard i've heard a number of other people who have an opinion of bachelors of i.t doesn't mean anything like because because i t itself is such a huge range of possible careers what could that possibly mean even like a bachelor's of science in i.t security is so vague that what do you actually know as a result of that right um so yeah when i was if anybody wonders mine is a bachelor's of technically a bachelor's of i.t i think the full title is bachelors of industrial technology with a concentration and information technology something like industrial technology because the way ecu um they ropes that in i think with like a few other different kinds of engineering like uh engineering like textiles or something along those lines there were different verticals under that department that offered that um but it's it's a bachelor's of of science for i t is is basically what it is cool so did you study a lot of different like i so i took some community college courses in 2018 when i started over right and i only got as far as like one like one and a half semesters um but the courses i took for that first semester of i think it was a in associates in information technology right the courses that i was required to take were introduction to programming introduction to computers which is like this is how excel works and then introduction networking so those are all like very broad that's like a very broad spread i feel like you basically have like computer science and then you have network engineering sort of and whatever intro to computers was like that's all very different um yeah so my associates the associates of applied science was um basically the cisco netacad course spread out across like for courses or whatever i think there was no that wasn't in that one uh there were some computer sciency ones like oh here's java here's um i think c either c or c plus plus was one of them and then there were a few others i think there was like a databases one it's just it's just entry-level stuff of a few different verticals within it just to get you started um and then the bachelors was the same thing except at the intermediate level so like ccmp level courses um from through netacad there were a handful of linux courses that i took in both degrees now that i think about it but i think the um the ecu courses were more red hat oriented they're through i think red hat has a curriculum um i think there was another database as well there was a storage one which i thought was interesting because now i know what flogging means that's about all i know about storage um it and yeah i did not like that class i was like why aren't you just doing ethernet ethernet works we know what ethernet is um so yeah it's it's kind of even even those classes you can see are kind of spread out amongst multiple different disciplines which on one hand means oh well you're not specialized so you know what do you know but on the other hand um kind of know like a little bit of everything right so i can choose you know obviously you can choose what courses you want so if you like the networking then you can take the networking specifically but you still kind of are forced into doing a little bit of linux and a little bit of programming and this and that and you you widen your experience um and that's super helpful in the real world i think um even within tac i can't tell you vmware classes is another example that just come that i just thought of i have solved a number of vmware attack cases i don't work for vmware um they called into cisco and i through that i you know the the esxi host connects to the nexus and they had some issue with it so i said well show me the esxi host i'm not scared of vmware you know just because it doesn't have cisco on label doesn't mean i'm here to solve your issue right um and figured out oh yeah misconfigured subnet mask or you know whatever the case might be um so having that that breadth of experience can help out even when you choose to specialize in one specific thing like networking yeah absolutely i can't tell you how many times i wish that i had you know taken a broader you know i don't know broader discipline yeah i think it definitely helps you open your eyes and that's why i think we we're all kind of big proponents of when you're starting out don't necessarily shy away from the service desk the help desk the desktop support because it really helps broaden your your horizons to different things um there is one thing that i wanted to highlight christopher and it's that the the school that has the mascot of a pirate told you that you didn't have the right degree to get in the first time around but just kidding ecu go pirates but um do you mind despite the fact as a teenager that i had pirated a lot of content by that it's really weird i'm the so do perfect mind you talked about the uh tech engineer position being high-paced being exhausting do you mind kind of unpacking the uh just kind of the whole experience you don't have to get into individual cases unless you've got some juicy ones that you don't mind uh sharing but i just kind of want to know how did how did you psych yourself up for that role did you know what you were getting into what was it like i so through because i had um had the co-op experience and was able to work get very close with a lot of these tech engineers um it got to the point where i was shadowing some of them on cases during my lunch break as a co-op where i would just be like hey what you're working on and you know they'd be on a call or something along those lines so i more than most i got i knew what i was getting into right so i was i was mentally prepared to get into that right um it is a it's a difficult job i'll put it that way um if i was a like a if there was a greek god of i.t and i was somehow chosen to be and that was omnipotent ignition and everything and i just had my way i would make everybody in it work two jobs the first one would be outside of it it'd be retail or food service because you just everybody should work one of those jobs the second would be working as a support team for a vendor wherein that vendor owns that solution or that product or something along those lines meaning the buck stops with you if a customer opens up a case with you you are responsible for figuring out that solution you will learn more about how that product that solution and how that vertical and i t whether it's networking or security or whatever you will learn how that stuff really works outside of what you might learn in a certification outside of even what you might work as if you were a customer in that case if you were in a managed service provider or a partner or or whatever the case is um because until you're put in that situation that high pressure situation where the buck stops with you and you have to have the answer even if you don't necessarily know not only the answer but even how to get to the answer um that you'll learn a lot through that experience i would not trade my time intact for anything i think it is like more than anything else in my career i think it is responsible for propelling propelling me forward and my knowledge of networking forward did it help you um did it help you like develop a process for troubleshooting like a yes oh yes you know yeah like a troubleshooting stack basically um one day i want to write a book on that i'll put it that way because there is definitely like a very problem or platform or technology independent method for troubleshooting a problem um that i think a lot of people go through and they don't realize it or they never they never go through it and they never realize that there is a process right um a lot of people that i've a lot of i want to be careful here because i don't want to insinuate that customers are bad at troubleshooting or that anybody is bad at troubleshooting right because i've been put through my experiences in a unique position where because the buck stops here i've gotten to see a large number of different problems and had to figure out how to troubleshoot them right a lot of people have a troubleshooting approach that i like to call darts at a dartboard they're given a problem and they say all right well let me try this all right let me try this let me try this there's not much rhyme or reason to it a lot of it is just kind of based on past experiences where it's like well i tried this one time and it solved some other problems let me try it now i'm going to try this they don't really analytically tackle the actual problem itself something that i like to tell a lot of junior engineers going into tech um is that as a tech engineer you're not actually like you might think you're a tac engineer you're not actually a tech engineer you're something else that something else varies in this case you're a lawyer you have quite literally a support case you have a problem put in front of you and a series of facts and or evidence supporting that problem and it is your job to go through and make sure that all of that evidence is true is factual and actually corresponds with the problem and how that corresponds with the symptoms the customer is seeing and you need to be able to prove without a shadow of a doubt that all of that evidence matches such that a specific root cause for a problem is like undeniably like i said without a shadow of a doubt that is the root cause right um and there have been so so many times where um i think it was like john adams who said that um facts are are inconvenient things they're fickle things they don't they don't go away right there have been so many times where i've thought that the root cause of a problem is something and i have like nine different pieces of evidence that all point to that one thing but there is one other piece of evidence that was like that just doesn't it does not fit into the puzzle it just doesn't fit so it can't be that and you have to kind of go back and re-analyze everything and do some more research and do some more testing and try and figure out what the that missing piece is and figure out what you're missing um there's a bit of a bit of a mind game that goes on there too i can imagine just one big puzzle wow yeah i'm attack sounds like the ultimate place to learn how to troubleshoot for sure yeah yep and under pressure yeah well yeah you're definitely under pressure because i think i think like i said before sometimes you need to solve these things in a matter of hours sometimes it's days sometimes it's months i want to say the oldest case i ever had don't know if it was start to it wasn't start to finish for sure but the oldest case i ever closed was two years old wow was it like fairly active like were there updates on it like you know yes there were posit like multi-month pauses because we're pending some action or something along those lines so that kind of you know it's a little bit of an inflated number but still it you know it is what it is and i would say in those scenarios props to the customers honestly because not only are they having to deal with the actual issue but they're for lack of a better word putting up with the process the support process and um the the the rigorous finite state machine of finally getting down to what the root cause of a problem is hey a1 fans aj here for net ally you ever heard of net ally sure you have they came from the same group of engineers that brought us network tools from fluke networks netscout and now their net ally they know networking i'm a network engineer for a partner and when i go to customers and see they use netaly i know it's going to be so much easier to troubleshoot issues we might run into the name may have changed in an ally but the way they build tools hasn't changed a bit they ask what would a network engineer want to help make their job faster and easier and then they go build it just like this etherscope nxg netally is here to help net ally simplicity visibility collaboration visit netally.com today now back to the show i'd like to get your thoughts on on uh on handling pressure uh and and maybe some advice because us in the in the enterprise space you know we hopefully don't have to deal with those downtime bridges that we got to be on people looking over your shoulder all the time we hope to not have to deal with those too often you could be dealing with those intact on a daily basis with a bunch of different customers so what are your what are your thoughts what's your advice on on folks handling that stress and still being able to function yes so what i like what i've told a number of customers is that whenever they're in that kind of high pressure outage situation is basically what you exactly what you said i tell them this is probably the worst outage you've had in the past six months in the past year two years for me this is a tuesday i love that like it is just this is what i do every single day right um as something that i preach to to all of the engineers that i've trained is a sense of ownership right and there's a um a sense of ownership and a sense of presence when you're on a call when you're attack well and i like to think especially coming from cisco the reason why customers tend to choose cisco is the um the sense of ownership that attack engineer can have when they're driving a multi-vendor call or any call really but um when when you get a good engineer you can tell because they take control of the call doesn't matter if there's five people in it one person 10 50 200. it doesn't really matter right they are driving the troubleshooting they're driving the analysis of a problem from start to finish they're getting the facts out of people and and sometimes they have to do the very uncomfortable thing where they have to hold a customer just to to the wall and say i need you this information and i need you to make sure that it's accurate because this is going to skew our troubleshooting for the next hour two hour three hours and your business is down and you're losing money as we speak so i need you to make sure that this is accurate so sometimes and i'll admit this readily sometimes i've been stern for lack of a better word with a customer who has tried to distract the troubleshooting to go look at some other problem that seems unrelated or something along those lines or we'll be troubleshooting like a specific traffic flow that's that's broken and they'll say oh well here's another flow that's broken and i'll say no no no we're gonna focus on this one flow and we're gonna trace it all the way through we're going to isolate it we're going to figure out what's going on because chances are pretty good if we fix this one flow we're going to fix all 10 000 of the other broken flows right and if we don't that's fine we'll troubleshoot the ones that are still broken afterwards right but you have to sometimes you have to be very stern and really own and drive that issue to its total completion um and in my opinion that includes when other vendors are need to troubleshoot something so i've been on troubleshooting calls with vmware with dell with arista that sort of thing where even if they're troubleshooting their box i am still actively engaged in the troubleshooting process so i'm still giving them the facts as i know it if they ask questions i'll try to respond with with the answers as i know it i'm there ultimately i'm there to solve the problem sometimes that problem is cisco's fault sometimes it's somebody else's fault i don't really care i just want the problem solved permanently not like oh let's reload the switch and hope it goes away kind of thing um so i've heard a lot of people in my time complain about providing show texts to tech and and i just want to ask you straight from the horse's mouth i like i've been given the answer before but i think it'd be fun to ask anyway why do we need to provide show techs with every tack case that we open excellent question um i don't want to say every single tech case okay but i would say most tech cases please do open them with a show tech because for for a few different reasons one being able to see that information from a single device gives me an idea as to obviously what that device's configuration is but also the general health of that device at that current point in time as well as depending on the product some point in time in the past so it helps give me a little bit of context behind what uh how the device is feeling for lack of a better word at that moment in time um so a showtek is a collection of commands essentially right that yes usually a large bundle of a bunch of different commands and the important part is that it's a selection of commands that are going to be important to both me as attack engineer as well as to cisco's engineering if we ever get up to that level right so it's just a general good snapshot of we'll just you know grab that and we'll get an idea as to where the box is at so the next question is probably like okay well if you're just looking at you know the log and the config and cpu utilization why can't i just run those commands and give them to you um trying to think of a way to say this as nicely as possible no there's actually a really good way i can say this i if you give me a some arbitrary problem uh i can probably tell you five six seven ten commands that i could use to troubleshoot that off the top of my head what i can't tell you is what commands i'll need to look at after those ten commands so usually a showtek will give me that ability oh the cpu you know the pro just using an arbitrary example this interface is down why is this interface down we can go look at the log okay and then based on the error in that we can look at a specific log from a specific process and so on so forth you can do that that sort of recursive investigation within a single file from a single device and it's all there for you right as opposed to okay you know show run and show interface cool i can see this interface is down hey can you get me the cpu utilization oh look it's this process so you know and it saves some back and forth uh is it going to save all of the back and forth depends on the problem sometimes yes sometimes no um is it going to save some back and forth most of the time yes it also helps out a lot when engineers um when ownership of a case moves from one engineer to another i always really really liked it when there was a set of show texts from a bunch of different devices in a case that i took ownership from somebody else because then i can look at those ahead of time without having to ask the customer for to grab them for me because that can also be a frustrating experience from a customer's point of view where it's like okay well you're the second or third person i've worked with why are we just now asking for these show texts um and i know some people listening will be like oh well you know attack has done that to me before where they've asked for a fresh set of show texts and yes that happens and i would say i've done it myself not because i didn't look at previous show texts but because i looked at them and i wanted to see a fresh snapshot of the issue because the issue has evolved since the case was initially open so i want to get another fresh snapshot just so i can refer or re-establish my contacts with the issue so it's a long-form answer there's not a simple answer unfortunately um the real solution to this problem the real networking nirvana we should all hope and pray for is a scenario where a um and amazon has actually done this it's a scenario where i have direct access to your device and i can pull whatever commands i need whether that's you know wherever your device is it's sort of like an always-on telemetry back to me where i can just fetch whatever i need to um i say amazon has done this because i mean they run aws you got an issue with aws they have access to all of that stuff it's their stuff right um if you open up a case with cisco it's our stuff but i can't ssh into your box right and i'll say this just as a you know free plug for cisco um some of our products do this um if you run ucs servers ucs manager or any of those ucs domains we have a product called intersight which does precisely this it can hook into all of your it will help you manage your ucs stuff as well and it can hook back into cisco's network so that we can dynamically pull show text and stuff from any of your ucs domains without you and actually it's done automatically whenever you open up a case if your serial number matches the um a product in the ucs domain that is being managed by intersight and so on so forth very cool so the future is now you said a few minutes ago that you want to know how the device is feeling i love that we're personifying the nexus now that's excellent oh i do that all the time yes yes i i give i give my devices pronouns so we are we are kind of getting close to the end here are there any any funny stories or or any anecdotes that you want to share from tack i have so many stories um that's it the one so there's one towards like i want to say the last six months or so of my of my career there was one case that really like um it really got me and i can't tell you all the solution uh but i can describe something that we were considering when troubleshooting so remember earlier i said we're we're not actually tac engineers we're lawyers right um given a problem we have to look at a bunch of different evidence and rule out what could be related and what could not be related there was this one customer who was having uh seemingly random links on their nexus on a bunch of random nexus switches which spanned multiple different like factory batches and stuff both the the um the nexus switches themselves and the transceivers uh cables were like all different it seemed like totally random every once in a while links would bounce and sometimes they would bounce a couple of times within a week sometimes we you know replace those cables and they would be three four five six months before they bounce again so from a customer's perspective this was very unusual because they'd open up a case it seemed like just the link was unstable we'd do some troubleshooting problem would go away just as a goodwill we'd replace some transceivers okay cool three months later same issue happens again but somewhere else some other set of links some other different set of switches same thing and because that issue is so intermittent it was really really difficult to track down what was related what was not so on so forth it got to the point where we had geographically identified we had asked the customer basically where these data centers were where these locations were and they were not data centers but offices actually spread out within a certain city and we were wondering maybe maybe the switches are like being exposed to some environmental factor right maybe the temperature is too hot or too cold or maybe it's just right you know like the three bears that causes these transceivers to flap and we're like we tried to look at temperature data nothing was really matching up like okay well it doesn't seem it because there's like no there's no data to support that what other environment mineral factors could there be what about vibration and we were like well there's no earthquakes recorded near that area it's not like a proper data center so i wouldn't expect them to have like you know all the tolerances that a data center would have to that sort of thing um they were in the midwest so it's like you know earthquakes maybe maybe not but either way there's no recorded activity but what else could cause like you know switches to vibrate um there any trains nearby we're like we're really entertaining this and we we go and look on google maps and within a mile of one of the offices there's a set of train tracks and we're like no no way and then we go and look at the second site because they were having this issue at two sites only and it was in the in their backyard basically was a highway and at this point we're like no way it's there's just no and so we had to we were talking with their account team and we're like this is a theory that we kind of have to entertain we want to run it by you first before we ask customer but i think we ended up we did have to ask us for any known like there's a truck going by or is a train going by whenever these links happen to fly god's telling me we were trying to track down the train schedules we were trying you've got to be quiet during this time we were trying to yeah luckily that wasn't the case it wasn't that i can't like i said i can't tell you the solution but it was not that um but we have historical precedence for this sort of thing happening there's a good story one of our uh vu escalation engineers likes to tell where he had gotten um the customer switch was having a similar kind of weird issue where um you know links were flapping and they had gotten the customer switch in the lab because the same switch that had the issue they put in the lab they were trying to reproduce it with like similar set of transceivers and cables and they weren't able to um and then the the engineer he had put a cell phone on top of the switch and while he was like on the cli or whatever his wife called him and the vibration caused the link to flap wow yes so in that scenario what happened was um i think it was the transceivers were manufactured and this was a specific like a one specific transceiver or set of transceivers they were manufactured to the minimum specification and the slots in the switch were manufactured to the maximum specification and this was all within like tolerances from the factory i'm pretty sure cisco does the whole six sigma thing so it was within tolerances acceptable tolerances but because they were at the very edges of both whenever there was vibration just the slightest vibration the transceiver would disconnect a little bit in the it would jostle inside of the slot and disconnect it momentarily that's nuts so we knew that that had happened in the past and so when we were talking about vibration we're like i mean we have to entertain this like as a possibility yeah damn choo choos so yes we had to yeah we had to ask the customer like hey any trains go by recently yeah it's um how did you even prove that out with like just look at the train schedule and it didn't line up like they they had told us like no because sometimes they go by like during you know like 3am or something like that and something i i don't remember how but they had basically proved out like yeah no there's no that that can't be it we were like well that's disappointing yeah because it'd be really cool for me to say root cause analysis the train went by right but lexi don't don't we have to chalk that up to solar flares i thought it was solar flares usually solar flares right before yeah i i okay okay back in the previous job we we i i heard of the solar flare like root cause as like a joke everyone would joke about it like oh what caused that ha ha solar flares right but then i started working in a rocket company and they're like oh yeah that happens that happens all the time absolutely errors are the absolute worst um i've had a major escalation on that that i can't talk about but uncorrectable parody errors are are absolutely terrible they're a thing these cases sound so interesting and you can't yeah yeah so christopher to kind of round this out um you obviously recently made a career change from tac into your new role what was it was it just time to leave tack for you did this role come up and it just felt like the right fit what was the scenario a bit of both um we within tech we always like to joke that like tech engineers have a shelf life um and there are some exceptions to the rule but usually it's like around the two to four year mark or so um so it was it was about that time this opportunity came up um something that i had um in my my last year intact was a little bit weird i deviated away from from taking cases every day to more of like a kind of sort of helping out with escalations kind of sort of helping out with like systemic um systemic product quality issues and just generally like improving stuff and some automation initiatives in terms of the company and that sort of thing um so i had already kind of been a little bit stepped away from the support case role or just you know taking cases day in day out and then this opportunity came up i had really wanted to get into hardcore network automation which this role allows for a lot of our testing and such is fully automated and so i part of my role is building that automation to automate that testing from end to end um that and i wanted to expand my skill set a little bit more so i know quite a bit of routing and switching i'm not afraid of any any route or any switch anywhere i can usually hop on them and do the best i can um but in the data center space there's there's a limited set of technologies and eventually you get um you know you get to know them right this would let me expand in this role let me expand into other technologies like sd-wan i can have the opportunity to learn like aci um it's another one right now i'm working on a a project with nso which i've talked to a few people in the in the community about um and cisco frostwork there's just a lot of different stuff that um that'll be able to dip my toes in and get a little bit more familiar branded not at the level that attack engineer would be able to but at some point like you can't it's actually not a good idea to heavily specialize in a specific product for a very long period of time um because you can't uh you can't do anything with that skill set um most customers don't want to hire somebody who knows how to troubleshoot asics they want a network engineer right and there's a very different skill set between those two so sure i can tell you exactly you know what the forwarding pipeline looks like for a specific product line but that's not useful information to a customer they're like dude i just asked you to send a message right so yeah this role helps me expand that the networking skill set to include some more stuff and um that way i'll be able to give better advice and and so on so forth on how to design things how how to implement solutions that sort of thing that sounds really exciting yeah yeah thank you so is there any any last-minute things that you want to say anything we should have asked you that we didn't plug yourself your website any projects yeah i mean advice-wise i would say i mean i kind of um said it before ownership ownership ownership is just so important um so something i've noticed is that a lot of people um tend to i don't want to say like not do their job but they'll only take their job so far or they'll say you know oh that's not just outside of my skill set or outside of my responsibility or something along those lines um own what you do and just focus on becoming excellent at that and then you know go from there um i i've always prided myself when i was when i was in tech and even now on the level of ownership that i would have where it's like like i said before if if you come to me as a customer with a problem that your problem is now my problem it's not just my problem until i can prove it's not cisco's problem it's no you know let's let's get this fixed within a reasonable level of course i'm not gonna troubleshoot some weird bug on an arista switch or something along those lines um yeah ownership is a if you demonstrate an extreme sense of ownership um i like to think that people will recognize that and you'll move forward in your career pretty rapidly that's excellent advice yeah great advice uh where can where can people find you on the socials on the internet blog anything like that oh yeah um yeah so i'm most active on twitter you can find me at underscore chris j hart um linkedin you can try to connect with me i'm on uh i think it's christopher j heart95 and if those don't do it for you chrisjhart.com has my links to everything that's my blog that um all my stuff gets posted on excellent well thanks it's been great having you yeah thank you not a problem thank you for having me it's been awesome well that love hearing about tech stuff yeah this is this is really cool and i really love to get into that mindset because it's tech engineers are people that we all nit interface with some you know somewhat often and i always kind of high-levelly try to think about what's a day-to-day for somebody like that because it's just got to be nuts uh some days if not all day so really appreciate the perspective this was great also absolutely christopher i just want to say thank you for everything you do for the community you are a very helpful presence um i have definitely asked you several questions before or like a million um and you know you've you've helped me find some interesting at least dig deep into some interesting problems so i really appreciate that it's been fun and i know i know you do that for others as well so you're you're a great person my philosophy is i got where i am in my career thus far because other people decided to publish stuff content was before blog posts or answering questions on stack overflow and that kind of thing so i figure well i can do that too anybody can do it and hopefully i do it well you do you're very much appreciated we love you well christopher hart thank you very much for joining us lexi great show tonight thank you very much and to all the a1 fans and patreons thank you for joining us as always you can always check out our patreon program at patreon.com art of netenge we will see you all soon take care hey y'all this is lexi if you vibe with what you heard us talking about today we'd love for you to subscribe to our podcast in your favorite podcatcher also go ahead and hit that bell icon to make sure you're notified of all our future episodes right when they come out if you want to hear what we're talking about when we're not on the podcast you can totally follow us on twitter and instagram at art of neteng that's art of n-e-t-e-n-g you can also find a bunch more info about us and the podcast at artoofnetworkengineering.com thanks for listening you