this is the art of network engineering podcast in this podcast we'll explore teams technologies and talented people we aim to bring new information to expand your skill sets and toolbox and share the stories of fellow network engineers in a world where layer 2 extension reigns supreme spanning tree pushed to its limits and no overlay in sight where phones buzz with constant meeting notifications and you work right through lunch because well your office is in your kitchen now amid the ups and downs the trials and tribulations a podcast has emerged a show about network engineering and other stuff the cast of characters include the fearless leader the guy you find wherever books are sold the lovable beard the avid barbecue john and the guy whose greatest contributions are goofy introductions premiering right now and if you're lucky you didn't even have to listen to this because hopefully andy cut out this intro entirely this is the art of network engineering rated r for ridiculous welcome back to another episode of the art of network engineering thank you tim for that wonderful introduction i love how we just like do that live and roll right into the episode it's not like spliced in there there's no reason in 2021 to do anything absolutely at all live but here we are like a three-ring circus that's aaron aaron's back welcome back aaron how you been man i'm hanging in there like that poster uh with the two kittens on the tree branch hanging in there excellent it it's great to have a full crew tonight i love it it's been so long since we've had everybody here uh so welcome back aaron dan how you doing i'm doing good man yourself not too bad getting ready to you know rack up some more air miles tomorrow gonna go get some more hotel points this weekend got some more work to do in a data center somewhere in an undisclosed location i'm gonna play with some nexus vpc should be a lot of fun there you go vpc and let's see let's see andy how you doing good choice uh i'm good man i i i'm very happy that my pool is finally up i fought it for weeks there was every equipment involved there was mud there was pain it was a hard-fought dad battle and it's finally full and up so it's going to be 90 degrees in northeastern pa this weekend and what yeah well 93 so we we will have a pool this weekend uh so that is my highlight of my life right now thank you yeah there's like something like like super super hot or super cold the super super hot just like yeah there's no like spring or fall anymore it's either freezing or hot as hell and i don't know when that happened but it's the same thing happening up here in vermont so like like i just said i'm heading out for the weekend and my wife is like you need to put in the air conditioner before you leave because it's going to be stupid hot this weekend next week maybe it's going to be like in the 90s like starting tomorrow through the weekend oh yeah it felt pretty good in this in the afternoon yeah today was nice and last but not least tim how you doing i got in trouble this week aj oh no what'd you do well you remember on our time management episode last week i made the comment that i wasn't a big fan of when certain people put things on my desk he said maybe and maybe i didn't put it that nicely but you get to just you said she wasn't a listener right she's not a listener yeah she's not a listener um so i i went upstairs earlier this week uh yesterday and uh she's standing outside watching the kids play and she's got her little earbud in she's like hey i'm listening to your episode luckily i haven't seen her since 24 hours down in the basement are you at home or have you moved to an apartment yeah that's why i use the backdrop yeah i'm not willing to get into that right now he's in the back of his van down by a river i'm sorry things are great thank you that'll teach you that i'll teach you what about you aj i'm doing well i'm doing well yeah yeah you don't sound too bent out of shape but the traveling thing these days it was kind of really a thorn in your side there for a minute you uh kind of itching at it now or what i'm i'm at peace with it okay it's just a thing that it's just a thing now i can't fight it uh everybody on my team's doing it okay so i i don't feel like i should warm the bench anymore less than you know anybody else is so you know gotta do what i gotta do okay i can't can't beat them join them huh that's right that's right all right look at you adapt and overcome modern problems call for modern solutions excellent uh let's let's talk about socials getting social so um let's see we have our instagram page we have a facebook page we're definitely on twitter we've got our youtube channel and we've got a new look to our youtube channel so if you haven't checked it out yet please go over to youtube.com forward slash the art of network engineering and uh you will see all the new updates to our youtube videos so we got a beautiful animated logo and uh dan has done some nice work in photoshop we've got a beautiful overlay you're going to see in all of our videos going forward so uh we're just trying to you know crank things up a notch but just please subscribe follow like uh tweet and and i don't know whatever else you do on all those other social pages because all of those numbers help us out quite a bit uh this episode is sponsored by us we have a merch store if you haven't checked it out yet you can go to artofnetenge.com forward slash store and you will find lots of merch with our logo on it every purchase has a little bit i i've trimmed the fat i've taken as much profit out of every every product so that way we're giving you the lowest price and we're getting the minimal amount back for each product that you buy and with the product uh profit that we do get we're going to use that to cover our costs which are really not a whole lot for recording the show we may make some improvements from time to time like buying an animated logo uh and then everything else we're going to take and we're going to put it right back in the community as uh hopefully if you've followed us for a while you've seen us do before we gave away a devnet associate kit that included the books and a raspberry pi that was a huge hit we want to do that again someday uh so please if you want to help go buy some merch so really what you meant to say is this episode is sponsored by you yes potentially potentially yes that's right hopefully that's right please pretty please there's a new addition to the store there's there's a couple of new additions so we've added some tapestries you can get a beautiful tavis tree with the art of network engineering logo you can get a fleece blanket so if for some reason you're feeling cold when it's 93 degrees outside you can wrap yourself up in a fleece blanket with an a1 logo and feel extra toasty all right uh and there's now pint glasses oh see yeah i got five of those on order fellas and when they come in turn them right back around and send them out to you all and then we'll be able to see y'all gonna give us more for that nice it's very versatile those plant clothes yes you can drink water or other water beverages you need water absolutely you know what else do you know what else you can drink water water or water if you drink water then you can get an a1 pint glass i'm excited for that class do you drink salsa water salsa so you know i i think i think that's how somebody in massachusetts might say i'm not sure i'm not from boston man i know i know you're not but i'm thinking you know i'm trying to mesh the two together you're closer to boston than i am and thank god okay same team guys he started dad both of you two new england nerds keep it keep it down over there i don't know should i be concerned that right after i started on here full time aj goes out and gets pint glasses uh are you catering to me or something you just happen to you know also benefit from it yeah so lots of great new merch please please support uh and uh like aaron said this episode could be sponsored by you and when you buy something if you remember the discord you can put in your discord handle and a nice little shout out and we will see it it'll pop up in the uh a1 fans channel every purchase that gets made through the teespring store our merch store will pop up in that channel so everybody can see what you're buying i think we should add something new like when the winning goat gets their name maybe we could do like an aoni winning goat t-shirt or something like that give people an incentive to go buy something because they already have something we got some stuff in the works yeah nice once the goat has the name i want a winning goat t-shirt personally i still want to find where we can get some of these with our logo on it well if it uh if it gets the nickname that's in the lead right now um i wouldn't no i wouldn't either but it could it's just gonna be real interesting you know what i mean like well i have to like abbreviate it yeah right or just i like how do you like just cleverly like slap i just i don't think you can you know i just it is just is what it is that was great when i saw somebody like recommend that name i'm like that is perfect i know dude of course i voted for it and then i pumped it up but so if you haven't voted for that too by the way uh is it closed already it is not close well it so sometimes here's the conundrum by the time this episode yeah you know posts it will be closed but uh all right time travelers here yeah we are we are weird but um all right next on the list we are coming up on our uh anniversary date we've been doing this for 10 months now i'm very excited to share with you all that as of tonight moments ago i checked the stats we are at 83 000 total downloads since we started 10 months ago yeah you all made it happen that's amazing trying to hit that 100 k mark before the one year anniversary i think for the one year i'm not sure i'm not a betting man but i feel like it's gonna happen and last month was our best month right it kind of blew previous months out of the world yeah absolutely it was like 14. so tell your friends you know you listen you're at work tell them check it out you know i don't often give you recommendations you know me i'm a stand-up dude you can trust me go listen to these guys they're smart sometimes they just talk for no reason you know you'll love it i swear pint glasses anyway so uh coming up for episode 50 we want to do our first ask us anything episode if you have a question you're dying to ask either all of us or one of us uh here's what we want you to do we want you to record yourself asking the question you can do it audio only if you're feeling daring you can do it on video let us know that you're going to record something you can send it to us and we will work that into the episode and answer your questions so please get us your questions you can contact us through twitter dms on any social platform or you can find us you can also email us directly at art of network engineering at gmail.com you can also go to our website ardentworkengineering.com and there's a contact page just follow the little contact form and it will get us uh right to our inbox uh so please again ask us questions let us know who you're asking the question to whether it's you know one of us or all of us and then ask your question away and we will answer it on our ask us anything episode anything at all anything anything if you want to know how big andy's john is oh my god john is our sponsor i john like the way you use john it's like everything well the way you use it is a little resistance how you decide how i use it is up to you i can't help you the viewers just saw the panel's faces when you did that so that'll speak for itself i can't help that you guys are a bunch of dirty like people that's not my fault i mean could be talking about your chair could be talking about the monitors behind you i you use john in such a you universal fashion i don't know it just sounded dirty coming out of your vermont mouth that's awesome so much that's how you interpret it andy i feel sorry for you again folks go to youtube watch the episode and watch the panels everybody's faces went i mean it was hard to hide we we have some fun here so ask us anything episode 50. this is uh episode 47 help us out ask us some questions and uh we will get those in the show can't wait any longer it's time for some wins this week we have a young man from the discord he passed his comptia network plus exam congratulations a big one a big one a one day 2.0 past their ccna oh yeah lockdown congratulations well done so huge davey87 david alicia was named a mentor in a new program at devry university which he is alumni uh and it's a program for women and tech so he's going to be a mentor for women in tech so congratulations that's pretty cool nice yeah he's perfect for that too oh yeah really perfect for that i always forget about devry i need a mentor did you go to devry no yes i just need a mentor oh okay i'll be here anybody willing to mentor andy youtube on any bbq i'll teach you uh whatever like mentor him on anything you know that's an interesting possible new sponsorship opportunity finding a way to mentor andy he's gonna ask you how do you do it how do you do it this is crazy man all right one one more winning before we get too far behind here mike vest recently accepted an offer uh for a help desk position so congratulations mike nice finally breaking in yeah good job get your foot in there and that's the end of the winning section for this week congratulations everybody if you want to hear your name on the podcast join our discord and drop your wins could be anything in hashtag winning and uh every week just before the show i collect all the wins from the previous week and we read them off right now during the introduction of the podcast so you want to get out on that join us so i'm happy that the goat is catching on but i'm simultaneously sad that now other people are ringing the winning goat that used to be my thing you beat me to it now now and i don't have a thing dan has helped other people are naming it yeah dan had howdy i had the goat right all right well i mean i i guess i'll keep my go-to no no it's good it's i mean this is what happened keep your goat to yourself this is what happened you sounded a little salty no i just i'm a little sad i you know like you know what though i i love that we're seeing so many of these pop up yeah people are posting pictures of it they're writing it are they oh yeah oh yeah yeah posted on the discord earlier that they got their own oh my god this goat is very therapeutic we got a flock of goats maybe that's a new name for the a1 fans like if you're an a1 fan maybe you're an a1 goat i don't know what are they goats is that what they're called flock is it a flock i kind of figured goat was like a sheep or super flocks now we gotta look it up you keep the show going i'll look it up yeah is a good one i feel like a goat is is it not a family maybe i know it like a young goat's a kid yeah and i only know that because i used to work with somebody who would raise kids my aunt has like 100 and they wanted to be able to access a remote webcam from the work network that would let them see their codes okay they're kids yeah yeah the kids you're not going to believe this what is it a group of goats a group of goats is called a trip a trip spell who does this why can't we just say i mean i think we just keep it flock of goats but it says a herd tribe or trip is what a group of ghosts yeah that's a trip man but you know that that's the first hit the second hit is it's called a trip so there you go interesting so you say like that's a trip a trip of goats the gathering of goats is officially called a trip nah it's a gaggle of john so there you go but you know if it's not that trippy if you love something set it free and i'm glad everybody has goats now and i'm just gonna sit back and enjoy i guess yeah it's gotta be pretty violating you know not only does everybody like have it and then abuse the hell out of it but just the simple fact that like the crowd is all going to voice their opinion on what his official name is going to be i think that's the best part and the most it's stripping you of everything and the dumbest possible name is in the lead absolutely dumbest impossible hot buttery udders exactly that that to me that's the best part too because it's like i wish one of these would andy hate the most like oh something like aaron there you go a nickname that was revealed in an episode aaron and i did yeah there you go his wife didn't even know this high school nickname so these are the kind of treasures we unearth in this show it's good it's good stuff i i don't know if you guys saw tony e on uh on twitter it was just like did i miss some conversation like where did hot butter i did see them it's just like oh yeah you missed a good episode with uh yeah somebody said i'm straight they're like nope you gotta go listen you gotta go it's a weird episode it is great i mean it's why i love this show and you guys like just you know it's just ridiculous like that's now gonna be part of the lexicon hot buttery utters like there's no other there's no other tech podcasts that are this much fun what were we supposed to talk about oh yeah we're getting there we're getting there we finally made it through uh all the introductory stuff so let's let's talk about on call i'm sure all of you guys have been oh yeah on call some of you are probably on call right now yeah that's where the money's at man it's worth the stress it used to be it used to be there was a point in my career where i was on call and i was still hourly and i got the the double time hourly rate the extra kick in for being on call the yeah uh i don't know so much extra for carrying the pager and and then i made it to salary land and there was no extra kickbacks for me see i've never been on call in in a salaried situation that sounds weird i mean they don't pay you extra to do that no no not when i hit it manager for a company and then i just still participate in the on-call process yeah and usually it was just to like relieve somebody on my team from having to do it for a week yeah but like you also knew that it wasn't just like one day out of nowhere they were like like like somebody would say to me like hey by the way you didn't know this probably but that that 65 000 you signed on the dotted line that also includes like 40 extra hours on work every week you know in the middle of the night right exactly like i don't think you can't exactly spring that on somebody you know what i mean oh they can no no i just mean like like to that extent you know what i mean like like oh i'm not getting paid extra for it type of thing right i mean i've heard people i've only i've heard people get paid you know for on call and people don't salary to non-salary for both i mean you should definitely get paid extra for being called you know woken up at one in the morning into a stressful situation well hell yeah i would say into a stressful that's the one part that like i think is just is always kind of overlooked it's like it's one thing to get woken up at like two in the morning but when it's like hey everything's not working sometimes my kid wakes me up because they have to pee but like when the building's burning down and people are calling you right like what oh man that's a nightmare yeah yeah but i would say if you have the opportunity and you know that there's going to be some sort of on-call involved to make sure that you knew that ahead of time yeah i guess maybe that's a takeaway before we get too deep absolutely definitely part of the interview process like is there an on-call and how do you treat that on call my first phone call was as a cable guy yeah that's a same and i believe we did get a bump like when your own call whatever it was you'd put it in your time sheet and i think he got extra and my my so you know when you're calling the cable company like unless you're you know a big wig somewhere typically they're not gonna call the on-call cable guy to go out and like look at your internet like it's two in the morning but i the two times i remember being called on call one was for a company executive so the the power was out i think and we had a generator at his node which is what powers the cable plan out there and they had somebody sitting out front of this guy's palace 24 7 keeping his generator full of gas so that his cable didn't go out so that was my first on call was to keep an executive's generator full of triple play right and the other one was also generator work it was after a hurricane i don't remember if it was sandy or what but we had generators everywhere because the you know the power lines had gotten destroyed when it moved through and generators were everywhere powering the plant you had to keep putting gas in them every so that was kind of like my first experience he never went out like uh for like mainline issues in the middle of the night i mean you know the linemen kind of took care of the the big stuff that went down they really didn't need us like yeah they're not going to send you in somebody's home at 1 30 in the morning because like on demand isn't working you know what i mean exactly yeah yeah it was always out of here specials like big stuff right two like morning yeah like if the lineman needed us you know like hey there's something big going on we you know we need help but yeah i think it was really just keeping generators running you know after storms and stuff yeah it's like two degrees outside and there's like a hail storm and all this other stuff happening like this wintry mix going on and it's like the power lines are covered in ice and they're sagging down like six feet and you gotta climb up there but you know what's nice about that is you can see it coming like you know and your boss is telling you like hey you're probably gonna you know probably gonna need you tonight and you could be like hey look like for this foreseeable future because no you're right that didn't always happen right so it was like yeah it was very in a way i guess it was predictable to that degree though right because that was like oh crap the storm of the centuries coming that's for me that's how it always happened it was like oh this is a storm that knocks out power for like five days and it wasn't stressful like yeah i mean like it like it was as stressful as like disaster cleanup can be because that's what it was very methodical and slow and repetitive and you know it was just like never ending like you know 19 20 22 hours a day type thing like but yeah i've never done it in a salary position because i i wouldn't even know how to approach that but like i you know in any salary position i've ever been in i was never like tied to like a nine to five or an eight to five or something so it's like okay so if somebody's scheduling you to be on call two nights a week let's say but i'm traveling two nights a week for five hours at a time you know i mean yeah i mean like one is a lot more work obviously than another but in a way but like is it though because i mean i could i could make a case for both being kind of equally a pain in the ass you know right to in their own way aj you probably obviously know what i'm talking about because yeah but you know i've been in like i think both of those kind of on-call situations where it's like organized and you know people take turns there's rotations there's kind of like define like okay you're on call from this time of this time and then somebody else is going to take over kind of thing and then there's just the perpetual call like you know forever you know just because that's what the position demands right like you you get issued a phone there's no like stipend or or extra kickbacks it's just if something happens and it's in your domain you're going to get called that's part of the deal though like it's weird like if they don't call it out like you're getting paid extra for this or like you know like andy alluded to earlier you know i think we got like it would give us like 23 dollars a day or something i think that brings itself yeah yeah if you were on call but you know the other thing that i think is often overlooked too it's like on calls of pain in the butt because it's so unpredictable in that regard because like you're actually almost guaranteed to not get called as much as you are to get called and i never talk about that like it's just the anticipation you're going to bed like gosh i could be called it's like it's almost as bad like you might as well call me you know i was just kind of laying there like anytime something bumped over here you know i'm like and now i look at my phone did i miss a call that i missed a text you know that kind of thing so is it safe to say like every network engineer position has on call with it like other places i would i would have to right most i.t positions that i've worked in have had some form of on-call because even if you have a knock which i always have they still escalate and they need you know higher tiered people so it seems like it seems like an evil uh you know it comes with a job right yeah it comes with a network engineer you're gonna be on call right yeah and i also one thing you know you guys were talking about you know getting 23 a day or something like that there's other ways that a company can compensate you on maybe not just straight up pay so what our company did for a while and they still kind of do it but they they kind of call it comp time right and so if let's say if i'm you know working a problem at 11 o'clock at night and i have to work you know for three hours straight or something like that well we have this you know honor system bank of hours where it's like hey i had three hours there i'm gonna take off three hours early friday or something oh that's pretty sweet and and it's just kind of understood that you know hey this is the kind of work we deal with so you know you're you're still getting in your 40 hours a week type thing it just might be at different times of the day right uh so i don't know i think i think that's a culture thing to feel out too when you're going in to an interview that kind of thing is is you're asking yeah what what is the on-call rotation like um yeah how often and and how do you feel about you know handling if i've got to work multiple hours overnight you know just just to kind of feel out what that culture is like i think is important and i think one thing too is you probably should ask them like okay so i'll give an example at our place we don't have a rotation right we we're just all on call 24 7 basically and uh and so you probably should ask about that right you should probably ask hey what what does it look like is it is it a rotation is it 24 7 how does that work um did you get called often dan i mean you're on call 24 7 365 it sounds like so when i correct and it helps having a team right because you know if we know someone's going to be going on vacation and stuff they're obviously not going to try to bother them that kind of thing um or like if i'm going out of town i let my team know that that hey i'm going to be out of town so i'd rather not pick up the phone basically but so why wouldn't you do a rotation like why are all of you guys on all the time that wasn't our call yeah that's just the way management's done it um sure so do they call everybody at once like no it depends on what the problem is so the way the way that we figured this out is like certain systems have certain owners right and so if that system has an issue they'll call that owner basically that's good um and then but but one thing too is so when i first started we used to have a lot of issues so i got called a lot and it kind of sucked right but uh as we've gotten our as we've improved our network over time those calls started going down and down and down right and and now rarely let me let me knock on wood real quick uh i i mean i don't get called that often anymore so that's nice but uh so that that's that should be a uh that should be um an incentive to try to you know design your networks right and don't do crazy crazy stuff just just because it's cool i try to keep my network simple man yeah i had a similar experience staying on my old team when i was on the lan team the the wan had been um uh i guess not cared for as much as you know you might want to and there were a lot of landmines and yeah there was three of us on the team so every third week you were on call and you just never knew when that phone was gonna ring because stuff would just get you know triggered all the time but they they they brought in an architect and we got it all cleaned up or most of it cleaned up over a couple years and you're right that the calls went from like almost you know non-existent yeah i think the worst i think the worst calls i i used to not like getting was so you know you'd have a knock and you'd get woken up and you'd hop on and you're you're getting the download what's going on and you would ask on the bridge like okay so what's been done and like you just find out nothing like nobody did any troubleshooting nobody did anything it's just somebody call somebody called they saw there was a problem and they just sprayed and prayed and like call everybody get everybody on the bridge yeah and again like tim you mentioned culture we were able to work with our noc resources to kind of like hey guys you know here's here's basic network troubleshooting let's let's go through this before we start you know paging everybody out um so between fixing the network like you said making it more reliable and then working with your teams you know hey don't they'll just page everybody all at once because you could do a couple things for five or ten minutes to dig around you know and because then it helps the engineers that get on the bridge like oh okay what what's going on what do you see as opposed to i don't know something's broke everybody's pissed check it out you know they're not funny when you let your users be the warning system that's that's not the best of uh approach or policy we used to call that the scream test yeah when we were trying to like shut something down you weren't really sure if people were using it yeah just do it do a screen test see if the phone rings yeah we can ring it we turn it back on a smoke test oh yeah okay yeah smoked upstairs also one thing too is uh you know when i when i first started my job we didn't actually have uh work cell phones we had little pagers oh yeah yeah yeah i used to work at a place where we had to we we had one alphanumeric pager for the team okay and it was a rotation and we had to like take turns like carrying this damn thing yeah and it like every friday i was just like all right who had that thing last yeah where is it like well you know because so we would go a long time without going off and also it would just like go off yeah yeah so we we all had our own so they could just page whoever they wanted but uh it was really funny every once in a while if all of us were kind of like in a group or something like that and some something did go down it was like what you were talking about before andy the spray and pray and so all of our beepers started going off you know it was like oh gosh here we go but jeepers i was just thinking back when i was in the comcast knock we weren't allowed to page engineering we never once in the years i was there like you've you fixed it you know there was no i wouldn't even know how to page engineering so it's interesting culture to culture like no it's we're break fix this is our job this is what we do and that was from management that was the culture there we're not we're not paging people you fix it i remember the uh like when we were in the field andy did you do you guys ever have the push to talk next house no but the contractors did i remember okay they like by the first of all i get it but like awful idea like like everybody gonna hear everything right people yeah you just gotta understand like the type of dudes you're hiring like we like i work here for a reason like have you seen me you know what i mean like uh i i just remember like i was installing like somebody's dvr once and and my buddy just started yelling like he's like holy you know expletive like he's like you should see this mother expletive and it's it's like volume like 12 right just coming out of my butt at somebody's house yeah because i'm on the ground like on my knees like you know like pressing the buttons or whatever my butt is just yelling at this dude's face and i'm like ah i feel so bad but but why that sucked well of course in that moment it sucked but for the most part why it sucked was that was how you found out like if you were getting called out was that same noise like they would just start talking to you like you'd have the next tell just like sitting there and all of a sudden somebody'd be like dude aaron hey aaron like it's freaking terrifying like to wake you up like something like that too like you know disembodied voice screaming at you right yeah it was always somebody different right like whoever was working in the attack that night like you know like get up ready to fight kind of thing yeah they're like hey hey he's gonna make you up yeah they're like do you want to see if this clears and i'm like slowly putting pants on i'm like yeah she's like it might clear in like 20 minutes i'm like all right i'll get the car started maybe 20 minutes i'm like so what's up with that average what's going on with that yeah uh yeah it's it's still it's still going well good because i'm like almost there i'm definitely almost yeah i andy probably you should be glad i didn't work with you during that time because uh i'd be like hey andy how's them hemorrhoids doing oh dude god i can only imagine what what a saboteur nice that's how you get somebody in trouble like you don't even have to know him you should be like hey what's this push-to-talk number you know it was like it was like star or something something something you know yeah and you can just start talking you start talking to strangers like what you watch your buddy walk up to the customer house you beep in like yo man i got your butt cream beeping it was like that constantly so bad like you had to have it on speaker though because like if you were like climbing a telephone pole and you were doing something and you're like trying to get somebody to help like it was the best thing ever because you could just they'd be like hey okay come down now you know and you're like okay you didn't have to like grab it you could just like hear him you know i mean it so in ways it was like ahead of its time but like to me it was just an annoyance when i was on call because like god what an awful device yeah thank god they went away aj do you have alcohol at your place because like you guys build things for customers right like you wouldn't get a call that something you built six months ago is down right like how does that work right so there's definitely periods of time where i could be on call and it's usually immediately after we've delivered a solution right so yeah and it's usually for me on calls is usually during business hours so like you know we we deliver the solution we implement it fully it you know we move into production whatever and then the next business day if there's you know a problem then i'm on call i'm unhooked to you know help them out is that defined in the contract like we get x amount of time after the install to you know bug these people for support depending on the the project we usually build in like so many hours of post project or post implementation support um a lot of our customers are also our managed services customers so yeah after after the implementation it moves directly to you know there's a handoff between the professional services team and the managed services team and you know we walk through the solution say okay here's here's what we installed here's the device ips here's you know usernames and passwords they they go into the devices get them tied into the monitoring system so that way they can start to monitor and watch the health of them and react if something goes wrong so you can sell them that service yeah yeah okay yeah that makes sense wondering if maybe it would be helpful just maybe in our own words collectively for anyone that maybe hasn't had the job yet you know like pretty much every week we're like yo so-and-so got a new job got a new help desk job but like a lot of people haven't been on call or maybe you haven't even like encountered what that's like maybe we should just kind of define it in our own words as a group like what what what is it right so like when you're going to get the job and you are you know negotiating your hours oftentimes with this position or it positions in general you can end up doing shift work right so these are important things to point out so it's not easy to just say like okay normally your job is eight to five on call is anything but that because it could be you know you could be running a swing shift or you know you could be working overnight or whatever because then if you're on call that would mean during the daytime so really it's kind of like the offset of what you are out like anytime you're out of your normal working hours and typically um i'll just add kind of a couple pieces here and i want you guys to just kind of sprinkle on some color to this as we go just so everybody kind of has their own little version you know what i mean um in case we leave pieces out but so you're on call sometimes you get paid for the on-call right sometimes you can get like a daily on-call pay um sometimes they're in rotations um sometimes they're always there uh what am i leaving out here help a brother out sometimes uh there's slas right like if you're on call you get a call you have x amount of minutes before you're kind of like expected to be logging in triaging reporting back kind of thing sometimes i i've not worked at a place where this was a thing but i have ever heard that when you're on call you had like a travel radius like you were only allowed to go you know okay so many miles away from your home or the main facility wherever you were providing coverage i don't like it interesting no no i don't like that like i said i have ever heard that that was a thing um i've not personally experienced that myself when i'm on outside of the house my work laptop is in my backpack i got my hot spot like yeah yeah i might be at a baseball field with a kid like hey okay so that could be a part of it too right yeah so anytime i travel anywhere i take my work laptop with me right yeah now okay maybe not as extreme as what andy's saying like um i like if i if i go to a restaurant or something like that i don't take it with me but if i travel out of town even on my vacations i always take my laptop my work laptop with me just in case i need to jump on real quick and look at something because here's the thing if my team is drowning and i'm on vacation you know i'm not going to be that guy that just says nope i'm on vacation you're like i'm gonna help my team out right but yeah so i always just take it with me now luckily we haven't had that happen uh but i you know i i definitely i'm not gonna let my team drown so yes i would say the frequency of being like when you're on call would be huge if you like when you're interviewing for a place right because like you said dan you're you're carrying a damn lap time around with you in case you gotta sign in you're basically working 24 7 when you're on call so if you're on call a lot if it's a small team if it's two people yeah and every other week you're on call and their network's unstable and you're always carrying your laptop everywhere and the phone's always ringing that's you know there's a price to be paid for on call aaron made me think of it with the shift work i worked mids i worked at midnight shift at the knock for a year and i don't you know i don't want to say it almost killed me but like i couldn't sleep i cannot work midnight to eight and come home and go to bed i tried i tried everything they say but i did so that the price for shift work for me i just couldn't do it it was it was killing me and it's kind of the same with on call like if you're getting woken up a lot if you're carrying stuff around you never know when the phone's gonna ring that's that can be stressful and it's hard to find it's how do you find that out in an interview you know like you're if that environment is that bad they might not tell you right because they might have people i think the goal is to like know that that's kind of like this right now right like like aj brought up some points that i i'd never heard of and in fact like he's never even encountered before he just knows that like that kind of stuff is out there i think the more that we can bring up right now it might be helpful you know so these things are in the back of your mind so you know i'm not saying like red flags necessarily like you know because it's hard to pry that deep into one subject without it even especially making it look like you're like trying to like get out of something you know like that's pretty sus like so tell me more like so what happens if i don't answer my phone right away you know like that type of thing like i don't want to go down that road buddy but but you know what i mean so like bringing up you know other stuff like that aj you said you were going to say something yeah so so i worked at a small company that they didn't have on call but the person before me that did it took phone calls outside of normal working hours and assisted people they just didn't formalize it and call it on-call and and that expectation wasn't set you know when i was hired on kind of thing uh and so we we had an it team we we then you know we got the pager we did the rotation we we set up some slas you know so we set up some applications for people you know when it was okay to call like if if you're working a weekend and you know your printer's not working save it to monday you know kind of thing like but if you're working on a critical project you get locked out of your account you know whatever like you know call us we'll help you figure it out kind of thing so maybe like what does on call even entail right like right like what constitutes like an uncallable situation exactly i feel like in the job description parameters i feel like in the job description you always see like occasional nights and weekends right yeah yeah i mean that that that could mean maintenance windows could be could mean you know the job descriptions are usually general in nature right like it's like fishing let's get them in the door so it's a good point it's it's tricky man to know what you're getting into right that's it so i guess in that vein aaron like let's let's kind of discuss compensation for for that like some people are salaried they don't get any additional compensation i've heard situations where people that were salaried did get extra compensation for being on call they got like just a kickback for being on call regardless of whether or not they got a call yeah okay and then right they got a call you get more then they got more yep like they got paid basically an hourly rate on top of their salary for having to be on call so kind of like a bonus then i guess yeah yeah right interesting because like i always wondered how they would do that because typically the being hourly it was what made that so easy because you know once you go over the 40 you know i mean obviously time and a half type of thing but it wasn't like you were gonna hit 40 so it's like okay so if i was on call tuesday and wednesday like i might hit 40 hours by wednesday right but i wouldn't be getting overtime during the on-call period right i would actually be on overtime on my normal working hours on thursday and friday well and they would just total up and also yeah um so we just got a new cio and he came in and like we you know our teams met with him and everything like that and that was one of the things he actually said was you know hey i understand he he actually came from infrastructure side so or his background is in infrastructure he was actually a network engineer which i thought was kind of cool um but uh he he said that he understands you know how the infrastructure life is and so he is not like a clock watcher um and and so what he and i think what he meant by this was let's say the same scenario right 11 o'clock i get a call and i've got to work on it for two or three hours or something like that and then i get it done well you know what if i come in a little bit later the next day you know they're they're not going to worry about that because they know that hey dan had a you know an on-call that he had to do fix a production issue or whatever and uh they're not gonna they're not gonna be upset with that right they're not gonna sit there and be like hey you know it's it's 9 30 and you're supposed to be here at eight you know that kind of thing um so i think i think that's something that you need to fill out in a in an interview right and i don't think that's like a um a shady question to ask because like when i was doing some interviews with people uh they would ask that right they were like hey like i had one guy come in he was like hey how would you rate your network like do you think it's you know zero being unstable and 10 being you know super stable and and i was like that's an interesting question because i've never had anyone ask me that because you know i think i think a piece of us are always like like let's say you know you're a tennessee voss fan and someone asks you hey how good is your team and and there's those voss fans that are like oh they're the best team out there when they really suck you know that kind of thing nah so what what was your number and did you hire them i don't i'm trying to think what i i said i think it was about a six or seven you know it wasn't it's it's not you know there's things that are missing but i was also i kind of actually went over with them what i just told you guys about you know when i first started you know it wasn't in the best shape and we got called a lot but now it's you know not as much was that his goal when he asked you that though you think to figure out if like on-call was going to be a pain oh yeah for sure and oh yeah he started listening to it yeah that's what i'm saying like i was really smart yeah but i thought but you know i think one of the guys that was in the room i think they kind of were like like afterwards they were like why would you ask that and i was like well i think what he's trying to see is like you know is he walking into a train wreck or or is he you know walking into where man you can learn a lot dan how have you been hiding that question for this long from us i almost i now we were talking about the goat earlier and how andy feel about now i feel violated you've been holding that away from us for this long like what a friggin gem that is like like how stable like what that are you kidding me because like we're talking about it in the context of like how crappy is my on-call experience gonna be but like gosh dang like let's just name them off now like what kind of things can i gain from you by asking that question right yeah i can see what kind of budget you guys use like how much money you spend right like how frivolous are you it was really good how forward thinking you are like you know how smart all you all are right like how much freedom do you have like gosh dang that question is endless man yeah and but but then i think to to dan's point like you want to believe the best in your network and you you might rate it maybe maybe a little bit higher you know than than so then maybe maybe ought to because you overlook certain blemishes about your network because it's your baby yeah so yeah but you can read it why not ask it differently how someone answers yeah why but ask it differently no yeah like not ask that like what do you like about it be like okay like what three things would you change about your network that's a good way to do it yeah now now we're getting tricky now we're getting i i want to i want to hear some notes i wanna hear from tim before we yeah we get run out of time here tim what's uh have you been on what's up in your network engineering career thus far yeah so i've actually seen uh multiple sides of it in the industry that i'm in so i in the at the place i'm at now i i started on the end user computing side so that was very much the hourly um when you were on call you got you know x amount per day and then when you got called in you got you know whatever for x amount of hours and then for the last eight years now i've been salaried on the network engineering team and yeah it's a rotation basis but i'm kind of in a position now where i'm kind of one of the unspoken escalation points so even if there's times where i'm not on call if there's a big enough issue they're going to pull me in which is completely fine i mean like dan was saying that that's this is my network this is my baby i want to help um so yeah i i've seen both sides of that um i do kind of want to go through kind of a list of tips um and then have any of you guys add on as well and a few of these are going to sound really familiar from the soft skills episode so a little bit of repeat there but some things that i think are really important when you're on call especially when things come in after hours especially when you are getting woken up and it takes time to figure out where you even are and you're trying to you know get information on what's broken and yeah who are you how did you get this when you're sitting in your underwear eating cheetos you know that kind of thing right in my bean bag chair dan yeah that one so some of the things i want to talk about um it may sound simple but it's not always when you've got a million things coming at you at once is remain calm you wanna you wanna be that cool head under pressure because when you're trying to fix a large issue like that you may be on a call with a bunch of people you want to keep the cool head you want to set the tone for the troubleshoot and it's only going to make you look good number two communicate because a lot of these things happen uh when they do happen after hours or on weekends you know your whole team including your your management isn't working um along with you always so when there are issues i think it's important to especially communicate to management and your team that hey i got called in on this i'm working through these steps i spent x amount of time we got it resolved i think management really appreciates that so communicate i think a really important one especially if you're in a uh if you're in a team environment is know when to ask for help so and that's kind of that back to that prioritization thing and that when an issue comes in if it's something that uh came to you it is an on-call issue but it's not a network down emergency but it's something you haven't seen before you can kind of gauge i think in how much time you're willing to give yourself to troubleshoot through it um but if there is a massive issue going on and it's something you haven't seen before you need to know that you need to ask for help probably pretty quickly in those in those situations um and then when you do have to ask for help know how to be grateful to those people who you had to ask for help at three o'clock in the morning i think that really helps the team environment too to not just shrug that off thank them do what you got to do to show that you were grateful and then when you do get that call in the middle of the night do the i feel like we bring her up in every episode now but do the tanea method take your time give yourself your five seconds to collect yourself uh i almost guarantee you whoever's calling you on the other end will understand that you need a few minutes to to figure out where you are who you're talking to that kind of thing and then the last thing i got is it can be stressful so so try not to let it consume you and and ruin your day your week whatever your your period of time that you're on call especially for somebody like dan where you've got um a set of applications or systems where you own them and you're on call at all times just do what you can to maintain a positive attitude and just don't let it run your life great great points tim i want to echo what you said about asking for help so you know swallow your pride uh there's no shame in asking for help you know i'm a senior deployment engineer and i bounce stuff off other engineers all the time i would much rather you know i i will give myself a period of time to troubleshoot it learn about it you know whatever and if i get stuck i can't figure it out i bounce stuff off other people like hey you know here's what i'm working with and if you if you are going to run something up to someone bring that information to them you know you don't want to be like okay i'm stuck okay well what'd you do yeah what'd you try uh so so start lead off with that information like hey i'm looking at this problem here's what i've done and i still can't figure it out what say you you know what what are your thoughts on this um and if you know there's there's definitely the i would say the opposite of that right like always asking for help you know if you if you constantly get or not constantly but if you um if you're on call you get a call and the first thing you do is punt it to somebody else and say yeah i can't figure this out like well did you look at it well no i just thought i'd ask you because you know you're really smart you know you're you're never gonna get that experience yourself and uh you know learn if you don't attempt it and give yourself the opportunity to try to take care of it that's a good point there's a certain amount of pain and stress you have to put yourself through on call because you don't have the answer and you have to dig and you're not sure and there's pressure there's if you're on a bridge well that's like in your skin too like the more you you get in those situations the easier and more gracefully you can handle them in the future yeah yeah the first one's rough i i can't echo what you just said anymore but the pride thing i that's one thing that i preach to people on my team about is don't like dude the way i operate is if i'm starting to have an issue and i and and i know that i'm gonna have to start doing some research to figure it out right like i'm talking about not just a quick google search and then i got it i'm talking about like i've gotta dig and dig and dig kind of thing yeah uh yeah dude all right let me get that tech case open and get that sent off because you know tech takes a little while to answer their emails now but uh that's not that's not a dig i'm missing you got a call man yeah well yeah especially if something's down right and then you got a p1 bro get on my website now i will say if there's any attack engineers that are listening to our podcast monster well yeah monster but uh i always i feel absolutely horrible selecting that p1 like i i was working with the guy a uh contractor and dude that that cat wanted to do a p1 on everything and i'm like dude this is not we haven't even installed this yet like crying wolf all the time they're not going to believe anybody dude i believe so much in karma when it comes to that like i don't know dude even if it's like you know stuff's on fire i'm like i still don't think everything's down so it's not a uh p1 just yet but uh so i you know this is i'm just saying that but uh yeah no take care well the wireless still working so there's not a problem here yeah but the internet's down wireless doesn't matter exactly so um but yeah take take the pride out like don't you know that that's one thing that bugs me i think is and and i understand we're in a field where it takes some intellect to figure this stuff out and i i feel like we get you know you get alpha dogs i guess if you want to say that in our field and because i definitely have heard uh people just belittle people for not knowing why something's not working and that right there is just that's just insane i it's not it's not helpful nope you're all you're doing is you're making the situation worse you're not helping the situation that's how you get a hang up from me that's for sure exactly like i'm not even going to talk to you anymore so you start doing that so um yeah i take that pride and just hang it up on the wall and like let's just figure it out kind of thing so andy what does it look like for you now since you've moved away from the lan team and you're on more of that what i'll call enterprise team that's probably not the right word for it but that's called neckbone backbone okay is it different is it is it the about the same size is the on-call experience different uh oh man you put me on the spot here well you're from what you've been talking about you've been doing more like maintenance windows than on-call stuff right i mean there there is an on-call schedule um i have received one i've been called once on call on my new team um you know what's nice is the knock is amazing so i've built places where they're less amazing so they're super knowledgeable and very good and i'm really only getting called because they need somebody to make a decision that they don't feel comfortable making like like a design decision like hey this thing broke it broke because of the design we can fix it but we need to talk to the people who designed it before because if they're going to change the design to fix it right that's that's kind of so it's different it's they're more devices it's bigger it's global or different technologies but the knock is so good that it's really helpful i don't know if i'm answering your question i'm kind of dancing around it but no you are so so it's not that problems aren't happening but you have an actual network operations center that's fielding those initial calls and able to do a fair amount of the resolution yeah very sharp and very enabled okay and maybe that's culture but yeah they top down get it working as fast as you can like you know we it probably sounds obvious like yeah nobody likes outages and like everything has to be working but you said enabled right like giving people the power that can fix it like because in other teams i've been in other areas they didn't have the power so they just page everybody and let other people figure it out fix it oh you know we don't own that gear so you know they look in the database and say oh these are the owners so kick it to them as opposed to where i'm at now which they're enabled they know what they're doing and they can they can get it fixed which results in much less calls for us like you can have a knock who isn't the best and i'm very fortunate where i'm at that the knock is brilliant um that all the macho that we had on came from the knock and that dude's just next level so apparently he was surrounded by people like him so i'm fortunate that i don't get called much i guess well i think it might be important to note also that i think the that might be dictated by the size of the organization as well uh you said that you've been in some organizations that have a great knock an organization that you know maybe didn't have a great knock i i've never heard dan even talk about a knock so i i don't even know if he's got one you know i just think it it depends as well and that's again back to the interview process i'm really glad that aaron brought that up at the beginning is um that interview process is just as much about you getting to know about what you're walking into as them getting to know you so yeah and that's important and it's funny the different stages of your career because when i interviewed for the job you know my first network engineering job out of the noc they could have told me that i'd be on call forever and i'd be like yeah great let's do it you know like when you're trying to get that first gig i think i'd have a different answer now like uh i think when you first get in like there's this like element of excitement you know like you got your first id job and you're on call and you'll do anything because it's so hard to get that foot in the door and get that first gig like yeah sure you know you know you're the only one here you'll always be on call we'll always be calling you great what do you know when do i when do i sign up yeah but then you get some experience it changes like you know you get older yeah it's just that's the priorities like you know you don't want to be a call and then you talk to people and plug into the community and you talk to people like oh yeah you know i'm hardly ever on call and when i am i don't get called like wait a minute that's a thing yeah like what am i doing here so well so for me like when i when i first started i wasn't married and i didn't have a kid right and so being on call really when that that crazy because i didn't i don't want to say it like i didn't have a life back then but i kind of didn't have a life back then compared to you know my family time i have now and so yeah there was a difference there it didn't but it did not bother me that much back then but now it's like all right dude i've got other stuff i got to do now so i think our priority changes as we get older and our family starts growing and stuff like that so so i got a question for aaron uh you know as an se i've heard of plenty of situations like you know when you have to call tac like oh call the account manager call the sc like they can help escalate you know our case and help us get resolution um as in in your role as an sc is that anything that you've had to to be involved in i would make sure that whatever organization that you worked for that you asked that question up front did you or not and that's where i will leave that yeah no not really i don't i don't that's not like a thing i i it could be right but like that we know what like the direct line would be there's nothing that i could do sure in any of the se rolls that i've ever been in there's nothing i could have done to make something go faster than you calling our engineers directly sure okay you know because i don't have access to any of those systems you know it would just be a tribal thing like hey i know so and so i know so and so but like you're calling the same place i would call you know i actually call the same number you know but even though you so even though you can't i would see like managers of places with stuff broken like call everybody remember that guy aaron who sold it to us yeah it was the worst when i was in the field quite honestly because then they because you know you'd be like hey you know if this breaks again today like just call me directly like i'll come back over here because i don't know what's going on no bad idea yeah you only do that once or twice you start you learn like don't ever do that again but you're just trying to save yourself from having a repeat i did that all the time and it was awful yeah because it'd be like dude 24 months later like no joke like like hey i don't know it stuck it right next to the thing so when it broke they know exactly what it is it's like you you've already changed jobs but they're still calling your cell phone number oh that happened that happened so many times like when i worked at charter especially like i was like i am no longer in the field actually um and then i would be escalating like that like oh let me let's call the guy i actually know the guy still hold on let me just actually let me just push to talk him real quick and start talking hey dude what are you doing um i was with a customer sorry but yeah excellent uh any other thoughts on on call before we wrap it up guys i think that's all i got man i think we've about covered it yeah yeah i think we should i think we hammered it pretty good it's a dirty word it's a necessary evil none of us want to do it um if you can get extra money while you're negotiating that would be smart because yeah know what you're getting yourself into absolutely you know it's it's time right time is our greatest you know asset so when you're on call there's more time you're going to be working potentially and it'd be even if you never get called then just think about what that does to limit your behavior and your extracurricular activities and mentally knowing that phone can ring at any moment when when when you're on call so get get extra money like dansa get some comp time they're all really good ways to don't let them own you with that salary and then just be under that stress and duress and exactly yeah good party excellent well uh super excited to announce next week we are recording our sponsored episode with open gear and we will be doing some very exciting giveaways we will be giving away some books and some additional swag that will include items from the podcast as well as some swag from opengear so look for the announcement of those giveaways on our social media accounts and uh while you listen to us well we'll let you know how you can enter to win those exciting giveaways very cool thank you to opengear for helping us help you to open gear for that yeah we're going to have a super exciting guest from those guys yes but more importantly not just specifically to open gear we get to talk about something that we've never talked about before absolutely yep we we have been working very hard on this with our sponsor opengear on how do we stay true to our format while still doing sponsored content we think we've got it figured out we're super excited to get this episode recorded and get it out to you guys and await your feedback so keep us honest make sure we're staying true to our our uh our type here and let us know what you think i think you won't be disappointed i'm excited yeah i'm pumped yeah excellent well thanks a lot guys uh have a great night and we'll we'll see you next week see you hey everyone this is aj if you like what you heard today then make sure you subscribe to our podcast and your favorite podcatcher smash that bell icon to get notified of all of our future episodes also follow us on twitter and instagram we are at art of netench that's art of n-e-t-e-n-g you can also find us on the web at artofnetworkengineering.com where we post all of our show notes you can read blog articles from the co-hosts and guests and also a lot more news and info from the networking world thanks for listening you