this is the art of network engineering podcast in this podcast we'll explore tools technologies and talented people we aim to bring new information that will expand your skill sets and toolbox and share the stories of fellow network engineers welcome to the art of network engineering where we are complete no-nonsense group of highly professional podcasters who always give well-thought-out information and opinions i am timbertino and joined with me tonight is someone who may or may not agree with everything i just said andy laptop andy how are you my friend hi tim awesome happy to be here good to see you friendly faces anything new or exciting this week um taking a deep dive other than the fact that you're either standing or floating i'm at my standing desk um i've taken a deep dive marty mcfly you ready for this this is what i've been learning this week it's a mouthful multi-protocol bgp evp and vxlan so that's what i'm up to yeah and you got it all figured out no oh you're the expert now i'm getting there yep yeah so yeah things are good tim uh work is good life is good how you doing it's good to hear i'm good i'm good i uh i'm getting close to the point where i'm starting to dig into uh wand design in the in salad portfolio if only i knew a seasoned veteran of uh large wan networks that could help me dude give me a call i'd have somebody to ask but i missed the weigh-in i'm not in wayne anymore and i'm all lost and confused again that's that's why i'm poking at it i miss knowing what was happening hold on before we move on i want to thank your wife for this beautiful piece of artwork that she sent me which is our logo this is very nice and i wanted to i haven't had a chance to officially thank you on the show so thank you mrs bertino yeah those were cool we uh she she found someone that does those and she wanted to get something signs made for a couple of kids teachers and she got those made and she gave the person our logo i didn't even know about it i got one for christmas and uh they were like yeah we'll give it a shot i think they turned out pretty cool yeah they're really nice man so thank you no problem also with us tonight is dan howdy packet richards what is happening dan nothing much nothing much let me tell you something though anybody with kids out there they have these places uh that have trampolines inside and they're in all crime kinds of crazy like you know some of them are like slanted and then they've got rows of trampolines and everything like that i tell you what if they had if if we had one closer to the house i could probably lose some weight doing that and have fun with the kid at the same time because i am hurting well hang on hang on did they serve beer you know i didn't see any there but okay i don't know if maybe i could bring my own if you like oh yeah granted beer and trampolines is probably not a good idea yeah probably not might spill your beer maybe maybe it's the ultimate test how good are you at holding you're not spilling your beer no i'm good tim uh you know i i actually i have my uh little plaque with our logo on it right there and the last recording i had it leaned up against the wall in the middle of the recording it just like slid down so i've gotta figure out a permanent mounting solution for it so that i can uh i can put it on the wall right there everybody can see it show the pride right right exactly so yes thank you and now it's time for some wins winning on our discord channel this week is a deal khan he got three certifications this past week alone what my goodness stop it stop i i it's it's the truth he got his aws ccp his google cdl uh i don't exactly know what that is off the top of my head i'm not too familiar with the google certifications uh and you got the oracle cloud infrastructure so somebody has been studying a lot of cloud lately congratulations a deal stop do you stop yeah my goodness uh network hug passed the pen test plus nice congratulations nice joey passed the a plus there you go there you go and river i haven't seen that name in a while river has passed the avery etrics certified engineer in multi-cloud networking operations congratulations good job and pet frong i think i i feel like i should go back and look at that in the discord but they recertified their ccna after six long years of being away they are ccna certified once again congratulations nice freshly minted lots of new patreons have joined us in the past week welcome welcome welcome uh to tim lowesbury michael hilton jared monk mock uh entry level jj mclean and richie thank you so much for your support of what we do here on the art of network engineering podcast uh and if you're listening to this episode as it releases it's not too late to register for the very first a1 live we are going to be in asheville north carolina on april 9th from 4 to 7 p.m at the high wire brewing and you can come hang out with us uh and we're gonna record an episode live right there and we're gonna have a great time you can find the registration link in the show notes for this episode again that is april 9th come and join us also we're doing a pop-up shop with limited swag that we don't have on our regular merch store uh and this pop-up shop is tied to the asheville event but the great news is if you're not able to make it to asheville you can hit our pop-up shop order your swag have it shipped right to you uh however if you are going to be at the vashville event you could uh select the option to pick it up at the event we will have the merchandise shipped there and available for pickup and you can put it on or do whatever it is that you do with the schwag that you got right away and so look for that pop-up link as well and as always thank you so much for your support of everything that we do here on the podcast we couldn't do it without you and we do it because you support us and show us so much love so thank you so much now back to the show all right let's jump right into it this is actually a sponsored episode of the art of network engineering we have with us this evening cto of path solutions tim titus tim thank you very much for joining us on this episode certainly glad to be here how are you this evening ah doing well doing well so you have i don't want to undersell you by any means but you have quite the storied past do you mind uh let's start off with kind of who you are who path solutions is at a high level because i know we'll get deep down into the weeds here pretty shortly but kind of who you are what path solutions is and then let's uh take it back to what got you into what pass solutions is today well when i was born i was very young so i've been a network engineer for over 30 years uh wow for those of you curious that's back in the days of coax arc net uh thin ethernet a little bit of thick ethernet that i dealt with uh but also novell netware i used to have to compile the operating system by flopping out swapping out floppies to actually compile the operating system uh back in the days when if you had a missing terminator on the end of an ethernet cord all of the packets would fall out of the cord and effectively you wouldn't have a network um would that be flooding flooding i'm just curious yeah bits spilling out all over the place you'd have to come with a mop it was it was terrible i mean honestly i t it hasn't changed from doing janitorial type of work from back then to now i mean it's nice um but what i found is is that back in the day it's like troubleshooting was the bane of of your existence it's like you could say gee i could spin up a new server i could do these really cool engineering projects but you'd always have users saying hey i got dropped out of a database hey the connection's slow hey i'm having problems and all of these problems was a mystery where you figure i have to go start assembling the clues to find out what the problem is to find out where the problem is find out why the problem happened and that whole sleuthing activity i'm gonna say hasn't really changed over the years and i found that kind of sad because you look at networking and technologies and going from those really old bus networks to star networks to ethernet ruling the world and then going to fast ethernet gig ethernet 10 gig you know 50 gig just the speeds going like crazy uh capabilities all of the the the the v landing the routing the the capabilities of the the networking services just through the roof yet troubleshooting really hasn't changed a whole lot and i i kind of looked at that world and said that world needs to come to an end um there's there's too many cases where people are stuck saying gee i don't know where the problem is i have packet loss somewhere in my network and i just don't know where and i figured all of this should be automated i dreamed of the day of saying i just want some software that i can splat at the network and say tell me all of your secrets tell me what's going on so that i can start troubleshooting and proactively solve the problem before users are affected so that was really the the idea that caused me to say gee if companies aren't moving forward doing this sort of automated troubleshooting uh i got angry i got frustrated and said gee you know i'm doing the same manual troubleshooting that the company across the street they're doing manual troubleshooting everyone on this call you guys are all doing manual troubleshooting and it's like why is this not better and after a little bit of frustration i figured that means that i should actually do something and solve that problem and eventually i i got the hoods put together to say okay i'm going to quit my job i'm going to found a company and and create a product that specifically targets automating that troubleshooting to tell you the secrets of what's going on in the network isn't it that's just a little bit of my my you know i needed to concoct this to solve my own insanity type of problem see so first off back then when you got your start was there even anything that could give you some sort of a baseline of what quote good was and how slowness was determined you had some ability to do measurements you'd have you know forever people have had bandwidth graphs and say gee what's the bandwidth look like well in order to get throughput you need the bandwidth of every involved link along the path okay that's a little bit more complex how do you find out what all of those links are well that's typically an engineer with tribal knowledge putting together saying well i know it's this link this trunk port on this switch uh this link on this router uh you know all of these links and and and connections all the way through and then you have to figure well if we have a router that's underperforming and it's having some cpu spikes and we're doing cpu-based routing because of a mis-configuration or there is a spanning tree topology change going on in one of the lands where it's causing glitches and that's causing packet loss and causing the you know tcp sliding windows to slow down so all of these things put together just means troubleshooting is darn difficult especially if you don't know what you're looking for because before you find it you don't know what it is um but it's that there are clues out there that a senior level person can manually put together if they're able to catch it at the right time at the right location at the right spot and i figured that rarely happens and i think we've all encountered this where you say gee the problem happened and the user complained and you look through and say okay i can't find what it is and you just tell the user maybe the next time this happens we'll get a little bit closer oh the intermittent issues you know the intermittent issue and at one time last thursday at the worst possible time your company's going through end-of-month processing and they're trying to ship extra stuff and all of a sudden the problem happens again okay and you go and look and it's disappeared by the time you want to look so it's this sort of troubleshooting that just leads folks to be frustrated saying i wish i really knew this network i wish i knew what every piece of equipment knew i wish i knew what every interface knew if i had ultimate knowledge on this network i could have an analytics engine tell me what's broken you know what's interesting the knowledge the information's out there you know what's inside are you saying oh sorry andy go ahead sorry dad what's interesting to me is like this is the art of network engineering right but i've thought in a couple years we've been doing this like the real art to this job is in the troubleshooting right because how do you teach somebody how to troubleshoot i mean there's some basic things you can do you know like divide and conquer and try to isolate and but like you said it takes it takes a long time to know all the intricacies of a particular network right and then like you had said you know like tribal knowledge like we used to call that where the bodies are buried right like oh yeah there's there's that one thing we built for that one bu because it was a super rush and we had to go out of the scope of whatever you know and every once in a while this weird thing happens as a result like if you're a new person going in there and this thing kicks off that other people know about you know you're the junior guy who gets paged in the middle of the night like i i don't know and my troubleshooting from the time i was in the field up until recently was just you know somebody calls and you're really just going there's some kind of intuition almost that's like okay well where where could this be right where device should i start in you know let me look at the like i would always look at logs right like i guess if you asked everybody on this panel we probably all troubleshoot differently and that's i guess why we're paid and hired is because we're the intelligence i guess right like hey guys figure this out what's going on fix it and it's amazing to me it's taken decades for this industry for somebody like yourself to come by and be like there's got to be a better way why are we doing it slow like andy why are we looking at each log in each interface and and trying to find it ourselves i've always wished that that intelligence existed you know because it's been up to us and the equipment has it right i mean there's there's counters there's logs there's all sorts of information trapped up inside of these devices that if you could just point a computer at and say go collect the information analyze it and you tell me what's broken and you sit back and you wait 10 seconds and the computer ends up saying oh your problem is is that you have a bad patch cord plugged into this this router over here go check the interface and you go look at it and say huh cat 3 patch cord nobody ever checked the cable jacket on this one where did this come from but you know how long that would take for me to troubleshoot that in one of my data centers to actually finally find the one patch cord that like nobody assumes it's physical right like oh what's going on did anybody do a change is this a code bug right like we over troubleshoot things sometimes like just i wouldn't send a data center out person to go and just start plugging in unplugging cables and swapping cables right but you know your software would have the intelligence to tell us like go look at the patch cord it might take me hours to find out that it's a physical problem you know so i'm really excited to hear about it yeah i think we all hear the go check the physical air that's where you should start yeah but in shared infrastructure when everybody's riding you know say a pair of wan routers in an mpls cloud with 10 000 clients connected you generally don't start unplugging stuff right like you know at least in the data center but if we have an intelligent piece of software that says hey go check the physical i've been baselining this network for weeks and i think the problem's there you know it's it's easier for me to get a dc ops person to go and unplug that thing he's like i don't want to plug the win you know yeah yeah well if it's if it's physically functioning and passing passing packets and you're not having any physical error issues like fcs errors you have no alignment errors then you can kind of rule physical out and then say let's start looking at data link errors let's start looking at network errors let's look at cueing errors so all of that information is still trapped up inside of the devices um and that's really the part that i looked at and said is magical is the devices no uh whether you you have cisco you know a juniper you know whatever manufacturer of router or switch you have even down to the lowly you know d-link and netgear as long as it's a business-class device it has all sorts of snmp information trapped up inside of it it's a question of do you want to go manually collect the error counters manually go collect routing tables bridge tables arp cache spanning tree performance information configuration information off of all these devices and then sit down on a huge network and say how do i make sense of all this data or do you want something automated to go through and say here's all of the clues assemble it for me tell me what's going on and that's really kind of what we've designed is something that will automatically collect automatically instrument automatically analyze to say here's the root cause of your problem here's why you're having slow downs when you're connecting to that one service gotcha now can we can we ask um so so when you say analyze is it is it all based on snmp or do you guys have the config so that it has some intelligence there that it can look through with you know how the device is configured versus some of the snmp traps versus maybe some syslog maybe some something else like like how you know how does the brain work i guess on this one how does the brain work i can't tell you the secret sauce yeah that's about this i don't even know how this brain works but so so our inputs are we're going to do snmp and we can do snmp v1 v2c v3 as well as cisco's high encryption uh v3 of triple des aes 192 aes 256 so we'll support the world on snmp we also take in syslog we also take in netflow and we will take in configs okay um so in one sense what we're going to be able to do with all of this knowledge is analyze all that to say here's what it would take a ccie four hours of analysis to find out what's going on and we'll give you the plain english answer and that tends to end up saving a lot of organizations a lot of time it also means that you know i'll admit i haven't found a senior level engineer yet that loves doing trouble tickets oh i get to do tickets no they want to work on strategic level projects they don't want to be given this ticket saying here find some packet loss because we have call quality problems and so if these problems can be solved by the junior level person saying this interface over here has a vlan tagging fault because it has an mtu mismatch that's where the problems are happening they can fix that one problem and it never needs escalating also the problems get fixed a lot faster because it never needs escalating and all the information is brought to bear so that problems get solved more faster by junior level folks gotcha god that sounds amazing wait wait so i'm thinking back when you were a junior level folk right like you said way back in the you know the coax thick net days because because i don't want i think you're a fascinating it's not that much it's not that many times we get like a guy with 30 years experience right as like a network engineer on the show and i i definitely want to circle back to like your your product and your solution and but i i'd love to get a little bit into your into your history of of your job if we could just just for a brief moment because you you said like yeah i've been in 30 years and the stuff was old and all but like how did you get in because a large part of our audience is like how do i get my first job how do i break in do i get a certification so if at the very least we can just hear how you got your start you know we're your computer science guy perfectly happy sharing the story um although i will freely admit that i think the world is different now than it was 30 years ago but this is maybe one entrant you know one way of getting in is that there was a company the semiconductor company in silicon valley that was growing like crazy and they looked out and said hey we need to build a new building and we need to get uh cabling done and they realized that to pay a cabling company to do the cabling infrastructure they said we're not willing to pay the the 500 000 to end up getting a cabling infrastructure built and this company was kind of scrappy in one sense is they said go hire some some college kids and go teach them how to do cabling and i was one of those college kids that just said sure i'll do anything and i dove in i learned about cabling cabling infrastructure uh uh learned about 66 patch system 110 patch system um learned how to crimp uh crimp enzyme cables really learned everything about layer one in such extreme detail and by the time we finished the building it was great is we found of i think it was something like a thousand network connections that were made there was only one connection in the building that was miswatching wow uh so that's impressive we ended up doing the record right there yeah yeah especially for a bunch of college kids that didn't know what they were doing when they started right that's impressive well okay so i'll i'll add fear helps when you're scared saying i don't know if i'm doing this right or not and you do it slow enough and methodical enough that you say i i want to make sure that none of mine are bad so okay if it's bad it's the next guy and actually i i don't remember who it was that had the one bad connection i think nobody nobody really told whom who did that one bad connection because it was just pretty amazing but uh you know going slow and methodical and just doing it right the first time meant uh we don't have to go back and redo it and i'll admit i learned from that is is that if you hire willing and dedicated college kids to do things and say you're going to learn a skill from this that is really going to aid your career i mean it wasn't just cabling for network we did cabling for cctv systems back then it was a separate system then the networking we did the phone system also back then separate uh wiring system separate cabling system we also installed alarms and so it was really we did everything for that physical plant and it was just amazing the speed and things that we learned uh and yet they got us to work 10 12-hour days weeks on end because we figured the amount of learning we were having was incredible so i guess my first suggestion to somebody who wants to get into this career is find a company that they might say gee we want you to work 8 and 10 hour days it's like yeah that's fine but make sure make darn sure that that is a company that is going to teach you skills and they're going to have the proper mentor to teach you those skills so that at the end of the first year of employment you are worth double of what you were paid previously and that just means that at that point you can say okay if they don't offer me a a raise or offer me a new position that's higher level based upon the fact that i am now fully skilled and also i have the tribal knowledge of of knowing every bit of cabling that went into that building i was incredibly valuable to them and actually the team of four of us were incredibly valuable um because we could go find any piece of cable uh in a second so make sure you get started do you do you feel like learning all that physical infrastructure helped you become a better network engineer as your career progressed so yes absolutely so in one sense um you look at the seven layer model and say wow there's a lot of complexity here and i i don't even know how to attack it by the fact that they took us in and said we're going to teach you building cabling systems cabling infrastructure uh learn how to count uh uh count the wires you know you have the uh you know the color codes of the wires and also do that for 25 pair uh do that for you know larger cabling infrastructure i look back at all that and it's it's really was foundational to me being able to say i really know everything i need to know about layer one and it made it so that when i started looking at back then the novel network saying okay i'm gonna start learning about hubs and switches and what happens at that layer uh a lot easier and then eventually it's like okay now some larger networks and you get into routing and what is the purpose of routing and so learning in stages is was so important in some senses you get you get a lot of stuff just force fed down down your brain and it's like you know packets and and frames and and you know routers and switches and all that stuff just jumbles up in your brain if you could just focus on layer one for a week and then focus on layer two for a week and then focus on layer three for a week it locks it in so that you get it as to what happens at each of those layers and can say okay is this something that is likely to be a layer one problem or is this something likely to be a layer two problem or layer three four through seven it allows you to compartmentalize and troubleshoot a lot better what were you yeah i really appreciate that methodology i i do have a question though i know you said it was it was this was a while ago but it does sound like you made a lot of cables have you regained the feeling in your thumbs yet yeah because they yeah you didn't have the uh the easy ones did you the ones that go through the uh through the end of the rj45 connector no no it was it was manual trim and you're trimming and cutting and trimming and cutting and you'd goof it up and then every single one you had to do a cable test because there was just you know the so those there doing the cabling was uh yeah you had had to do a cable test of those because it was just too much chance you're going to go wrong that's all i've ever done i i know that they have the ones the cables go through but i had a trick with my thumbnail that that's how long the wire needed to be when it went into the ice cube so that it would be the room right yeah be the right length so that the jacket would catch in the thing and yeah and maybe that's just my background tim but you know i i hear people i i try to pound on like you know physical lab learn the physical it's important it's helped me in my career and and it gets poo poos like ah you know who needs that that's ridiculous i have data center people for that but there's been so many outages i've worked and i have a dc ops person remote somewhere and it helps me because i'm trying to get into their head see what they see because we're working the problem together and they may or may not be all that skilled so it to me i can always lean on that physical infrastructure background i think it's made me a better engineer well agreed physical layer is always a place to start and something i'll share as well is when i left that job i think i spent seven years at that company when i left that job and joined the next company i worked for a far smaller organization and i got to say you feel real insecure and real scared when you start your first day on the next job and it's like oh hey this is our new it manager and it's like oh they're going to fix all the problems and you're like yeah i'm going to fix all your problems oh i don't even know your network and i figured what's the first thing you do you step in the door and the very first thing you should do is meet the people that's actually the most important part but the second thing you should do is visit every cabling closet and start understanding the physical infrastructure and if you start documenting things and saying oh okay there's a switch over there uh this is this closet it services this building it services this floor and you start just taking notes and you get the first week of your work is getting that physical layer understood your next week should be okay let's get some switches where we log into switches start understanding what's going on at layer two and then layer three and you start working your way up the stack at that point i was able to say gee i feel like i have some confidence of understanding this network because i know what we have i know how things are connected i know where the tribal knowledge is the skeletons are yeah you finally see what you're working with basically you must have had so much trouble shooting pain as a network engineer because for you to be incited to create a company and build software to solve it i mean you must have some great stories i'm thinking back to my longest i think it was 26 hours of a maintenance one so so here's one here's one that's in one sense it's insidious but it's it's a crazy story so we had a user that was uh connected to the network and to say yeah they're connected but it's incredibly slow for just them and i figure okay well let me find out what's going on go back to the physical layer find out okay there's link light in the in the the closet uh check the switch port and say okay this is linked at 100 meg okay that's not too bad because they're just doing doing web browsing 100 meg for web browsing should be fine um but they're still having a really slow connection i check the utilization on the switch port find out gee it's using like two percent so there's no problems with utilization okay end up spending a bunch of time thinking about it eventually i say well let me go to the user's pc and see what's going on maybe they have a mini hub in their office or something nope no mini hub they're plugged directly in and i stumbled across the fact that their network connection showed as connected as 10 meg well wait a minute they're connected as 10 meg with their network connection they're connected as a hundred meg on the switch port how does that happen and i sat there and stood for probably five minutes until i realized that somehow this connection goes back to something up in the plenum got a ladder popped into the plenum looked around found a switch hiding out there a little 10-100 switch and it's like oh my god who puts a switch up in the plenum i've seen that happen i've seen it yes that's sad it's sad that multiple people have seen that happen yeah that's uh that's crazy um so so going back to uh path solutions just a little bit um you were talking about you know you use a little bit of snmp use a little bit of syslog a little dusting of the config you know um how do you so what i'm trying to get at is what i've experienced with typical likes you know network monitoring systems and stuff like that is the alerts from them right like you get an alert saying that hey ping is down or you know this power supply is acting weird like it's it's not seated or something and you have to reseed it right but you're what you're kind of saying is that the past solutions will kind of figure out what might be the issue and send that alert to you rather than just saying hey this interface is down just thought i'd let you know you know that kind of thing yeah so so so that to me that's like okay now we're going into the next level of a network solution right or a monitoring solution so can you explain that a little bit how does that look to to an engineer right like if if if it's figured out something's wrong with the way i configured ospf or or like you're saying uh weird errors on an interface or something like that that might be linked to a misconfiguration of some sort uh what what does that look like to us so i guess what i'll do is before i specifically answer that question i'll back up a little bit and talk a little bit about how we do what we do okay so we're going to go out and scan the network we're going to find all of the snmp manageable switches routers gateways and firewalls that happens really within the first maybe 30 minutes of deployment okay as we find those devices we're then going to start asking what are you okay this is a cisco nexus switch this is an asr uh this is extreme you know an extreme black diamond from years ago so we find out what that device is we found an old three-comm switch so we find the stuff we then find out what is it and then what is it capable of supporting and really what that means is we start digging around with snmp to see what mibs it supports what oids it supports what information does it have that allows us to learn from it and if it's a cisco nexus great it probably has qos counters let's go see if it has qos setup on interfaces so there's a huge bridge a branching path that we take based upon the os or the manufacturer the os then the configuration and so we'll dig at the device to find out what does it have for us uh at that point then we start putting that together with syslog information netflow information and say okay these are the clues we can get off of this device based upon those clues we have a heuristics engine now the heuristics engine is actually this huge tangle of code that's really we've had around for uh really the inception of the company we started the the heuristics engine really early on because we figured okay we want to start solving problems let's just keep adding problems to this so that it gets smarter and smarter every single year we do a release and at that point we're able to end up splatting a network where we install it's scanning the network it's finding out what you have it's finding out what the problems are because it's able to properly interrogate the devices to say okay you have this type of device with these types of error counters here's what it's trying to scream at you here's what it's trying to tell you it has a vlan tagging fault and you have an mtu mismatch and you're dropping 30 of the packets okay in that case we're gonna give an alert and send an alert out that instead of saying gee you're you're dropping packets somewhere in the network we're gonna say along this path you're dropping packets on this interface because of an mtu mismatch that way junior or senior level engineer the problem has already been found and diagnosed resolution can now occur i have two questions that's incredibly valuable is the discovery automated does it just go out and crawl and lldp whatever he's doing okay again again the dream is you splat some software on your network and say tell me all your secrets and actually we've had some of our customers they use us in a what's called our tech expert mode what that means is they deploy it on a customer network let it run for an hour they come back and say okay here's everything that's broken and they can print a report out sit in front of the customer and say okay let's go through kind of like when you bring your car in to have it fixed by a mechanic let's go through everything that we want to fix uh it'd be 30 minutes to fix this problem an hour to fix this problem uh five minutes to fix this problem and you go through with the customer and say here's our our billing do you want us to go do this now that we've found all the problems do i have to allow you with dty apples or like how do i allow your just snmp and and ssh okay so in one sense i'll admit we are going to replace a standard network monitoring platform that was my next question folks that have you're not running and they have old school you're not running with spectrum yeah they're tearing something out and putting this in yeah i mean you can run us alongside of it because in one sense most of the old-school monitoring solutions they just don't go deep enough to tell you what's absolutely yeah they'll say they'll tell you about nowadays you're just going to get a trap they're great yeah a trap or or an outage it's like they're great at outage detection the problem is the world has moved so far beyond outages that it's like uh why am i having a route flight okay so so you do all that plus all the intelligence and going deeper yeah so we'll do configuration management we do network diagramming we'll do all of your netflow uh path mapping uh we have a voice module that does call simulation so we can do apm types of tests we also have a work at home solution so i can end up measuring all of your guys wireless signal strength i can tell you about you know uh uh problems with ssids i can tell you about your neighborhood uh so it really kind of automates all of that sort of stuff so we can say gee do you realize that you know you're in your neighborhood everyone's using wireless channel six and uh nobody's using channel three and if you switch to that you're gonna have it all to yourself you know this sort of stuff so is that a is that an endpoint agent at that point that for that it's an end pain endpoint okay yes okay um something that i that i want to jump into you'd mention configuration management and that's that's something i'm pretty pretty passionate about i think it's really important can you kind of dive into that a little bit am i with pass solutions configuration management am i getting reports when there's changes in the config so i can see the side by side config difference and who did it is that is that part of that solution yep yep okay so we'll back up the configs on a regular schedule so like friday at 8 pm we'll do the full backup also if somebody goes in to make a configuration change we're going to detect that change immediately go perform a backup do a diff against the previous backup and say here are the two here's the two new static routes that were added to the core router and who did it and then send an email out to the team saying here's who just made that change and what the change was so that if there's problems everyone knows what needs to be backed out so it's a nanny there's nowhere to hide it's gonna tell you finger point dan did it blame it no i i think that that is that's such an important solution because i i'm not going to spout off a fake percentage here but i mean how many times does something break or something become affected because a change has happened in the environment or you just you want to know if there's going to become configuration drift over time so i really think that configuration management backup configuration differences is is super important i got a question but it's also it's also people lie oh no no changes were made yeah yeah i was about to say here's another thing too is if uh me myself if i know my changes are being logged and uh and people are getting alerted that i made a change you know i might be a little bit careful on what i do you got you guys know me i'm a send it kind of guy and if but if i know like in our meraki setup we have it set to where uh anytime we make changes and stuff like that it emails the group saying that that you know such and such network changed here's what changed and it was changed by this person except when it's your sign in right you've got exactly that's the that's the beauty thing of being the admin now so just being just knowing that you're being logged on that kind of uh on that level right and it's being alerted to people you know that kind of makes you wanna you know related to configuration changes like that um back when i was a network director i had a team of network engineers and at the end of the year there was one engineer that i awarded the fat finger award oh nice oh my god oops they typed the wrong route in on the router for the the new route to go in and just goofed up routing for an entire organization it happens nice hey i i want to go back just a little bit you you said um your scan will look at the the oids uh or however you say that i i think you had a i think you said it uh you know what how would you say that oh wait no there's no standard go ahead go boyds yeah but yeah anyways so let me get this right do you have to manually put in oids for things or god no oh my gosh well you've got to be sold right here because the whole purpose is okay if we look at technology as it moves forward technology tends to make other technology disappear yeah do you guys worry about how to apply the choke on your car to start the engine and how to apply this no technology has been done to automatically deal with that whether it's cold or hot and the engine just starts so if you can have technology that makes other technology disappear then you're moving forward and so that's what we're doing is we're eliminating all of the snmp we just need an snmp community string or credentials to be able to get access to the box and we will automatically interrogate the box and tell you you've got this problem on this interface you've got a missing default route on this device and it's sending packets off into the loopback adapter you know this sort of stuff okay am i the only one who has struggled with the trying to figure out what the oid is for certain devices or is that a common thing i don't know i feel like it's a common thing it is common and the thing is is that having people have to learn where do you find the data i mean i'll admit i spent half of my career digging around to say where do we find the data oh this this set of oids or oid is deprecated so you have to find it in the new location oh this isn't provided in the the mib2 standard section you have to go in the private enterprise section and you're just digging around forever hoping that you stumble across it doing massive google searches to try and find out where do i get the data and i figured again that day and age needs to come to an end where all this information is brought back and normalized to say oh if you have a cisco router we know how to collect the information off of that if you have a a a an entire system old and terraces box we know how to get the information off of that and so whatever the equipment is we can dig at that and say here's what it's trying to tell you so that you never have to deal with with specific oids ever again yeah so the magic is just built in yeah that's that's awesome because i i i personally don't have time to sit there and try to dig and find oid numbers for i need to know what the temperature is on a cisco switch kind of thing you know and uh so knowing that this kind of just does that automatically that's all right i know he's got it you gotta check for me right here we're checking off of things on my list here dan doesn't have time for that he's got he's got anonymous config changes to me yeah exactly so tim i i think we can all agree that that having you know as complete network visibility as you can get is awesome and it's really important actually our moniker is total network total network tim it's a sponsored episode i can't get it but while that that total network visibility don't turtle your network by the way i'll add that plug as well while that's really important i think there's also something out there that's called alert or alarm fatigue so and that's when you're just getting so many alerts throughout the day that you don't know what to focus on so how does pass solutions how do you how do you handle that like in an access layer switch i don't need to know if somebody walks up to an access port and plugs it in and unplugs it i don't need all those alerts but i definitely want to know if you can't reach a loopback or uplinks go down so how do you handle that alarm fatigue with past solutions so in one sense setting global alarms is the thing that people are scared about doing because you're right you set that global alarm and say tell me if any interface changes status and i will admit we do have some government entities that use our software that want to know if a single interface anywhere in the infrastructure changes status because that's a security concern um but do they have a person that just sits there and looks at that because i can't keep up with that exactly like what tim was saying no they have networks where nothing changes and nothing should be a spy oh okay yeah jason bourne just plugged an access layer i got you that makes much more sense within 10 seconds he's saying i'm in that's the magic hollywood you know hacker you know 10 seconds and then i'm in yeah um so the global alerts are something that a lot of organizations fear and what we've done is we've made it so that you can squelch you can still have global alerts yet they get squelched because you can say let's make it so that if any interface in the infrastructure has more than 10 percent packet loss i actually do want to know about that that is something to say help desk go out fix that problem because that user is going to complain about a problem because 10 is enough that you can drop a connection you can have slow downs and so that's that threshold if any of my infrastructure interfaces meaning trunk ports uh router to router connections links anything that is part of the infrastructure has more than four percent or three percent packet loss i want to know about that because generally that means that you're going to start having voice issues unified communication anything that is real time protocol you need to be concerned about do if i drop any packets out of a high priority queue your queuing is set up so that you should have enough q space enough q depth for all of your high priority protocols and if you're dropping packets out of that it either means you have a misconfiguration or you need to add more queue depth or you need more bandwidth so knowing about that is something that you can say globally i want to know about queue drops or q drops only on high priority queues so you can do some global things with these alerts that are a lot more smart i guess compared to a lot of other products where sadly a lot of folks install network monitoring and they say well what does it get to you well it pings things so i know about outages and i got a nice utilization graph on my wan link but they don't know much else and that's because of the amount of effort needed to build out collecting information and then interpreting the information manually and i figured it's kind of like the the the the new toy that you set up and say hey i spent 15 20 minutes setting this up i want to play with it if it takes four hours to set the toy up nobody's going to end up having fun with that nobody's going to benefit from that because the amount of setup time right that's me is all of your ikea furniture unpacked and actually assembled so this is easy to set up is this like an appliance run as a vm somewhere so it's actually a 90 megabyte windows installer so this will install on any windows machine typically our customers put this on a virtual machine one of the benefits is that it's far more efficient than a lot of other products there's no bloatware here in our development lab we have giant science saying no bloatware um meaning that it's all coded in c and c plus plus so it's actually very fast on lesser hardware so there's a lot of other products that they say oh well you need a four-way box or an eight-way box and a separate database server you need 24 gigs of ram all of this resource for their monitoring software and we can actually get away with like a two core processor four gigs of ram and 20 gigs of disk space and we're going to be a lot snappier in performance based upon the fact that we're just far more compact and more efficiently coded wow that's crazy so do you have a do you just have a single monitor do you need to have multiple monitors does it does it depend on the design of your network like how does how does this design so typically i'm going to say a single monitor is really everything everyone needs um so you don't need to have multiple monitors um even across wan links you don't need to have multiple monitors and the reason is is we designed this back in the day and i'm going back to history again of fractional frame relay links the 64k fractional t1 links the 128k links we can collect information across those narrow bandwidth links and not flood the link and the reason we can do that and other products can't is that other products will send a flurry of snmp packets to collect their little bit of information we figure huh we have a five minute window let's slow collect the information go get us all of the oids for interface number one and then we wait go collect all of the interface number two information and then we wait interface three information so within a five minute poll window we're gonna get all 48 ports on that switch but we're not gonna flood the link and we're not gonna overpower the device by hitting it with a whole bunch of snmp requests at the same time so we're really designed to collect the information the way a network expert would want it collected so it doesn't cause cpu spikes on the devices and won't cause bandwidth spikes and yet you still get far more information than you get off of other products so we're well behaved i guess nice is it easy to manage like if i have to install this and now it's my baby right like andy that thing's yours now you know like are there upgrades i have to upgrade the code is it is it just set it and forget it it's pretty much leave it alone so we've i've been in operations for too long and so it really is it's set it and forget it if anything if if one of my engineers comes up and says hey i just want to have this manual routine no nobody wants to run that there's so eliminating the maintenance of the tool is is kind of that's paramount you don't want to have a tool that sucks five hours of your time each month just to have a payback of maybe you know 20 or 30 hours that means you only get 15 you know 15 20 to 20 hours worth of benefit i figure if we can reduce that to as close to zero as possible and make it so that it's helping you solve problems that would normally take days or weeks to solve and you spend five minutes saying we found the cabling fault we fixed it and the problems just disappeared that's really the magic that i want our customers and engineers to have is the ability to solve the problem before the user complains about it or to be the magician so that when the user complains about it they have it solved within five minutes because they know their network speaking to me right now because uh one of my co-workers she's she's the one who's like working on our monitoring system and when you're talking about taking time to just sit there and maintenance the tool and figure out the tool and get it you know tuned the way you want it and everything like that that's what she's dealing with right now so and what's sad is that tool that they they probably have i could hide a half duplex hundred meg connection somewhere in the network and it would never be aware of it yeah probably i could have a qos mis configuration never be aware of it i could have spanning tree topology changes and never be aware of it it's all of these monitoring solutions there's so much they don't know that people end up getting trapped by they a user complains about a problem you check your monitoring solution and it says everything's healthy what's the problem the problem is there's a disconnect between what's actually happening on the network and what the monitoring system is aware of and if the monitoring system is aware of only five elements of information and there's a hundred elements of information available that's a lot that goes missing that you have to manually go collect yeah now let me let me also ask this you were talking about a a work from home uh agent that i would assume does it run in the browser does it run on their machine we have two versions we have one that runs on the machine it can collect all sorts of depth of information because it runs on the machine but we also do have a browser uh solution as well to help troubleshoot the work at home users now do you guys have like a server edition uh where maybe you can like monitor server to server kind of traffic or is that not something you guys are looking at right now so we can monitor windows servers we can monitor services um did i i don't know if i'm going down server and services because we do also in ingest wmi information so you don't need agents on servers yeah so we can monitor the servers uh it's really a full-fledged fully capable network monitoring and management system okay gotcha we're just kind of there's also another go ahead go ahead there's also another module i didn't even mention which focuses on security we kind of realized uh a couple years ago is that the amount of knowledge the depth of knowledge we have automated brought into the network would really be fantastic for security so we we figured we'd create reports and capabilities that allows us to have a security module as well so we're able to tell you about vulnerabilities policy errors policy violations i should say we have soar built into the product so we can do soar activities so geographic risks exposures rogue devices so really all of your iot devices where are they who do they talk to so a full-fledged security package as well nice and i guess it has its own alerting and everything like that that does it have like a separate dashboard or anything like that yep yeah it has its own security dashboard gotcha this might be so i wanna i wanna keep digging into these what i'll call kind of non-traditional network monitoring features that you have uh within the pass solution suite and i wanna start uh next with you have a feature that deals with voip troubleshooting can you kind of dig into uh what kind of value you get out of that feature so if you have two phones on a network and somebody comes up to you and says hey i had a call quality problem 2 30 in the afternoon what happened most network folks will sit there and say uh well there's nothing in my logs and my utilization graph on my wan link looks pretty good at that time and that's as far as they can go that means you can't solve the problem and you have to tell the user maybe the next time this occurs i might see another clue and realistically this means the problem has to go on and on and on until they escalate with their boss and then you're having your boss breathing down your neck and now you're logging into switches logging into routers randomly searching the network hoping you're going to stumble across what the problem is so we figured automating this means that we can tell you between that phone and the other phone at 2 30 in the afternoon here's what was happening on every link every switch every router so that you can look along the path and say oh we had a qos miss configuration on this interface and it was dropping six percent of the packets out of the high priority queue and two minutes later we were dropping packets due to a bad station cord for the far end phone so this was two problems that were happening along that path that if you solve one of them yeah you'd solve some of the problem but recognizing that some of these troubleshooting issues and i think we've all encountered this is is the troubleshooting issue that's actually multiple problems working in concert those are the ugly ones we can pinpoint that and say here are the two problems that are occurring so you can get the root causes of the problems solved really within minutes of the person saying i had the problem or better yet we can send you alerts on that so that you can tell the user hey i'm already working on fixing the qos issue and here's a new patch cord i'm going to put it in on that far end phone and thus you tell the user we're ahead of the game because we know what the equipment did to mangle your packets also on the voip side we have a call simulator that will simulate voice traffic across the network so you can find out where do we have latency jitter packet loss out of order packets and by the way i'm on a huge soap box about out of order packets in that uh networks are far more resilient than they've ever been outages are far less but the reason the outages are less is there's more paths to get to the destination more paths at layer two more paths at layer three that ends up creating more chance for out of order packets well out of order packets for data is fine the transport layer will just reassemble them out of order packets for real-time protocol sound like yoda we would you can't have out-of-order packets so finding out where do you have the out-of-order packets when do you have the outer order packets so that you can say okay let's put some proper uh uh per stream load balancing on those layer two links let's put some per stream load balancing on layer three links so that you're not having those out of order conditions uh so we have a call simulator that can help you find where those problems are happening gotcha now will it actually hit real phones or or how does that it can target real phones okay effectively what we have is we have a patented technology that uses uh ping packets that are wrapped inside of a g711 codec packet or you can choose whatever packet you want and we then mark that packet with dscp so according to the network it looks like a g711 packet according to the destination it looks and says oh this is a ping packet turn it around and send it back that way we can target midpoints so we can target a switch we can target a router we can target the far end router so you can test just across that wan link then you can test all the way end to end and you don't need a remote agent very nice yeah that is really cool uh the the next thing that i wanted to talk about is something that i know dan and i at least have gone back and forth on multiple times with our mad vizio skills and that is uh static network diagramming is difficult and if you don't keep up on it can quickly become a waste of time because it just it's meaningless after a certain period of time if you're not constantly updating it with changes pass solutions has a network diagram tool um within the product how does it work and what does it look like so again it's it's that we realize we have this incredible data set of information that we've automatically pulled from these devices what can you do with it okay we can do path mapping we can do root cause troubleshooting you know we can also do diagramming so we added diagramming figuring we have all that information we have the routing tables bridge tables arp cache spanning tree uh of all of that information to put together to say here is the topology of your network and you can just download it to visio you can use it to move elements around lock elements in place uh search for things if you have a particularly large network and you're thinking what is the purpose of the 10.51 subnet where does that exist and it will help search and say oh that's located in the austin office and that connects the round rock switches so you always understand where things are in a large and complex network that's pretty cool especially for like a let's say we're talking about like a new junior engineer or something like that and they don't have that tribal knowledge of a even if it's just a medium-sized network right um and they need to figure out where something's at this could be a search tool for them to not go bug the senior engineers right yeah back in my day i had to cdp the hell out of everything yeah at notepad plus plus and cdp everything yeah that's another thing is we take in cdp and lldp and that's another way we learn about things in the environment so we'll we'll pick up the cdp information and say hey there's a switch out here uh or there's a device out here and that's another thing we looked at a lot of other monitoring software packages is they don't benefit from that cdp or lldp and we figured it's knowledge bring the knowledge to bear so that somebody can say oh that's what's connected out there so basically what i'm hearing is uh past solutions you're soaking up all this data that's out there on your network and and you're putting like a ccie mindset uh on what might be causing that issue right is is that is that fair to say that yeah okay yeah all right so the the tr you've really discovered the trick is that if you have a junior level engineer they can start solving problems within the first hour of deployment because we found and diagnosed them a senior level engineer is going to say oh happy day i don't have to go through all of these logs i don't have to sift through snmp i don't have to spend hours and hours and hours i've got the problems found and i can verify that yeah it made the correct analysis and go solve problems today catch you where the hell have you been in my whole career tim yeah right is that how y'all do pocs like like splat the network and then be like look you got this this this and this wrong buy our product it actually is so we we do proof of concepts like that and what happens is people are shocked to find that with their existing monitoring software they don't really know their network because what we will do in many cases is we'll deploy and then say did you realize this this session border controller is running 100 meg half duplex and there's a little bit of embarrassed conversation but you see it magically full duplex and it's like okay that got fixed in you know 30 seconds because you were embarrassed about it yeah but on the other hand the poc means you're going to start improving your network from day one right and that's really the value know your network i was gonna ask my my follow-up question was gonna be and how many butt hurt engineers do you get whenever it is just silly uh misconfigurations like that because i know i would be like oh crap dang it so yeah there is a little bit of embarrassment that happens occasionally yeah um but on the other hand enlightenment is actually a far more valuable feeling than embarrassment so that tends to work to our favor nice that's that sounds like an awesome strategy honestly tim are there any of these feature sets um that we haven't touched on that you think are important that we should warrant some more conversation on um i did want to finish one last uh follow-up on the automation part if you do have large-scale automations like for example you have 100 lines of acl that you want to put through on all of the cisco nexus switches in the chicago data center you can do that with one fell swoop so large scale deployment changes can can be done with the product as well so does that go along with like um yeah i think tim was hitting on earlier uh basically like a config audit uh like if if um i i don't i'm going blank but like if you if you if you had certain vlans on a switch right and then when it does its check it's missing vlan 30 right can it go back and push vlan 30 back out to that switch or so we don't do gold level checking gold level there you go but that is that is actually on our road map so okay are you sneaking into our documents no no that is on our roadmap that is something that we figure we're so close to that we can taste it because we have all of the other information yeah so having that gold level config is one of our very next features okay all right so hold on this this is this is exciting so it's not just monitoring it's not just intelligence looking at all the big data and intuiting things from it but if i have to go out and change an sm well let me think of a better example if i have to change i don't know a banner of the day on you know 500 devices i can push that change out with your solution so and and you can do it to select switches or switches that have a certain operating system on them or uh slice and dice however you want so instead of learning ansible and python and yaml and playbooks well it just sounds so much easier i've experienced the old way of managing a network and then experienced the automation stuff coming in and having to learn all that i didn't realize that you could push config changes with your platform as well that's it sounds easier than learning all the other stuff that i just said yeah and in one sense there is a purpose for ansible if you're doing super complex changes where you need huge if then branches and switch cases and you're honestly doing programming at that point then yeah that's that's perfect for ansible for most i'm going to say most meaning 95 of what a network ops person needs to do we're going to cover those bases that's awesome yeah i'm excited to see this uh gold config feature that you put in there so you can be like okay this is what i want my configs to look like now make sure it stays that way and uh you have to change it here if if you if you want the actual change to happen kind of thing i wish we could do a demo you know like the audio podcast i get all excited and they're like oh there's all the like i want to see it you know show me the magic yeah this thing looks sick are there a lot of colors 100 of the front end is is all restful json so any piece of data you want to pluck out of our out of our tool and it's all scrapable very easily um we've skinned it so if you want dark mode you have that i mean it's really it's designed designed by network folks for network folks so it's it's really this is what uh uh folks want i'm so happy you said that it has a dark mode i i i know that's so petty and and like you know whatever so important yeah it's so awesome whenever a tool has dark mode to me at least you're muted tim yeah trying to think of something to say before i said oh i got you i thought i thought you had started talking already no no i thought he had a stroke i was waiting i was waiting to see if anybody else could say something oh yeah no i mean i think we hit it all right end of day brain is empty this has been hugely informative i'm i'm yeah you know i think people are nuts not to use this i've never heard of i mean i'm guessing i don't know if you want to put this in the episode i'm guessing you have competitors i mean i've never heard of a solution that's doing what you're offering so in in one sense i'm going to say and this this might sound a little arrogant but in one sense i think we're as a company we're down the path where we're a little bit like palo alto palo alto over engineered their firewalls forever and they really didn't go to market in in early enough time and so in one sense we're over engineered and under marketed as a company okay so uh as more people start learning about what we're doing they'll start realizing this is a better way of running a network yeah gotcha and we'll we'll we'll cut like back and before tim unmuted or whatever uh but you're hitting on something i want to ask have you heard of crap what's that one called monitoring solution yeah yeah but they they do sort of the same with what you're talking about like monitoring all the different links between two devices okay so there's a couple companies in the market there's stat seeker is one okay and i'm forgetting what the second one was they're both australian companies um and they're monitoring every single interface okay the problem is they're not collecting the error counters they'll give you performance and utilization they give you utilization yeah but you could have a collision and not know about it you could have an fcs error and not know about it a runt packet a giant packet all of those different error counters uh symbol errors and still not be aware of it so i figure you're not getting the full value with that type of product yeah and that's where we figure we're going to pull all the error counters we're going to be graceful on the network as far as pulling all of that information and yet put a heuristics engine on top of that that says let's make sense of this data so that you can solve a problem today gotcha this this paper so yeah in one sense we're like we're like stat seeker combined with uh uh uh net brain for doing the diagramming combined with uh net scout for the analytics combined with a fluke meter for the fact that we can find physical layer errors yeah i mean we're really a combination of all of these best of breed solutions built into one software deployable capability that's a lot less expensive than all of those other solutions put together what is this software called total view total so this may be a goofy question but it's based on an experience i had so does total view monitor itself like what monitors the monitoring solution and and what's behind that is we had a router fall over years ago and we were using ncm network configuration manager and i don't know if that's proprietary or not but that's what we had and when i went to pull a config we realized oh this hasn't backed up a config in six weeks uh-oh yeah and there was some reason those systems tools gave us like oh you know i guess that agent stopped running and we restarted or whatever so i i just wonder you know we rely on these tools and monitoring and backing up configs and stuff but how do you know the tool is working or doing its thing are there self-checks like i don't know if that's a fair question but yeah no no it's perfectly fair is is in one sense something i've learned a long time ago is certain things you need to have positive as well as negative response you need to know did this successfully work or did this fail um so backing up configs if the password changes and we stopped getting configs oh yeah we're going to alert you and say hey we can't get into this device to back a config up if somebody changes a community string on a device we're not going to send the outage alert outage alert you can't talk to this device anymore we're going to send snmp communication failures and say the device still responds to ping so it's still up but this is more of an administrative error in the fact that we can't get our data anymore and so there's different levels of alerts that happen so that you know it's not an outage it's just oh yeah i changed all those community strings but i didn't change it on total view and another thing is we have it set so that if you do want to change every community string you know you go through and say i'm going to change every community string on the system within five minutes and say click and you go you can also go into total view and do a bulk change where within five minutes you can say change all of our community strings to use this new string save done so you're not going in edit device change community string yeah save edit device change oh my god a lot of tools they really hamstring you with the administration of the tool yes and that's something where it's like no don't do that people don't like that add a search and replace capability add bulk change capability good i mean in one sense i'll admit i'm an engineer i'm not a marketing guy i'm not a sales guy i know where we fit in the market um if you need to do extreme diagramming of a network then yeah net brain might be appropriate but for 99 of all of what network folks need we're going to cover the base yeah um ansible there's a there's a position in the world for ansible there's a purpose for it and it's incredibly valuable incredibly powerful do most people need to learn how to use it probably not with our tools they can end up solving the problem and they don't have to go through a lengthy training session right oh that's huge i i know we're probably getting toward the end but i wanted to mention so it's all about network visibility right and and something that keeps coming to mind and and we had earlier today we spoke about this but i wanted to bring it up here is uh you know we we had i was in an environment that had csrs and their licenses expired for reasons right but nobody knew right we didn't have visibility into the network and if you had gone in the logs there were thousands of messages saying license expired oh my god bandwidth has dropped help me but we weren't looking nobody was calling right like networks are big they're complex there's a lot going on there's other things you're doing so and it reminded me of something tim said earlier i've never heard of like alarm fatigue or whatever it was called so can i would would what's the name of the product i forgot or i'm sorry totally would total view know that if a license expires on a csr or any licensed device does that just automatically you know because if it's sending me every darn alarm it's crazy right now i'm fatigued and i'm not looking but can i set thresholds like would you have caught that and let me know because it was down for a month the the bandwidth because it expired we didn't find out until somebody pushed the big file and then it was a big nightmare and then people like well why didn't we know but we just didn't have the visibility and this was a big old company so would would total view have caught that would i have to set a threshold for that like for licensing type thing so so i think there's two ways not knowing all the technical details but there's two ways of of catching that and one would be syslog alerts is being able to say if we see any syslog that comes in with the word license i think that's something that you'd want to know about yeah so if those came in and those were just splatting like crazy you're going to say why are we getting these where we weren't getting them before and so that's one part of the picture the second part though is if those interfaces you know you say gee these are gig interfaces they should be able to pump a gig but i think you mentioned that they were only passing one meg worth of traffic okay we're gonna see the packet loss counters on those interfaces go crazy and we're going to say hey as of this time and date you're going to see every one of those happen and it's all going to be correlated so you can say as of this exact time we started seeing syslog messages that all had these license errors and we started seeing packet loss counters on these interfaces that went through the roof like crazy and those things correlated just means you're going to look at it and say oh i now know what happened because you have the data in front of you now in that case it sounds like that's probably not something that we would have automated saying you have a licensing problem and it's going it's dropping the interface bandwidth on these interfaces but we will have the correlation to make it so that you can spot the two just by looking at it right yeah that it's that's it's so valuable because when you're in a big environment managing hundreds or thousands of devices you're not going and looking at everything digging around for problems and if nobody's calling and and your monitoring solution hasn't thrown an snmp alarm like okay i guess everything's fine so i really wish i've had this tool over my career we could have caught so many problems before the customer called it and called you know it's really all about knowing your network to a broader and deeper level than ever before yeah it's awesome all right tim as we start to round this out is there any last-minute thoughts you want to give us on yourself or on total view um if you're interested in learning more about it you can go to our website dub dub dub path solutions.com uh we have videos that show more of what we do we have a sandbox that you can go browse around and actually play with a product uh you can contact an engineer and we'll work with you to set up a 30 day proof of concept and during that proof of concept we're going to show you things that you don't know about your network that your other monitoring software is simply unaware of and really prove our worth during that first 30 days so bring it i was going to say somebody i don't know don't get butthurt whenever they show you that i was going to say bring your safety blanket with you your i think we got dan's uh parting thought there andy do you have anything uh left to wrap this up no i really appreciate your time tim it's it's been really fascinating learning about this i sincerely mean i mean i know this is a sponsored episode but i wish i had access to this network visibility tool in my network engineering career because it was it was all very reactive it wasn't proactive and i don't feel that we had the intelligence or the visibility and our tools um to to find things like that so i i can really see you coming in installing this and like i said earlier like oh guys there's you know even in a park right like hey you got some stuff going on here you might want to look at you know it's i mean it is egg on your face but you'd rather your company come in find that for us and help us than a customer calling someday with a problem so now this has been really informative and and honestly i want to see a demo of this um it's just i'm fascinated by it well thank you all for joining us here this evening and to the listeners if you want to support us some more check out our patreon program at patreon.com art of netenge tim titus thank you very much for joining us check out pass solutions and don't turtle your network take care now hey everyone this is aj if you like what you heard today then make sure you subscribe to our podcast and your favorite podcatcher smash that bell icon to get notified of all of our future episodes also follow us on twitter and instagram we are at art of net eng that's art of n-e-t-e-n-g you can also find us on the web at art of network engineering.com where we post all of our show notes you can read blog articles from the co-hosts and guests and also a lot more news and info from the networking world thanks for listening you