00:48
This is the Art of Network Engineering podcast.

00:59
In this podcast, we'll explore tools, technologies, and talented people. We enter for new information that will expand your skill sets and toolbox and share the stories of fellow network engineers. Welcome to This Old Network, a LAN improvement show where we try to help you by showing you how we build and handle networks. Frankly, it's your choice whether or not you listen to us. In fact, we should probably have a legal disclaimer somewhere. Anyway...

01:26
Have you ever tried to troubleshoot an issue with nothing more than a few pings in your pocket? Perhaps you've spent hours troubleshooting an application when the issue is just a link flapping. We'd better flip that around since we know the issue is rarely the network. Maybe you spent hours digging through route tables when really a certificate had just expired. Hey, we've all been there. You're among friends here. Now grab your console cable and enjoy the show. Very nice.

01:54
Welcome to the art of network engineering. My name is Andy Lapteff. You care to look, you can find me at permitipandyandy.com. And I am joined tonight by Rocket Girl. Lexi, how you doing Lex? I'm good, Andy. I'm learning a lot about pronunciation of words this week. So.

02:21
As an example, for example, according to five million men from TikTok, the proper way to pronounce this word is. Ethernet, not not ethernet. Ethernet, either or ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet, ethernet

02:50
Ethernet that's how you're supposed to pronounce it. So I have learned from my mistakes I realized this was a cardinal sin that I've committed and I am Dedicating my life to making it up to the people. I'm so sorry for those whom I have offended With my pronunciation of this that is the most sincere apology. I've ever heard in my life. I Love I mean it I love what Pete Lumbus

03:20
It was a little, you know what I'm talking about? Which one? It was like a funny video thing, you know, in response to your pronunciation thing of like, I forget what it was about, you know, men having to tear. Was it well actually? I think, yeah, yeah, yeah, yeah. It started out like well actually. It was just really, it was a really funny take on. Are you a man who feels the need to correct a woman on something she knows more than you about?

03:48
Maybe you should try just fucking not. Yeah, that was that right. I don't remember exactly. Yeah, but that was the, that was the gist of it. No, that doesn't sound like it was all. It was almost like a pharmaceutical commercial, but the name of the medicine was like just fricking not or something. Like it was, it was, it was great. But I love how you pronounce words, Lexi. I like, like, I, like I shot back at you. I have some words that I say pretty funny because of where I'm from. So you, you, you say you call ethernet, whatever the hell you want to call it.

04:18
You know more about Ethernet than I do. Call it whatever the hell you want to. I appreciate it. You know what? I've changed my mind. I've reversed my position. I am not going to change it or make up for any of this that I've committed in pronunciation of network terms. I implore you to please call it Ethernet for the rest of time. I really, really... I will. It's already my default, so it shouldn't be too hard to keep doing that. Well, Lex, I'm sorry that...

04:48
People are giving you a hard time, but keep doing what you're doing. I love your content. Did that. Thank you. And also tonight, Mr. Tim Bertino. How you doing buddy? Andy, I'm fantastic. You know, everybody, Andy was just giving me shit before we recorded that, uh, of my, my bright sunny disposition all the time. So I'm going to fluff it up even more. Andy, I am fantastic. So happy to see you. It's, it's nighttime, but I'm pretty sure there's a rainbow outside.

05:17
Things are fantastic. I don't know if you caught it in there, Andy. You were partly the inspiration for that intro. Cause you had recently told us cause I'm old. Oh shit. Sorry. That wasn't even it. Without Dan, I didn't even bring up your age. That was awesome. You just bring it up yourself, Andy. But we had something, uh, we had something in our group chat. I think a few days ago we were talking about

05:46
something that you had fought in the past where you were on this huge troubleshooting call for a long time and it ended up just being an expired certificate. Oh, yeah. That shit happens, right? I mean, dude, hours upon hours. You know, the app people, we're going to get an app person on here and just eviscerate them.

06:10
I can't tell you how many bridges I've been on with a crap ton of people and it's not the application. We didn't touch it. Everything's fine. Nothing's broken. It's definitely the network. People can't get to this app. We're losing a million dollars a second. You need to fix your network. What's wrong with you people? Why don't you people, you know, 15 hours later, oh, our certificate was expired. Whoopsie. Sorry. You know, but they don't even tell you that. It'll be like three days later. You're like, whatever happened to that outage thing? They never bothered us about like, oh yeah, their certificate was expired. Sorry. I'm triggered.

06:38
And now that Andy's triggered, Andy, how are you? I was fine till just then. My job is complete. I'm in a good place, Tim. I'm in a good place in spite of the fact that I quit caffeine about six days ago. This is news to me. Yeah, well, I've been quiet about it because I'm trying not to say much while I detox. I don't want to say anything stupid to anybody, but...

07:07
I am like a hardcore coffee guy, because I'm tired, Tim. But if once or twice a year I get into this migraine thing I can't get out of, which is caffeine induced, so I have to go cold turkey. It's this ridiculous cycle. So right now I'm off caffeine. Will I stay off of it? Probably not. But in spite of a caffeine detox, I'm really well. I went out to Sunnyvale, California.

07:35
for my first trip to Juniper headquarters a week or two ago. It was fantastic. It's sunny and warm out there every day. I got to meet my team in person, which was really... And listen, take this for what you will, but I love being remote and working from home and having that freedom. But man, it felt really good to be together with my team.

08:05
I don't want to move to California to be with my team. I don't want to drive three hours a day to be with my team, right? Like, I used to commute three hours a day to get to my old team in Philly, but it was just really nice. And I'm looking forward to going back out. I was in the lab, we were doing some cool stuff. And so that was really nice. And I'm decorating the house for Christmas. I got to get up on ladders tomorrow and put a bunch of damn lights up. That will look pretty. Careful. Careful with that.

08:34
I just walk on a 28 foot ladder. Didn't you fall off of one or did I make that up? She fell off one in the old jokes. Just keep on rolling. He was young then. As soon as he you all need to know, it's never young. As soon as Andy started talking about that, Lexi steps back. You bet. You better be careful, Andy.

08:59
You're too old to be doing that for too much longer, Andy. Oh, Lex, you're lucky I like you. Listen, before we jump into this episode, I just want to put out in the universe that I want us to have an in-person meetup next year. I don't know why. I don't know how. I don't think we have the money for it, but I would love to see you beautiful people again and try to get together with some fans and have a beverage and. Cheeseburger.

09:28
Dare to dream. I'm just putting it out there. If any listeners are out there that want to help us. Seattle? Yeah. I'm down. It's a beautiful airport. Is that is that the most you've been to Seattle is the airport? Yeah. I mean, I remember I get a text. I get a text from Andy on a random weeknight that's like, hey, what are you doing? I'm at the airport. Come hang out at the airport.

09:54
You got a bar somewhere out in the airport? I think I actually did have something, but I was seriously considering it. I just couldn't get out of what I was doing. I'm in the lounge, man. Come on. Yeah. Did you end up in a lounge? Damn right. I'm like, I'm like a fancy fire guy. The Alaska lounge or what was it? First class laptop, I think is what they call them.

10:18
All I know is my whole life I've been sitting at the gate and man, you, you, you go up to this lounge and it's just like leather reclining chairs everywhere. Every food you could imagine an open bar. This did you see the picture he took when he went out to California? He had like this little like pod. He was sitting in. I'd never seen anything like that before. Wait, what? Oh, that was, that was in Philly. Yeah. They have these minutes suites. So Philly is such a ghetto airport. That.

10:47
They don't have a lounge and it just put you in a little closet by yourself. You think about what you did. So yeah, I, I, there's this, these minute sweets that you can go in and it's basically just a fancy closet that you sit in with a, with a desk and a sofa, but it was nicer than the gate because the gates are full of crazy people and it smells like urine. So it was a nice closet to hang out in, but I, I digress. Let's talk troubleshooting tools.

11:17
All right, come out of the airport into Seattle sometime. Maybe that's what we'll do next. I'll give you more of a heads up. I'll give you more than an hour lead time. Sounds good. Sounds good. I thought I thought it'd be fun if it was spontaneous. Like, OK, I'm here. You know, I was I was almost going to do it, but I had I could not quite get out of whatever it was I was doing. It sounds like a bullshit answer, but I don't remember what it was. But I remember being like, fuck.

11:43
It feels like a bullshit answer because the text response didn't seem like you were upset about not coming. Well, it might have been something better I was doing. I'm sure it was better than in the air. Anything's better than sitting at the damn airport. Hanging out at the airport. All right. All right. Network tools. Let's go. Network tools. I'm going to check the chat. This episode is a brainchild of our buddy Jordan Villarreal. I found it in our Asana dashboard.

12:13
We are talking troubleshooting tools this week for, I'll say mainly for network admins, network engineers, but different people could get use out of these. What I wanted to kind of start with was some really low barrier tools that are really available to anybody and everybody. And then we can just kind of see where this thing blossoms from there. So let's start with just kind of our basic troubleshooting commands in different operating systems.

12:43
And the first thing I wanted to bring up was was we teased it in the intro ping. So there are different operators that you can add to the ping command and I'll specifically talk about routers, Cisco routers, Cisco switches. In this case is they have the extended ping command and there's different minus T minus T that'll get you going forever. Right. The only one I know. So

13:11
On the router side, the extended ping command, one of the things I wanted to call out that you can add in there is you can change the default MTU. So if you're troubleshooting an issue and you're not sure why traffic isn't going across the link and you think you suspect that might be an issue, you can in the extended ping. One of the options in there is to change the MTU so you can change it from the default 1500.

13:39
go as high as you can before you stop getting replies. So I think that's really been, that can be helpful when you're troubleshooting things, especially across the service provider network, when you are kind of suspecting that might be an issue. Question. Yeah. Do you source your pings from like a particular interface or a loop back? Ooh, good question. Thank you. That's actually, I think that's in my list somewhere.

14:07
But I do want to talk about that. It does, right? When we troubleshoot, that ping command is what we go to first a lot of times. So when you're on a router specifically and you have different interfaces, let's say somebody says they don't have connectivity. And you go on the router and you ping that device, and it replies. And you're like, no, you're fine. You're up and running. If you just do a default ping, your router is going to pick the source interface.

14:37
from its routing table and that's going to be the closest interface to that device. So it's going to ping essentially from the default gateway. If that client can't get off network, can't get off its local network to another client, you're not going to find that issue by just doing the default ping. You would have to source your interface to another interface on the router to see if that client is able to route to you. So yeah, good call out Andy. Thanks. It all depends on what you're troubleshooting.

15:07
I think sometimes you go into it wanting to source it just so you know the behavior you're trying to see. Real quick, do you have quick tangent?

15:23
Do you have a set troubleshooting methodology you go through? Because I don't. And to me, troubleshooting is an art, which when we came up with the art of network engineering, the first thing I thought of was troubleshooting because I've tried to teach troubleshooting to people. I've had people teach me troubleshooting. Even as a cable guy for Comcast, they had all divide and conquer and they had all these ways to like... But there was usually 10 ways to do it and there wasn't one definite way.

15:53
So do you guys know Andy? Yeah. Yeah. Do you have one? Yeah. It makes me really excited. Um, because I just did a tick. I could see it in your eyes. It's like, uh, yeah. So this won't work for every single situation obviously, but, um, when I was first starting out, knock break fix, um, I didn't know what the heck I was doing. So I was taught to start at the bottom of the OSI

16:23
model and move up. So start with layer one. So we'd start with like, okay, well, can you ping it? Right? If you can't ping it from where you're supposed to be able to ping it, which could be different places, right? Like, you know, ask the data center tech, Hey, is this plugged in? Maybe the link is actually electrically not even up and there's something wrong with the cable or the SFP or whatever. Right? So start with layer one, then move up to layer two, you know,

16:49
Is there STP involved, I guess, things like that? Like, is there a weird, like, is there a broadcast store? I don't know, you know, different things going on than move up to layer three and so on. We rarely ever get up to like layer seven, but when it's not actually the network that's causing the issue, I guess that's when you're up there. I have spent an inordinate amount of time troubleshooting things. It's such a complexity that wound up being a physical problem.

17:13
It's got to be this or what's this protocol doing? This looks like a software bug, like just jumping to these silly conclusions. Respect the physical layer for sure. And I remember in the CCNA, they, I don't know if it's true, but they taught, in my experience proves it out that something like 60 to 70% of all network issues started are the physical layer, the physical stuff, what they taught me. And you should, you should start there. You set up to layer seven, Lex. I wanted to, well, Tim, do you, do you have a standard?

17:42
troubleshooting methodology before we jump back in. I do the same thing. And I think maybe when people hear that, follow the OSI layer, they may think, oh, that's just overly complex. I have to think about all these things. But really, we're not saying you have to sit down and make sure you write all this out. I make sure I check this, that, and the other for the physical layer, and then I go up a layer and check this, that, and the other. It's really, after you do it, go through it multiple times, troubleshoot things over time.

18:09
That stuff is just going to become a quick thought where you think, okay, we just want to make sure we got physical connectivity. Yes. Somebody sees a blinking light or you're on a switch and you can see the ports connected, that kind of thing. And you just go through it quickly. So yeah, I follow the same thing and I do got to say, Lexi, I think you need to follow Jordan's chat comment here. You need to do a video where you call it the OC model.

18:38
I thought really hard about that. I agree with that 100%. The OSI model is damn near perfect for troubleshooting. Again, you don't have to follow it to a T. It just structures it to where you won't forget things that you should check.

18:59
So for the new people listening, can we just walk up the stack real quick? Maybe up to layer four. So layer one, I'm glad you were talking about blinking lights and stuff, Tim, right? So if you can see it, see the lights, great. If all the lights are yellow on the switch, maybe you got a spanning tree loop, whatever. But look at the physical first. Also remotely, I would just do a show interface, right? To see if I'm up, up, up, down, down, down. I've had a lot of up, down stuff that was like a unidirectional fiber break and there was traffic getting black.

19:28
Show interface status, you click for CRC errors, right? That's like physical-ish, right? If there's a physical problem, you'll probably have the cyclical redundancy check errors. So there's a lot to check there at layer one. Layer two, what? Show MAC address, right? Like, the hell's happening at layer two? If your physical's up, check layer two, right? Yeah. MAC address is at it. Check your forwarding table, honestly. Are you, and our entry, it depends on like how. Yeah.

19:53
You know how appropriate that is to your network topology and what's supposed to be going on there. But yeah, check your forwarding table. Make sure you have valid ARP entries for whatever you need and that can bridge you over to layer three then, right? I like how Lexi framed it originally when you're going into something that you're just trying to see if you have connectivity to something. You you start with the ping because you know, if you get a ping reply that tests layers one, two and three right off the bat in one command.

20:22
So if you don't get replies, then you know immediately you can jump down into layer one, layer two, but at least you're not starting with, I need to dig into why this application isn't working and you're completely bypassing physical layer routing, all that kind of stuff. So yeah, I think while ping doesn't necessarily start you at layer one, it's a quick test to see if you need to check layer one, layer two or not.

20:48
And it can also narrow down again, depending on your topology, what's going on. It can narrow down the section of your network where the problem is, right? When I was working, um, my last job, we had like global activity, right? Like this enormous network. So like something in Singapore, like somebody's server or whatever, can't reach somebody's server in like Germany. Okay. Well, there's like, there's like a whole distribution and you know, core network.

21:16
going on in between those two servers or multiple ones. So you want to paint, like does first server, can it ping its default gateway? All right, can the other server ping its default gateway? And move up slowly, like into where they meet in the middle. And you slowly get more complex too, because you're getting into like from the layer two part of it maybe to more layer three and like MPLS weird stuff and all that going on. So you start, it helps you start.

21:45
with the least complex thing a lot of the time, move into more complex. And you're also along the way, like narrowing down, you know, where the issue could be. And then once you sort of find a section of the network to narrow it down to, you can then maybe start at your OSI, you know, lower layers and move up a little bit. It's just like a way to, you don't wanna go too crazy and wild and all over the place and forget where you started, right? So it's a good way to organize your thoughts as well.

22:13
I just want to go layer three, four real quick. And Lex, I think you hit on that, right? Like, and it depends on the environment. If you're at an edge, you know, router at a client site, it's might be different things than you're because I was a WAN guy in a data center, like my physical is probably fine because the whole world would be on fire if my physical stuff was down in my data center, right? So what I would do is I would usually go my distribution layer when something was broken and look for the routes to the source and desk, try to ping them like, okay, great. But then the source ping comes into play and all that. And that was.

22:43
just the bare minimum, but I want to talk about layer four because I can't tell you how many times it's something I'm kind of blissfully unaware of. I'm not really good at layer four. That was a firewall problem that other people managed. But when an application, connectivity is fine up to layer three, but a port isn't open through a firewall. But how do you test that? I remember somebody showing me, was it like Telnet? Can you open a specific port through a Telnet or something like that? That's like third on my list. Yeah, I did.

23:12
I did want to frame that because I wanted to bring up Telnet and say no, not for connecting to devices, but for doing just what Andy said. We always get blamed, the network, the firewall, whatever, for, hey, you're blocking this port. This isn't allowed through. And that's one of the first things that I'll do is I'll open up a command prompt and just

23:42
The device space the port that they're saying is blocked. And you know that once you hit enter, if you get something that's not an error, right? If you get just like a blank screen or just some gibberish letters, whatever, you know the device is open and answering on that port. You can say right away, yep, the device, the destination is open on that port. It's listening. It's answering.

24:11
It's just not configured correctly for what you're trying to do. I did not know that. It's a good one. It's fantastic. Yeah. What if your network allows Telnet? Yeah. What is it about Telnet? Do you know what allows it to do that? Like, why wouldn't you use, I don't know, SSH to do that?

24:33
We'd have to ask smarter people. Yeah, I don't know. I'm just going to speculate. I mean, I know I don't know. I'm just speculating. I'm looking in the chat. I don't know if anybody knows. And you're just telling it to function on a different port. It's just a way to connect. Because Telnet is, let's see, Dave is saying clean TCP text based no crypto.

24:55
So what does that mean, Jordan? It's historically just there. It's magic. What does that mean? We used to use Telnet to troubleshoot up to layer four and then the environment I worked in shut Telnet down everywhere on everything. And I'm like, oh, I guess we'll just escalate the firewall now for layer four stuff. So sorry, Tim, I didn't want to take us down the rabbit hole, but I thought it's important because it's a troubleshooting episode. Oh, I love the troubleshooting stack. I mean, we're talking about tools. I love that we're framing it up with walking through the stack to...

25:24
to get us from point A to point B when it comes to troubleshooting. Ping, Telnet. I have one that isn't a tool I'd like to mention, the logs. The logs. Yeah. Checking the logs. I know it's not a tool, but it's a source of data you can share. I'm just thinking of things I used to use in troubleshooting, you know. Something's broken. Splunk. Somebody's compeloding. I mean, yes, Splunk is great, right? But trying to correlate times too. Like this customer couldn't connect from meow to meow and then I could check my logs and if I can do a time correlation of like, aha.

25:53
You know, these tunnels went down at the same time. They couldn't get in. And sometimes it's really hard, especially in a complex network. Like, I mean, there's so many things that- And in those situations, NTP saves lives. I mean, you gotta make sure everything's on the same time. But I always checked. I used to do, so I didn't have a really standardized methodology troubleshooting. When I was new, Lex, like you were talking about in the knock, I'd be terrified.

26:18
And what I just type over and over again to show IP interface brief, show IP interface. I just, I would just type one or two commands I know to like, just get around. Nothing's down. Let me check it. Show logging type include ethernet three. So it was show IP interface brief show log, you know, and then maybe a show ver to see if it rebooted. But yeah, show log. There's a lot of data in there in the log. I know it's not a tool necessarily, but people.

26:46
I would come into a troubleshooting expedition and nobody had looked in the logs and I was able to find, I could correlate events in my network on devices to times things happen. If somebody said the WAN was down for an hour and I look and I'm seeing the logs, the device should tell you if you have your logging set up right if something bad is happening. Yeah, but you got to have your logging set up right. And we're talking in the chat about time zones.

27:16
That can be tough. Hopefully you're all running the same thing. Everything should be UTC. UTC, yeah. Everything should be UTC. It isn't anywhere that makes it easy. It isn't anywhere. I want to call that out though, Andy. You talked about looking at logs and we were talking about the different layers of the OSI model as we go and troubleshoot. And really looking at logs can apply to every single layer as you go through that step. It doesn't really belong to any given one. So yeah, I think that's important.

27:46
Going down kind of the the ping vein, I wanted to bring out trace route. So traceroute basically just leveraging ICMP with different TTLs to show you the path that a packet is taking. Has anybody noticed? Just using default traceroute like in Windows, how incredibly slow it is to go through each hop?

28:15
Sometimes sometimes trying to think so. What what that is? Why it takes so much time each hop is by default. Traceroute is going to be doing a reverse DNS look up at every hop to try to show you a name for what that IP address belongs to or maps to from a DNS perspective. In Windows, if you do the. Tracer or trace RT.

28:45
I know Lexi, you get shit all the time for pronouncing things apparently very incorrectly. So, yeah, so I don't know if I'm saying that right. So hopefully I take some of that pressure off you this week. We'll just say tracer. So if you do in windows, if you do a tracer space minus D Delta for DNS and then space your hostname IP address, whatever, it will skip that DNS lookup process. So your tracer routes, if you don't care.

29:14
What the name of every hop is. You just want to see what IP is. It's hopping through as it goes through the network. That minus D will disable that DNS lookup, so it'll just go a lot faster than it would if it had to do the DNS lookups. So I wanted to call that one out. Which command gives you the response time? Is it Tracerouter ping? Because I know like if you're having a latency issue, you can find the hop. They both will. Like you know everything's. Okay.

29:38
Because I've used that too, like every hops like 10 or 15 milliseconds. Yeah, trace route will show you that. 4,000 milliseconds. You're like, let's look there. Something seems funny there, right? Yeah, there's even an application build off of that. I think it's called like Ping Plotter. I don't know if anybody's used that before. But it's like it shows you a graphical representation of it's basically a graph of a trace route. If you're trying to see latency from point A to point B, it'll plot that out for you, pun intended. That's really cool.

30:07
Now we're getting into app territory because I remember using a few different things at previous jobs that were built that were supposed to be all-in-one network monitoring tools and troubleshooting tools and some of the functionalities included. You can ping using this exact same... You don't have to go anywhere to ping. You can just ping from the web interface or whatever and do a trace route and lookups and stuff.

30:37
All from here. So easy, so simple. Why would you ever use anything else? And that can be good and it can be not so good as well. So in talking about DNS, I did want to bring up a couple of tools, both NS look up and dig, depending if you're Windows or Linux. I've got more experience on the NS look upside, so I'll talk through that a little bit. And really what that is, is it's it's just a way to

31:06
look up different DNS records and you can point to different servers. So a couple of use cases are if you're creating DNS records, for instance you you're creating public facing DNS records on your local DNS server that's going to propagate out to the world, you may want to check different DNS servers out in the world to see if they've received those record updates yet. So you

31:36
Any server you want one that I use often is just 8.8.8.8 because I know it's a DNS server out on the Internet and you just put in the records you want to look up and it'll return the answer by default. It's going to look at a and quad a record so IPv4 IPv6 DNS, but you can change the record type to one that I've used in the past is changing the type to the SOA or the start of authority record.

32:04
And the reason behind that is so you can see. For the record, you're looking up the domain name. You're looking up. You can see what server on the Internet is authoritative for that and a use case for that is. Let's say you've got a firewall and you're doing geo blocking and all of a sudden people start looking up trying to go to a website and their DNS lookup is just returning nothing. You can.

32:31
Look up this start of authority record to find that authoritative DNS record with an NS look up or a dig and then you can take that IP address and look in your firewall logs to see what is it doing with that. And that's just kind of a quick way to try to get to the bottom of something like that. So different bunch of different use cases for using things like NS look up and dig. That's amazing Tim, you you're earning that architect money buddy. Thanks.

32:59
That's a bunch of stuff I don't know anything about. I am embarrassed to say I have never had a DNS problem in networking. I know everybody's like, it's DNS and it must be- You've never lived, man. I know. I mean, I've broken it at home because I was doing nefarious things and I forgot to change your back, but like, yeah, I've never like had DNS burn me abroad. Well, I've only had one DNS problem to deal with in my entire life. So I'm-

33:28
I'm close to you, I guess. Andy. You're better than me, Lex. It's okay. I mean, we could do a not well, not me. I think we need to get an SME, but I think a deep dive on DNS and the distributed environment and how it really works. The whole recursive setup and everything. I think an episode on that would be maybe boring as hell, but it might be good. I remember studying it for.

33:54
Maybe CCNA, maybe Encore, I don't remember exactly. But like that knowledge, I don't use day to day. So I've lost a lot of it. So it'd be a great refresher in my opinion for everybody. Andy, Andre just said it in the chat. You've been looking for a way to quote use Russ White on this show. He can explain DNS to everybody.

34:19
I had a really cool topic for us. Yes. So you gave me a yes, we will have Russ on He has he has agreed So DNS is something I would not want to talk about and you gave me another Idea for a good episode that I wouldn't want to talk about is v6 IPv6 We we have to have so there's a couple people. I love trolling on Twitter And telling them how useless v6 is and how networks fine and we don't need it

34:48
and they get so butt hurt and upset and start yelling at me. Like they'll yell for three hours if I just like throw a little nugget out there about like V6 is dumb. So I think we should have, we'll have a DNS episode for sure. And then I think we should have a V6 episode as well, but I think we should bring on a couple people that are V6 experts like Smeez so that we can cut. It's almost like bringing the app owners on to yell at them. Like I think we should bring the V6 people on to be like, all right, dude, give me your pitch.

35:16
It's been 20 years, nobody cares. We're 30% global adoption. Like I'm a board with v6. They don't have to pitch to me. It's dumb as shit. We don't need it. It's not dumb. Very useful. I've seen one use case for it that made any sense in my 14 years of networking. One. The fact that v4 is working fine. v4 is just over. Isn't the use case that makes sense? The world is running on v4 and we nat everything and we don't need the addressing for six.

35:46
That is an argument. It's how every it's how every global network I've ever worked on runs. The only use case I've seen is when two companies merge with overlapping IPs. Well, now you're screwed because now you're double netting stuff to try to get two company networks talking together. So if you go to v6 that solves it all beyond that v4 is fine. Sorry, I think the biggest thing I've seen Andy is what Dave saying in the chat is mobile networks. So sell networks.

36:13
Every phone has a public IP address on a cell network. I think that's that's probably the biggest use case. I have seen

36:22
Why does every phone have a public address? Every client that ever connected to my data center, we now had to do a private. So it didn't matter what they were coming in as because we re-addressed them. Right, but a phone on a carrier network, like a Verizon, like an AT&T, you're gonna have that public address. So it's the sheer number of devices, right? I mean, that's what they say. IOT and there's just not enough. IOT has ruined IPv4. Just like millennials have ruined so many things. V6 is so critical to the infrastructure of the internet that we're at 30%...

36:50
global adoption 22 years after the protocol was released in the wild. And the shit's working fine. Sorry, let's move on. This will be a future episode. We're talking about tools, Andy. I haven't been on my soapbox in a while. Tim triggered me. Although actually tying a little bit of IPv6 into it, I have seen issues in the past where a device is reachable over v4 but not v6, and it's cool to be able to tell that easily by just specifying that in extended paint.

37:19
You can say ping it IPv4 or ping it IPv6. And that's an easy way to tell whether or not there's something that only affects IPv6, believe it or not. Not because of anything inherent in the protocol itself, but just like something might be set up all janky on the v6 side that's not on the v4 side because...

37:41
You know, like many of the network engineers who did it perhaps were afraid of V6 and check all their shit. I will also completely backpedal everything I just said and say that when I was studying for the MP and V6 was a required chapter, the shit was fascinating. Like it's amazing what they did. Yeah, it really is. Yeah. Just hex, guys. Don't be afraid of it. It's just hex.

38:04
Ha ha!

38:07
Just here is a one, bro. Everyone is afraid of hex. That is something I have learned in my time as a network engineer. Hex is very cool. We can't even remember IP addresses, so we need DNS. Now you want me to remember a bajillion character long hex address? Like, I just can't. No, I mean, that has to be managed, but you know what I really like about yeah, this is becoming IPVC. Hey.

38:32
Cisco textbooks, Cisco press textbooks, they will have like hilarious Mac addresses and IPv6 addresses in hex. Yeah, the dead beef. That's a good one. Yeah, dead beef. I love it. Anyway, enough of that. All right, where are we going after on this lookup? Does anybody do any of you, that was kind of the major host operating system tools that I wanted to cover. Do any of you have any that come top of mind that's just something in either a...

39:00
a guest operating system or router commands or just something that you think is really helpful but not maybe top of mind for a lot of people. Absolutely. But Lexi can go first because I'm a gentleman. Well, I was just going to say I have some software in my mind that I've used before. The efficacy of it varies. But right now, I don't want to.

39:29
I don't want to speak out of turn about vendors and things, but I'm mostly on the hardware side or I'm very much into layer one right now. So a lot of my stuff is actually like physical tools that I was going to talk about. So before we maybe pivot a little bit to that. Sure. Happy to open it up. Like an oscilloscope. Oh, yeah, that's one of them. That's a tool. Mm hmm. It is a tool.

39:53
All right, let's go to Andy. Give it. Give us something that's operating system or router or whatever. And then we'll jump into some some L1 stuff that. Lexi can drop some knowledge on us. Cool. So my favorite troubleshooting tool. Well, let me back up. And I can't believe we haven't said Wireshark yet, but I wish. Yeah, it's probably Lexis. So I won't steal the thunder, but I will say that I wish I spent the time.

40:23
I don't know anything about it, other than what Chris Greer told us, so you can talk about it. But what I learned from Chris and what I've heard forever, and when I was talking earlier about a standardized troubleshooting methodology, when the heavy hitters would come into those adage calls after we were fumbling for hours, they knew what they were doing, they had a thing they did at every single thing, they were calm, they weren't feeding into the chaos on the call and the people, oh, you know, they were...

40:50
Super chill knew what to look for blah blah blah, but they'd always do a packet capture Packets don't lie and and and that would give you the you know the problem So I wish wire shark is a tool and it's still not too late and Chris got me excited So I'm gonna try to run a capture at home but That's not the one I was thinking of because the only one I felt I had in prod that was worth any salt was net flow Because I worked in the way and so I was a transit network

41:20
customers would come in, they would come in and out of my data center. And when something would break, it was always some IP can't reach some other IP. So I'd get the source, I'd get the destination, even if they were prefixes, like it might be this block can't talk to this block, but people can't get the applications. And you check your routing and that's usually there. And the only way I could figure out to prove out my transit network was to look at NetFlow, because NetFlow...

41:50
will give you the source IP, the destination IP, and the protocol that it came in through on your ingress and on your egress. So I could say, hey, your traffic came in this IP, this source port, is that the one you sent? Yeah, uh-huh, okay, that was yours. And it came in, and then look over here. Oh, look, it left on this one. So my transit network.

42:16
Past your traffic, which is what I know you're telling me my transit networks broken and I did a whole I did a whole like 15 minute thing on YouTube about NetFlow and I built a lab and all that stuff to show you the table and all but NetFlow is magical. It's amazing. And you can also leverage collectors. We had see a performance manager, Performance Center, whatever the hell it's called see a suite. But so you could look at historical like, hey, a customer said their stuff didn't work two weeks ago, so you could go in.

42:43
And look at it and it would give you the graphical stuff. It gives you top talkers, right? Like who were the most busy talker things on there. But I love netflow because it was up to layer four, kind of like what we talked about earlier, um, you know, source destination, IP protocol, and the, and the device would tell you, yes, this thing left my interface, like it wasn't sitting in my buffer, like this thing's gone. Um, so anyway, that's, that's the only thing I wanted to say is I love netflow because usually my routes were there and usually I could ping them, but

43:13
NetFlow was the only thing I had at my disposal because I didn't know Wireshark to prove that that stuff entered my network, exited my network, and you need to go bother somebody else, bro. So I love NetFlow. Yeah, that's a good one for sure for not only troubleshooting, but can also be ingested by that data can be ingested by security tools as well for investigations, looking at traffic flows, that kind of thing. So yeah, it's definitely a multi-use.

43:41
multi-use tool. I think the best tools are the ones that you can say it's not the network, right? Yeah. And NetFlow was one of them. Like, yo, your stuff left my network. It entered here, it went out there. Do you want to see? I'll show you the CLI. You want to see a pretty graph of the picture? But it was slow, Andrew. Well. Actually, that is another issue. If it's slow, it still could be the network. Could be. Never know.

44:10
What do you got? What do you got Lex? I was gonna say netflow sounds awesome I like the tools that very quickly can tell you whether or not it is the network Because if it's not then you can move on with your day and if it is you can move on to troubleshooting very quickly, you know But I've never used netflow. So I will send you a link to my super awesome. Yeah, be fun time. I will watch your video That sounds great thing. You won't do I need to put it on to get your attention

44:36
That would be cool, but no, I will watch it. I'm on YouTube too, man. Gen X don't do TikTok, man. Grunge forever. Sorry. So old Andy. Experience, babe.

44:58
Okay, whoever's editing this episode just cut out like most of that. Even the NetFlow part? I'm really sorry. What tools you got Lex? You going to talk about Wireshark? Fucking, yeah, so Wireshark is badass. I will never say anything.

45:16
Listen up, shitbirds. It was a gerund. Heart a sipple, whatever. Yeah. Wireshark is the coolest. The only complaint that I could ever have about Wireshark is that I don't know how to write like custom plugins for it and I wish I could just snap my fingers and have all of the custom plugins that I want. Well, do you need them? Because Chris made it seem like everything's there. Oh, I do sometimes. You're working on like super fancy special stuff. When you're working on stupid, silly shit.

45:44
you do and when you're working on shit that most people don't do on their networks, you probably also but no for the most part. Yeah, Wireshark has pretty much anything you could dream about. So it's really great. I use it all the time constantly forever. And if you have an issue on your network, if you can take a PCAP on the device you're on or whatever, you know, various different ways you can connect up and take a packet capture.

46:11
It's extremely, extremely useful to throw that into Wireshark. And if you know, Wireshark gives you a lot of options to like, you can have little profiles for, you know, like one profile for like, I'm troubleshooting this part of the network and another profile for like, you know, some software dev, like said some shit and like, it's probably not the network, but I'm looking for some other stuff. And in those profiles, you can like customize coloring rules and.

46:37
the different columns of information that you have that show up and how it shows up too. You can also filter by like certain IP addresses. You can create little buttons that will filter by like protocol IP, whatever you want, and have that all contained within like specific little profiles you can switch back and forth between. And it's just a really, really handy once you have it all set up, it's so easy basically to just like eyeball exactly what you need to see.

47:06
or you know exactly what you're not seeing, which tells you other information, right? So it's a very good like at a glance troubleshooting tool once you're able to set that up. I have like a profile for one lab and a profile for another lab, where like I'm looking for certain things in one and I'm looking for other things in another one and everything is different between them, but I can switch back and forth and I have the coloring rules so that they like pop out at me. And it's very obvious, like I wanna see these IPs showing up.

47:35
Are they there or not? It's very obvious. Wow, that's hardcore. It's just so extremely useful. And depending on like the kind of hardware that you're using for networking stuff, sometimes you have more or less visibility into it. And so if you can hook up Wireshark, that gives you a lot more than what you might've had before. Another thing that gives you visibility, more visibility than maybe anyone would ever want potentially is an oscilloscope.

48:05
Which I'm not about that like yes, I've never heard anybody using an assault every time that gets brought up I I need people to explain it to me again. So Let me tell you all I understand is that Lex is measuring pulses of electricity that apparently are auto negotiation and she gets really pissed off when You turn off auto neg and it still sends the pulses

48:30
Or if you turn off AutoNeg and it stops setting the pulses because everybody else sends the pulses and why doesn't this one send the pulses? I mean, you've discovered that AutoNeg is really not a standard that's deployed normally across stuff, right? Well, I won't get too deep into it, but I got obsessed with Ethernet not too long ago for reasons that it doesn't matter. I went way too far down the rabbit hole than anyone ever should ever want or need to go. It was just because of curiosity. And

49:01
Looking into 1000BaseT specifically, which is what everyone knows as like copper wire gigabit ethernet, the one that's most commonly deployed today. When you look at how auto negotiation happens, basically the way auto negotiation occurs with ethernet, you have what's called link pulses that each device sends each link partner, as we call them, sends to the other one and

49:28
They send these initial ones that advertise their abilities, speed, duplex, et cetera. And they'll essentially have a little conversation, negotiate with those pulses. Is that like a frame? Like, is it a source destination Mac? Or right, it's not that. If you look at the 802.3 standard, it is specified exactly what those bits are and how far apart they're spaced and how big the entire pulse is. But they're essentially- It's not a frame coming out of the switch. It's like a separate-

49:56
No, it's not like a layer two thing. It is a layer. It's pulses. So but they represent bits that mean something. And so depending on if it's 100 base TX or 1000 base T, which are both are fasty and gigabit like copper standards that everybody sort of knows. They actually have slightly different information in them. And they're sent slightly differently. And so if you can actually with an oscilloscope, hook that thing up to a switch or whatever,

50:26
And if auto negotiation is enabled on the port, you will actually see the fast link pulses in the exact pattern that they are supposed to be in, because it's a standardized thing that's laid out. You can look up the standard RFC or whatever it is, and it'll tell you what they should be. You can literally, I have an oscilloscope screenshot of fast link pulses, and you can literally just copy, paste, like you can.

50:51
overlay it on top of the standard. And it's almost exactly. Question, how do you connect an oscilloscope to an ethernet port? Is there like a connector? So, no. So, well, it has its own connectors, but it's got, and I am not an EE. Okay, so I apologize to everyone listening to this. None of us are here. Yeah, but somebody out there is for sure. Listen to another show. She's not saying it right. Yeah, but I'm not an EE. So all of this came from like trial and error and learning from people who know way more than me. But basically,

51:21
We have, what, four pairs on your typical cat 5, 6, whatever cable. So what you actually have to do is, my setup at least, is cut that thing in half and then split out the wire pairs. And then I actually got assistance with this at work. I had somebody actually with the ability to do this put these little metal nubs.

51:49
on the ends of each wire pair. Okay, so with the oscilloscope, you have a grounding clamp, and then you have a little hook, and you, I forget which one now, one of the, either the striped pair gets one of them, and the solid pair gets the other one. Like gator clips or whatever? Like something that clicks on. Yeah, gator clips for the grounding, yeah. So you hook it up, and really, for my purposes, I only really, if you need to look at gigabit really intensely, you need other special equipment. But...

52:17
If I just want to look at the fast link pulses, I can still see them just looking at the orange and the green pair. And you tell us why you were looking at fast link pulses. Were you troubleshooting something? It, it came from, it has nothing really to do with what I do at work. All it has to do with is that I was using ethernet for something a while, like eight months ago. And then I went down an insane rabbit hole of, wait a minute, if I try to turn off auto negotiation on a switch, I don't think it's actually turning it off. Is it? And then I realized,

52:47
Somehow someone probably told me you can look at it with an oscilloscope You can see pulses and oscilloscope is very cool. You can't look at literally everything what it does is it measures You know voltage It measures the electrical currents on a wire and that's it and you have to be very careful with an ethernet cable It's fine You're not gonna like screw anything up But you gotta be very careful when you just like clamp it onto anything right because you can short stuff out So be careful with them, but if you have the opportunity to learn

53:14
No, because I haven't I haven't touched anything but ethernet cables. That's it. So But yeah, it's an interesting tool and it's very cool for at least that one use case that I have Do we have anything else on troubleshooting tools thing left out I'm just I'm researching quickly how to jump start a car with an ethernet cable. Hang on. I Highly recommend making your own cables if you haven't

53:43
You're amazed. You talk about the art of network engineering and the art of troubleshooting, you know, the art of trying to... And not the cheat ones where you push it all through. And I've seen those, like the actual old school ones, because all four pair have to be the exact length and the right length. So when you push it in, the jacketing gets caught in that little, whatever you call it, the arrestor that keeps everything in. And it took me a long time as a cable guy to get...

54:11
Ethernet, right? It's a very satisfying feeling though to do that. I mean, once you can feel your fingers again and you actually test that cable and get the pass, I mean, that's pretty cool. It's the whole Tom Hanks thing in Castaway when he makes the fire and look at what I've created. You know how I test mine, I plug them in, Tim. Yeah. Cool, works. No, I don't really have anything else.

54:39
Oh, can I say one more thing? Please, anything you want. Tap aggregation is really, it was a new thing for me, didn't know about it, started using it, very cool. Now everybody has their opinions. So when you say tap aggregation, I've heard of taps, which are physical things, like, so at my old job, we had taps, which would then feed into like, packet collector stuff, right? Is that what? That's tap aggregation, yeah. The actual protocol that I was learning was like, how to do it on a switch.

55:09
So if you have a lot of taps and you want to aggregate the data, you want to bring it all together. So why would you use a physical tap instead of a mirrored port or whatever they call it? Well, it was a question why. Well, yeah, like can't you do the same thing with a mirrored port out of a switch? You can, but it doesn't use up switches resources and it won't take up another port that you don't want it to take up. And if you have a lot of them for whatever reason, it's just so much easier to have.

55:37
Basically a dedicated switch that will connect all of those, you know, be the like port density for it. And then that switch can just feed all of that aggregated tap data to like a PC or whatever. And the protocol that sets that up, that's tap aggregation. So do they exist to capture traffic and put it into like a wire shark and see what's going on? Is that- Yeah, like the switch that you configure to be your tap switch basically has to be a dedicated.

56:04
tap aggregation switch. So you put it in like a mode, I'll forget exactly, like tap exclusive or something like that, tap aggregation mode exclusive. And you just- But is it like one port that comes out of that switch and then goes into the tap? Yeah, you have a tap ports and a tool port or multiple tool ports if you want. It's very easy setup. You just specify which ports are the tap ports by giving them, it uses 802.1 Q tagging. So it's just using the V, it's not technically VLANs but it's the exact same thing.

56:34
and place, I think, in the frame or whatever. So you just specify the tag and you name the tool group, and then you just tell the tool port, hey, send this tool group out, just like you would a VLAN trunk, right? And you connect that to your PC, and it'll know, okay, I'm going to take the tap data from all these ports and send it out my port to the PC every time it comes in. Boom, done.

57:04
Right? Yeah. Yeah. And there's different kinds of taps. You can have passive taps, which work for under gigabit speeds. So if you're doing fasty stuff that you're tapping, that's you would use these passive taps, which just means they don't need to be powered. They don't need to do any extra stuff. A lot of people use like the throwing star land taps is what they're called for gigabit speeds and higher. You need to have something that's actually powered and it's an actual like bigger device. But you can definitely put that in a rack and mount it and do whatever you want.

57:35
graphic. It's very cool. Troubleshooting stuff.

57:44
Take take us home, and if about tools me. Oh my god. I wasn't prepared hold on I'm afraid to open a window. Give me a second Getting there vamp for a second would you Yep That's it that was the vamping yep I'm still hold on. I have no idea what that means. Oh vamp like

58:13
Talk for a while. I got it, hold on. Oh. Oh, I thought it meant like dance. I thought that was like an e-cigarette or something. Mm-hmm. All right, well, Tim, Lex, it's been awesome talking about all the troubleshooting and troubleshooting tools, and God, there's just so much to know and learn, and so many things that can go wrong in the network. So the more tools you have at your disposal, better equipped you are to do your job and prove it's not the network.

58:41
Thanks so much for listening to us. You can find us on Twitter. At art of net ends run Instagram at the same handle art of net ends art of net ends on Facebook on LinkedIn guy. We're everywhere websites art of network engineering. Dot com we have some new merch up on our merch store art of net ends dot com forward slash store. Also, we have a discord study group. It's just fantastic. There's like 5300 members. I think in there now and.

59:07
You know, if Twitter's on your nerves because of all the things going on, you're looking for a community to join. That's at artofneteng.com forward slash IAATJ, which stands for It's All About the Journey. Do we have a mastodon? Is that something we're supposed to like? We're going to. Okay, so Alex is working on there. Alright, so we'll be there and announce it when it's time.

59:33
Well, Tim, Lex, it's been a pleasure. Thank you everybody for listening, for joining in. Thanks to the folks who joined the public room. And as always, we love talking tech and networking with you guys. And we'll see you next time on the art of network engineering.

59:59
Hey y'all, this is Lexi. If you vibe with what you heard us talking about today, we'd love for you to subscribe to our podcast in your favorite podcatcher. Also, go ahead and hit that bell icon to make sure you're notified of all our future episodes right when they come out. If you wanna hear what we're talking about when we're not on the podcast, you can totally follow us on Twitter and Instagram at Art of NetEng. That's Art of N-E-T-E-N-G.

01:00:25
You can also find a bunch more info about us and the podcast at art of network engineering.com. Thanks for listening.