Speaker 1: 0:00

This is the art of Network Engineering podcast.

Speaker 2: 0:23

Thanks for shopping the A1 Emporium. Did you find everything alright today? Yeah, for the most part, but there were a few things I couldn't track down. I was looking for a new Mac table for the living room. Ah yes, those would be up on level 2. If you reach the art tables, you've gone too far, thanks. How about a router? I'm looking to make some new cabinet doors. Ah, routers would be up on level 3. We have OSPF and BGP brands of those. This is a big place. You got an app store? Sure do. That's up on level 7. There's apps on apps on apps up there. Wow, I heard you even offer educational programs here. Do you happen to have a management course? Oh yeah, that's up on level 8. Good luck up there, pal Well, maybe I'll check that out next time. Thanks for your help. I gotta go. Gotta catch the newest episode of the Art of Network Engineering.

Speaker 1: 1:15

Welcome to the Art of Network Engineering. I feel like it's been forever since I've heard of Tim Intro, although we did release an episode this week, and it was that Nacho.

Speaker 3: 1:23

Man, andy Savage, try to top that one.

Speaker 1: 1:28

That was awesome. That was an instant classic. I think Andy said that earlier today and this one's great. I can't wait to edit this one. We'll put in some shopping sound stores, Maybe make it sound like an IKEA Costco something like that If it's got seven or eight floors, man, it's gotta sound like an IKEA.

Speaker 1: 1:44

I think I am AJ Murray at no Blinky Blinky. Thanks for joining us tonight. We are live streaming this on YouTube actually. So we have not ever, or we have not in a while, live streamed on YouTube, so we're going to try it out, see how it goes.

Speaker 2: 1:59

You can say it we haven't successfully done it. We did it once and it was a miserable failure.

Speaker 1: 2:05

But you know, the technology has come a long way. Our platform has done some great updates. Things have been a lot stable in the last year, so we're giving it another try and we'll see if we can have some more success on YouTube. A lot more people tend to hang out there than in our live stream, so maybe we can get some bigger audiences doing it this way. So, as a reminder to our listeners, if you want to support the show, you can do so by going to wherever you find our podcast and leaving us a like subscribing. Smash that bell icon to get notified of all of our future content.

Speaker 1: 2:41

Leave us a review. The review goes a long way in helping other network engineers find the show. Those are all things that you can do for free. We're not looking for any money from our listeners. We do appreciate everybody that's been reaching out, looking for ways to contribute and just showing general appreciation for our content and if you want to, that's the best way to do it. Share the show with a fellow network engineer. All right, let's get to the roundtable. Tim at Tim Bertino on X or Twitter. Tim, how are you doing?

Speaker 2: 3:12

Good, aj, it's good to see you and Andy for this one At the time of this recording. We are kind of coming out of the holiday season 2023 into 24. And I got to say it's been relaxing. I've actually gotten the better part of two weeks off and it really just been getting to be immersed in the family and it's been incredible. So going back is going to be interesting, to say the least.

Speaker 1: 3:39

Now now, Tim, I don't know if you've officially mentioned it here on the show, but you started a new job just before the holidays, and that's that's part of the reason why you have two weeks off.

Speaker 2: 3:48

That's right. I finally jumped into vendor land and I am now with Cisco since about December of 2023.

Speaker 1: 4:00

Thank you, andy, bringing the goat back. Congratulations, tim, that's great. What's your role there at Cisco?

Speaker 2: 4:06

So I am a systems engineer, so I'm actually part of the sales team, so I will get to have technical related sales conversations with customers. And I think the biggest thing that's exciting to me is really getting to see other industries, because, as you both know, and probably everybody that's listened to this, I've really only known one industry, one organization up to now my entire career. So I think it's going to be really cool getting to see some different verticals and and learn how other people have done things.

Speaker 1: 4:39

You really went to the dark side on that one. No, that's great. I you know you have worked with, I think, everything that Cisco sells Ice, aci, firepower. You know it just did all of our passing conversations. So you are going to be one hell of an asset to your Cisco customer.

Speaker 1: 4:59

So that's, that's great, that's a great great, great snap up by Cisco and you know I look forward to hearing about your experiences there. That's fantastic, andy. Over to you, sir, I saw that you were doing some photography in the city the other day and I am dying to see how your photos.

Speaker 3: 5:16

I'm working on the edits right now. Nothing crazy, but yeah, I have a little bit of more leisure time on my hands. So I had a doctor's appointment down the city and I'm like you know, let me bring my camera, I have it mounted up here in the studio and I just leave it there. But I'm like you know, I miss, I miss photography, I miss shooting. So I brought it down and just some cool angles with buildings and some smoke coming off stacks and stuff. But yeah, man, this is a photography. Like Tim said, the holidays were amazing. I had such a nice, you know, the whole family was off for a week. We went to like Hershey Park. Did you know, before Milton Hershey created Hershey Park, chocolate was an expensive delicacy only found in Europe? I did not know this.

Speaker 2: 6:01

I did not know this in Hershey Park, I didn't know that, and I didn't know that his name was Milton.

Speaker 3: 6:06

So the cat yeah, the cat's like. Why can't we have affordable chocolate? Here, so anyway, we had a nice trip to Hershey Park, went down the shore I I read a 5k race, oh yeah.

Speaker 2: 6:20

Oh, come on, man, it was it was ridiculous.

Speaker 3: 6:23

So my, I have runners in the family my in-laws and they're always running these races and they asked my son if he wanted to race one and he's like, yeah, well, I can't, you know, have my boy race in and me not. So, yeah, I ran the race. It was. I've been running for probably a month or so prior. So I'll tell you that, like for somebody at my level in my age, like 35 to 37 minutes is like an average time for like a 5k. 5k is 3.1 miles and up until like before then I was running like 11 minute miles, 10 minute miles, kind of slow. I don't know what the hell happened to me in this race, but I ran like 8 minute and 30 miles and I ran the race in like 26 minutes and I did like way better than I thought I would.

Speaker 1: 7:08

Yeah, man, that is so funny, so anyway that was a highlight.

Speaker 3: 7:11

Now my knee is killing me and I don't know.

Speaker 1: 7:14

I don't know how long my running career is going to last, but yeah.

Speaker 2: 7:18

Anyway, the holidays are great. I was going to give you crap about your time too, because it was a. To me that seemed like a really good time, and wasn't that New Year's Day? And I saw you were up till midnight ringing in the new year we were.

Speaker 3: 7:31

I heard you guys talking about that before the recording. So, yeah, my, the kids wanted to stay up till midnight for the first time and we did. Yeah, we were up banging the pots and pans and we were down the shore. It was awesome. One other house on the block Everybody else is asleep. My end law is the next day. My mother and father-in-law are like yeah, we heard you guys. I didn't seem too happy about it, but yeah, yeah, new Year's was fun, the race was fun. It's been a nice holiday. Sounds like you guys had a good time and, yeah, I'm super excited for you, tim.

Speaker 1: 8:01

This next career journey is just going to be amazing, man. So it's the pots and pans thing. Like a philly thing Like I've never.

Speaker 2: 8:09

I wanted to ask that too, because I didn't know that was a thing. Oh Did you invent that or something.

Speaker 3: 8:16

I grew up with my parents like you go out at midnight and beat pots and pans at midnight. I don't know if it's a Philly thing, if it's Broke-ass police families, you know, maybe we couldn't afford the, the, the, the, the ways makers.

Speaker 2: 8:32

I mean, I've seen people doing that before, but it's usually to scare off bears.

Speaker 3: 8:39

Well see, I learned something new on this show every day. I didn't know that, uh, that that wasn't a, that wasn't a thing nationally. Yeah, that's funny.

Speaker 1: 8:47

So what do you?

Speaker 3: 8:48

so, aj, you were up at midnight. Did you go outside?

Speaker 1: 8:51

No, no, we did not go outside.

Speaker 3: 8:53

Does anybody like I want to know what other people do if they go outside at midnight and like make noise if they don't beat pots and pans?

Speaker 1: 9:00

So if I go outside it's to watch the fireworks. But we don't. I can't see the fireworks. So they usually do fireworks at like the Burlington waterfront, but I don't live anywhere near there anymore, so I don't go outside and watch those. We just watched the ball drop on TV.

Speaker 3: 9:13

Chris Denny says they call them Johns Johns and pans Got Alex in the chat, Denny Nice and some local celebrities in here.

Speaker 1: 9:27

Nice, awesome, all right. Tonight. We are talking about critical services, these critical services that we really depend on as network engineers. They're they're not necessarily routing protocols, but they are very critical to the function of the network. Tim, this was, this was kind of your baby. Where would you like to start?

Speaker 2: 9:51

Yeah, I just to kind of frame it up I wanted to highlight some of these what I've deemed critical services and not just me, I'm sure tons of other people really just are just there and have to be there. And why I want to highlight them is because I don't think we always think about the inner workings of them. I kind of like in these services to Utilities, things like water, gas, electrical things that are completely critical to our day to day lives, but they're just there and they just work, so we don't always think about how they work in the inner workings of them. So I really wanted to highlight at least a few of these that I think are critical to network operations for clients, both in an enterprise and even at home and anywhere else.

Speaker 2: 10:41

So the first one that I want to talk about is the dynamic host configuration protocol, or dhcp. And first off, for our ipv6 friends, I gotta give a disclaimer here we're really only gonna talk about ipv4. I know that there is dhcp for ipv6. I also know that there are functions and ipv6. That remove the need for dhcp, but this guy doesn't know a whole lot about those. So I'm not gonna try to spout like I do so for the sake of this conversation.

Speaker 2: 11:15

We're talking about ipv4 and tim, if I could interject quickly, please do regard regarding ipv6, nobody cares, moving on moving on, in stepping back a second before we get into dhcp, I did want to give kind of a spoiler alert. I don't think that we're gonna cover anything here that's really earth shattering or groundbreaking. It's really just highlighting those day to day things that are essential to our networks but we don't Always necessarily think about. So, dhcp, why do we need it? So every host, every device talking on a network, needs an ip address and if we didn't have dhcp stand for Dynamic host configuration protocol, I think yes

Speaker 2: 12:02

I nailed it, thank you. I don't know why I didn't put that in my notes, but hey. So if we don't have a way to dynamically assign these ip addresses in an ipv4 world we would have to go around and statically set ip's All devices, and who wants to do that right? So we have dhcp and one of the things I really like to talk about when it comes to dhcp, because I like to know, and I think a lot of us other network engineers like to know how things work and the dhcp process to me is really cutting dry. It's a. It's a four step process and I think a lot of us call it the dora or dora process. That consists of discover, offer, request, acknowledge. So the first and third letters of that are from the client and the second and fourth letters are from the dhcp server. So let's let's step through this a little bit.

Speaker 3: 12:56

When a device needs an ip address, it does a dhcp discover message and yeah, sorry, quick note, so Talking to an audience that might not be as familiar as we are maybe working on the ccna or something like that, the device that needs the address. These are end devices. Right, these are yes, these are clients. These are pc's, these are networking devices in a network that needed address. Right, this is an end device.

Speaker 2: 13:20

Try to get a call. We're talking in device, yet Can think phones, printers, computers.

Speaker 2: 13:27

Yeah, great point and yeah, not the network, but people trying to consume the network right right people trying to consume the network hundred percent device will send a discover message and that discover message goes to what's called the broadcast address. So if you've seen it, layer three, the ip layer, it's all two fifty fives, which Is an interesting point because, as we know, broadcast messages don't get routed. We would either have to have a dhcp server on every single segment, every single vlan, or we have what's called an ip helper that sits on the router for that vlan and basically it's just a statement. If you've ever configured one that that says basically, point any dhcp messages to a centralized dhcp server. Once that discover message gets to a dhcp server on, that dhcp server will have a dhcp pool, so a grouping of addresses that's used for dynamically handing out ip's to devices. So the dhcp server will send that offer message Back toward the client. The client will receive that and say, yes, I want to use that address, assign it to me, and then to close it out, the dhcp server sends an acknowledgement saying yes, that's yours.

Speaker 2: 14:43

And now that Device, that device, that computer, that phone, that printer now has an ip address that it can use to communicate on the network with. Now that's just dynamic, that's just having like we talked about a pool of addresses that can be used. Anybody who is running enterprise network before and has had to bring on certain devices from different vendors for some reason. A lot of times One way or another vendor will come in and say, hey, I'm bringing this device on your network. I need a static ip. And I always shuttered when I've heard that because that can bring a lot of issues. Because whenever you're talking about static ip's, somebody has to, like we said earlier, somebody has to manually configure, type in that I p address, the subnet mask, the default gateway, all into their device and Work humans, right? We never get anything wrong. We never type in anything incorrectly, right?

Speaker 3: 15:41

so there's no, it's not just the fat fingering, but like the house keeping of like what addresses are in use and what are you call?

Speaker 3: 15:49

yeah, what I am well because I've, we've done that in production. Like you think, you look in your ipam and you think you have an available address and you might do some modicum of checking to make sure it isn't available anywhere, like ping from a jump box or something. But there been times in our network where we assign the same ip to two different places and it broke all the things right. So like dhcp just manages that all for you, there's your pool will assign. You know it'll never, as far as I know it won't double assign. You know an address to two hosts. But your point, you could do that when you start with statics.

Speaker 1: 16:21

I've stopped using excel for my ipam and I've upgraded to google sheets.

Speaker 2: 16:28

I totally, I totally thought he was being serious man, that was. But no, I didn't think about that from a documentation standpoint, andy, because you can have a great ipam, you can keep track. But if somebody comes in and just says, okay, this is my address now, you have no way of having documentation.

Speaker 3: 16:48

Human are like you said. They might look at the ipam and see it's available when it isn't there, you know? Yeah, it's definitely a mistake. I've seen that mistake made.

Speaker 2: 16:56

So one thing that you can do to go back on these vendors are people that are requesting to have static ip is a lot of times they, in my opinion, they don't necessarily need to manually or statically set that I p address on the device. They just need or want an ip address that is not gonna change. So you can still use dhcp for that. What can be done is you can have your pool of addresses and then in that same subnet you can have what's essentially call a reserved range.

Speaker 2: 17:28

So this range of ip is is never handed out dynamically by dhcp, but you can use them for I p reservations and all you need to do is to basically pick an ip address in dhcp. You set a I p reservation in that subnet by the mac address of the device. So anytime that device request an ip address from dhcp, it's gonna get the one that you reserved. So you can get the same benefit of getting having the same ip address every time, but nobody has to manually set an ip address subnet mass default gateway on the device so that that can come save the day. And in those kinds of situations you just gotta get people that you're talking to to say, hey, yes, I know this is dhcp, but you're gonna get that same ip address every time that works great for copiers and some other device like I think.

Speaker 1: 18:24

A lot of times people default like I'm just gonna go set a static ip address. Nope, just just get the mac address off of their, pop it into the dhcp server with that reservation and every time it boots up it's gonna get the same address. Now, a long time ago on some hp printers.

Speaker 1: 18:39

They used to have those what I think they call them jet direct cards, and if the net card died you could pull the card out. You weren't necessarily replacing the entire printer, but you're pulling the card out, you're putting a new network card in a network interface card and then, of course, the mac address changes and then you gotta do the whole thing over again. I think that there is one caveat there are some devices and some services, just for whatever reason, don't like being set to dhcp, and one that comes to mind is you know, I used to be a big microsoft guru and a lot of the microsoft services were not happy unless they had a static ip address. Because you could do the same process on servers, and maybe on some not a central servers you could probably get away with this. But, for example, if you're trying to spin up an ad domain controller, if you were to try to not put a static ip address on the domain controller, it would say, hey, wait a minute, you really ought to put a static ip address on here.

Speaker 2: 19:42

Yeah, that's a good call out.

Speaker 1: 19:43

Domain controllers are that way yeah, yeah so, and I'm sure there's there's some other central services that that would probably say the same thing.

Speaker 2: 19:51

I do want to call out a couple of really good questions that are being asked in the chat. First is from zitherian asking about when you set up Reservations based on a mac address. Does that need to be removed from the dhcp pool? And I'll say no, because the you're usually setting up Two different ranges in a given subnet. You're setting up that dhcp pool. That's always dynamic. That's just being handed out by the dhcp server and after a least time it gets released back to the pool. These addresses, these reserved addresses, are in another range, outside of that in. The client doesn't have to know that, doesn't have to do anything. It's just a reservation set by the, by the mac address, and then had another one from chris denny. Now I can't find it.

Speaker 3: 20:39

What is the case for?

Speaker 1: 20:40

you for what's a use case that's a.

Speaker 2: 20:42

That's a good question. What I've seen before, chris, is that it depends on the application. I've seen Application servers that need to configure the clients that are going to be talking to them, or vice versa. That configuration is set up in the server by ipea dress, right, wrong or indifferent. You know it's twenty twenty four now, but we do still see stuff like that. There are systems, servers, applications that still configure things based on ipea dress rather than domain name Door M Y Lang, again right, wrong or indifferent. There is still stuff like that out there.

Speaker 3: 21:22

I just put something in the chat I saw. I know you're so a spoiler alert. We're going to get to DNS in a bit but I saw that DHCP can be integrated with DNS. So it says automatically update DNS records when an IP address is assigned or changes. This ensures DNS remains synchronized with IP address assignments. It's just something I never heard of, never thought about.

Speaker 2: 21:43

Yeah, so I know that is referred to. Like you said, we're going to talk about DNS here in a little bit, but what I'm thinking that is, andy is what's called dynamic DNS. So that allows clients to communicate with the DNS server and say, hey, this is my hostname. So when a client gets an IP address from DHCP, one of the options it can get in there is that, hey, client, here's your DNS servers. Then the client can reach out to those DNS servers and say, hey, this is my current IP address, this is my hostname, and that's what's referred to as dynamic DNS. So if somebody else needs to talk to a host, they can hit them by name and with that integration between DNS and DHCP, we can keep that information updated. So one last thing I wanted to talk about in regard to DHCP was a little bit on the security side, and there's a feature that has really come in handy before on the network infrastructure side is called DHCP snooping.

Speaker 2: 22:48

So there is an issue that can happen where a rogue DHCP server whether it's on purpose or not, somebody being malicious or just there are devices that you can plug into a network that can act as a DHCP server. Commodity residential routers are like that right. You plug your router into your modem and your router acts as your DHCP server as well. What DHCP snooping does is it's able to and it usually will run on a switch and it's able to look at messages going through the network and see which ones are DHCP. And basically, once you set it up, it can block certain DHCP messages, so somebody can't plug in a rogue DHCP server.

Speaker 2: 23:34

It takes the concept of trusted and untrusted ports. So how you can think about it is that a trusted port, in a DHCP snooping sense, is essentially an uplink. Whatever port is facing the DHCP server, you want DHCP messages to flow through that port. Anything else is an untrusted port. And when I say anything else is any other port that plugs into an end user device. You don't want anybody that plugs in a device that could be a rogue DHCP server. It plugs into an untrusted port. Those messages get blocked. So that's a pretty cool and helpful security mechanism that plays with DHCP in the network infrastructure. All right, now let's talk about one of the favorite topics of people, that this thing gets blamed for damn near everything and sometimes it's valid, but I think other times it's not and that is the DNS or the domain name system.

Speaker 3: 24:36

I want to go on record as saying I've never had a DNS problem in production.

Speaker 2: 24:40

Yell that from the mountain tops my friend.

Speaker 3: 24:42

Well, yeah, I don't know if it's because I was like a data center WAN guy and maybe it's just not as relevant there. But yeah, I know DNS is a thing. Right, it's always DNS and it's just a meme I never got because I'm like I never had this problem.

Speaker 2: 24:55

Well, Andy, you couldn't have teed that up any better, because the big reason why I wanted to talk about DNS not only because it's a critical service, but I don't think it's always necessarily managed by the network administration or network engineering team Like you just talked about, Andy, you were a network engineer for a long time. You worked in a WAN In some organizations. I think DNS is more a systems administration responsibility rather than a network team. I think it's just kind of a culture thing as far as who manages DNS. But I will say it's not always the network team and I think that might be why you never ran into it or part of the reason why you never.

Speaker 3: 25:37

When we reserved IPs, we would submit the DNS request and the only problem I can think of we had is if you'd try to hit a host name and it wouldn't resolve. Then you just have to use the numerical. But I guess you're going to walk through some of the DNS problems you guys have had over your career. Right Pain that it's caused you.

Speaker 2: 25:54

We certainly can, because, yeah, there can be issues and I would say that I think a lot of and I could be way off base in AJ, I'd like to get your take on this. But I think a lot of potential issues with DNS can be. Go back to what's called the DNS time to live or the TTL, in that when DNS queries happen, when a client gets a response or a DNS server gets a response and is able to resolve it, it'll cache that response and it depends on how long that time to live is, which is configured on the authoritative DNS server, so the server that owns that domain. It could say, hey, my TTL is 10 minutes, so basically you know that every 10 minutes you're going to get a fresh record, or it could be eight hours. That's why I think people have seen before that have managed DNS.

Speaker 2: 26:48

If you're making DNS changes that are internet facing so you're changing an IP address or a name to IP address resolution for a DNS record that you own, you could see that worldwide propagation take a day or better. You could change it and in certain DNS servers that you query it could reply back right away, depending on when they refresh and pick it up, or it can take a long time for propagation, and I think that's where some DNS issues come from. It's not necessarily a configuration thing. Like I made the right change, it changed. It's going to take a while, depending on where in the world you are.

Speaker 1: 27:28

Yeah, so I've done a lot of migrations of various services. So in recent years it's been migrating from an on-premises exchange server to Office 365. Right, We've got to update the MX records and a bunch of other mail-related records to point to the on-premises server over to Office 365. And one little trick I've learned over the years is cranking that TTL down as low as it will go prior to or leading up to that migration point. Right, so the TTL will actually tell the DNS server consider this valid in your cache for eight hours, 24 hours a week, whatever the TTL is set to. So if a DNS server has that record cached, when it looks it up, if it says, well, this is still well within the TTL, I don't need to go check for a new record. Here you go.

Speaker 1: 28:19

So I've seen some people want to move services and they don't change the TTL before changing the record. And now you have a TTL of 24 hours or seven days or something like that, and so you can go ahead and make the DNS change. But it might take a while before it's propagated around the world. There are some really cool websites that you can get and I'll try to find those and link them in the show notes for this episode, when we go live with it, when we post it. But you can actually make a DNS record change and then query it on these websites and it will pull from DNS servers literally all around the world and you can see what gets returned for any given record you're trying to look up.

Speaker 2: 29:00

Yeah, and I've done the same thing, aj, where I've known that we're gonna make a change public facing or internet facing and I'll go in and drop that TTL down.

Speaker 2: 29:13

And the reason why I've set it higher so like eight hours, and maybe people do even more than that is that then drops down the amount of queries that'll hit your DNS server, because servers will be able to cache that and won't have to ask your DNS server every time. So if you leverage a cloud service that does DNS for you, you may pay for that service by the amount of DNS queries that hit that service. So you may not want a low TTL because that means there's going to be more hits, more queries to your DNS server. So that might be why you set that TTL higher. But, yeah, if you're going to make a change to a record, you may go in depending on when you set the TTL a day before, drop that TTL down to five, ten minutes, make your change and then, after you know your change is propagated, you can bump that TTL back up to whatever you want it to be.

Speaker 1: 30:09

So I think some of the people in chat are kind of asking why DNS can present itself as an issue. And so if for some reason, a DNS server is unavailable or if somebody is not connected to the appropriate network in an internal versus external right. So we have always had the issue where somebody's trying to reach an internal resource they're working from home, they might not be connected to the VPN and they're trying to hit like SharePointwhatevercom or whatever the internal URL is, and they get the age old 404 not found. Oh, it's broken, it's not working as well. No, you're not connected to the VPN, so you're not going to resolve that. You have to. That record is only available from our internal DNS servers or for some reason, dns servers aren't available.

Speaker 1: 30:53

So if any DNS servers not available, it can certainly give the impression that the network's down, even though you have full end to end IP connectivity. If the client can't either reach the DNS server, the DNS servers, otherwise are that available it's going to present as if the internet's broken because it can't resolve those queries. You know, I know this kind of famously happened with, you know, a service provider, xfinity. You know everybody by default gets Xfinity's DNS servers and they had a problem with their DNS servers, both the primary and the secondary, and everybody thought their internet was down. But it turned out it was just a problem with the DNS server. Simply changing to a new DNS server solved the issue.

Speaker 3: 31:38

Speaking of names, their name is Comcast. People just like them so much they thought that if they renamed themselves Xfinity, people would forget how awful the experiences they were having with Comcast.

Speaker 1: 31:52

It's like calling cramp poo or shit.

Speaker 2: 31:54

It's like it's still the same thing, you know.

Speaker 3: 31:59

Well, that makes a lot of sense, you know, like in my context I'm thinking the wild west of the internet, like when was the day you couldn't get to Googlecom or Tiktokcom because the DNS record was? But yours makes sense of like getting new internal resource somewhere not on the VPN. So okay, that's some meat on the bone for me.

Speaker 2: 32:15

And there are issues that can look like their DNS issues. But there are other things. So I've run into issues before with firewalling. So let's say you have an X generation firewall and you're doing some geo blocking where you don't want traffic going to certain places in the world and somebody may be trying to go to a specific site resolving a DNS name to an IP address and that DNS query has to hit a DNS server that's in another part of the world to be able to resolve that. If that's blocked by a firewall, that's going to look like you just can't resolve the name to an IP address and people are going to think, well, that's a DNS problem. So you kind of have to follow the bouncing ball and go through your troubleshooting process to end up finding what it was.

Speaker 1: 33:07

I think it's important to note that, from a security standpoint, dns is like an important first line of that right. There's been plenty of vendors Cisco included, of course that have created security solutions that revolve around DNS. And if you can prevent the client, you know if the client's reaching out looking to resolve a domain name for a known CNC, then it can just not resolve the request and then the client will never get there. So it's a great way to get some visibility into the traffic coming from the clients and to stop threats before you know the request actually reaches the threat. So there's lots of tools available to help you do that. I think Open DNS is still a thing, even though Cisco bought it and it's an umbrella. You can still use umbrella and Open DNS for free. You don't have to actually buy into it. I mean, of course, if you want to get a lot of the advanced security features, you do need to, but there's some benefits that you can get.

Speaker 3: 34:03

I open it, I use it at home. Yeah, the kids oh nice.

Speaker 1: 34:07

Yeah, yeah, so good stuff there. All right, tim, what's next? We've covered the DNS.

Speaker 2: 34:13

Well, we really just jumped into what the issues are with DNS.

Speaker 1: 34:17

Oh, true, true.

Speaker 2: 34:19

Let's talk a little bit more about it. We can kind of fly through this, but so the reason DNS is needed is we kind of talked about it in DSP. Every device needs an IP address and nobody can nor wants to be able to remember IP addresses for getting to different places. We kind of know how the human brain works. We can recognize names right. So DNS is there to. When somebody types in wwwgooglecom, on the back end it can translate to an IP address and the traffic can flow. So that's, I think we can all understand and agree upon why that is now a critical service. We already talked about how it.

Speaker 2: 35:01

Basically, if that's not functioning, you for the most part don't have internet access because you're not going to know IP addresses of the different sites that you need to get to. So we need DNS. Again, like we said, it's not always managed by the network team. So you know, network engineers even may not always understand DN or workings of DNS. So I kind of wanted to talk about the hierarchical structure to DNS and that at the very top we have what's called the root zone, and really how I see the root zone is servers in the world that are really just pointing at in resolving where the top level domains are to be able to resolve. And when I say top level domains, I'm talking about things likecomorgedu. So we have the root zone top level domains. And then I need to know why Andy's laughing at that.

Speaker 3: 35:54

Just for the people of the East Coast of the United States. Tim is saying root.

Speaker 1: 36:01

Oh, okay.

Speaker 2: 36:02

On my, on my roof, yeah.

Speaker 3: 36:06

Sorry, I couldn't contain myself.

Speaker 2: 36:07

Okay, and then the the other two levels we have are the second level domain and the third level domain. And how I see those are those are owned by the organization. So second level domain, in terms of this podcast that would be in the domain name Art of Network Engineering. So Art of Network Engineering would be the second level domaincom would be the top level domain and then if you have any sites or subdomains within there, that'd be the third level. So things like wwwshoppingblogs, whatever. That's the third level domain. So I really just kind of wanted to kind of cover how DNS is based on a hierarchy and we can now put DNS to bed, AJ.

Speaker 1: 36:51

All right. So what's? What's next in the list of critical service system?

Speaker 2: 36:54

Next one I got is NTP, or network time protocol. Love NTP, and this is really, I mean, on the face of it, it's. It's simple, Because what we're really talking about is just syncing time, and we want to make sure that all the devices on our network both end devices that Andy brought up earlier, as well as and probably you know more important for us as network engineers and network infrastructure we want all of our network devices to have a synced time.

Speaker 3: 37:25

Good luck troubleshooting with like inaccurate timestamps in your logs. Good luck, I run into that. And then, like you see, the NTP server wasn't set up or it's wrong or the addresses aren't reachable, and you're like great, these logs are useless. Yeah, something happened around 2 30am last night. Can you help us? No, my devices have no idea. I see stuff that happened, but I don't know when. I love NTP because of troubleshooting, so I have two.

Speaker 2: 37:51

I have two main reasons why I think NTP is important, and you just called out the first one. It's troubleshooting, and for this use case you want to make sure that when you're tracking something down, an issue across multiple devices, that the time is same, so you can correlate, because, like you said, andy, if you don't have that, then it's it's meaningless. And then the second is for security. So, similar to the troubleshooting use case, when there are security incidents, you will also want to be able to coordinate events across multiple devices. So that's that's why I think security, or NTP, is important. You got anything to add there, aj? Yeah.

Speaker 1: 38:31

I mean device correlation, or event correlation rather, is huge. And I guess, to kind of tack on to NTP, you know, having trusted NTP servers, so you know is it is it. Is it necessarily a bad thing if you have all your devices going out to a public time source? No, not really. But maybe you know you would rather have a couple of devices go out to the Internet, like your core switches maybe, and then have all your other devices look at an internal address for for time, thinking so there's, there's plenty of public, publicly available. Google has one. I'm sure all the big cloud providers have one. You can go to timegooglecom to get NTP source. If you're working in an enterprise environment, all domain controllers will act as an NTP server. So you can point any network device to a local domain controller and it will get NTP. And then there's even some organizations that host their own clocks, their own atomic clocks, gps clocks in their own networks and have everything.

Speaker 1: 39:35

Get time off of that time. Time for some services is, like, extremely critical. If time wasn't like atomically synced, cell phones would never work. It's just crazy to hear how cell phones work and how much they absolutely rely on time If you've ever been to a cell tower site. You'll find somewhere on the site there's usually some sort of like helical, like looking antenna pointed towards the sky and that's that's the GPS clock Looking for its signal to get that super accurate time. Now that now there's, there's another version of NTP. I don't remember what it's called but it's like even more accurate than NTP and I think it's usually used in like multimedia networks for that like super extra accurate critical time.

Speaker 3: 40:22

precision time I heard that called before. I do need to call out some comments in the chat from our buddy, Just ignore them no no we can't ignore them.

Speaker 1: 40:30

That's why they're here.

Speaker 2: 40:35

Because, people are people in the chat are talking about how great NTP is and they made reference to Andy saying I've never had a DNS issue and NTP is the best. What universe are we?

Speaker 3: 40:52

in. You know what, though? It's why I loved the best part about studying and labbing. I can still remember how excited I got Sounded like such a nerd the first time I configured NTP on a router and saw it sync up and like the stratum thing happened. I'm like, oh my God, just when I was labbing CCNA, like before I even touched the production thing, you know, and then years later get to implement it and, like you said, tim, these are all things that, like you forget about. You said it and forget it. It's gone, it runs in the background. It does become a utility, but without it, you know, you're kind of screwed in a lot of different scenarios.

Speaker 2: 41:24

So, as a Therian, asked a really good question asking if latency is calculated when requesting time. So NTP is. So it's a client server technology. Clients can either set NTP servers that they want to sync with or you can use dynamic methods like Microsoft or with DHCP that can hand out that information. But yes, I do believe you can see, like if you're setting it up on a network device, you can look at different show commands to see the different statistics of NTP. And there's also a concept of stratums. Somebody called it up, called it out in the chat earlier and that basically your quote, atomic clock is a stratum of zero and that is the absolute source of time and anything else that syncs with that. The first server that syncs with that is stratum one. And then if a server syncs with a stratum one server, it's a stratum two server and etc and it's. I do believe there are ways to go in and see, like what the time offset is and how off it is from the time source. Yeah, I think it gives you a latency.

Speaker 3: 42:37

I kind of remember in the CLI latency measurement. If you do like, I think, show time servers or show NTP servers.

Speaker 1: 42:44

It'll tell you what stratum that your network devices source is. And then the offset, the calculated offset based on the stratum.

Speaker 2: 42:52

Yeah and Zapdos calls that a good one too. Stratums are extremely important when doing things at scale, when troubleshooting yeah yeah, because I think you can max out the stratum at 16.

Speaker 1: 43:02

So you can't go beyond a stratum of 16.

Speaker 2: 43:06

And Chris says Andy banging pots and pans, that New Year's Eve is stratum zero.

Speaker 1: 43:13

Love it. So do you want to dive in? Like like, network time protocol is really interesting in that where all computers are basically just counting seconds from a known point in time, from, like you know, 50 years ago now, I think at this point I forget when like second zero is actually considered. So, like, all network devices are counting seconds and then they're just doing the math to make the clock look pretty in humans, except the face value of what time it is. Yeah, that's crazy.

Speaker 3: 43:42

I mean, time is completely made up guys.

Speaker 1: 43:46

It only exists for us.

Speaker 2: 43:47

I mean it's relative Time's relative man.

Speaker 3: 43:50

If you put your router on a spaceship going close to the speed of light now, never mind the deers only know when it's rut season by.

Speaker 1: 43:57

You know what's going on with them biologically. They don't look at a clock to know when it's time to rut.

Speaker 3: 44:04

You know what I just thought of rocket ships. Well, I mean, how critical is time right On, like you know, all that stuff, the seconds of things happening, and this you know. You better not jettison the thing I'm about before the rocket stops spewing, you know, and if your NTP is off, man, that's probably precision time protocol, right that?

Speaker 2: 44:26

they probably need really specific. Yeah, I don't know. All right, so there was. There was one more critical service that I wanted to talk about, and I'm now debating whether or not I want to talk about it, because if I miss step here, tim McConaughey is going to rip me to shreds.

Speaker 3: 44:43

Well, that's going to happen anyway.

Speaker 2: 44:46

And I say that lovingly, tim, just because I know that if you don't do this, last critical service.

Speaker 3: 44:52

We're going to sing a song, so you better do the critical service, I better do it.

Speaker 2: 44:57

And I say that because Tim has an AJ will have to find it and link it in the show notes. Tim did a Cisco live chat in Barcelona, whole presentation with a room full of people. It was great. I watched it when I was studying for my CCMP and it was fantastic. So we'll have to link to it.

Speaker 1: 45:19

I believe he has a plural site course as well.

Speaker 2: 45:22

We had a plural site course right, so we'll quit teasing it here.

Speaker 2: 45:26

It's at a high level.

Speaker 2: 45:27

We're talking about multicast and from a critical service standpoint we're really talking about the rendezvous point component to shortest path or PIM, sparse mode.

Speaker 2: 45:39

So what the rendezvous point is for is really to be able to do that initial distribution of that multicast traffic. So let's say you have a media server and you are pushing out a multicast stream of a live event and you have clients that could be dispersed throughout the network that will subscribe to or join that multicast group and need to pull that feed. The routers of the devices or sorry, the routers that devices connect to that want that stream, basically point toward the rendezvous point and the media server will push its stream toward the rendezvous point and that's how the clients get that initial feed in a sparse mode, multicast deployment. So if that rendezvous point goes away or if you only have one or a router doesn't know how to communicate to it, you can have an issue with devices getting that feed and you can have a major multicast problem. That's why I've seen rendezvous points. I see them as a critical service if your organization relies on multicast for anything.

Speaker 3: 46:47

You know what else? They're called Ronda, never run out of jokes. I was gonna say rendezvous, single point of failure. I didn't know any of this. This is pretty amazing. So is the rendezvous point a router, some kind of networking Usually usually running.

Speaker 2: 47:02

It's usually a service running on a router.

Speaker 3: 47:04

I'll say, or later switch, yeah, so that kind of syncs up or is like the gateway for all the multicast streams, right?

Speaker 2: 47:12

Yeah, yeah, you can see, it is that, and Tim, if, if any of this is incorrect, please hit at permit IP Andy, andy, I actually know it's at Andy laptop.

Speaker 1: 47:22

Sorry, tim, I can. I actually did this drum the stream.

Speaker 2: 47:26

He's on here, the guy joins he heard good, now we're done talking about it.

Speaker 3: 47:31

He heard multicast, then the force ripple he just showed up, yeah you just appear this computer.

Speaker 1: 47:37

Just somebody said something wrong about a rendezvous point the multicast beacon out the multicast beacon.

Speaker 1: 47:43

I love it, that's great. Yeah, you know, tim, it's funny. Every single one of these services we've talked about tonight are absolutely critical to what we do, and from a from a device configuration standpoint. I've been in a ton of networks as a pro services engineer, right, so a lot of this is sometimes, or most of the time, like an afterthought, right like I've seen there on very rare occasions where somebody thoughtfully planned out and design their multicast and whenever I see that kind of stuff, it's like, oh yes, because you'll have some people that are like, oh yeah, we don't, we don't use multicast. And then you do like show IPM around. It's like, oh yeah, you do.

Speaker 2: 48:26

Oh yeah, you do, something is so. Those were the major ones I wanted to cover. Did you guys have anything that popped into your heads as we talked through this of something we haven't talked about?

Speaker 1: 48:38

It's. It's not like a critical networking adjacent service, but I would say logging right like having some sort of centralized law for sure system and not just relying on the local buffer to dump all your logs to, because, depending on what's going on, you could like completely dump the buffer pretty quickly and then not have any logs, or so having some sort of centralized log system to help you troubleshoot always good authentication.

Speaker 3: 49:07

Sorry, buddy.

Speaker 1: 49:10

You know active directory is popular, so having an understanding of how active directory works, or you know if you're using tack, acts or irate, radius or whatever just under understanding how authorizations work on your network is always a good one.

Speaker 3: 49:24

Well, I'm leaning on my like knock days. I mean, is SNMP considered a service?

Speaker 2: 49:28

because that a critical service for I could see considering SNMP a critical service in terms of monitoring, if you're relying on SNMP to make sure things are up down under capacity, that kind of thing. Yeah, I would say that's critical service.

Speaker 3: 49:42

I mean, is there any monitoring without SNMP? And I'm being kind of snarky, but isn't that most of what everybody's using SNMP?

Speaker 2: 49:49

it'd be tough to see what the hell is going on, right well, yeah, I mean, there's the whole concept of streaming telemetry to try to. I knew you were going to say telemetry. I was forcing you to say telemetry.

Speaker 3: 50:00

Yeah, telemetry is like a unicorn, right as anybody actually seen telemetry. They got rid SNMP and they made streaming telemetry happen. I mean I know there's companies out there doing it, but People say telemetry. I'm like, yeah, okay. I mean I mean, have you guys seen that like a networks you've run? Have you not run SNMP for monitoring? I haven't met anybody that hasn't run SNMP for monitoring. But when I push on it and they say telemetry and like here we go, tell me about your telemetry, sorry.

Speaker 1: 50:30

Awesome. Anything else, tim? Before we wrap up.

Speaker 2: 50:32

Oh, this has been fun and I've really I'm really glad you brought up pushing this to YouTube, because the engagement we're getting on this is fun as hell.

Speaker 3: 50:42

Yeah, jordan Villareal agrees with me that SNMP is totally critical, so I got one guys.

Speaker 1: 50:51

Awesome. Yeah, this has been a ton of fun, very successful evening of live streaming our recording of this episode. So I think you can count on us to do this in the future on YouTube so you can join us and chat with us and interact. We do thank everybody that participated this evening. It was a lot of fun to hear from a lot of our listeners and interact with you all while we recorded this episode and get you to chime in on what we're doing here. Thanks again for showing up, guys. Thank you so much for for joining us tonight. Tim, thank you for a wonderful idea. It's always fun to dive into topics like this and we'll see you next time on another episode of the arts of network engineering.

Speaker 2: 51:31

Happy New Year to you all that see this in April.

Speaker 1: 51:34

Hey everyone, this is AJ. If you like what you heard today, then make sure you subscribe to our podcast and your favorite podcatcher. Smash that bell icon to get notified of all of our future episodes. Also follow us on Twitter and Instagram. We are at art of net inch. That's art of net e n g. You can also find us on the web at art of network engineering dot com, where we post all of our show notes. You can read blog articles from the co hosts and guests, and also a lot more news and info from the networking world. Thanks for listening.