00:00
This is the Art of Network Engineering podcast. uh

00:11
In this podcast, we'll explore tools, technologies, and talented people. We enter through new information that will expand your skill sets and toolbox and share the stories of fellow network engineers.

00:28
Do you all smell that?  There's something in the air, but I just can't put my finger on it. Rather, maybe not in the air, but out in the ether? Or the ethernet? You know, Lexi reads that book about ethernet. It's got an octopus on it. That'd be pretty wild if that's what ethernet smelled like, wouldn't it? Did you know that a group of octopuses is called a consortium? They're just so dignified. What am I talking about again? I think I'm freaking out, man. Wait.

00:57
What's that up on the screen? Oh, that's what's wrong. Hey, who left Wireshark running again? I've been in here sniffing packets all day. All right, well, while I air it out in here, enjoy this episode of the art of network engineering. know, Tim started sniffing and I got real worried. I thought we were in the same room again and I was like, oh man. Welcome to the art of network engineering, Tim. Thank you so much for that intro. That was that was awesome. Very.

01:26
very well done intro considering our topic this evening.  I am joined tonight by Tim at Timbertino. Tim, how you doing? I'm good, AJ. But I mean, even if we were in the same room, I've got all my shots. I'm up to date. I don't have to wear the cone thing anymore, so we're good. I wasn't worried about you sniffing that.  Oh,  oh, this is a different kind of show.  OK.  Oh, gosh.

01:53
I'm doing good AJ. It's uh at the time of this recording. It's getting to be the the fall time of year uh I'm kind of a college football nut. So I got oh, I'm in my I'm in my happy place. So you are That's awesome. I'm good. Things are good. I how you doing? I heard a stat last Saturday. There was like 58 college football games playing like that day How did you watch them all Tim? Like I know I know you got like more than one TV in your basement. Did you have 58?

02:20
Not last weekend. We were actually out and about, I will be in my command center on Saturday. I'm ready. It's really the knock for where you work, but you just put cable on all the TVs. I'm going to start calling it that. like that. go. There you go. All right. And he is back. Dan at Howdy Packet.

02:44
If we were live streaming  his arrival this evening, uh he's been gone all summer. He's had a lot on his plate. uh He was playing Back in Black by ACDC as he joined our stream here prior to going live. So was  very,  very good intro song. Dan, how you doing? Good to see you again. Hey, Jay. I'm doing wonderful. I got a new baby girl. uh Congratulations. We sold our house. So I'm in a family-owned rental.

03:14
At the moment.  So, yeah, that's kind of like my life right now. Hang on. He's he's been gone for a little bit. I wasn't going to say it was too long, but he's been gone long enough that he didn't say howdy.  He said, hi, I  was going to say howdy. And then AJ kept going on. So I slipped my mind. I'm ADHD. So if like I don't say it,  I forget it.

03:44
Excellent well Dan it's good to have you back and last but not least mr. Andy lap teff Andy. Do you have a pool yet? It's been all summer I Do yeah, all right all right I will direct you to his YouTube channel to see at first hand  You put up a YouTube video on how to scrub a Cannonball man. Oh All right.  All right.  I think it's uploaded it yesterday come on

04:14
It's been a long, it's been a long time coming and yeah, we opened it and we got the swim in it and you really got to brush plaster a lot, which for C, you know, I wanted a big house till I had to clean it. I wanted a big yard till I had to mow it. You know, I wanted to be cool until I had to scrub every square inch of it twice a day for a month. It takes me an hour each time. you know, like, but now we got in at the other day, we had some friends over. It was just, you know, it was all worth it. Uh, the kids.

04:44
Got home from school today and like daddy we want to go in a pool and great. It's  so it's yeah it's it's been it's been really nice. Other than that everybody's good. Happy to see Dan. It's good to be on here. How you doing AJ. What's going on with you. Doing good doing real good. uh Man things are so busy at work but I love it. It's a good busy. I have  my first Vermont customer. We kicked off a project with a few weeks ago  so it's very nice not to be driving down to the Boston area for once.

05:13
Not that I mind that. It's just really nice. I appreciate having a customer in my own backyard. And I've got a few projects lined up that will keep me in state for a little while. So pretty excited about that. Drone stuff's really taken off, pun intended. Yes. I see what you did there. Doing a lot of drone photography, having a ton of fun with that. And yeah, it's a good time. Yo, I got to give you props. Oh, thank you. Appreciate that.

05:42
So listen, I'm an amateur photographer, right? I've even been paid for a gig or two. And I saw you get into the drone stuff and I'm like, oh, he's adorable. Look at this, isn't this cute?  You know what mean? This is really cool. He's going to be photographer, right? And then some of your shots are like, okay, yeah, but dude, the ones you just sent me from your mass trip, like next level, like you're  locked in, dude. So yeah, really. uh

06:05
I can now say rock on dude, you're doing some really nice, really nice work. That one  sunset with the silhouettes of all the boats. Yeah. And I think there was like little, like dude, it's like a work of art. So I'm really enjoying it now. Yeah. I really appreciate that.  I do a little uh Instagram separate from my own  No Blinky Blinky account. If you're interested in checking out my drone photography, you can go to at drone journey VT for drone journey of Vermont. That was the intention.

06:34
ah But yeah, as Andy said, I took a trip down to Massachusetts this past weekend for the long weekend ah and  yeah, brought the drone and uh my wife grew up in the Rockport area. So took some really cool shots of the Rockport Harbor and  had a great time doing it. So lots of fun. But that's not why we gathered here tonight. I am very excited. We are talking about capturing packets, as  Tim alluded to, and we brought no other than Chris Greer to join us and lead the discussion. Chris, thank you so much for joining us.

07:04
It's a pleasure to be here, gentlemen. Thanks for having me tonight.  When we usually interview guests, we want to go over just a little bit of your career. Give us a little bit of background, how you got started in network engineering, and more importantly, how did you get to focus on capturing packets  and  kind of what you do today?  Just the  cliff notes, if you will. Sure. Yeah, no problem. And I think like a lot of you guys, I've heard on previous shows that

07:33
Just getting started in networking, having a passion for it, getting excited about moving traffic around, uh making things work end to end.  It hooks you quickly.  So that's something that I got onto  early in my career. I was hired by a company to come in and just do some internship stuff on just building small networks and getting things up and going. um I started going to trade shows. So specifically Networld Interop was one that I...

08:02
uh started to help set up those event networks and that's where things got to a much larger scale and I started getting some more certifications, vendor certifications, Cisco and otherwise.  But when I was at those shows, I started to see a bit of an interesting,  just a niche within the IT sphere and that was that we started to have these really weird issues on the network. And of course the network's getting blamed. I mean, this is nothing that's gone, right? We still all have weird stuff happens.

08:32
Oh, sure. But uh at that time, what was interesting to me is you had top dogs from Cisco and Foundry and Extreme Networks at the time and  all these other big players. They were there and they had some of their best guys and a problem would strike and everyone's just hammering away at their keyboards and this problem is just continuing on. And then these other guys would be over here in this corner and they were capturing traffic and looking through things at the packet level.

09:03
And sure enough, 99.9999 % of the time, unless it was some bug that needed to be patched, they would come back and they would be able to pinpoint the problem, just boom. If they couldn't fix it, at least within minutes, they could get it down to the, like maybe we don't know exactly what it is, but that box over there, that server, that device, that client, that something is spending 20 seconds responding to whatever. That's the device we got to focus on. I was hooked.

09:32
That's awesome. And what did you call them? Oh yeah, they got, those people got a name on the team and it was the Packet Heads, right? Packet Heads. So we get a problem and then boom, Packet Heads. So if you're watching this on YouTube, you can see my Packet Head shirt. That's a, or if you follow me on social media, just go to at packet pioneer and you'll see my logo there. But that's something that for me, it was such a career changing moment to go, well, you know, I got a,

10:02
Building the infrastructure, the highways and byways and the speeds and feeds is fun, hands down. But understanding the traffic that is traversing those speeds and feeds and how it actually works and the protocols within  that  carry all this important traffic. Once I really started to get into that, again, it was that those aha moments started clicking and I really got passionate about analyzing and troubleshooting with packet captures. That's awesome.

10:32
You know, packet captures is like one of those things that I did a lot when I was studying for  CCNA and I do it  not a ton, but every so often, but every time I do it, I'm like, why don't I do this more?  It's just so insightful.  I was telling the team uh a few weeks ago, I troubleshot a BGP issue with uh Cisco endpoint and NSX edge tier zero.  And uh we did a packet capture and we saw it's setting up the TCP.

11:01
connection, it's sending the BGP initiation from the Cisco side, but then we're getting a TCP.  What was it? uh

11:13
like a TCP drop or something, right? And it's just like, well, that's not the right response. And so  we didn't know exactly what was causing the problem, but we knew the problem was over here on the NSX side, right? Because Cisco was setting up the BGP, uh but NSX wasn't responding properly. So we had to have another team member come in and deal with the NSX side, and we got it figured out eventually. But without doing that PCAP, we wouldn't have known what exactly was going on. Oh, 100%. Yeah.

11:40
And I think now, you know, for a lot of my career, I've been on the performance troubleshooting side. So now as a trade, what I do is I fix stuff for people that send me PCAPs.  Right? They send me the capture. go, Hey, here was the general thing. We're disconnect, uh, slow course. The network's getting blamed. Just fingers pointing around the data center.  We just got to get this thing fixed. Here's a PCAP. So help out. So, so I do that. And I also on the other.

12:08
Half of my world I do training for Wireshark University through the Wireshark Foundation. So  I love teaching it too. So that's why I started up my YouTube channel and  it's fun helping people get started with it. Nice. What's that email address? I might have to send you some packets.  Oh  yeah, right. Hit me up, man.  For sure. Like I know a guy. Hold on. Is that a paid service? Like people will pay you to look at their PCAPs and tell them the problem? Yeah, absolutely. Wow. That is awesome.

12:37
I want to point out two things. One, AJ used the past tense incorrectly of troubleshoot. He said trouble shot. Oh, sorry. I believe it's trouble shat. uh That sounds right.  Sorry. shart.  Trouble shart. Oh my God. Well played. But the other thing I wanted to mention, we were talking before the show, AJ to your point, you've solved problems.

13:03
doing packet captures and you know, people are paying Chris to like look at PCAPs and I somehow spent a decade working at very large networks, very large companies and I've never run a packet capture myself. Now I have seen the wizards, like you said, the packet heads Chris, like in our company there would be like one or two people at the knock. You know, we're all banging our heads against the wall and this weird thing and it's been going on for days and it's intermittent, blah, blah, blah, right? We just can't catch it. And the people in the know, right? The people who can see the matrix, which is what it's like to me, you know, you guys can just see.

13:33
What's happening behind the, you know, behind the scenes  and yeah, invariably they always figure it out. Right. That weird. So I'm really looking forward to you talking about and showing us some stuff tonight. Cause this is definitely a blind spot of you seen the, uh, the shame meme? The shame shame.  That's me right now to you, It's,  it's deserved.  Can we, uh

14:00
Can we kind of take it back to basics? I want to start off, Chris, if we can kind of break down some terminology.  When it comes to this kind of topic, I think it's easy  to use multiple terms to mean the same thing.  sometimes that's not always the case. So can you kind of walk us through,  at least at a high level,  what is a packet capture? What's Wireshark? And what's a PCAP? So it's a great question, because it really just depends on who you're talking to sometimes and which one they'll use.

14:30
The industry typically is going to call it PCAP or trace file. Like for a long time I used trace file. Like, hey, send me a trace of that. And then  I had a student in one  class. said, do you mean a PCAP? And that's usually the extension on the end of em the  trace file or the packet capture. is that where PCAP comes from? It's the file extension? Yeah. Yeah. Well, that's one of the... There's probably some amazing history that someone  knows deeper than I do on the origin  of that.

15:01
um Yeah, so so pcap or pcap ng  you'll see a lot these days, but yeah, so so  Either trace file packet capture so wire shark. Let's talk about that for a moment um so wire shark is a the world's  basically the world's best open source packet capture tool  and  Underneath it it uses a driver called NP cap, which is actually from the end map organization  so uh it it's a

15:30
GUI-based uh packet analysis tool that is free. You go out there to download it and you install it. And what's cool about Wireshark is because it's open source and it has a very large contribution of developers. A lot of times when you'll see experimental protocols or maybe somebody with the IETF is tinkering with something new, a lot of times you'll see support for that protocol come to Wireshark first.  say, for example,  the QUIC protocol that's uh come out just in the last year.

15:59
as a standard,  you saw that the disectors for it show up in Wireshark first, because the guys that are actually writing the protocol need to see it, right? So they use Wireshark to dissect what they're working on, then that all goes back into the actual tool itself. So that's one of the reasons why it has such a solid tool, because it has such a community involvement to it. So the tool that we'll use to capture would be something like Wireshark.

16:28
That's the one that I use the most. You also have other vendors that have their own packet collection and analysis tools. A lot of times those cost quite a bit of money because they also have hardware on top of it. So purpose-built stuff that's designed to keep up with a hundred gig stream. So, uh but packet collection is its own em section of our industry. But I think the most common tool in most network engineers toolbox.

16:56
four pack collection with the Wireshark.

17:00
We'll be right back. This episode of the Art of Network Engineering is sponsored by METER.  METER delivers network infrastructure for the enterprise because every organization deserves seamless connectivity.  Whether you have a large team of network engineers or an IT team of one,  METER makes it simple to get online and stay online.  METER provides a full stack integrated platform that combines hardware, software, deployment and support.

17:27
So enterprises can ensure their networks have the performance, security, and reliability they need  without the inefficiencies of juggling multiple vendors.  It's enterprise networking re-imagined, giving IT teams the ability to spend less time managing complexity  and more time driving business strategy.  Go to meter.com forward slash uh A1 to book a demo now. That's M-E-T-E-R dot com forward slash A-O-N-E.  Now back to the show.

17:58
How do I get the packets to wire shark? Like I've wanted to run a packet capture at home and I just have no idea. Start like, know in big organizations, there's like a tap, right? And it just sure. It's everything over to somewhere and they capture it that way. Like, how do I, I have a laptop, I install wire shark.  How do I get packets to it to look at? Yeah. So that's a great question. Um, so the first thing.

18:24
The simplest way is if you can generate or break or experience whatever it is you're troubleshooting from your machine itself, you can start up that capture locally, break whatever it is, and stop the capture. And you can do that with Wireshark. Hit the blue fin, break it, hit the stop. Boom, you've got a PCAP off of whatever interface you selected. So a lot of times if I'm working with one of my clients, that's what I'll have them do. Is there any way that I can get them to just experience locally from a machine?

18:55
Now there's a few things, there's some gotchas there. Whenever I'm capturing from within the device that is generating the application, there's a few uh one-offs I got to consider. Like it might not see it the same way as it is on the network, right? The NIC can change things, the TCP stack can change things. But Andy, the next way  that we can get started, and I actually literally just did this on my home network.

19:21
I bought a $50 little managed switch. It's got a span port on there.  So now I go from my home router through that switch, copy everything to and from my edge router over to the span port or  the switch port analyzer port or the mirror port, whatever it calls it. And from there I can plug in. I went ahead and bought a Raspberry Pi and stuck a little credit card size SSD on that thing. And now I'm streaming to disk 500 gigs.

19:51
At my, my house, everything coming and going from my network is being captured.  So it doesn't take a lot of, I mean, I built the whole thing for 250 bucks. I'm actually going to talk about it on my YouTube channel, but it doesn't take a whole lot of hardware to get started. You know, we can just start with the laptop. just so I, just so I follow. So you have your edge router at home and you connect it to a switch.  all your traffic is coming through that switch  in and out. And then.

20:21
You just create a span port off of that, which takes everything coming through that device, sends it out this other port called a span port. And you connect that, you connect your laptop into that span port. How does Wireshark, okay, so that's coming out of that span port, boop, I'm in, Wireshark grabs it. Okay. I'm with you. And then I can have maybe, let's just say I had a USB to RJ45 connection, right? To make it cabled, because a lot of devices don't have RJ45.

20:51
So if I use that to plug into the switch, then I can manage that device wirelessly. That's what I'm doing with my Raspberry Pi. It only works though, Andy, if you actually make the boop sound. you don't do that, you missed it.  So  that's good question. So that's the second way. You can either locally capture to and from your local machine. Another way is you can use a span or mirror port.

21:20
You have to in a switched environment, Because switches are going to isolate frames to, uh especially unicast frames are only going to go between those two interfaces. The third way would be a tap. And that's when I want to spend a little bit of money and I got a bit more,  you know,  more power,  you know, behind it. That's hardware, right? Is it tap hardware? Yeah. And that's where I actually literally physically break a connection,  snap it in  physically.

21:48
snap in whatever else, let's just say a server array, pull it out,  pop in my tap, plug it back in, it's back up and running, and that tap is now physically in line.  Then it has  analyzer ports that you then plug your analyzer to.

22:07
So I'd like uh Chris, I'd like to get your take on this because I'm  I typically work with Cisco, right? So uh one of the things that I've been using lately uh and I've got actually an interesting use case. uh So  like you had mentioned earlier, the collective we as is network folks, we always seem to have to prove that the network's not at fault. Right.  we  troubleshooting is a huge use case for uh

22:37
packet capture, we actually had an issue where there was this print server  that had to uh print jobs to these little ticket printers to print out orders for different things.  And every now and again,  these  printers would just stop. And the people that were expecting the prints uh just had no idea  that there were orders that were supposed to be spitting out  and they weren't. it was

23:07
pretty detrimental for them when this didn't happen. And their fix was that they just had to basically power cycle the printer and  it would come back up  and none of the jobs that were supposed to it  out, but any new jobs would come through. So it was obviously the network's fault, right? We were just gobbling up them packets and eating them.  So I am not by any means a packet capture expert, but I'm halfway decent at.

23:34
click it on shit till it works. uh And one thing that I really like about the newer Cisco switches, and  I'd love to get your opinion, I'm guessing you have worked with multi-vendor switches, is that they have basically an embedded packet capture feature.  So you can just build  a packet capture on the switch, pick your source port, your destination port, or your source VLAN, whatever, and it'll just create the file locally and then you can

24:04
ship it off, SCP or whatever and view it in Wireshark. Are you seeing that with uh that kind of feature with other vendors as well? Oh, 100%. Yeah, that's coming. it, well, it's been here. And I would say like in beyond even switches,  like firewalls  and you know, being able to do TCP dump on a box and so on.  So here's the deal guys. I've been bit hard by that. Okay. oh So  I've been bit hard by that. And that, and what I mean is, em

24:34
This is why I write my statements of work real specifically. So if there's any unexpected outages that happen because of some weird bug on  some device, it's not my fault. You can't sue me. So um basically like  we need to take into consideration. Well, first of all, if the network's getting blamed for slowness, now we're adding another job for that thing to do. So, and then the other thing is at what point will Cisco prioritize

25:02
the packet capture function versus the forwarding function. Sure. Because I've done some benchmarking with just tinkering. These are articles that I've written in the past. Maybe we can link them or share them or whatever. where, OK, if a, let's just take basic switch, two interfaces, packet comes in. I expect switch goes, whoa, packet important, forward pocket onto next link.

25:30
But when I give it a job to do, I can't trust that now. Is it going to go receive packet, make copy, save to buffer, then process, then do what it's got to do and forward it on? Is it going to delay the forwarding? Hopefully it won't. But where I've gotten bit in the past is when buffers like that can fill or even unexpectedly become taxed, I've seen it.

25:59
dump switches and completely reboot them on massive application arrays. I haven't seen it recently, but I'm gun-shy enough that I have a bit of a policy. If I can, now this is where I get to be a little bit of an elitist, right? If I can. I want the firewalls to firewall. I want the routers to route. I want the switches to switch. I want the taps to tap. And I want the analyzers to capture that stuff. That's what they're designed to do.

26:28
If I cannot do any of the things that I just said and I have no choice and my client lives in  Germany and they have  no packet capture or taps or span capabilities, then go down the path of what's the next worst option.  So  I think it's cool that these options are coming to the infrastructure, but uh it wouldn't be my first way to capture it if I had a choice.  let me ask you. uh

26:58
So I've done a lot of captures on like ASAs before. uh I've done a lot of that. don't know. Are you saying that that might not be the best way to do that? Like directly on the box? Well, now that device,  sure, sure. That's a good question. um Any time you capture within the device that is involved in the path, it's going to give you its perspective. Right.

27:22
So how did that box shape that um stream? Did we really see it as it was on the wire? I gotcha. Right? So there's nothing, there's nothing, I would never discourage people and say, I don't make blanket absolute statements because then as soon as you do, you find the exception, Right, yeah. So  I never say never because tomorrow I could get a client that says, hey, I got an ASA capture, what can we do with it? Okay, give me the capture. Yeah. You know, but.

27:51
But best would be uh external to the active device.  And uh tap is always going to be the best. If you can't, span would be my second best.  And then if I can't, then physically on that device. Because again, people always blame the firewall, right? Yeah. Oh yeah. Well, if I'm capturing on that firewall, how do I know that that process  is They're not going to believe you.

28:15
Well, that's what your firewall says, but  yeah, that yeah, I already don't trust that. Well, and actually I, I had a, um, I'm sorry. I didn't mean to, who did I jump on just now?  Uh,  I was just going to say, I mean, to, to finish off the, the, uh, printer story,  I cannot tell you how good of a feeling it was to  have that packet capture going, have the, uh, the end user tell me, okay, it happened.

28:43
At this time, we're not getting any prints at all right now.  Because basically we told them, you know, when this happens, let us know when you uh have to power cycle that printer. So look through that packet capture and in the feeling I got to be able to see that that the printer received the job  and acknowledged the fact that it received the job and just never printed it was was huge. Because at that point you're just saying, OK.

29:12
We can prove to you that the printer got it.  Yeah. So figure out if it's, if it's firmware, if it's hardware, whatever,  we can prove that the packets made it through the network and got to your printer. And not only that, but your printer said, yes, I got it. Yeah. That was cool. And from what you just said, that's where the terms, uh, or the term packets don't lie came from. Yeah. Packets don't lie.  I know my stuff got there cause it acted. Yeah.

29:42
Network's done, homie.  I mean, I actually have the capture. was going to show you guys just, but  I don't know if I can talk through a screen share, but just to give you the high level, um a client of mine was like, we're getting network guys. We're just getting beat up on and shredded because of this issue. It's slow. The clients hit this one tab and it's just crawling. Like, man, just, can you help get the blame off of us? And they were getting hammered.

30:12
with the blame like the network sucks and that works out.  We found out that the clients were connecting to this server, TCP connection, boom, boom, boom. A lot happens in that TCP handshake. There's a ton we can unpack there, but I'll withhold the excitement to go down that road. um But that at least tells me my network round trip time right away. I don't need a ping. If you give me a PCAP with a TCP conversation, boom, off the first.

30:40
handshake, can tell the network round trip time, boom. So now I know the latency. Latency wasn't too bad.  No retransmissions. Request goes to server, server acts it immediately. Exactly the same round trip time. So I know the network's not congested, because latency is not shifting. It's not dropping anything, because I have no packet loss. So request goes, act comes back immediately.

31:06
45 seconds later the client checks into the server like it sends a TCP keep alive Mm-hmm because the application was taking so long to respond TCP had to say whoa I haven't been told to shut this down yet. So hey server you still there. Oh wow Server hits back immediately network latency never changed. Oh, yep still here So client waits another 45 seconds

31:33
Check soon again, TCP keep alive. Server's like, yo, I'm here. It's like if you ever call into a call center and they're just like, thank you for calling, please hold. And then 45 seconds later they come back, you still there? Good, please hold. And then 45 seconds again, you still there? Please hold. So that's what's happening. Well, finally 108 seconds was the application response time before that application actually sent data back to us. In seven packets we were able to prove that.

32:04
The network had no congestion at all. Latency was like a clock. No loss, no packet loss. It is not the network. It is 100 % that application. So who the hell, this is a guy who knows nothing about software development, who in their right mind would write an application and say that two and a half minutes is an acceptable time to wait to send a response to a request?

32:31
They didn't write that in. It was a back-end congested process. So it wasn't even the server that was responding  quote unquote slowly. Something behind it was. It was trying to talk to somewhere else before it responded and something was hung up on the back end. Okay. Now here's what's frustrating as a client. I was hired by the network guys. As soon as I got the blame off the network, they're like, okay, bye, thanks. It's not us.

33:00
What is it then? Oh, that's not what we're paying you. That's not what we're paying you for. So get out of here.  You don't you don't get any closure. Oh man, curiosity kills me. That happens. That happens a lot though. You'd be surprised  as soon as I get the blame off of whatever whoever is paying them the money. It just depends on if the other  silos motivated to actually fix it. They're like, okay, bye.  Right. Exactly. Thank you. So we are. We've talked about uh we've talked about troubleshooting being a big.

33:29
reason why you'd capture packets. What are some other use cases that people would want to do this? Oh boy, cybersecurity. Just 1000 % like in fact, um even for me um in my  teaching, I used to do just network performance troubleshooting. I was always listening for two words, intermittent and weird.  If I hear either one of those, let's go. Let's get it. Wireshark out and let's.

33:56
Let's figure out what's up and let's teach you to do what we are doing here. You know, make sense of the matrix like Andy said earlier, which is actually something I say in my courses. So we're thinking the same way Andy. Yes. Um, yeah. But, uh, over the past few years, I've had to pivot hard into the cyber conversation in that, Hey,  um,  Hey Chris, are we getting hacked? Is data being exfiltrated?  Do you see anything strange? Any anomalous behavior? Any, any malware?

34:26
Are getting scanned? And this has happened. It's starting to pick up pace where they hire me to come in and do a performance gig. Hey, this application is slow. And then I find out,  well, that's weird. Your IoT device over there is sending out some Nmap looking scan activity. Why would your light bulb be scanning your network?  That's what I was going to ask.  You get asked those kind of questions.

34:55
Hey, do you see anything weird or anomalous? When you're looking through a packet capture file, when you hear that, what is there something specific you're looking for or just things that don't look right? What are you looking for? That's a great question, Tim. That's the money question really. It's what does weird look like? I think. Exactly.

35:15
I ask that in the mirror every morning. You mean you answer it in the mirror. Well, yeah, he talks back. And that's why you're on a podcast, right? a video stream. Thanks for radio for sure. No, I think one of the ways that I go threat hunting is I remove...

35:41
And that's where we always have to make some assumptions even though we can be wrong with anything with cyber. Let's get rid of what we expect to see. And then let's come in with what's left. Now that's a place that hackers can lurk. So just because things don't look weird doesn't mean they didn't make it look normal. Right? So you always have to ask yourself questions like,  should Dan be talking to AJ and sending him a gig of traffic at one in the afternoon?

36:11
You know, those are questions. um Why is Dan talking to XYZ country?

36:18
You know, if we don't have any clients over there and  does he, is he even aware? You talking to North Korea or?  You can spit out any country you want to.  But you know, like those are the things that we start to look at and go, Hey,  and this goes a little bit beyond packet capture Tim. There's other tools that can help us to do this. Cause the writer or the original author of Wireshark, Gerald Combs says it best. He's like, you're.

36:47
When you break out Wireshark, you're investigating an elephant with a microscope. Like, why don't we back up and look at where the micros... If the elephant's limping, then we can hone in on a certain area. But we can't run in and just look at that guy with the big old microscope and just hope we find the right thing. We got to kind of start 100,000 foot and then dig down, which there's some other tools that help us to do that. But packet capture... So back to your original question, what are some other use cases, cyber's...

37:17
the conversation out there right now. didn't even consider there were other use cases. I thought it was just people trying to broken, weird networks. Do you remember that story years back where a casino got hacked through a fish tank thermometer?  No way. Yeah. But it was the same kind of thing you're talking about. Like hackers got in through an IoT device, got a foothold into the network, started digging around and somehow got into like a high roller database and

37:45
hold the data through the network, through the fish tank thermometer and out to the cloud and got information. It was a big thing, that IoT stuff, but that's so smart that you can look at the captures and see why is our database talking through this thermometer  in a fish tank? To your point, yeah, it doesn't make any sense. But do they have software that does that?

38:13
And people just don't have the money for it. Like do they have stuff that just watches like cyber? Right? Yeah. Yeah. 100%. And I mean,  now that's what our, you know, ideas, IPS systems intrusion detection is designed to do to watch. Hey, I on any given Monday, Andy talks to Tim, Dan and AJ and that's it. Why is he suddenly having a conversation with Chris who lives in?  Zimbabwe, I don't know, but you know, like the

38:43
This is not typical of Andy.  So that is now behavior based analysis and all that comes from the traffic patterns. As a human being, that's hard for me to do, right? So that's why we got to use some of these tools because we can't keep up with 10 gig feeds and just with the naked eye. But that doesn't mean that Wireshark can't help us go digging and threat hunting. So I bet you Andy that happened at DEFCON probably. What you're talking about with  the fish tank.  It might have.

39:11
That's why when I went to Def Con, man, I turned everything off. I left it all.  I didn't bring a thing, man. No. That's what I've heard. If you're going to go to Def Con, you just kind of leave your personal stuff at home. Like if you absolutely need a phone, get a burner phone. Turn everything off. Yeah. It sounds like a bad idea.  Tim, another use case I would throw in there is for anybody learning. When you're studying for any sort of certification or if you're just trying to learn this stuff.

39:41
You know take OSPF for instance. You're trying to learn the protocol Look at a packet capture of two OSPF routers trying to form an adjacency and you'll you'll get it. You'll see  right on so it's a great way to learn Hey real quick. We have a question from the patreon's Eric Doogie 312 is asking question for Chris as you use the hack 5 packet squirrel And it is a is it a good tap device? Hmm heard of that one. I've seen it  I've

40:09
just held it in my hand but I've never practically used it.  So,  sorry I don't have much on that one.

40:17
Hey everybody, Lexi here. Our friends over at Netali just announced the next generation Etherscope NXG, the first and  only analyzer of its kind that fully supports Wi-Fi 6 and  6E.  This powerful all-in-one instrument can help you quickly test, verify, and troubleshoot technology upgrades and base T, PoE, 10 gig, and Wi-Fi 6 and  6E networks. So check it out.

40:43
Go to netallied.com slash either scope to request a virtual demo.  we going to actually see like, don't know what it is. do  it. Let's,  let's see what  do you have some PCAPs teed up to show us? Do you, you ready to do like a  packet capture in real time?  I think it's time to show us what, what Wireshark looks like here. I'm currently hacking Dan. like  I could give you a tap if you want. I wouldn't be a

41:13
I'm wearing this shirt if I wasn't ready to break open my shirt.  This might be a good time to tell our listeners, know, this is an audio podcast, right? But we also have a YouTube feed. we're about to get into a screen share and some really cool pack of capture stuff led by Chris. And we'll try to describe to the audio folks what's happening, but... Lots of numbers. Picture of the matrix.

41:39
That's what I was just going to say. It's like trying to describe the matrix. So there's red, green. see. This might be a good time to go check out our YouTube feed if you want to see what's happening. But sorry, Chris. Go ahead. No, it's all good. I was going to say anybody that has seen my content knows what green means. Those are sins. So I know that was an attempt at a connection. So no, actually. So something that I am very passionate about doing is going back to the matrix example. Like, like, what is this? I look like I'm looking into an airplane.

42:09
cockpit like,  what are all these buttons? Like, what do I actually do? How can I turn this into actionable answers? Well, a of that comes down to taking some of the fear out and the confusion and the overwhelming thing. Right. This screen is why I think I ran with my tail between my legs when I first saw a I did too. I did too, man. No, I mean, I never will forget how that felt.

42:35
So let's go ahead and take this one little bit at a time. It starts with getting a very, very clear picture of what was being experienced. Okay.  That will guide me getting into the packet capture. So I start before I look into the packets. If I get a blind PCAP sent my way, hey Chris, there's a weird problem here. I'm not opening that thing up.  I pop right back. Okay. I need to know.

43:03
Who, what, where, why, when, how often is it reproducible on demand? Where was this captured from?  What was the capture perspective? Are we confident we even captured it? How far into the capture did you experience it? Did it go away immediately?  Did you see it several times during the capture? Did you stop right away? So you need context to make sense of this  mountain of data, right? That'll make it easier on me. Yeah, okay. Right. Now, I will say, and Chris, correct me, but sometimes uh if you get too granular though,

43:33
you kind of miss what the problem is. Because I've looked in here before looking at source destination and what port.  And when they tell me the port that they expect it to be working on, come to find out backing up out of that port, it was using a different port that they weren't. Sometimes developers, they'll use just kind of random high number ports for some of their applications. Well, instead of it using 443, we found it using something completely different.

44:01
Oh, yeah, and so it was like but I was so focused on trying to figure that out that I missed the what the actual issue was so I just want to kind of throw that out there, but that was that's a good observation because that that explains You know a lot of I mean if that still happens to me, of course You know, I don't know I don't open up a PCAP and just BAM find the answer just within seconds every time right nobody does you know, so

44:28
uh It's asking myself questions. What was the symptom? So let's be clear on the symptom on this example that I have open here. I have, just for the listeners, I 900 packets on my screen. The symptom was there was uh a client that was having connectivity problems to the internet. All so they open up a browser, tab one, go out to news.com, whatever, works fine. Tab two, go to the weather, no problem.

44:57
Tab 3, check their bank, no problem. Tab 4, YouTube, no problem. Tab 5, Spotify, cool. Listen to music. Tab 6, white page, didn't work. Tab 7, white page, didn't work. Go back to Tab 6, hit refresh, works.  Now suddenly Tab 1 isn't working anymore.  Okay, shut the whole browser, start over. All seven tabs, no problem.

45:26
Now then, you know, as you go throughout your morning, intermittent random connectivity problem affected everybody. It wasn't just this one individual. Network was being blamed, 100%. It's the wireless, it's those dumb APs, it's the this or that, oh, we need to upgrade our backbone to 400 gigs per second, even though we have 20 employees.  You know the deal. It's okay.

45:53
Intermediate connectivity. So things are not establishing. It's that's different than I got there. No problem. I was in flight and then the site took a dive Okay, so I just put that I plunked that in the back of my mind. Let's go kind of take a walk I'm just gonna take a walk down the PCAP right now I'm just dragging the slider on the right top to bottom looking for anything that just jumps out from my eyes This is called just taking a walk down the PCAP Chris. I'm sorry. Was this like from a span port at the?

46:22
at the business with the company? Like that's a good question. And in this case, this was taken directly on one of the clients because they didn't have access to change the network during broad daylight change control.  This is that first example you gave us earlier, like the local capture. OK, I said, do you see it? Can you reproduce it on your client? And they said, yes. I said, can you get it on a span? Not now during day. We got to wait till tonight to change. OK, just do it quick and dirty and we'll see what we find. Yeah, you're not seeing the wire, but this is a good. Yeah.

46:52
Yeah, pretty close. All right, next I see something in the PCAP that I call striping. Someone already talked about it and that's where I have this is one of the coloring rules that I set up on my copy of Wireshark here. Let me just show you guys what the default is first. Let me just come back to default. If you just open this up in Wireshark regularly, this is what you would see. And the gray kind of  doesn't jump out quite normally to me. So this is just my eyes. um

47:21
Some packets are gray, others are red, and the gray ones are SINs. Immediately following each SIN is this red reset.  Okay. So just going to flip back to the one that I like and that's plain. And it's my way or the highway. Is that what RST means in that bracket?  This is a SIN, this is reset. SIN means synchronized, this is the beginning of the TCP three-way handshake. Now, almost every time when I see a new SIN, I see a reset right after.

47:50
And what my eyes are doing here is over here on the right of the screen in the info column, I can see the client side of femoral port because that's usually unique. So 49974, for example, this is a SYN going to that server and then 49974. Here's the reset that coincides with that SYN. So just for maybe young bucks who were tuning in for the first time, TCP should get a SYN and a SYNAC then an ACK, right? That's normal.

48:18
Correct. It should be a three-way handshake if a port is open and available. That establishes a connection. The whole reason, not to get too deep in the weeds, that it's called SYN is because we're synchronizing sequence numbers, which allows us to figure out if any data got missing. But here a reset is basically TCP saying talk to the hand.

48:38
If I hit you on a port you're not listening to, let's just say Andy, you're not listening to port 100. I say send 100. You're going to pop me back a reset. Right. So that comes from the other side, not me, but whoever I'm trying to reach out to, they'll send a reset and say, go away. Correct. Resets can happen for other reasons, but that's a common one that you see. You're trying to hit a port that's either not available or there's a handful of other reasons, but that's a big one.

49:07
Okay, so I see resets on new sins, but here's what gets weird. What my eyes do,  and let's just do this. So resets, right? So let me just come up here to my filter. And someone said earlier um that filters can be a pain to remember, and that's absolutely true. The only one I know is ip.addr == and then the IP address I need.  Hey!

49:32
That'll get you far,  That's why everybody's got to go out and check out my brand new Tri Hack Me room. I just released it last week.  And it's all on Wireshark filters. Okay. Oh, nice. Hands on. It's free. Literally. It's trihackme.com slash jr slash Wireshark filters. That's the room link. Let's put that link in the description. You can,  and that'll pop right to that. And it's just a, it's a whole lab all about Wireshark filters. And you can check that out on my YouTube channel too.

50:02
So all right, so check us out now. Let's just go down the likelihood road Here's a client 192 168 1.1 is receiving resets from all these different sources Look at all these different IPs guys Are these all coming from the same network? No Nope,  nope, are they coming from the same range?

50:27
Let me come down to my source GIP information and I'm just going to plunk in.  I've got a country code here. So why don't I just say source country? I'm going to right click that. Apply as column. I'm just adding a column upstairs and I can see that there's different countries.  United States, Germany,  Japan.  Quick question source desk. So I would have assumed source was the client sitting at the laptop, but looking at these addresses that doesn't.

50:57
The resets are coming from these different IPs. these are addresses that the resets are coming from? Yeah. See, the, those IPs are coming from the source column. The client IP is on the destination column, but that was a good note. That was something good to note. It's the source of the packet. Okay. Correct. That makes sense. That is correct. Let's remove this. So I can see they're all coming from different places. Now the symptom was I go and I hit YouTube and then I hit

51:26
weather.com and I hit whatever. Now is it likely that all of those sites simultaneously get together and decide to reset me collectively? They're just picking on me and I'm having a bad day.

51:39
Some days it feels like that, but no, that's probably not the case.  They don't have an orchestrated reset that all of them are like, do it now guys.  So I'm like,  okay, resets coming from a lot of different quote unquote places. That's one symptom. Just going to remove my filter. Now I'm going to do is I'm going to come up here to the top. This is this packet that this is a sin, which is a TCP initial connection.

52:08
packet, right? It's at the first one in the PCAP. I'm right clicking this because right click filtering is way easier than typing it all out. I'm go conversation filter. I'm going to base the conversation on the TCP for tuple. What that means, both IPs and both ports. So now show me that TCP conversation. I can see here, so I've got just in my initial handshake for all the listeners.

52:36
SIN goes out and I immediately get a reset back. If I look at my delta time, this is the time between packets, one millisecond later I get a reset back. I try again, half a second later, SIN. Not even a millisecond, 233 microseconds later, I get a reset back.

52:57
Third time. The client tried one more time. It was probably like the TCP stack was like, look, try three times and then give it up. They're not listening. Try a third time and then boom, 44 milliseconds later, I get a TCP SYN act. The connection establishes the TLS client. Hello begins. And now this connection is off to the races and things work.

53:21
So why do I hit my head twice on two resets but then the third one works?

53:30
Now some people,  ooh, well, the firewall's blocking that port. Well, you guys know this,  a firewall rule is drop or not. Right, so if it was a  rule doing this, well then it would never work.

53:46
Are they single home, the one ISP?  You know, that's a good question. Yes, in this case.

53:54
I a theory. But keep going. No, let the packets prove it. Get out there. This is actually something that I like to do with other analysts. Sometimes if I'm just  stuck on something, I'll call a buddy and say, what do you see that I don't? We'll just start throwing ideas at it and let the packets prove or disprove what we are thinking. This sounds like a really good drinking game, by the way. uh Well, sometimes the answer comes to you after you've had a beer or two sometimes.

54:24
So, all  right, go ahead. Was there another  guess or thought or question? I mean, this is my whole career troubleshooting. I get a hunch and I say something and it's generally not correct. And then I just go back to watching the smart people. Welcome to my life.  It just seems weird. Like, so you're getting these resets uh geographically. They're coming from all over the place.  For some reason in my head, my head said that if you're single home, your ISP is under like some weird DDoS and

54:53
You're just experiencing weirdness because of it, but I'm sure that's not it That's where my head went like why is this stuff coming from all over the planet, right? Why would you get what I what I find out about this  is that?  If I remember this right, I'm kind of trying to look through it the first two resets were almost instantaneous. Hmm and then the third uh Sin it took what 40 milliseconds before it got the the sin act

55:22
Very good eye. In fact, my eye caught that too. Right? I don't know if that has anything to do with it. Hey, no, that's good. The IP never changed, right? Look at the target. Let's just say 201. 201, 201. That's the server's IP in this case. So we're talking to this 201 box. And it resets, resets, then boom, we work. And a big indicator was I had

55:50
1.3 milliseconds and then sub millisecond  resets coming back and then 44 milliseconds of network round trip time if I pinged that server it would have been 44 milliseconds Okay, that's my latency. Yeah, I was gonna say but why are you getting  denied? You know right at one millisecond All right. So now that here's the question This dude is not sending these resets, homie

56:19
He's not he's not even getting them. He's not even getting them for sins.  It's too far. Yeah, because you're saying because what you're saying is the latency between the server and the client in this situation was around the 40 ish milliseconds. How in the world could that server respond in one millisecond? Exactly, yeah, some sort of a local proxy or  firewall man in the middle kind of thing. Let's try and see if we could figure that out.

56:48
So, SYN goes out and a big thing that  I look for now, this is actually one of the troubleshooting exercises that really helped to form  my troubleshooting process. And now I pay a whole lot more attention to a beautiful field in the IP header, sonicwalled.ttl.

57:13
I'm going to add this as a column. Okay. And I mean, down here, just so you know, Sonic wall is in the ones that work too. Right. So it's a, it's the Mac, but yeah, let's come up here and take a, check out these TTLs. So basically time to live. So what do we know about time to live at the IP header? So that's the number of hops that I'll live before I, a router will drop me and send back an ICMP message saying, Hey, your time to live expired, right? Prevents things from living on forever in loops.

57:43
So when  a stack starts  a packet, this SYN is originating from this client. It's what we call a full count TTL. This is what that stack uses. Now by 128, you can actually  fingerprint or  start to fingerprint that operating system. Now know Windows for a long time used 128 to begin with. This is likely a Windows box I'm starting with. Okay, might not be.

58:12
possible it's not, hey, just kind of an interesting fun fact. Okay, so client is sending 128. I know that this is an un-routed packet because it hasn't gone down. Right, if I caught this at 127, then  I know that there is a router between the sender and my capture point.

58:34
If I captured it at 126, there's two routers between the sender and my capture point. 125, 124, right? So when I see these resets come back with 64, here's the possibilities. Either. Now the three big numbers this starts with is 64, 128, 255. Those are the big three bit boundary numbers that T-Tails begin with most of the time. So.

59:04
This tells me this is an un-routed packet because it's a 64. This reset in the IP header is showing TTL64. It's not a 63. This packet didn't begin one router away from me. Or 62, two routers away from me. Another possibility is this started at 128 as well and it decremented 64 times. But I look at the round trip time in a millisecond, no way.

59:30
So I know this SYN is going and an un-routed device on my network is actually sending this reset. Now  the MAC addresses matter. Now I can look at layer two.

59:47
Now come down to the  sender at layer two, the sonic wall is originating this reset. This was coming from the firewall.

01:00:00
Now let's do this. Okay. This reset's coming from a firewall. SYN, reset. SYN, reset. SYN, SYNAC. Now just for the people that are listening, the SYNAC that actually came back through from the firewall has the same exact source MAC. So it's not like one  sonic wall was resetting and the other was properly forwarding. It's coming from the same box.

01:00:26
And in that case, my time to live is 244. Now remember what I talked to you about. TTLs only start at 64, 128 or 255 and they don't go up. They only go down. So I know the true, the true box on the other end. If it's 244, I just do the math. This is 11 hops away from me. The true box is 11 hops away. Okay. So that's what works. This is with the rest of this actually works, right? So coming up here.

01:00:57
Now I can see, okay, this sonic wall is sending out resets and then letting stuff through. And you know what I find? It's interesting because if I remove my filter and this goes probably a little bit beyond what we're going to have time to do here in this conversation, what I find is that other connections were finishing between  the one that worked, between the SYN and the SYNAC, I found out that other connections that that client was having

01:01:26
out to the internet were actually finishing in that amount of time. So the SYNAC came back through and in between that, between the SYN and SYNAC, I had another connection that was either thinned or reset or it terminated. So now I'm going, wait a second, it's almost like the firewall has a limit, like a connection limit. Like Tim, you get 100 and

01:01:54
Dan and Andy, you guys get 100. Everybody gets 100 connections. Once you hit your ceiling, I'm resetting you. That's what it to feel like. So would you believe it? We go and go to the  Sonic wall. We take a look and dig through the configuration. We find connections, TCP connections permitted per user. And somebody set that to 100.  We call Sonic wall. I say, it's this box. It's this.

01:02:24
this much utilization, it's this version, it's this, this, that, that. You tell us what this should be." And they went, Chris, why on earth did someone put that to a hundred? I said, I don't know. And then I asked the client, like, who put this to a hundred? Oh, probably Jimbo. He's gone way, he's off to somewhere else. I bet he is. He left you with this burning dubster fire. It's actually kind of a fun prank. He set it to a hundred the day he left. Yeah, why I locked down your stuff,

01:02:53
So anyway, so I said, what do you recommend we use? He's like, oh, you're  comfortably able to do 300, if not 500 connections per user easily with that kind of load. We flipped it to 500, why not? More power and problem gone.  All new connections were now able to  be established. This is absolutely an example where the network was being blamed and, okay, I guess by all rights it was, but TCP is what led us to the symptom that brought the answer.

01:03:21
And that's definitely a weird scenario too, you know, like that's not like, you know, you don't just go, Oh, you know what it was? Or what I bet it is. I bet you it's their TCP connections on the firewall. I bet you it's our, like that's not something you're just going to think of. So a hundred percent. Well, you did, uh, you did leave out the fact that later on you found out it was actually DNS. Yeah.

01:03:49
Actually DNS. Well, I told you I told y'all there's two reasons I get called it's either weird or intermittent. Yeah. Yeah, so this is a weird one. This was an awesome example. I appreciate you showing us this. Sure, and I think for me what this speaks to is as a network engineer what I realize is I don't know Jack about the transport layer That's what I'm thinking this entire time you were showing me that I'm like I do not know So so through the window of

01:04:18
Wireshark.  What I've tried to do is just grab hold of the transport layer horns and just  force it to,  I'm gonna learn you, you know? And I think that's a, it's a missing layer in everything that we do. Cause I mean, if we think about it and I'll just, I'll stop sharing for just a minute. um If we think about it when a problem strikes, right? Like. uh

01:04:43
As network engineers, we go and we check our cables, we check the Wi-Fi, we check the switches, we check the routers. We might tap into the transport layer with a firewall, maybe just, ooh, the port open. But that's not really, really the full transport layer and everything that happens within that TCP stack. If flip that to the  application developers, yeah, they're in their coding. They know their stack a little bit that they're interacting with, but they know the application. Might know a little bit about the kernel that they're using. They don't really understand TCP either.

01:05:14
Network and developers, there's a big old gap with the transport layer. And that's why 99 % of the time I begin there. that's probably going to lead me, within minutes, I'll be able to figure out is it that way or that way. Is it the network? How's latency or packet loss? If there's no latency, no packet loss, everybody go home. No, it's not the network. All right, application guys, I'm seeing a 30 second delay off of this database. Why?

01:05:42
But for that reason, do you ever get calls from the application guys or do you only hear from the network engineers? Actually, I just did. just had an email when I was jumping on here with from a client that I'm working with. It's their developers and they're developing their own TCP stack. And they called up and they're like, this thing is not working with this game. And I'm like, let's check out the TCP stack and just the way that it was implemented. They didn't know a lot about the options that they were.

01:06:12
Configuring so they left off a few really important options that the server was expecting or wanting to work with So that impacted the way the kernel would accept that connection. Very cool. I'm nothing the network can do Why  because we're talking about the transport layer. Why would developers create their own TCP stack? I Just think that they hate life.  I mean  TCP works just fine like they're

01:06:40
They're taking  the standard and saying we want to do something special. No, yeah, they're doing something really unique with it. They're building their own interface with its specific tunnel and a little TCP wouldn't serve them. So they're like, we're to do our own thing. Yeah, in this case. So so let me ask you, oh when when you first got that problem and they sent you that packet capture, like how long did it take you to  figure that out? Because  usually when I start pulling out Wireshark and getting people to get captures,

01:07:10
Pressure's on  and I feel like I make more mistakes in that moment. uh You know, just trying to miss reading  a capture and that kind of thing. So I'm just curious, like how long did it take you to actually figure that out right there? I think that's a good question too, Dan, because I just basically, I call it the cooking show.  Yeah. You you ever watch a cooking show and you just go, oh, this is pre-done and boom, boom. Dang, it's already out of the oven. Look what we have. Let's all eat.  Like those are usually when I'm demonstrating.

01:07:39
something for you guys, you know, and we're whipping through it, you know, I see it like that. um This took  a few hours. Okay. But it took me going, hmm, is that real? Are all these servers really resetting me? No. Gotcha. Something's resetting me and it's not them. You know, so then I just started asking myself those questions and I do go for a bike ride um and I do go for a swim.

01:08:08
And you think about it sometimes.  my poor wife,  if I'm really on an analysis gig that's just burning my brains,  I'll wake up at 2.30 in the morning with the answer.  It's the congestion window and she's like, what are your dreams  about? I better be in one of those dreams. uh

01:08:33
No, it's like I'll wake up with it or,  you know, and I think that stress is real, Dan, what you said. It's usually high pressure, high lots on the line.  And I  think  I'm kind of an adrenaline junkie that way. I love that.  know, so  I like it, except for the fact that like when you have management sitting behind you, like, what is it? What is it? You know, and you're just like.

01:09:01
I don't know.  know?  Well,  and I use the fireman example, like don't put that poor fireman out there the first time with a hose on a burning building. You know, I mean, let them go out to the parking lot and figure out how this thing works.  Yeah, right. So that's why in my courses, you know, I have a course that's coming up in just a couple of weeks. Everyone should check it out on  September 19th and 20th. We go into a sandbox and we learn how to drive. learn.

01:09:29
how to set filters, how to  make these coloring rules I'm showing you guys. And here's the, honestly, the legit top 10 things that I do every single time I open up a PCAP. And that takes some of that fear away  and the overwhelmingness away. And I give them examples that help them to successfully find the answer. And then they go, oh, okay, this is fun. And once they say that, I know

01:09:57
that they're going to be okay because now it's like, all right, see how fun this was? Now here's another example. Now let's look at your network. Now you tell me within the next week what you found on your network and keep that excitement for it because it's fun stuff. Yeah, I'm going to start some packing captures now.  be like,  cool. taught me the TTO. need to, there's only a couple of them. Yeah. Nice.

01:10:25
So Chris, I wouldn't want to leave you without looking at another example where you disprove that the network  was the problem. So  can you walk us through another PCAP?  Another one?  I'd love to show you one more. How about another one where it was slow? It's the network thing? Sure. All right, problem  set up. I do have some cybersecurity examples, but I think for the networking audience, this is a better way  to go. Agreed. Right. I mean, there's like hacks and attacks and how an ARP poison works. And that's just fun stuff.

01:10:54
I think it's fun. anyway, okay. Problem. Buddy calls me. He's a guy that I've done a bunch of work with. he's become a friend. He's in Seattle and he's responsible for moving a database from one server to another in a data center. Servers are right next to each other. No latency at all. Gig attached, gig attached. Okay. Primary server, backup server. He's literally just sending 60 gigs over to the other side.

01:11:24
That's it. Copy. Go.

01:11:30
Hours later, it's still going.  As this database, so every night he'd back it up. As this database grew and grew and and grew and grew, it would actually make it to where it would take all night to copy it. Like literally it was still running and finishing when he came back the next morning. What are they blaming? Network.  It's the network, right?

01:11:55
Like, that shouldn't should be fiber. should be a 10 and 10 gig or a hundred gig or per, you know,  it's like, give me a break. You're on  take it to gig.  Can we just do some math here? Let's just make this very simple.  If it's 50 gigs. Okay. That's 400 bits,  400 giga bits, right? 50 gigabytes is 400 giga bits, right?  If I'm moving one gigabit per second.

01:12:25
It should take 400 seconds. This thing should take seven minutes to move.

01:12:35
Yeah. Not seven hours. If everything was optimally being used. Okay. That's our backdrop. Let me go ahead and. Did you say these are two servers in the same rack? Same rack. the same top rack switch and. Yep. Same ASIC even. Does it matter how they're moving it? Like is this like a vCenter whatever? I don't know the v stuff but. That's a good question. to play or no? At the time it was an application that was moving it.

01:13:03
You know what's funny? I don't remember the name of it. And this PCAP's a few years old, Okay, there's only two IPs in this whole packet capture. Okay, so one of the first things that I do is I like to go up to statistics and go to conversations. And I like to go over to TCP just to see what are my carriers here? What am I really looking at? And this is one of the questions that was asked earlier. How do you whittle down on problems quickly?

01:13:32
Well, here I can say I only have one IP conversation. And if I come over to TCP, I have two TCP conversations over that one IP conversation. So now look at my port numbers. If I look at the number of packets, I've got 5,000 on one of those connections and on the other, there's only 12. This 5,000 is the one I'm looking for.  Non-standard ports, 472 and then a high-side ephemeral port number. Okay, these are all notes to self. This is what I'm talking my brain through right now. Like, okay.

01:14:02
But come over here to close. All right. Again, what I'm gonna do is I'm just gonna start at the top and just scroll on down to the bottom. Now, what do you not see that we saw before?

01:14:17
The sins resets the knack. Yeah. Nice. TCP conversation. Good job. I don't see any TCP dot analysis dot flags.

01:14:32
These are resets. I see a couple of window updates. whatever. But I see,  I don't see any resets,  due packs,  black lines, red letters.  Okay. And if I look at the Delta time up here at the top, this thing is absolutely ripping. This is all sub millisecond here. So if I, I have a column on my copy of Wireshark called Delta and it shows me the number of, the amount of time between packets, right? And

01:15:00
I see 247 microseconds, 123, 123, 122, blah, blah, blah, you know, but I don't see, I mean, the biggest one is 200 microseconds, big deal. That's not causing hours of delay, right? So another thing my eye does, time to live, this is all on the same routed, remember I said about 128, right? So both sides are saying 128, so we're not going through a router. Look at our IPs, which I actually changed from the real ones, but.

01:15:29
101 is talking to 102, there's only two boxes. 101 is the sender and 102 and 1.2 is the receiver. I can take a look at the fully loaded packets are coming from one to two and then the empty acknowledgements are coming from two to one. Data is going, the sender is 1.1, the receiver is 1.2. So now all I gotta do is figure out which one of you two takes your sweet time and why.

01:15:57
People don't call me because things are too fast.  People call me because it's slow. And they also don't call me when it's a couple of milliseconds, unless they experience a couple of milliseconds 100,000 times. Death by latency. So let's go ahead and do this. I'm just going to start to just take a walk down my little thing here.  My eyes are looking for...

01:16:25
And I can do this faster. I'm just being realistic on how I first started out here. What my eyes are looking for is a jump in time that's bigger than a few microseconds. All right. Let me just come down here. Just do, do, do. And look at packet 140. Now that doesn't seem like a lot of time, but look at it in context. That's 19 milliseconds. That's surrounded by microseconds.

01:16:56
Let's just say that this decimal point was over three points. Okay, so now I'm looking at 19 full seconds. 20 seconds. Let's just say, let's just put it in that context. The rest of this would be sub seconds. This would be 123 milliseconds. 54 milliseconds. 20 seconds. 71 milliseconds. 52. So in context, this is way the heck slower than anything else around it. Okay.

01:17:25
Note to self, 1.2 took 19 or 20 milliseconds to reply with an empty ACK. Hmm.

01:17:36
All right. Keep going. Let's see if this was just a one off. Let ahead and scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll, scroll,

01:18:06
Scroll scroll scroll. Do some people don't like scrolling? I do. uh Just for reference, walking down the PCAP was Packethead's debut album. Yeah.  It all started right here. You guys heard it first.  Aren't you fortunate?  Okay, so here I got another one. Here's another delay, right? Dot two.  Empty AC. Okay.

01:18:35
So now I'm actually going to sort this delta. Let's just sort this and look at this. All these delays, well, most of them, 14, 14, 14, 14, 14, 14, 19, 19, 19, they're all coming from dot two to dot one. So this is not a long PCAP. This is only, this is actually less than a full second. If I sort this, put this back in order, if I scroll all the way to the bottom, this whole packet capture is a whopping 873 milliseconds.

01:19:05
But of that 873, I've got all these pauses. In fact, something I can do is use a sweet little graph. I'm going to pick a packet. This is one of the loaded ones. Going to go up here, statistics, going to come down here to TCP stream graphs,  going to go to TCP trace.  And here, this shows me the sequence numbers increasing over time. Now for the listeners,

01:19:33
I'm looking at a line that's very jagged. almost looks like a stair step. Looks like I'm going up and then  I have a flat line like a stair step and then it goes up and a stair step and then up and a stair step. Now my goal when I move in data from one TCP endpoint to another is I want this line to be as  steep and unbroken as possible because that represents full data being sent with no delays.

01:20:00
Just like if I was sending you, if I'm your neighbor and you ask me to help you fill your pool and you say, hey, give me the hose, turn on the hose. I want to, you want me to crank that water on and just let it flow. Right? You don't want me to be over at the spigot, turning it on and then off and then on and then off and then on and then off. You'd be like, really dude? That's something Dan would do to me. 100%. Dan would carry buckets over.

01:20:29
Let's keep that in mind because I'm going to use that analogy a little further here in a second. All right. So this is a stair step. I call this stepping. We have a persistent delay. Every single 20 milliseconds that you saw in the PCAP at 19 milliseconds is that flat line you see.  Now who was causing that? Was it the sender or the receiver? You remember? The receiver.  The receiver. Receiver was saying it delayed. Now the sender stopped.

01:20:58
Sending too.

01:21:02
Something made the sender stop. It waited for something and then it resumed.

01:21:10
But the receiver is the one that we were seeing that delay on. So let's do this. I'm going to close this. I'm going to look at something called, and this is one of the things that network engineers, everybody out there has to learn this because this is stuff we get blamed for 100 % of the time and it's not our fault. You can tell I'm a network guy. All right. I'm going to  look at.

01:21:37
this packet here. I'm going to go to the one just before it in this direction and I'm going to come down here to this little value just collapsing everything here and  reorienting my screen, TCP. I'm looking at the TCP header in the packet just before the point of pause. And I'm coming down here and I'm coming to this little value called window size. Now in this thread I didn't get the handshake so I didn't know if this had some

01:22:05
other values that were involved here. But I'm just going to tell you, take it from me, there was no window scaling here for anybody that knows TCP well enough to know window scaling. That's not involved. There was no window scaling on this connection. So this number that you see here is real. What this number means, this is window and it says 2299. What  the receiver is saying is, hey partner, I only have enough room in my receive bucket for 2299 bytes. That's all you can send me.

01:22:35
Sender sends one more full-size packet, then waits.

01:22:41
Looking at that delay, this window just cleared out. 65, 535. Now it's back up to the full size window. So it was 2299, and then it was 65, 535. Let's go ahead and add this guy as a column. All right. So I'm gonna come up to, let's just keep our eyes on this column now. Gonna go up a little further. Okay.

01:23:09
All right, so the 65, kind of pay, we don't have to pay attention. In fact, you know I'm gonna do? I'm just gonna take, I just wanna look at the ax. If I just dragged the 1.2 from the source column up to my display filter, dropped it and it set the filter for me. Oh, nice. Quick tip everybody, stop typing. That's what I do. Yeah, no, I did too. So check us out. 65535. Now what this means,

01:23:39
When this is 65535 at the beginning of this connection, what this means is, oh okay, Andy, you're sending me water and I'm drinking it as fast as you're sending it.

01:23:50
I'm keeping up as fast as you are going back to the neighbor in the pool analogy. You send me water and I've got, let's just say a bucket to catch it in. I catch it in the bucket and I dump it into my pool. I catch it in the bucket. I'm doing that so fast that I'm keeping up with the ingress water. Not a single bite dropped. But then this starts happening. You see this value here, this number 64915. This number starts to drop.

01:24:19
This represents the amount of space left over in the TCP receive buffer on the receiver. That's not an infinite number. When I have a TCP connection, I allocate resources to it and those resources are finite. There's only a certain amount of resource that one connection can have. The TCP receive window is one of the things that is not infinite. So that's basically our catch bucket for ingress data. Hopefully the application

01:24:48
empties that guy out as fast as data is coming in. Otherwise data pools and eventually that buffer fills and the sender has to stop sending. So let's see if that happened here. 65535, 64915. Look at this number, it's dropping. 59, 47, 38, 30, 21, 19, 2299. We get one more fall size MSS, maximum segment size, and then we wait.

01:25:19
We clear the buffer. The application goes,  scoops out all the stuff out of the buffer. And then we can go ahead and advertise a full, an empty buffer. Go ahead, keep sending.

01:25:33
The sender goes, sweet, let's send. Let's watch this number here. 35535. Okay, look, it's starting to drop again. Did you say this is not windowing? This is windowing. Oh yeah. This is a receive window that's congested. So in this case, it went down to 1459, which is one byte less than the MSS. If you notice the other size, it was 1460s. That's a common MSS.

01:26:01
because the TCP standard header is 20 bytes and the IP standard header is 20 bytes. That brings me to my MTU of 1500. The 1460 is basically the max payload on IPv4 TCP connection. This is one byte less. The sender can't even make use of it. Okay, we wait 14 milliseconds. We clear. Now we're back off to the races. So this is a window that keeps dropping and then I see the delay.

01:26:30
and then the window clears, then it drops,  and then I see the delay. Every time we go down,  it drops, we delay, we clear, we resume. So this is an example where the client was getting congested. It was not clearing  the TCP receive buffer out as fast as data was coming in.

01:26:55
So the sender felt like this. The sender was like, know, brrr, here's some data, stop, okay, I'll stop. Here's some more data, okay, I'll stop. Here's some more data, okay, I'll stop, because you're full. Like, here's some more data, right? Yeah. So it kept sending, stopping, sending, stopping, sending, stopping, sending, stopping, sending, stopping, these delays just got worse and worse and worse. Why would the receiver fill up? Was the server busy doing other stuff? Like it was just constrained resources?

01:27:24
Correct. guess we're getting there. oh yeah. It was a congested,  it was running out of resource on the server side. It had other processes going also. Hardware constraint.  On the box. Yeah. This, this receiver wasn't optimized very well as, as well. This application wasn't making good use of,  um, the, the PCP stack. wasn't  optimized for this level of ingress.  I'm kind of like, you know,

01:27:52
Maybe it was better at smaller files. Like if this is a website, you probably wouldn't notice it because we're not moving a ton of  like when you go and connect to a website, you're just pulling down a bunch of small things.  But when you're when you're putting pedal down and moving 60 gigs, that's a different story, right? You got to be optimized for that. But if this receiver wasn't doing anything else, if its only job that night was to receive that big file, but you think it would have been fine? Possibly.

01:28:22
Possibly so I showed him this problem and that was my question like what else what else is going on? He's like yeah I do have a couple of the processes, but Man this thing's slow. I was like yeah, okay, so let's do this. Let's tinker with the application So we did we got in there and we found that there was at the time an option to be able to multi-thread This application so instead of you notice that everything is going over one TCP connection So I was like well, why don't we try offloading some of that pressure?

01:28:51
into other TCP connections and see if we can basically lighten the pressure on this one poor connection. We did that and the problem went away. We didn't see, once we spread the workload over eight connections, things moved a lot faster.  In today's world, you probably wouldn't see,  you would see the TCP stack more optimally used. But one way that I used to be able to force a problem like this is I would open up like,

01:29:19
30 tabs on a browser and just have a bunch of stuff going and I could force a zero window. could force the client to be busy and the browser doing so much that it can't clear out all those TCP buffers. But here's an example of something that was the network was absolutely being blamed. The problem was 100 % at the TCP kernel level within the receiver. There's nothing the network could have done about it.

01:29:47
What caused it all to go over one single  connection? Is that the application? It was  the designer of the application.

01:29:58
So there you go. said you changed that. mean, did  the app owner have to change something on their end? There was an option within that app where we could tinker with that setting, thankfully.  right. But I mean, right there, how  many developers know how a TCP window works? Right?  When they go, oh, one thread. You know, mean, now I think there's a bit more awareness around it. But you know, this  type of issue, the reason why I like this PCAP is

01:30:28
There's absolutely nothing the network people could have done. Right, yeah.

01:30:34
eh All the fiber and upgrading and reducing latency to the nanosecond wouldn't have  changed a thing here.  that's where again, weird, slow. uh That's a corner case for packet capture. Beautiful. Cool. That was awesome.  so badly don't want to be like, and there we go.

01:31:03
the blame is off of the network,  like you just, you just like nailed it on that part.  Well,  I think there's something important out of that is not only  in this case, and I'm sure in many other cases, not only can you use packet captures and wire shark to  find that, the network isn't to blame, but you can also help by saying, I think you need to look here.

01:31:31
Let's take a look at this. It's not just a wash your hands of it and walk away ah You're able to  find that next I don't want to say finger to point but  the next uh Avenue to explore. Yeah, but I think that's also like extremely valuable for you know even even your company as a whole right because if you're able to point You know a developer or whoever owns that application in this use case like hey

01:32:01
Look here, you've got one TCP uh stream right here going like  use more sessions  or or you know if they have that option or if they know how to actually you know make that an option in their application. uh I think there's a lot of value in that I could see  other teams wanting to work with you if you're able to help them on even their own issues, right? Yeah, well, and that's why in in. uh

01:32:30
There's been quite a movement, especially in the last few years about,  we can't as network engineers, we can't just say it's not the network anymore. Right. Like we own the highways and byways and speeds and feeds. We have the capacity typically. the, we're the silo within the organization that can capture.  that if the, the burden's on us to read those captures, because I've worked with clients before. Oh yeah, we took a B cap and we sent it to the developer.

01:32:59
Boy did that hit his bit bucket. Yeah, what's he gonna do with that? Yeah  He's not gonna open the PCAP and go. Oh, there's my problem. It's a TCP window congestion issue When you open that PCAP the first thought I had just because it looked so different like well Why aren't there any sin or sin acts like why is it all acts and I didn't because I'm like, well, that's just gonna sound dumb Like this is it's my second PCAP ever and I'm not you know what I mean? I'm not trying to like

01:33:28
put my foot in my mouth, but it was one damn session, right? I mean, that was the damn problem.  you're calling Andy. We're going somewhere here,  Well, you how I know I love networking like sometimes I'm not sure. right. Like I love networking because honestly, I could do this for hours. Like this is fascinating. You know what mean? It's late. I've been up all day. I had a busy day, but like this is so cool.

01:33:54
I  could watch you teach PCAPs all night. I guess I'm in right field still.  Yeah, well, and it's also understanding too  why the field needed to move forward at the transport layer like this. What I just showed you was a fundamental driver toward why QUIC is now delivering so many. mean, right now this conversation is happening over QUIC, you guys,  unless we're blocking it. This is over UDP.

01:34:25
He's about to do a capture.  He's going to do a capture. I think so. Did you see his eyes just go, ooh. uh

01:34:35
So we're UDP guys. Yeah. Look at them. Oh wow. The same TCP. Oh, this is, this is, this is quick. In fact, is it actually it's I just lied. It's not quick. Is it, it's not four, four, three, but well quick and run over a separate port, but whatever application we're using for this stream is being delivered. The media part is being delivered over UDP, but you see there's a few TCPs that flash by. That's probably yes. That TCP.

01:35:03
Burst, that's probably control traffic. It's probably like a TCP check-in. Okay. And you'll see stun. Yeah. So anyway, but that's a driver of,  what I just shared is what we saw in the industry and TCP. It just, we started to outgrow it. It was created at a time when all you needed was two bytes of receive buffer.

01:35:32
Now, window scaling has changed. mean, now you can have a gig buffer if you really wanted to, but you know, TCP,  it's got a little stressed. So that's why QUIC is out there now and it just became a standard and you're starting to see more and more shift over to the smarts. It's not that UDP is a better carrier. It's just like, UDP, I want you to not think, just do.

01:35:56
Just send. just want to send and full send.  Let's let the guy above you do the thinking. Speaking dance language now.  Hey,  that's that's all. That's what I'm all about. Click that button.  Yeah, I don't need no stinky can. me for man. Yeah,  yeah, yeah. I got really excited.  I got one switch sitting behind my router at home and I looked at the 3550 and I looked it up and it I can configure a span port on. I'm like, oh my God.

01:36:26
Yeah, that's my action item or my takeaway, right?  I'm going to hook up a laptop. I got to figure out how to set up. I tried logging in while we were talking and I don't remember the damn password to my own switch, but  once I could get into the stupid thing, I'll set up a span port and try to hook up some Wireshark and then bother you for filters. Heck yeah, that'd be awesome. You have ton of YouTube content, right? That teaches all this stuff. I checked out your  channel. You just have an endless amount of...

01:36:56
content on there to teach people.  So what I did is I have a series on my YouTube channel where I basically break down the TCP handshake and look through all the options, like literally  one feature at a time. How does, how do um dupacs work? How do retransmissions work? What is a window? What's a, you know, just breaking down all these ideas and the congestion window, what the heck's that? Just

01:37:24
trying to simplify this thing. And I find I get a lot of comments where people are just like, it's just that little understanding that little piece, one piece at a time has helped to take away the mystery. There's a lot happening at the transport layer, man.

01:37:41
A lot. we get blamed for it.  Which is why I was surprised now.  I mean, I was thinking over here on the network side, like, look, we're down here at layer one, two, three a lot. Why would we be transport layer experts? I thought in cybersecurity. Like these dudes are writing  their custom little tools and like they're doing packet injections and you know, interrupting TTP streams. These guys must be rock stars at the transport layer.

01:38:11
So I was like, okay, this year for Defcon, was like, I'll teach a TCP for hackers course if you guys are interested. And Defcon jumped all over it. They were like, we need to know the transport layer. None of us know it. And was like, are you serious? And I went there and they're like, no, legit. And it  was well attended. We had a great time. We just took a cyber security slant to everything that you guys just saw. What does a DDoS botnet attack look like? What does malware look like?

01:38:40
How do you can how can you tell if you're getting man in the middle? How does a TLS cert look when it's been adjusted like all of that you can do from the wire and  I realized like that is a missing gap in this  it's not missing but  There's some  headway to make on the cyber security side Which is why you you're gonna start seeing a lot more of that in my channel nice I learned more about layer 4 in this past hour or so than I've Ten years in the industry every every place I was at

01:39:10
You know, I got a route to it. could ping it. can trace it. All the hops are good. They're, you know, no latency, like firewall problem. Go talk to them. Right. Like I just, cause I didn't manage the firewalls I couldn't. And so this has been  super informative for me and like, Oh my God, how many problems can you find and, know, identify right at the transport layer. I've never looked at it. I got a hard drive, more examples.  super informative, man. Thank you. Cool. Yeah.

01:39:39
I'm glad you got excited about it. I certainly am. I hope that comes through. 100%. 100%. Cool. Chris, this has been a super fun conversation. I know we've kind of danced around your YouTube channel and your social profiles. Let's throw them out there so people can find you. What's your Twitter? Do you have a blog? What's the YouTube channel? Shout it out. So my company is Packet Pioneer, and that's also my Twitter handle. So at Packet Pioneer. On YouTube.

01:40:07
At this point, think I have enough. I'm not, I don't want to sound like full of myself, whatever, but you should be able to type in Wireshark and you better find me within the first five videos. If you don't, then I'm doing something wrong. But yeah, if you just search Wireshark on YouTube, hopefully a few scans you'll get, or scrolls you'll get to me. And the channel, I post tips and tricks about Wireshark, just how to do all of this and make packet analysis simpler.

01:40:36
both for network performance, troubleshooting and cybersecurity. Apart from that, you're going to notice in a lot of my uh YouTube videos, you'll also see down in the description that there's a Udemy course that I wrote.  And on Udemy, if you just search Wireshark, you'll see it there. I teamed up with David Bombal to  go ahead and assemble and deliver a Wireshark for beginners course. So  10 bucks and you'll have that. um Also, I teach live courses. So every

01:41:06
I do about one every other month and I'll do a two day come one, come all it's remote over zoom. And, uh, we go through a full sandbox box of PCAPs and learn how to set filters and demystify this art of pack analysis. So all those ways you should be able to, to find me, my website packupioneer.com. Please reach out, please subscribe, check it out. And, um, yeah, let me know if there's any types of interesting protocols that you'd like to see. I just searched you on YouTube and, uh, your.

01:41:36
Wireshark master class is the number two hit. definitely above me. No, I'm kidding. That's great. No, that's cool. Yeah. Yeah. That master class. So what I did is I that's like a mini course. So it's a playlist. If you go, I think there's nine lessons there, but it's just the how to set up a profile, how to set filters, how to capture, how to do this on the command line. We haven't even talked about the command line.

01:42:05
We can shred all of this stuff on the command line.  If all we can do is SSH into a box and do dump cap and then we can go at it with Tshark as terminal shark and then just that's great for the cyber side. So you'll start to see some of that stuff. But yeah, check out that master class. It's all free. And also the try hack me room.  I have a recent video on my channel about  I just did a try hack me room walk through. So I walk you through each filter and how to set it.

01:42:35
Yeah,  and find me everybody. That'd be cool to meet you and  see what other kind of content you'd like to see.  Awesome. Chris, thank you so much for joining us tonight. This has been an awful lot of fun. ah We will get this episode out as soon as possible. And uh if you're listening, make sure you go to the YouTube channel, subscribe, check it out. Chris showed us some awesome stuff here. You don't want to miss it. uh Anything we should have asked you that we didn't ask you?

01:43:03
When's next time? uh I like it. I like it. There we go. Awesome. All right. We'll see you next time on another episode of the Art of Network Engineering podcast.

01:43:22
Hey y'all, this is Lexi.  If you vibe with what you heard us talking about today, we'd love for you to subscribe to our podcast in your favorite podcatcher.  Also, go ahead and hit that bell icon to make sure you're notified of all our future episodes right when they come out.  If you want to hear what we're talking about when we're not on the podcast, you can totally follow us on Twitter  and Instagram  at Art of NetEng.  That's Art  of N-E-T-E-N-G.

01:43:48
You can also find a bunch more info about us and the podcast at art of networkengineering.com.  Thanks for listening.