00:48
This is the Art of Network Engineering podcast.

00:59
In this podcast, we'll explore tools, technologies, and talented people. We aim to bring you information that will expand your skill sets and toolbox and share the stories of fellow network engineers. Welcome to the Art of Network Engineering podcast. My name is Andy Lapteff. I am at Andy Lapteff on Twitter. And if Twitter goes away tomorrow, you can find me at permitipandyandy.com. Join tonight is one of my favorite humans in networking.

01:29
And maybe beyond Lexi Cooper, how you doing Lex? Aw, that was so sweet, Andy. You were just shit talking me like three seconds ago. So I wasn't prepared for that. Damn. I'm doing good. I got my ass kicked today by knowledge. So I'm a little like mentally tired, but other than that, I mean, it's the good kind of tired, right? So you work in tech just another day, right? Yes, basically. Yeah. What about you, Andy? How was your day? You know.

01:58
I think it was good. I want to tell you that my son just entered his first science fair. He is in second grade and he wanted to learn about rockets. And I thought, oh, I can't wait to tell Lexi. So yeah, it's like, how do rockets work daddy? Right. I mean, I wanted to be an astronaut and that didn't work out. So there's hope for the next generation. Is there something specific he wants to learn about rockets or is he just like all over anything about rockets? So he was just like...

02:26
So he's interested in flight in general and then when we start talking about is like well, how do rockets work and we You know, we googled it and YouTube and just like anything. There's so many cool resources to learn and stuff. They explode until they go up. That's how I explain it. Yeah, so so what we did very briefly is there's a two liter soda bottle experiment you can do You know waters the fuel the compressed air is the you know

02:54
kinetic energy that's going to push out the water. We put some fins on it. We put a ballast on the top to make it a, and it was fantastic, man. It's so yeah, I couldn't wait to tell you that we're making rockets and launching them at our house. How far up did it go? There's like a 40 foot oak tree out front and I think it went like as high as the tree. So the first one we did vertical and then I was reading that you're supposed to do a 45 degree angle for like distance. So then we started going across the lawn, but it went further than I thought it would.

03:24
And then we did a bunch of experiments with more water, less water. Um, you know, there's a cork in there that until the pressure builds, what I found out was that the less water I used, the further it went, which I thought was kind of counterintuitive. I guess that way. Yeah. So if I was like smarter, I'd probably go into rocket science, but I know that's beyond me. So here we are. It's all a matter of just studying the right thing, Andy, as I have learned. The only other thing I want to mention before we get to our awesome guests tonight, um, is, uh, Tom Hollingsworth.

03:54
Of NFD fame and gestalt IT gave us an awesome shout out on his blog today about the Mike Bushong episode that came out not too long ago around leadership and just how it really resonated with him and how it kind of solidified some of the concepts that he knows and practices. But so I just, you know, it was just very kind. Tom didn't have to do that, right? Like you could like an episode and not put it out there publicly. So I just wanted to give Tom a shout out and thank him.

04:19
for that and being Mike's my boss, it's just a good thing. Like, hey Mike, you came on the show and Tom liked it. So please like me. You heard it here first, Mike. You heard that one? Yeah, I did. Yeah, great job on that. Oh, thanks man. Well, that was all Mike. So the voice that you're hearing from the ether is our guest tonight, Jason Ginter. Am I saying it right, Jason? You got it. Should I call you Jay or Jason? Either one works for me.

04:46
After this episode, we'll be calling him Jay. My first interview with Mike Bouchang, I called him Michael for an hour and he waited till the end of it to say, next time you can just call me Mike. Good place to start then. Yeah. So Jason, thanks so much for coming on the show. AJ Murray knows of you and you're involved with...

05:13
USNUA, which bought some time on our show. Where do you work? It's a pleasure to be here. You guys do a great job, so I'm really honored to be on.

05:37
I do a number of things. We do managed and professional services. but it pivoted into data center, deployments, designs, Also, as you mentioned, volunteer for USNUA,

06:06
And we can talk more about that, but it's basically network user groups that are inclusive under the umbrella. talk about networking, and have fun. And I'm also a volunteer at the Ohio Internet Exchange.

06:36
It's Ohio's largest internet exchange, nonprofit, volunteer-led. Yeah, so I keep myself pretty busy. I did not ever realize that there were nonprofit internet exchanges. I find that really interesting. What brought you to a nonprofit internet exchange? Why provide those services in that way?

07:05
Yeah, so I'm in Ohio, actually out of Cleveland, Ohio, in that if you do a traceroute between networks, back to Chicago or Washington, D.C., like tracerouting from Toledo back to Toledo,

07:32
Chicago to Toronto, then back to Cleveland. It's just this securitas route. Like, you know, it goes all over the, it bounces around all over the internet. So I actually got involved back in 2016 and, you know, due to life, had to take a break from it. I got involved again in 2020 because had a lot more time on my hands with COVID and all that. So, you know, really the goal is to keep traffic local, you know, in the state of Ohio.

08:01
us being the Ohio IX. across our fabric, content distribution networks, the guys that are serving up most of the things so our main pops in Columbus.

08:31
the members to peer with one another across that switching fabric. We also provide what are called route servers. So there's two types of interconnection on an IX. There's multilateral and bilateral. Multilateral is you just establish a BGP session to the route servers, and then everybody else who connects to those route servers, you can get to. Then bilateral peering is when they want to peer one-to-one. So, say, for example, we've got Amazon.

09:00
on the fabric. I mean, if you want to establish a peering relationship with the Amazon, you just send a note to their peering department and then you exchange routes and you get peering directly with their network. So really, the goal is to, again, keep that traffic local in Ohio, give Ohioans access, better access to content, internet content in general, faster access, because there's just, like I said, there's...

09:29
Most of the ways that you get there from here is super inefficient. That's so interesting how inefficient it is. I never realized that and I'm seeing, you know, some comments in the chat even about the Midwest being a bit of a mess when it comes to internet connectivity. I had no idea. The Wild West anyway, like it's so I'm not surprised to hear that it's that the routing is inefficient, right? Because the internet's crazy town as far as I know, but

09:56
Yeah, pretty much true. Dude, you talked for like three minutes. Well, so here's the question I have is, why do you want to keep traffic local to Ohio? I think you answered it in your explanation, which was speed. Is that why it's a problem to go somewhere else, hairpin, come back? Latency, yeah. It all comes down to higher latency. The longer the round trip time to get to that data and get it back to you, it impacts.

10:25
access to that. And, you know, there's also a great deal of reliance on these, you know, central points. So, you know, most of, most people peer through Equinix in Chicago and Equinix in Washington, D.C., or Ashburn, Virginia. So, you know, you're, there's just a high density there. And so when there's, there's outages or there's things that are affected in those regions, they could take out huge swaths of internet access. So,

10:52
That's another thing is providing additional redundancy by keeping, you know, allowing those networks to peer locally versus having to go all the way up to those central points. So yeah, but you're right, latency is a key factor and lowering the latency where you can improves performance to, you know, internet applications. That access is a paid for service, right?

11:21
ASNs. Where I'm going with this is my understanding, like when I worked at an ISP, I believe people either paid each other or had some kind of agreement about connecting ASs to each other. But because this is a nonprofit, I'm just curious the dynamic between like, are these free? Anybody just come in and here's my fiber and put me on the internet for free? There's lots of different models. You know, and the commercial IXs, obviously they...

11:50
They're out to make money so they're they're they're charging usually monthly for you know, getting access to the exchange There's infrastructure to maintain to right? Like I mean, how are you guys keeping it running without charging, right? We are we chart we charge an annual fee So for you know to connect it with a 10 gig ports a thousand bucks a year If you want to connect at a hundred gigs, it's twenty five hundred bucks a year and then that's right It's pretty inexpensive and then that's just really to keep the lights on, you know, that's that's to pay for you know

12:19
got Arista and Juniper switches we have to pay for support on. We've got, you know, we have to buy optics whenever new members join. We actually run our route servers on VMware hosts because the route servers are actually just Linux boxes running the Bird BGP daemon. So, you know, we have to keep all that stuff up and running, but the people that do all the work, they're all volunteers. So none of us get paid to do any of this stuff. It's just, you know.

12:49
For the for the love of doing it. So that's awesome. Can we take a quick step back? Yeah. Yeah, absolutely a noble cause so we got right into it, but I guess just to take a step back for someone who might not be familiar with this or have done it you explained it really well, but Why so why Why does somebody need to connect to your internet exchange, right? So am I a company?

13:17
that wants to advertise my routes from my data center out. And this is a connection into the internet or it's because I'm a local Ohio business and I want to like, you know, I'm brand new. I'm studying for my CCNA and you're like, Hey, there's, you know, this thing called the internet exchange. Okay. Well I've learned about BGP. I use the internet, but what, what, what, who is connecting at this exchange, just all the businesses, Microsoft, the Apple to level three is

13:45
It's just the physical cross connects of the internet in Ohio locally. Is that what this internet exchange actually is? Like very high level, super simple explanation. Yeah. So from a high level, in order to join the exchange, you need to have your own autonomous system number and your own address space direct from. Yeah. Our address space and yeah, you have your ASN and you're right. You're up and running. So the reason you have that is because you want to advertise your.

14:14
Address us out to the internet because you have services tied to them, right? Or you're a content provider. Akamai, I mean, they're delivering content to users. So we've got a mix of service providers, and a lot of them are regional.

14:43
ISPs like Wisps and regional fiber providers. And then we have some enterprises. And we have an AWS region here. So US East 2 is actually in Columbus.

15:12
Join the exchange directly peer with Amazon and your four milliseconds of latency away from Amazon's network So for that for them the enterprise users to you know reduce that latency make user performance on their network better You know that just shaving that off, you know You you it would probably be around 50 60 milliseconds Typically to you get to those types of services to bring it down to single digits be significant

15:40
Here's the last question I have because I don't want to beat it up too bad. But, um, at my last job, we had, you know, we had circuits with telcos. So our data centers would connect to, I don't know, pick one Verizon. And there was a circuit that connected to their internet router. We would peer with them and advertise our stuff out, you know, publicly. And then you could check the looking glasses to see that it got to other places. That's not an internet exchange, right?

16:07
That's just a connection to telco. So Verizon would then somewhere have a physical connection to an internet exchange in my city, right? Like you can peer with telco and peer up your BGP and advertise your prefixes without being in an internet exchange. Does that sound accurate? Yeah, you definitely can. And usually that's, you know, what's known as IP transit. Typically cost money and you're paying.

16:32
you're usually paying a service provider per megabit per second. and it's all you can eat. So from a cost perspective, on the exchange.

16:56
You're going to be able to connect to the other members But you're still going to need to get that IP transit to get to everything else So you'll still need to pop pay for IP from a from a service provider What we're really doing is facilitating interconnection between all of our members It's awesome. The fact that you guys volunteer is really amazing like it that's Is the whole organization volunteer like it's just yes everybody. Yeah. Wow. Yeah, there's no people from yeah Do you love tech do it for free, right? Yeah

17:25
Yeah, it's pretty fun. Cologix, actually we're hosted in a Cologix data center So it's just a number of technologists in the region make internet connectivity better. That's the Seattle Internet Exchange.

17:53
That's actually one of the oldest ones in the US. I wonder if they need volunteers, Lexi. What do you do as a volunteer, actually, at a non-profit IX? What could I, I guess, I could contribute to one of these places? We're all a little different.

18:23
So we'll get a cross-connect in, Most IXs use- in really seeing how the internet works because there's a whole community around peering,

18:53
Content networks that they specialize in this sort of thing So you really get to see how the internet works when you get involved in the peering community She sounds really dope. You just got my brain firing. So We have a lot of folks We've harped a lot over the years of it's so hard to get that first job you know you want to get in the networking and you get a certification and you do and and how do you that and

19:20
So where I'm going with this, you know where I'm going with this Lex, is if someone's studying for the CCNA, let's say, as an example, or they just got it and they're interviewing and people are blowing them off and you don't have any experience, I mean, would they be valid entrants for, should they look for their local, you know, internet exchanges and maybe see if they're looking for volunteers? Because it sounds like a phenomenal way to get hands-on. Would you take someone with a CCNA and experience in prod to help? I would. I mean, I certainly would. You know.

19:49
Typically the volunteers are part of the member organization, so they usually work for them or are somehow involved, but we're always looking for extra hands. Because it being volunteer, people come and go. Because people get excited about it for a little while, help out, and then they're on to the next thing. Because there's no real... I mean, like you said, it's a great learning experience, but it is kind of a revolving door as far as the people that are volunteering to assist on the technical committee.

20:19
but it is a great way to get experience and learn more about BGP. And you know what the other thing is, there's a lot of good mentors on these teams too. So, in addition to myself, there's a bunch of people that work for service providers, municipal service providers, internet service providers, like a nutritional carrier or telco that volunteer. And we're always looking to help scale up a new engineer.

20:46
teach them new things, teach them how BGP works and all that sort of stuff. Yeah, we're gonna have to scream that from the mountains when we release this, right, Alex? Because that's a new, that's one I haven't heard before. I haven't either, this is awesome. Yeah, this is a great opportunity for human networking as well. Shit, definitely is. Yeah, and that's another thing. I mean, we usually do mixers where we get together socially and it's a great way to meet the people in the community if you're interested in peering and that sort of thing.

21:15
Well, thanks for telling me about six. I am definitely going to check that out. Yeah. Sorry. Sorry for cussing. This will be a good segue. I have a stress ball that I squeeze all day and it's full of water and it just exploded all over the place. I apologize for the Ness word. Nandy's stress. Nah, we're good. We're going to keep going. But we got these for Christmas and my kids blew theirs both up on the first day. Mine took about a week. So...

21:40
If you're watching YouTube and you see this stress ball, don't get it, because they explode very easily. I just made a mess here. So pivot a little bit, but to a similar thing, volunteering. So the USNUA, Networking User Association, you gave a brief description. And the question I wrote down as you were talking is how did that start? I mean, were you a network engineer or you wanted to get in and there was no community around you? And

22:07
You got involved? having a beer, and talking about Vmug, and then Lug, which is the Linux user group. why don't we have a user group? We've been running.

22:33
The, um, Oh Nug, since about 2017. So just a little over five years old now. Um, Ohio was the first one. That's where you're from. That was the first one. Okay. So that's where I'm from. Gotcha. Oh, no, see it on your shirt is another experience, right? Cause like you can hear this on the podcast, but seeing it on the shirt, it's so good cause it's O in parentheses and then it's N U G. So it's Oh Nug. I love it. It's great. You got it. Yeah. And we wanted to make sure like it wasn't like.

23:02
Because there's another ONUG, so we kind of want to differentiate it. So we've got INUG and MINUG.

23:21
Not yet. Hopefully the show isn't the reason they come and get you. Yeah, hopefully not. So that's so cool. You know, we've heard that a couple of times over the years. I mean, just this very podcast was just created by some people who needed a study group and didn't really feel like they had the community necessarily locally to do it, not that we were local to each other, but kind of found each other online, started studying and.

23:48
It became a podcast, but I love those stories of people filling a gap, you know, starting, starting a community if they didn't have one, or maybe, you know, maybe you're a certain type of person and you don't have a group like you and like, you know what, we should have a whatever, you know, group. And that's, that's really cool. So that was 2017. Is that when you guys started off? Yeah, I think right around then. So our first meeting was like 15 people. It was just, you know, the people we knew and we.

24:14
hung out, had some beers, and I think that was just speakers. so no specific vendor pitches. you're not coming in to sell your solution to somebody. So we've had people present on like.

24:39
EVPN and VXLin and Multicast and just talk about the technology and answer any questions that people have. So you got a presentation and we usually cap it with a panel discussion, which is just like, we ask, what are the things that are challenging you in your day-to-day duties? What kind of things are you encountering and the frustrations you have?

25:07
usually just those two, just a presentation middle and end for network engineers to talk and our biggest one in Cleveland, like I think it was fall of 2019,

25:35
We've got people coming up from all over. It became fun to see the same faces at all these events when you'd be popping out to them. Do you have formal membership for the USNUA? Yeah, if you go to the usnuaa.com page, you can sign up for updates on the page.

26:03
So we'll notify you when events are coming up. And we've got, I think it's 13 chapters now. So, and I think we've got slated another four or five that we're adding here pretty soon in the first quarter. And we send out those updates, let you know when one's coming. Typically, like in the state of Ohio, we rotate through four cities, Cleveland, Columbus.

26:31
Cincinnati and Toledo, every quarter we'll do a different one. But they're open to anyone who wants to come. No cost, just show up, hang out with your buds and the beer and the food's free. Love it. Really? Very accessible. Sounds great. How many areas are you in? States? I don't know how you measure your... Are you guys all over the place? Like if I come to Ohio, can I just drop in? I know you said before the show there wasn't one in PA. I'm going to California next month. There's a one there.

27:01
Could I just hop into a meeting or how does it work? Yep, you just pop in if there's one available near you. We've got Ohio, Michigan, Indiana, Massachusetts, Connecticut, North Carolina, Colorado, Georgia, the list goes on, you can check it out on the site. That's awesome. Each group, each NUG is run by a local team.

27:29
So, you know, there'll be a local set of folks that run the events. We do all the enablements. So we print the shirts like these. We print the stickers. We send an event kit, which has all the questions to ask for the panel and all of that stuff. When we rotate that stuff, you know, every quarter we'll mix up those questions for the panel discussion. But we work with the local teams. We've got actually...

27:55
Starting this year, we've got a dedicated employee who's just going to run these events. Last year, we did 40 of them. It was, again, just working on the side as volunteers to make that happen. 40 events is a lot to pull off. This year, we're shooting for 60 events, but we'll have someone full-time dedicated to it.

28:17
Well, somebody in the chat asked what the reason was, if there was like a specific reason for, you know, network user group in the name versus network operators group. That is a great question. Is there any particular? Yeah, it was intentional. So network operator groups usually attract network operators. So those content providers, ISPs, and the folks that we talked about before,

28:46
they are usually the ones that show up for NOGs. So most of the folks that show up to our events are enterprise campus technologies, WAN technologies, guys and gals that are doing the day-to-day on those sorts of networks.

29:15
IPMPLS and a lot of NOGs like ShyNog and then there's NYNOG and a few others that, like I said, I find that the discussions and in very, very large networks. of most enterprise engineers and not really the problems

29:43
That's a great explanation. That makes a lot of sense. I can understand. I didn't know about network operators groups, so I'm learning a lot today. Thanks. Sure. I have a random question. I was just talking your, so I guess you're attending NFD next week? That's inaccurate? CFD, yeah. Or CFD rather? CFD 16. There it is. So I was just looking at your bio. Are you a drummer? I am.

30:12
Oh no. Oh no, Andy. I love the drums. Do you? Lexi got to watch me play little drums in Askew at our event last year. Yes, I have a video of Andy just going nuts on the drums at a guitar center we were at, trying to find the cable. We were trying to find this specific adapter for mic stuff and we're all in there. And then I look back and Andy's just going nuts on the drums back there. It's so funny.

30:41
Yeah, I need to find that video. Post that to Twitter. Tell me about music, Jason. Is it a hobby of yours, love of yours? You've just, is that, you know, it's something, we don't really have a format, right? But I'm just thinking as we're talking, like, you know, we talk about self care, right, on the show and how stressful the job can be. And a lot of what you hear is like, well, these are the things I do for myself to, you know, dispel some of that stress, right? Like, I don't know if your own call for the internet exchange, I don't know if it breaks. I mean, I know you have a day job too, so I'm sure that...

31:11
I feel like anybody in tech, there's. He is a busy guy. So you got a lot going on. I'm sure there's some stress in there. And I definitely want to get into your business and SD-WAN and all that fun stuff. But how do you balance it? What does self-care look like for you? How do you take care of yourself, your mental health in a stressful tech world? Yeah, drums is definitely an important part of it. I've been playing drums since I was just.

31:41
Probably just out of high school and a lot of that's not, I have no formal training. It's just playing in like punk rock bands and stuff. So, yeah. Awesome. You know, I still carry on, you know, I guess like I did when I was that age, I still liked to skateboard. I still played drums with, you know, my shithead buddies drink beer. That's definitely my self-care routine.

32:10
But yeah, I mean I have two kids one of them's not a kid anymore. Sorry. Oh, yeah Oh, yeah. Yeah. Yeah, I do. I have I have What I just get a Toma Imperial Star kit

32:26
My wife would kick me out of the house if I brought a drum kit in this house. I've been looking at the electronic ones because I think she'll, you know, I put the headphones on and she won't be like, what the hell? Andy, my partner has an electronic one and it is, it's loud, but it is not like drum kit-less. You know, like it's tolerable for sure. I didn't know he was a drummer. Yeah. Yeah. Yeah. He enjoys it a lot. Stock just went up. So every once in a while, yeah, every once in a while I come home and I hear like...

32:55
vague banging from the third floor and I know, okay, Rick's on the drums. So I just had a- It's an interesting sound, like that rub hitting the rubber. Yeah, it's like rubber, yeah, really hard hitting on rubber. I much prefer the analog. You hear those in guitar settings too. Right, I much prefer the real, you know, I want to feel the kick drum in my kidneys. I just flash back and then we'll get off of music, but I just flashed back. I was like-

33:23
I was 16 or 17 and me and my buddy, like I'm a kid of the 90s and Nirvana was happening and I was awful at guitar, but I could play really loud and play some power chords. So we were all about it and my buddy had an old Roger's drum kit that was huge and loud. One day I brought his kit into my little bedroom. It was literally a closet and set it up in the corner, moved the bed out into the hallway. So I start playing and I'm, you know.

33:51
Five minutes in, my stepdad comes, he wasn't home, he was at work, I guess he came home. I'm trying to imagine now I'm a parent, what it must have sounded like, to get out of his car on the street and go, what the hell is that noise? And then the closer you get to your house, you realize it's coming from your house, and then go inside and see your moronic, 16 year old stepson like. Andy.

34:15
Andy, it's easy for you to find out exactly how that feels. Just give one of your kids or both of them. There you go. Easy. Yeah. Okay. I have to. Okay. So, um.

34:28
I'm interested in SD-WAN, Lex. I don't know if there's anything that you wanna jump over or talk about. I do, I know almost nothing about SD-WAN, so I think we should definitely talk about that. I've seen it at jobs, I wasn't responsible for it. I don't think we've ever done- It's one of those buzzwords that I hear and I think, yes, that is a networking thing. Now- And it's just out there. We've never really, I think, brought it up as a topic on the show. I think somebody did a blog post once or twice about it, but-

34:57
I'll just level set for my own, you know, where I'm coming from is I'm a WAN network person, right? I don't know much about the LAN. I'm just learning fabric stuff now, but I was peering routers, you know, out multiple services to the world, you know, DMVPN, IPsec, you know, MPLS, GitVPN, like, you know, you name it, it didn't matter. And my services were how people got in and out.

35:25
Of our data centers and into our services. And the way where I'm going with this is I believe SD-WAN is a more intelligent way to route traffic based on a bunch of cool stuff, but what we did, all we could do in the old school non SD-WAN is, you know, we would set routing metrics. So like I'd have an A and a B side router at the edge and the A would be primary, say AT&T, the B would be primary Verizon. And we would control traffic in and out with route maps and redistribution and all kinds of fun tagging and stuff. So that.

35:54
You know, this one's primary, this one's secondary. I want some traffic here, some traffic there. I don't know if it'd be considered traffic engineering, but that was pretty much it. We had two lanes in and out, and I could tell it where I wanted traffic, and if A failed, it would go to B. That was pretty much it. I was gonna say, that sounds like traffic engineering to me. Well, right, but then I saw SD-WAN later. They brought SD-WAN in, and for an edge client,

36:23
So here's where I'm going, Jason, and I'll shut up and you're going to school us, which is going to be great. But so that was in the data center. But like, let's say for a bank that was utilizing our services, they'd have two routers, one primary BMPLS and all the traffic go over MPLS. The internet router and the internet circuit would sit fat dumb and happy forever because unless the MPLS went down, it would never ever use the internet. My understanding is in comes SD-WAN like, yo, pal.

36:52
How about we use that B side? So Jason, that's kind of my dumb WAN, Why do we need it? Yeah, I'll shut that. I worked for an internet service provider at the time,

37:22
That was our bread and butter. doing SIP over MPLS. I ran into a customer saying, It's probably just some gimmick that whatever,

37:51
The deeper I started to dive into it, I was like, holy shit, this is pretty cool. he's like, we got this, right?

38:09
doesn't really compete with our enterprise-grade MPLS. This is a problem, guys. there was a lot of, that's kind of like the inflection point they started coming out of the woodwork, I mean, all the big names we know today

38:39
And we started looking at those options, And I talked to that new company about, we think it's a gimmick. Hold on, can we pause there for a second? but the problem in technology over the years.

39:07
I remember, I was just reading something like how Blackberry just disappeared because they were so tied to like, well, this is the thing and everybody's going to want this and they're always, they're never going to type on it like a glass screen. Like they just refused. Oh, I know what it is. I'm reading this book, Think Again, and Yvonne Sharp recommended it to me. And it's amazing. And it's about questioning your assumptions and everything and how it's integral to evolving and continuing to grow and succeed. And that's just a perfect example. Like these people are like, ah.

39:35
You know, MPLS is great. People love spending six grand a month on a circuit. Like you shut up, right? And that was a mistake. Like they were wrong. Yeah, what they've stood by is like, oh, you're not gonna run voice over that SD-WAN. You need an MPLS network to run voice. You know, we're talking about the business critical applications of running them over MPLS because you know what? MPLS was a lot better than the alternative. You can carry quos over MPLS, right? Yeah, exactly. You can carry the class service or whatever it is. I mean, that was the whole thing with MPLS.

40:03
You got it. And they really didn't. And then a friend of mine who, we've worked together, he hired me as a dial-up tech support guy in 99. to different companies together.

40:29
this, if this SD-WAN thing is going to be so cool, let's just do it ourselves. So we, we founded a WAN Dynamics and we do, you know, we started off doing managed SD-WAN, I mean, it's a slow start. I mean, it took a little while for it to catch on, but you know, you found an SD-WAN company that's, that's WAN Dynamics, right? That's right. Yeah. We, we don't make the hardware, the software, we, we sell other people's solutions, but we, we provide a managed service around it. Gotcha. And that was right when it was like lighting up like 2015 ish.

40:59
Yep, you got it. What made you, I mean it's just because you're an engineer But what convinced you when the MPLS gods were like, What grabbed you about it?

41:23
That's the platform we work with the most. and I think I had a 4G circuit. and it came right back up. I've never seen that before.

41:44
And then that just kind of blew my mind when I saw that. The other thing that I was able to do is, you could choose from. connection as a backup.

42:14
internet, just dump it out here and it can determine between the two. punt that out the internet interface, send that down the MPLS network. I couldn't say, this internet traffic here, send over here,

42:43
If you're connected to AT&T, go that way. being able to correct problems on the fly. and jitter buffering.

43:13
It could be over a single path. from the media, It'll start sending two copies across the same link if the packet loss is too bad, you could overcome it.

43:42
Go ahead, Alexar. but is it the same kind of FEC errors that you'll see Or is that something different? the fact that I'm talking about in the SD-WAN context lives.

44:10
What I wanted to ask, isn't SD-WAN constantly testing all available paths and then you can make forwarding decisions based on... So if I'm under these certain parameters, like I need to be under 2% packet loss, under 20 milliseconds latency, no jitter, whatever, send all my stuff here, as soon as that crosses a threshold, boom, go the other way. Isn't it constantly measuring? Which is amazing to me.

44:38
It's constantly monitoring the health of your WAN, right? you set and parameters, right? so over that MPLS network, it'll build like a tunnel mesh. So while it's tunneling, it's measuring.

45:04
and all that stuff. So if it sees that, oh shit, the MPLS path's looking not so good, let's redirect, go over a tunnel over the internet because that's better. You can also do like 4G backup where it will have that just sitting idle and then when all the paths are too bad or they've gone down, let's take the 4G. But you can say, okay, during that time, because you may have meter to access on that 4G, you might wanna say, look, don't do any.

45:33
I don't want any streaming services, social media, all the internet stuff. Let's just kill that, just black hole that for now, but make sure the business critical apps can still stay up and running so you don't get clobbered with a huge bill. Did people... I'm sorry, Lex, go ahead. One of the promises I thought of SD-WAN was you can get rid of your expensive MPLS circuits. Did that? So I guess I should ask first, was that a claim at some point, right? It was.

46:02
But you can't always. Did that happen? Sometimes. Sometimes you can. So I've talked to some companies who are like, oh yeah, we were able to scale way back. We didn't get rid of MPLS, but we went way less with what we had before. Cause you really can't, you can't get rid of your super expensive, dedicated MPLS circuit, go with two different cable company ISP internet boxes, and SD-WAN will make it just as good. Right? Like that's an overreach. Or is it not? Like, do you need MPLS though?

46:31
Kind of back to the latency thing, Again, those paths that are not as efficient now not always, but typically MPLS is just The internet, you might be looking great today, So that's one of the challenges.

47:00
You have had customers switch from MPLS networks So they were using Citrix remote desktop, It's a little slower. but you are sacrificing performance a little bit.

47:22
We do talk customers through it. Or if you're a call center and you live and die by voice applications, you might want to stick with that MPLS because the lower latency is definitely preferred on voice applications. Sorry, Alex, go ahead.

47:50
you know, the benefits of it. Optimization of those routes, Where are those decisions actually being made? How does that work for your typical, There's usually a, you know,

48:17
you log into to manage all of your policy. So all of those things we talked about as far as application rules and here's how you should do your failover and all of those things, you've got to manage in this controller that pushes that policy out to those devices. But usually the devices are making the call about, okay, which path should I send the traffic out based on the policy that I've inherited from the controller? And then there's, you know.

48:42
Every SD-WAN has a little bit of a different flavor to it. You know, there is no real, it's not as though you're talking about like BGP, which is a well-defined standard or OSPF. You know, these things that are standardized, that are spelled out and you know, well-known, well-understood, everyone kind of knows what they are. Each SD-WAN platform's a little bit different in the components they use, the problems they solve. So.

49:08
Like I said, we work with Bellow Cloud a lot, but Cisco's got their own idea on what SD-WAN looks like. There's other partners that we work with like Bigleaf. Bigleaf, they do an SD-WAN, but it's just public side. So they actually tunnel public traffic, public IP addresses across the internet. And the way the problem they solve is, let's say you wanna use multiple circuits at a site.

49:33
and you want your inbound services to have the same IP address, Bigleaf solves that problem in an interesting way, But as far as the components you see the most, that are making forwarding decisions based on that policy.

50:02
architecture more than it is. Yeah. that's a component. they use different tunneling technologies.

50:31
for their platform. And actually the tunneling runs over to UDP 2426 They wrote it from the ground up. They have their own proprietary tunneling. measuring the paths and also doing some of the things that

50:59
fix up things on the fly, you have to bolt that on to whatever tunneling tech you're using. Interesting. Do you need appliances at both ends? I know that you need appliances in your data center because I've seen that. The SD-WAN solutions I've seen. So if you have an edge client with two routers, MPLS-A side, internet backup B side, do they have appliances at the edge and where are they in the path?

51:29
I guess they're behind the routers? It's not all built in. And they can do BGP, they can do OSPF, Is that typical?

51:59
you'll have both and then the MPLS router can take over. If they like, they'll do VRRP, you know, it's a dynamic gateway protocol to fail over to like the MPLS router, for example, if the SD-WAN goes down, there's a lot of different designs, but usually they replace the router. And that's a great question. But bookending is required because that path measurement is really done in band in the tunneling, at least, you know, a lot of the examples I'm using is Velo.

52:28
Some of the other platforms might work a little differently than Velo does. Velo now VMWare SD-WAN. But it is bookended. On the internet, they have these things called gateways that they talk to. So even for talking over the internet, they're doing that measurement, checking things out, and always measuring each and every path. The internet-based paths, the MPLS-based paths, the...

52:56
And if you want to, if you have like an all you can plan on on like four or five G, you can have them run all day on that too and use it just as it would any other circuit. Just got a great question in the chat. Any cons of SD-WAN, any downside? What's the, why wouldn't everybody go to SD-WAN? Yeah, the one that we see the most is people just assuming we're gonna get off MPLS, we're gonna save a ton of money, it's gonna work the same, not always.

53:21
Especially if you use all internet circuits, that latency can be a huge challenge. A lot of- So do people not know that dedicated private circuits are more reliable and better than the wild west of the internet? Like they think that? Are these executives who know nothing about tech? Like we're just gonna save money and Q3 is gonna look great. And then they're like, oh God, why is everything slow? What happened?

53:45
Well, I think these executives hear from their other executive buddies, But their use case might not be the same. Yeah, right? That's definitely one of the bigger things that we encounter. I had another one, too.

54:14
Security. Because most of these, what I would call peer play SD-WAN platforms, you've got to integrate them with something else on the security side. Meaning a firewall?

54:43
routing or whatever you call it. transparent or bridge mode behind the SD-WAN appliance. For clients that haven't really had to do that, Doing those integrations can get complex

55:09
What I saw, and I just want to put it out there is another, what I would think is a benefit of SD-WAN. So I was at a place and we were rolling out SD-WAN, we meaning people smarter than me. I was watching them. But I was in a lot of the calls and planning and posts, whatever. And there was a lot of we're going to save money conversation up front, right? Because we had a ton of MPLS circuits. I know that we paired back on MPLS, but I believe...

55:37
because of the cost of the SD-WAN solution and whatever else might come with it, it kind of became a break even. But because of all the benefits that SD-WAN gave, we were providing clients seamless failover and load balancing out of it, like you were saying with the smart routing. So now the clients are using internet circuits they never used before and using the router just sitting there dumb and happy. So like...

56:05
It was happy clients. It was better fail over for them. It was more uptime for clients. So even though I think the cost was a wash in the end, the benefits SD-WAN brought made it a win. I was like, you know what? This is great. Now we don't have, you know, clients pissed off because our damn internet failover breaks 30% of the time for reasons, right? Like, so yeah, just, just another feather in the cap of, you know, if you think you're going to save a ton of money and go to MPLS.

56:34
It might be a wash in the end, from the intelligence of the system, Completely agree with you. when we talked through it, some of them, that wasn't the case. But almost always, performance was better.

57:02
Uptime was better. It doesn't get much better than that, right? Yeah, that's what we've been trying to do for 50 years. Cool. And I mean, is that still keeping you busy, SD-WAN?

57:30
SD-WAN still keeps us busy. which is integration of a few other technologies. Did you say SASE? I'm glad I'm not the only one who likes that name. So is that what...

57:56
SASE means, I like to define things very simply because I get lost easily with stuff. Yeah, people pronounce it S-A-S-E, don't they? So SASE software is a service, right? Yeah. SASE is S-A-S-E, secure access services edge, which... That's where I got lost, okay. Is that SASE? Is that something different? SASE could be a component of it, I guess. You could have a SASE stack that's part of it. But SASE ends up being really the...

58:25
the securing of the SD-WAN network. You're adding usually, I mean, really they're just firewalls Securing SD-WAN, it's tunnels. It's probably a dumb thing to say, but like, Are they just GRE?

58:53
happen with COVID. So you want to build this perimeter that all users can connect to if they're in the office, whether they're remote, There's also some components like

59:19
Zero trust access, so authenticating users at the edge So it's just really this, that those IT executives we talked about earlier. That's what it is. So we're doing more of that now.

59:45
It is. And they had a few other options out there. I wish I could recall them because I'm so glad that they didn't end up calling it some of these other ones. Like I think they were like Slappy or something like that. Oh, my God. Please, please can we call something Slappy? I don't care what it is. Please. Network engineers. I have to get on it. To be my new name. Yo, Slappy.

01:00:12
Thank you for joining us. Lappy laptop. So you just got me thinking with like a, you know, work from home person and Sassy, and I still don't think I understand Sassy, but what made me think of this question was, do you need two paths, two network paths for SD-WAN? So I'm working from home. I only have one internet connection. In my mind, there is no SD-WAN because there's only one road in and out of my house. Is that accurate?

01:00:42
No, it actually does improve performance like the forward error correction and jitter buffering. maybe you got a handset, maybe you got a soft client There you go, yeah, you got it. by doing the forward error correction,

01:01:09
Really, I find the most successful new technologies but it also does something about it. It takes an internet circuit and makes it business class as far as improving performance on voice. Then the thing they do is TCP optimization.

01:01:39
sending negative acknowledgments if it detects that there's been packets dropped in the path. So just those little fix-ups that they do over crappy DSL or cable connections that people have at home, a lot of that. That's interesting. It can actually improve the user performance even if you only have a cable modem at one house but you put an SD-WAN appliance in there. It'll actually improve your performance and experience. Yep.

01:02:07
Yeah, and you're starting to see that SD-WAN tech having a VPN client that does board error correction, that crappy DSL line or cable line, and make that Zoom call better, or Teams call, or whatever. That is very encouraging.

01:02:35
We need to get it for AJ because his internet sucks. Does it? I didn't realize that. Awful. I don't know if it's all of Vermont, but wherever he lives. It might just be Vermont, you know, the whole state. I know a guy might be able to help him. It's awesome if something. Yeah. We're getting close to the end here. Go ahead, Lex, did you have something? No, I was gonna say, you know, we are getting close to the end. What should we have asked you? What did we forget, right? Like, yeah.

01:03:05
I don't know. I don't know, man. I think you guys covered it all. We quizzed you enough on SD-WAN. You absolutely did. Yeah, it's been fun. Thanks for having me. Man, thanks for coming on. I really enjoyed the conversation. I learned a lot. I didn't know. And I really think it's cool that you, again, just to circle back to the USNUA for a second, I think it's so cool that you yearned for a community or some, you know,

01:03:35
I don't know, camaraderie locally and created that and then have been able to see it grow. Getting network engineers together in person is kind of a big deal. So I'm impressed that you're able to do that. Like 13 plus chapters much. So he's giving out free beer, Lex. Yeah. All right. Well, that is strategic. All right. Strategic.

01:04:01
It's not that complicated. Get them drunk and they'll come. Yeah. I mean, it's been awesome. We never expected to get this big when we started O-NUG, but during COVID, we kind of retooled it while we had all the time to really look at it and see if we could do something bigger. So I would encourage everyone to go take a look for their local chapter. I'm actually...

01:04:29
JayGinnard at USNUA.com if you have any questions for me or anyone on the team. If you want to start your own chapter, we're looking for folks to start them all across all 50 states. Reach out to us if you're interested. So I was thinking if I'm ever in a town with one, like if I'm traveling for work or anything, I'll see if there's a chapter and I'd love to drop in and meet some folks. We had a meet up last year in Asheville just through the show.

01:04:55
It was one of the most satisfying experiences I've had in my career of just, you know, I mean, it was post COVID. So like, I was thrilled to be anywhere doing anything, but, um, it was just so nice to be with people that have done the job or were doing the job and could commiserate and, you know, we've, we've all been in that together. Yeah. It was just, so, so I really love that. And I'm hoping I can drop in on a, um, on a USNUA, uh, meeting. There's a USNUA, uh, where can we.

01:05:23
Take a look again, usnuay.com. Got it. Yep, check it out. Feel free to sign up for updates. Uh-oh. We do not sell your information. I was like. Oh, he's back. We lost you for a second. We lost Andy. He better not be running us the way. Can you hear us, Jason? I can hear you. All right, Jason, where can people find you out on the internets? Bits and Flight on Twitter. Bits and Flight at mstdn.social on Mastodon.

01:05:53
LinkedIn, another good place to find me. Check out USNUA.com and hopefully you can see some friendly faces at the next ONUG event. Awesome. Thanks so much for coming on, Jason. I'll give you a couple of our links here. You can check out our Patreon at patreon.com forward slash art of net eng, if you would like to support the show in that way. We are on the Twitter's at art of net eng. Lexario Mastodon somewhere.

01:06:19
We are on Macedon, Art of NetEng at infosec.exchange. Cool. Instagram Art of NetEng, Facebook Art of NetEng, LinkedIn Art of NetEng, our website, artofnetworkengineering.com. We have some great blogs that have been coming out recently. Tim's been killing it with the COMTIA cloud certification that he got and a lot of great information there. And we have a merch store, artofneteng.com forward slash store. And as always,

01:06:49
You can, if you're looking for a community, some support, we are on discord at art of net ends.com forge slash I A A T J that stands for it's all about the journey. We have about 5,400 members in there. Study groups, resume reviews, just how you doing? Um, it's a great community. If you're, uh, you know, looking to connect with folks, um, as always, thanks for listening, Lex always great to see you and we will see you next time on the art of network engineering.

01:07:27
Hey y'all, this is Lexi. If you vibe with what you heard us talking about today, we'd love for you to subscribe to our podcast in your favorite pod capture. Also, go ahead and hit that bell icon to make sure you're notified of all our future episodes right when they come out. If you wanna hear what we're talking about when we're not on the podcast, you can totally follow us on Twitter and Instagram at Art of NetEng. That's Art of N-E-T-E-N-G.

01:07:53
You can also find a bunch more info about us and the podcast at art of network engineering.com. Thanks for listening.