Speaker 1: 0:00

This is the Art of Network Engineering podcast.

Speaker 2: 0:11

In this podcast, we explore tools, technologies and talented people. We aim to bring you information that will expand your skill sets and toolbox and share the stories of fellow network engineers.

Speaker 1: 0:23

Welcome to the Art of Network Engineering podcast. My name is Andy Laptev and I am joined by some monster brains, some super smart people, some IPv6 luminaries in the field, and we are going to get into the IPv6 discussion. What's taken so long for adoption? Why is it so important? What are the problems? Why do traditional network operators, like I used to be because now I'm new and evolved why do they have such a hard time with v6? So let's go around the roundtable here. See who we have. Joining us tonight is Kevin Myers. Kevin, what's up, man? How you doing.

Speaker 3: 1:00

Hey, Andy, Thanks for having me on. Yeah, I've worked in network engineering for about 20 years and service provider and enterprise and I guess in the last decade I've done a lot more work on v6, on building it, testing it. Ed and I have gone rounds with the IETF and lots of fun things there together and yeah, so I'm definitely a v6 nerd, maybe even a v6 zealot, like you said.

Speaker 1: 1:25

I mean that with all the love and respect in the world. Well, cool. Thanks for coming on, kev. I appreciate it. We're going to get into it shortly here. Ed Ed Horley. How you doing, sir? Who are you? What do you do? Where are you from?

Speaker 4: 1:35

Oh, wow, what do I do? I do v6 work remarkably. Um, yeah and uh, let's see what else do I do. I'm still the co-chair of the california v6 task force, so still involved with doing work there. I blog on howfunkycom occasionally, write a few things down very, very occasionally, and then, uh, let's see what else. Uh, I host another uh podcast. You might have heard of it, it's IPv6 Buzz Podcast over on the Packer Pushers. I don't know if I'm allowed to mention that on the show. Absolutely, absolutely, but yeah.

Speaker 1: 2:13

Ed and I met at NFD35 and Ed made v6 sound interesting and I'm like I got to have this guy on man. We got to get the good word out. Ed, thanks for coming on. I really appreciate it. Yeah, absolutely, chris. Chris Miles from Cables to Cloud fame. Tell us about yourself, buddy, how you doing. Thanks for hopping on.

Speaker 2: 2:30

Yeah, of course I definitely. I don't think I'd call myself an IPv6 expert, definitely according to this panel that we have today, but, yeah, glad to be here, chris Miles. Like Andy said, I also co-host a podcast called Cables to Clouds, so network engineering background for about 15 years and I moved to the cloud just a couple years ago and excited to talk about v6 today. It should be fun and there will be no hot takes, I'm sure.

Speaker 1: 3:00

Chris and I could be the v4 guys, and then Ed and Kevin will be the v6 guys and we'll just have a battle royal. Remember the old WWF battle royals cage matches.

Speaker 3: 3:08

We're going to duke it out. See who wins. I'll tag team Ed in on you. I'm not afraid to do that. I'm kidding.

Speaker 1: 3:25

The folks that are advocates of v6 are very passionate about v6, right? Every once in a while I go on social media and I'm like, ah, v4 and that forever right. And oh my God, the v6 people get so upset and so crazy and I mean, I'm partly kidding, but I'm also coming from a decade of experience in huge production networks where everything's NATed and you know they came up with CIDR and NAT when they started. I learned this actually listening to IPv6 Buzz Ed. I hopped on your. I really like your basics. I think you have an IPv6 basics series.

Speaker 3: 3:54

Yeah.

Speaker 1: 3:54

And I started listening to the first one and I didn't realize that, as they were running out of v4 address space. I guess that's when they came up with, you know, with CIDR notation, where you could break up subnets and then NAD, and that was supposed to be a temporary workaround. That was in the 90s, I think. So here we are 20 years later, give or take and I looked earlier. You guys can correct me if I'm wrong. I think we're around 47% global v6 adoption. Does that sound kind of accurate?

Speaker 4: 4:18

Yeah, that sounds in the right ballpark.

Speaker 4: 4:20

Jeff Houston just released a new blog about the IPv6 transition, so I don't know if you guys have had a chance to.

Speaker 4: 4:31

Well, it just came out today so maybe not a chance to see it, but I'll send it over handy so you can include it in the show notes. But he talks about exactly this, which is sort of the challenges around adoption and what's going on and what potentially could be issues around why the transition is taking as long as it's, as it's taken both now and then projection into the future of like well, if you assume what was true yesterday, it was more than likely going to be true tomorrow. You know why is it looking the way it is? So it's a good read. Um, and definitely you know jeff's jeff's uh, while still active in the itf and working on v6, uh, stuff is is very-driven in terms of looking and analyzing problems. So he points out some good gaps and maybe some learned lessons about why it's floundering the way it is, and really it comes down to what we thought were Band-Aids are really much more permanent fixes than we ever envisioned them to be.

Speaker 1: 5:22

Which is true with a lot of things. With the running chest networking, there's nothing like a. You know, there isn't a temporary fix, right, yeah, there's no, well, hey, we're.

Speaker 4: 5:29

You know, those of us in the US, we're still stuck on. You know freedom units. So you know, hey, the rest of the world moved on. We're still here. What us and two other countries.

Speaker 4: 5:41

So I think you're always going to have stalwarts that are going to be holding out in regards to older standards and what's happening there. But I think the trajectory and many initiatives that are going on right now, both in the US, but also in China and Brazil and in India, are really changing global footprint behavior around IPv6 in a way that a lot of organizations just can't ignore, including China behavior around IPv6 in a way that a lot of organizations just can't ignore, including China. So it's like, hey, if major countries with very large populations are making strategic moves around IPv6, do you really want to be the guy that says, hey, no, we shouldn't do that because we don't have anyone in that market, right? Or?

Speaker 1: 6:19

we don't interact with people Because they're running out of v4 space and NAT isn't working.

Speaker 4: 6:25

So partially, I mean. It really comes down to a couple different things. That's probably not true in the US at all. We got plenty of V4 because we hogged it all in the beginning. Let's be honest. There's the advantage of being first mover around all of that stuff and inventing the internet. Well, hey, guess what you get first reaps around all of that sort of stuff, reaps around all of that sort of stuff.

Speaker 4: 6:43

If you're in the Asia-Pacific region, you've got a much more constrained address space to deal with, given your populations, which clearly their populations are overwhelmingly much larger in terms of anything that the US has. So it's like is it an unfair distribution? Absolutely, and they're going to use what's available to them in order to be able to perform and do business, and one of the things you need to do is be able to connect and talk with other folks on a network, and you've got V4, v6 to choose from to do that. If you don't have V4 to use, guess what? You're going to use V6. That's just a practical reality.

Speaker 4: 7:18

Now, when mandates come down around innovation and around moving your economies forward, and you don't want to have the baggage that comes with v4, and there is baggage that comes with v4. We don't like to talk about it very often, but we can talk about it here. Then, yeah, you're going to see innovation that happens in that particular space, because you're no longer constrained in the same way that you were with v4. And I think this is a lost opportunity for many network engineers who are looking at it and saying, like, why do I want to learn something that new? Well, you've got a couple of reasons why. Number one you don't seem to be too shy about learning. Network automation or zero trust or SD-WAN or VXLAN, evpn, right, like those are all things that naturally fit. This should be no different for you from a skillset basis.

Speaker 1: 8:05

Hey, ed, quick interjection there, because V6, I had this thought earlier today kind of reminds me of the automation journey, something that's also been around for decades that was going to take the world by storm. That's somewhere around 30 to 40% of networks, and of those it's very minimal stuff, like they're not fully automated automated. So I don't know if there's a parallel there, but that thought hit me today and now they're good, there is resistance don't tell chris grunman that he has a business model generated right, I mean like oh, automation let's create a business, right?

Speaker 1: 8:38

so yeah, yeah, I mean automation's.

Speaker 3: 8:40

Automation's great and it's been around for a while. But you look, I mean we're talking about the fundamental protocol that everything else in the world of technology has to use. Should you use automation? Yes, absolutely. Shouldn't you embrace automation? But you don't have to. There's plenty of people that don't have to. But when you come to the fundamental protocol, that glues every other IT bit back together. You don't really have a choice there. You have to have that. And moving on to the new protocol is something that we've needed to do for a long time and I'm sure we can talk ad nauseum about the why, and we will here shortly.

Speaker 3: 9:11

But to talk about something that Ed mentioned, in Asia they're definitely hurting for address space and if you look around Asia, one of the things you'll find that is starting to happen here are the performance gaps that are there. It's like Japan. It's very well known in Japan that you're not going to see the same speeds on a V4 address because it's mostly going through a CGNAT gateway, which is congested, that you do on V6. People, it's the thing. You go look at the internet in Japan and people will post their IPv6 speed tests and they'll post their IPv4 speed tests and the CGNNet gateways are so congested in Japan and other Asian countries have similar problems that we enjoy. Like Ed said, we've got a wealth of IP addresses a lot of people have public, even the ones that do have to go through CGNet. We're pretty fortunate that most of the US carriers have been able to keep pace with the CGNet gateways.

Speaker 3: 10:01

But we won't and I'll talk about in the ISP space that I've spent a couple of decades in. Look at the broadband numbers, like look at what we've defined broadband and we said it's got to be this and it can only be a minimum of this. We've never done that before. That's new in the last few years. Now you have 10 gig and 25 gig to the home, like that's real and that's happening.

Speaker 3: 10:20

You're you're not going to put that through a CGNet gateway. You and all your neighbors are not going to do 25 gig and throw that through. The ISPs are not going to go build 1.21 gigawatts of CGNet gateways. That's the reality. Because it's super expensive. Think about a layer three switch that can forward IPv6 at wire speed. If I go put in a router that's got an ASIC in it and I don't have to NAT, that's cheap. I can go put in a layer three switch and go sling packets all day. If I go to the leading CG NAT vendors and I go get something that can do multi-hundred gig, we're talking mid six figures, maybe even low seven figures for those boxes.

Speaker 1: 10:59

Yeah. So question, and I don't know this is NAT done in an ASIC or not?

Speaker 3: 11:04

It can be, but it's also incredibly expensive. It's possible to do it in an ASIC, but the problem is not the NAT, it's the flows. You've got to track state and then, if you're a carrier, you've got to have redundancy for that or you're going to end up like AT&T did Like when that crazy guy bombed Nashville inville in christmas of like 2020.

Speaker 3: 11:25

At&t did not have redundancy in their bngs, which is where they uh, they put their sessions and any nat that they have, and so there were like three entire states down on the internet because they hadn't built that redundancy and, um, I remember because my mother-in-law was one of them, she didn't have phones, internet for a week. So anyway, you have to track state you've got sessions. All of that is very expensive in an ASIC, both from a cost of computing standpoint as well as the physical cost, like the financial cost, and just for CG NAT is carrier grade, NAT.

Speaker 1: 11:54

Is that correct?

Speaker 3: 11:55

That's right, yeah, and that's maybe one thing that we should disambiguate a little bit is that not all NATs are created equal. We talk about NAT almost always. When network engineers talk about NAT, we're talking about the good old port translation that we all learned in Cisco back in the day that you've put on a firewall. But there's a lot of different flavors of NAT out there that exist now, both in IPv4 and there's even you'll be shocked to learn, andy we even have NAT in IPv6. I will even make this admission. I and ipv6, that I will even make this admission. I have ipv6 nat, one of my networks because, um, I the way that the carrier hands it off, I can't route it easily, so I have to use nat on my t-mobile mobility connection and I hate it. I don't like doing it, but I have to because I can't route it otherwise, because they only give me a 64, I didn't know.

Speaker 1: 12:37

Nat could slow down your connection like this is why I love these conversations.

Speaker 3: 12:41

Right, I'm like'm like. What's the?

Speaker 1: 12:42

big deal. It's NAT, right.

Speaker 2: 12:43

But that's the big deal to me. If you're going to slow my connection Anytime you're introducing something that's stateful, there's trade-offs to that right.

Speaker 1: 12:51

But in my mind it's in an ASIC and it's wire speed. And again, a lot of assumptions that probably aren't true.

Speaker 4: 12:57

Well, I think so. Like all good answers, it depends. So there's certain flows in NAT that are going to work just great at wire speed and are ASIC developed, and server load balancers are a great example of doing exactly this right. So often they do NAT in addition. But to Kevin's point, just when you have flows for particular traffic types, even with NAT, because of just the rewrite factor, you're going to have issues in terms of the inspect that has to happen, other things and just getting that to operate at the rate that it needs to, because you have to remember, a carry-grade NAT is doing NAT for lots of folks at the same time.

Speaker 4: 13:38

In a central point Most enterprise architecture your NAT's distributed, right. You're doing it on a handful of serious sets of firewalls at data centers that distribute it around. You're not taking 60,000 people and cramming them through a single device, right? So just the scale problem is very different in terms of what you're trying to deal with and the amount of bandwidth that you're trying to do that for and the amount of state that you're trying to keep. And I would add a third one, which is logging Every location that you have to do NAT for anyone in a service provider basis. They have to log that session and know what it is for reporting purposes. Almost everyone has this reporting requirement. When you start introducing this scale of NAT, just think about how much storage you have to buy just to hold on to all of the data.

Speaker 3: 14:38

Oh it's nuts, They'll sell you a $100,000 piece of software just to track the logging, because it's like you said, it's something called CALEA in what's going on. And if you can't show that Andy had Andy's 192.168.10.101 mapped to this public IP to go to this website, you know when Andy was hacking the CIA. You know last week that you have to be able to show that and you have to be able to prove that legally in a court of law with chain of custody and evidence and all of that. And, like Ed said, that's really expensive to log in store with NAT.

Speaker 1: 15:12

Do customers complain about slow speeds and the cause is NAT? No, yeah.

Speaker 4: 15:19

There are some, I would say in the US it's probably not as common. I would say overseas it's probably a little bit more common. I would say the bigger issue actually domestically in the US. That actually was one of the reasons Microsoft invented Teredo and put Teredo out. There was first-person shooter gaming.

Speaker 4: 15:36

If you're playing your buddy down the street and you're both hidden behind the same CGNAT, it's impossible for that provider to really distinguish between who's session is doing what and how do you actually play first person shooter directly to each other, so you're bypassing having to go up to a central server and come back down because you want to reduce.

Speaker 4: 15:54

You want the same latency for all your game players, right? That's the biggest issue. So if you have everyone having to go up to a particular location but other people are able to go peer to peer, guess what? The latency then is a different metric and so you're going to get your shot off quicker and you're going to see that other individual faster in the gameplay mode, which is not fair, right? So there's a fairness factor around just gameplay and Kevin can probably talk about this way better than I can because I'm not a gamer but to any extent. But there is a performance factor that goes. That was a really big deal and so trying to solve that was really about using Teredo to be able to allow devices that are behind the same carrier grade NAT to basically directly pin up peer-to-peer sessions amongst each other.

Speaker 1: 16:36

Before we get off that, can you tell me what Teredo is? I have no idea.

Speaker 4: 16:41

Teredo is just a tunneling protocol that allows you to do basically V6, you know, nefariously across V4 session sets. It allows you to dynamically build those and use ports to be able to do that. So you can basically tunnel sessions back and forth across. So normally if we were in the same neighborhood and our kids were playing first-person shooter games, their consoles couldn't talk directly to each other because they're behind the same carry grade NAT.

Speaker 1: 17:10

There's no direct connection. Aren't all ISPs dual stacking and wouldn't they be handing out V6, and why would they be NAT?

Speaker 4: 17:14

Today? Yeah, that's true, but if this, more than a decade ago, 15 years ago, when they were trying to solve this problem, that wasn't the case. And so how do you deal with that? Well, you have something like Teredo. Now, if you just have a V6 address, you just directly peer-to-peer and you're done. You don't have to worry about this, which is one of the reasons to Kevin's point earlier. You get better performance out of V6 because you don't have to do that extra hop or the translation right. Neither one.

Speaker 1: 17:37

You're and you're all good and I don't care what my ISP gives me, right, I have a fiber hand up to my house. It's gig, symmetric, it's lovely and they can give me V whatever. It's fast, it works, it's great. But as someone managing, you know, production, networks and enterprise, like when I see that V6 address space, I'm like, oh my God. And then when I see that half a dozen to v6, I'm like, oh crap. I mean to your point earlier, if you can learn automation, if you can learn ospf, you can learn v6. But I think, because it's it feels so different than what I spent all that time learning before. For some reason, v6 just seems intimidating as as hell to me. And what I'd like to get to by the end of this episode, what I'd like to do is speak to the people like myself, basically, who were like I think V4 and that is fine, I'm in the US, it's kind of a crappy attitude, but I guess they can figure out in Asia what they have to do. Again, this isn't really how I feel.

Speaker 4: 18:33

No, you're representing yeah, you're representing the US.

Speaker 1: 18:36

I'm representing the 60-something percent of people that haven't gone to V6 20 years later and I'm trying to compel them. Well, this isn't like go v6, but like if we can dig into some of this stuff. So like, as far as I can tell, mobile carriers and content providers are like yeah, content provide.

Speaker 1: 18:54

Like you know, there's a handful of things you have to have v6 like. I'll give you an example, then I'll shut up. I worked at a huge company and then another huge company bought them and then they merged and they all have private address space. They're netting all their hundreds of thousands of clients they have, you know, from whatever the client is to 10, and then we have to get all that crap talking between companies. So we put two honking net firewalls between the companies and then you added the net and you know it's a nightmare.

Speaker 1: 19:19

Now that's the perfect scenario for v6. Sure they haven't done it? And I think they haven't done it because there's hundreds of thousands of hosts that would have to be readdressed and customers would have to take down time, like stock exchanges in new york, as an example. Like this was in fintech right. So I don't know if we can. We got real into the weeds quick and I love it, but I don't know if it helps at all at some point to kind of address the higher level things like is there complexity, are there migration outages?

Speaker 4: 19:47

um, like what do you think are the?

Speaker 4: 19:48

top three things stopping people from from going v6 right so there's there's a culture thing, and so I think the culture thing is is uh, you're always better in your native tongue, so we're all you know. So I sort of look at v4 to v6 translation is like when we went from French to English. So if you're natively French and you're speaking French every single day and we're suddenly coming around and telling everyone we have to speak English and you have to learn English, and if you want to continue doing your job, you got to learn English but you're still continuing to talk French with all your friends every single day and all your work colleagues every single day, and everyone else and everything you read in the paper is French, your compelling reason to move to English is pretty low. It takes time and it takes a set of business initiatives and pressures to make the move worthwhile. You're going to have certain people that are bilingual. They're going to make more money because they can talk to both markets, so they can talk to the French folks, they can talk to the English folks, they're going to have the opportunity and they might make money in the triage portion of working back and forth. And then you're going to have native English speakers who are going to have certain advantages around, just the fact that that's where the market's going. It's maybe not a perfect analogy, but it gives you an idea of sort of walking through the process. So don't hold anything against people that are V4.

Speaker 4: 20:57

And I think in V4 too. It's not like it's what I learned first. It wasn't. I didn't learn V6 first. I think I'm much more fluent in v6 now.

Speaker 4: 21:05

So it's really a skill to learn over time and it also depends on where you're at in your career and what you're doing. So if you're in a mobile service provider space or in your service provider space or your content delivery space, you need to know v6. There's just no getting around that today. It's just reality. If you're in a large enterprise network that maybe isn't in the Fortune or Global 2000 or something like that, you can probably run away and avoid v6. And if you're close enough to your retirement age, you can choose to ignore it.

Speaker 4: 21:34

And my ask of you is just don't hammer on everyone else who's trying to get v6 deployed. Don't do it yourself. Go learn something else. Go learn Python and do network automation. Go learn SD-WAN and do that and get your niche and finish off your five years and be good or whatever it's going to be, but you don't need to stomp on everyone else's project. Otherwise I'll stop on yours and say like hey, you know SD-WAN. That was stupid. We solved that problem years ago. That's just routing. Why can't you do routing as routing? Are you dumb? Do you not understand how to do BGP?

Speaker 3: 22:02

It's just BGP and tunnels, man Right, exactly so.

Speaker 4: 22:06

I laugh at all of those counterpoints around it. It's like everything old is new again, and for me I look at it and say how can you add true value to your business? For BGP global shops? Now there are mandate requirements from almost every single major industrial country that says V6 is where things need to happen, from both an investment, innovation. The ITF doesn't work on IPv4 anymore. I don't know if this is something that you guys actually know, but from a standards group's basis, they don't work on IPv4 as a standard anymore. It's deprecated, it's done. They only work on IPv6 as a standard anymore. It's deprecated, it's done. They only work on IPv6 as a standard basis. So do you really want to be working on everything? That's just older.

Speaker 4: 22:50

And I understand the industry and network and network vendors. Look, you sell what's on the truck. So don't get me wrong. I understand all their motivations around it and I understand why every single one of them says like oh well, that's not something that we're working on, because guess what? They're trying to sell what's on the truck.

Speaker 4: 23:07

The reality is is that mass majority of the customers will still buy what's on the truck. That's fine too, but the reality is is, if you want to innovate in certain areas. If you want to be able to meet requirements for contracts or if you want to be able to actually innovate in certain areas, you're going to need v6 in your tool belt. Does v6 have to be everywhere in your network? Probably not. That's a design choice. But if you don't know v6 well enough, you don't know where to put it and where not to put it, and that's a shame on you because you didn't do the homework, you weren't educated enough. That's like going out and saying, like well, sd-wan is going to solve all of our related problems and we never have to buy a managed service or an MPLS service or a VPN service, ever again. We're only going to buy public internet circuits and deal with everything that way.

Speaker 4: 23:49

That might be true for certain businesses, probably. If you're heavily regulated, probably not so much. If you're the US federal government, maybe not so much. You might be deploying SD-WAN on top of your managed service, right, just to provide security. So there's other things that go on with all of that, and not knowing something and then making design decisions or making architecture decisions is a poor position to be in. So my challenge to everyone is why don't you learn v6? Understand whether it's in or out, and this is really common with security teams. When we go in and talk with teams, they're like well, we can't handle v6. We can't do with all of that. I'm like, well, v6 is on by default and preferred V6 is already running around your network. Do you want it in the camp or do you want it out of the camp?

Speaker 1: 24:31

Are you able to unearth their reason? Why Is it just fear? Is it just lack of?

Speaker 4: 24:35

knowledge, it's fear, and they don't have time and energy and resources dedicated to being able to do that. Like, how much money are you paid to go learn v6? Probably not?

Speaker 1: 24:44

Is it a lot of time, money and resources? So, like I guess you do, migrations, right, Ed, as part of your job.

Speaker 4: 24:50

Yeah, we mainly do. We do training, consulting and education and that's. But I'll be honest, we're a small boutique shop. We don't do deployment and operations at all because because if Walmart decides they want us to do deployment and operations, we disappear. For five years. You never hear from us again. They're so large. We really want to transfer knowledge and information to customers like that so that their own teams can do it, because that's the only way you can scale. Their team already needs to know it because they're going to operate it anyway. So our best job is to help them understand the protocol. Help them to understand you know the protocol. Help them to understand the design and architecture decisions. Help them build an architecture. Help them build a proof of concept lab to get learning and skills and then work on a low level design, because that's what you use to deploy, right, but their team should know how to deploy. By that point, hopefully we've done enough. We've done our job right and done the knowledge transfer to help them along.

Speaker 2: 25:43

Well, I think I think one of the kind of beautiful things about it is that you know the OSI model is the OSI model, right? So the upper layers of the stack don't really change that much once you move to v6. You know there's some nuances here and there, but overall it doesn't change that much. But you know, I think so far we've spent a lot of time talking about the major benefactors to V6 being service providers, telcos, mobile providers. But how does this apply to the enterprise? Because I feel like there has to be some operational benefit for an enterprise to move to V6. That isn't just I'm running out of address space. There has to be some other optimizations there. So can we kind of diagnose what those are?

Speaker 4: 26:26

Yeah, kevin and I can go back and forth on this one, oh yeah.

Speaker 3: 26:29

Well, the first thing that comes to mind for me is that is this affects companies of all sizes is your cloud costs, because companies are starting to figure out that AWS is now charging for IPv4 space or more than they were Azure. All the cloud operators now charging for IPv4 space or more than they were Azure. All the cloud operators are charging for IPv4 space and even some of the transit costs. When you talk about cloud transit costs, even I've seen not everybody, but some of them will even v6 doesn't cost as much or is sometimes free, as compared to IPv4 transit. And so, because it costs them all at last, because, let's be honest, most of the large scale networks are all IPv6 underneath Large cloud operators, large carriers.

Speaker 3: 27:11

They've been IPv6 native in their backbones for a while. Ipv4 is just a service that overlays on top of it. Almost all the large carriers are like that, so there's big v6 only networks that are already out there and then they just put on top of what you want that you're going to use. So if you want to lower your cloud costs and you're a small company, large enterprise doesn't really matter you can put IPv4 on top of an IPv6 underlay on the public internet and lower your cloud bill because you can get as many IP endpoints as you want and there's enough ISPs, in the US at least, and I think, europe. It's fair to say as well that it's not terribly hard to go do that. Whether you're going to get a wired connection or a mobile connection, you can do a pure V6, underlay in WireGuard or IPSec, or pick your poise, pick your flavor of VPN that you like, and that's an immediate impact to OpEx that you're reducing just by putting everything on the IPv6 internet, even if you're going to put IPv4 on top of it.

Speaker 4: 28:05

Yeah, from a transport, because you don't have to pay for that public IP.

Speaker 2: 28:09

Overlays just fix everything, don't they?

Speaker 4: 28:10

They're so sick dude, Do you think the CSP should build?

Speaker 1: 28:15

Did they build their greenfield with v6? Because migration seems to me to be the biggest hurdle to try to overcome.

Speaker 3: 28:22

Yeah, I mean that's the thing I know, that you know, I think you know, I know Ed has done this. I've, you know I've worked for large companies, publicly traded companies, and I actually worked directly for one but way back in the day, earlier in my career, and I remember doing mergers and I mean we spent like years on mergers and half of it was just when you're dealing with these really massive, you know, fortune 500, fortune 100 type companies. Trying to iron out the overlap in RFC 1918 is insane. In fact. I remember I was out at NFD and I went to go visit Google. In fact I think it may have been the same NFD where Ed and I met, because Ed and I also met at NFD and were V6 kindred spirits and Google was using the carrier grade NAT space in their campus. They were in IPv4.

Speaker 4: 29:01

They were in IPv4. They were using 100.64.0.0.

Speaker 3: 29:03

Slash 10 in addition to IPv6, because they had run out of RFC 1918 a long time ago.

Speaker 3: 29:09

So I mean you look at this and if you've got to merge those together where you've got 10 dot overlap every which way, that's really really hard.

Speaker 3: 29:18

And so if you build a native IPv6 backbone and underlay in a company, then you can put your IPv4 as a service on top of it but you're no longer worried about your routing underneath having to worry about that overlapping, because that can be unique space. And one thing I think is a really important point to make that everybody really gets up in arms about is just because you go get IPv6 space from Aaron or whoever your RIR is doesn't mean you have to advertise it to the global internet. So everybody gets all up in arms like I will never, ever let this IPv6 touch my network because our security team will never allow it. It's more important that it's unique. You don't have to advertise it and make it public. It can just be unique space that you number from and use as much as you want. And I think that's something that everybody misses on IPv6. Just because it is publicly registered or registered within RIR doesn't mean that it's publicly accessible to the internet, right?

Speaker 4: 30:12

Yeah, good points.

Speaker 4: 30:13

I think for enterprises there's a couple of different things that really come to mind, at least from the discussions that we do with customers Strategically. Merchant acquisition is a big one that Kevin already mentioned, which is really, how do you use that to strategically solve certain problems and the problems that we solve for customers around merger and acquisition really is more of a finance thing than it is a technology thing to be for merging a company together and you've already calculated all the factor costs of closed data centers, closed circuits, transitioning people, transitioning all your services over, and it takes you an extra 24 months to make all of that happen because you're dealing with NAT and translations and overlapping address space and not enough people and resources to actually do the work. You have like a Delta in financial costs that has to be accounted for from the finance team to have to go back out to the street and say like, yeah, we actually didn't hit our marks. We have to reevaluate or revalue what we actually did performance-wise here in order to and there may be clawbacks that are associated with some of that. So there can be real world impacts from a finance basis when you can show them that tactically you can just leave the v4 alone and never touch it again and just hand them v6 resources, say please go, deploy these, redeploy your services in v6 in our, in our cloud portion. We never want to even talk to you over ipv4. You're only going to talk to us over the ipv6 allocation that we give you. That's how you talk back to the head in office. We're done. And that means your engineering team can concentrate on the things that are that good at, which is deploying new services, doing that sort of stuff. The v6 side should be pretty much taken care of If they have any translation capabilities that they need to deal with. Well, you're like, yeah, we've got NAT64, dns64, and we got CLI capabilities to be able to introduce that if we need to for the mobile side, and we're done, and they're able to walk on, and so the number of resources that you need to do that work versus renumbering across an enterprise on both sides of the organization, is really important.

Speaker 4: 32:10

The second one is for very large organizations, and this resonates for folks that are dealing with large companies. Is NAT is brittle. It breaks your routing, it just does. It introduces state and it introduces choke points within your network topology that don't allow for routing to do what routing is supposed to do, which is natural, failover capabilities, right Of routing around problems.

Speaker 4: 32:33

Pushing firewalls closer to the end client means that you know asynchronous, asymmetrical routing isn't an issue. You don't care which path it goes over and comes back to you because the state device doesn't have to measure or deal with that particular issue, because it's as close to the client as possible and in fact if you're doing zero trust, effectively the firewall component is really on the client itself or as close as possible to it. So you get away from a centralized firewall model. You get away from that centralized NAT model. It sort of allows you to have a little bit more flexibility to allow routing to do what it's really supposed to do, because you can't exchange 10-dot routes with someone else on 10-dot routes if all the routes are the same.

Speaker 4: 33:12

It just introduces brittleness around what you can or cannot do from a failover in data centers. And stamping out the same addresses over and over again in all of your data center sites because that's the template that you know how to use doesn't really scale and solve that particular problem. It also causes logging problems and compliance problems and a bunch of other issues that go along with it. So the global uniqueness around that address space provides that tool. And to Kevin's point earlier, the global uniqueness doesn't mean that you have to put it out on the internet. There's plenty of address space that you can just say like yeah, we just won't route advertise this outbound.

Speaker 4: 33:46

This is designed to be our interior portion of our V6 address space and the firewall knows not to allow traffic inbound for that particular address space or outbound for that particular address space, depending on what your rules you want to put in play. I think the other major innovation area for companies is the fact that with a plethora of addresses comes opportunities to build new services that you may not have been able to build before. So, andy, you worked in fintech before. How nice would it be to have, besides just a transaction ID, having some sort of unique global IP address that you use once, discard and never reuse again and it's tied to that session transaction and you knew what machine it was generated off of. You knew exactly what transaction it was tied to because it was tried per application, per transaction it's possible to do, because if you take something, like you know, a standard 64 in V6 world right to the 64th, you decide to do 10 million transactions a second, never reuse an address.

Speaker 4: 34:41

it's going to take you 58,000 plus years to burn through your slash 64. I think, just with that model alone, how many people think their data center is going to be here in a thousand years or heck a hundred years? So that sort of lifts that off. You know, 10 million a second. I don't think there's a lot of apps that can generate and use 10 million addresses a second. So I think you're going to be probably pretty safe in terms of your first 64, getting through that and being able to do some very unique things. Sure, can you do that with a GUID, absolutely. You could generate random GUIDs at 2 to the 64th and do that same way, or even 2 to the 32nd probably, and be fine. But the reality is, if I give you that as a unique transaction, going from a mobile device, a handset that's got a V6 address on it because all the mobile providers are V6 going to your service that you provide, you now have much better auditing and logging capabilities for the application, the session transaction, the end user who authenticated the mobile device that they talk to. You have all of these very key sets of metrics versus going through NAT from the public mobile provider side that has to go from v6 back to v4 to talk to you, talk to your v4 service that you NAT translate again. That goes through a server load balancer, that goes through three-tier architecture, probably all the way down to the database transaction.

Speaker 4: 35:53

Do you really think you have useful information at that point in terms of what's going on? Which is why, obviously, tls 1.3 helps solve some of these related. There's other things that we can work around, but why have less security measures and less information versus not right? I mean, this is one of those cases where this is a value proposition of adding more to the bucket, not less. And so more data, more telemetry, more information is more useful and it guarantees uniqueness. And it guarantees uniqueness and I think that's lost on people in terms of.

Speaker 4: 36:24

We're so used to associating stateful packet inspection and NAT as a one-to-one correlation. There's no requirements around that. You could have NAT with no stateful packet inspection. You could have stateful packet inspection with no NAT, which is the reality of where v6 fits. A v6 firewall is just doing stateful packet inspection, but it doesn't have to do the network address translation portion. One less piece of information to float around and keep track of. Much more efficient, less data storage. When you start adding this up over the millions and millions and bajillions of packets that are getting pushed across the network, that reduces storage costs. That means there's less data center build. That means there's less electricity used. There's actual financial benefits that you go in that direction.

Speaker 4: 37:02

Yes, at scale, this is all things that have to happen at scale, but the internet is an at-scale problem, right, that's the whole purpose of it. And so if you want to sort of talk the lingua franca across the internet, well, that's now moving from V4 to V6, and there's plenty of organizations that are pushing that need requirement that way. And more and more concentrated services from the CDM providers are happening over IPv6. They prefer them that way. They have more addresses to be able to provide services and uniqueness to all of their customers, plus to all those folks.

Speaker 4: 37:38

And the reality is that the mass majority of us running around are really consuming internet services off of mobile handsets. Is that the mass majority of us running around are really consuming internet services off of mobile handsets? So the enterprise fits into this weird category of like you're the left behind, right, even though everyone works at an enterprise or says they work at an enterprise, small businesses they're going to switch over to v6 and never know it, because their service provider is just going to do it to them and that's going to happen and they'll suddenly get v6 working.

Speaker 3: 38:02

It's just DNS, I mean them, and that's going to happen. And there's something. It's just dns, I mean that's. My mother-in-law has been using ipv6 on at&t for years and they don't know it. But when we go over there I get an ipv6 address out of the 64 they allocate into her at&t gateway and it's been there for like a decade and it just, it just works. And you know, I always laugh.

Speaker 3: 38:15

I think there's a big joke in the ipv6 circles is at least half the people that say I've never used ipv6 and I will never use it are doing it over IPv6 on their mobile handset. That's v6 and they didn't know it because it's been v6 for a decade. Right, and I think that's the thing is almost all the phones, at least in the US, mostly Europe most of the content is v6. If you look at the breakdown of the DNS entries of most of the stuff you use and we look at the traffic profiles in service providers, 60% of the stuff you go to in your phone it's already IPv6. It's been that way for years and you don't know it. Because you don't know the difference? Because the carriers have gotten really good at making it seamless for you. So I would say, if you were to go dig into your phone like Andy's we were talking about, hey, I'm good with IPv4 and NAT.

Speaker 1: 39:07

When you're on I and you're not on wi-fi, you're going to be using v6, probably for 60 of everything you do on your phone and you've been doing it for years.

Speaker 3: 39:10

How do I see that there's tools that'll?

Speaker 1: 39:11

show you, if you I don't know if you're android or iphone it's easier on android because iphone's all you know locked down and the reason I'm asking as you're talking on my computer, on my pc I go to what's my ip addresscom and I see a V4 handed to me from my carrier and it says V6 not detected. And I'm like, so is this my fault? Like what's happening? Is V6 happening? But it's transparent. But I'd like to do the same on my phone, like where I want to see V6. Right, but I don't see him have it at home, which I'm kind of surprised or I'm doing something wrong.

Speaker 3: 39:38

Yeah, Most phones if you do like what is my IP address? You go to one of those sites on a phone and you're on the mobility network. You're going to pop up on v6. The reason is that a little backstory in the carrier world when they designed LTE and 5G, they knew very quickly that it was not going to be an IPv4 problem. A decade ago they knew that when they designed the LTE standard, when they designed the 5G standard, the 6G standard, and I think Andy just got his first IPv6 address.

Speaker 1: 40:03

You're right. I just smile on his face. I got off Wi-Fi. I'm like, oh my.

Speaker 3: 40:08

God, there it is. Oh my God, I've had IPv6 for 10 years and I never knew it.

Speaker 1: 40:17

But to your point.

Speaker 3: 40:17

Everybody on a mobile device that's out and about off Wi-Fi is probably on a v6.

Speaker 4: 40:19

You're probably over 50% of the stuff you go to every day is just natively via v6 and you never know it yeah, I mean if in, in fact, to narrow that down for kevin if you're only using, uh, the top 10 apps on any of the given platforms, you're probably way higher than that yeah, 90, 90 to 100 percent. Uh, ipv6.

Speaker 3: 40:34

I think the only exception is might be twitter still yeah, I don't know if they've they've done it, but youtube, netflix, disney plus, I mean all the things so that I'm trying to get into the nuts and bolts of it.

Speaker 1: 40:45

So is that on the platform Like, say, let's say, facebook, because it's less inflammatory than Twitter right now? So do they make a decision on their platform and their software that, like, we're going to be v6 and we're going to get?

Speaker 4: 40:57

v6? Yeah, facebook was super early, so Paul Saab over at Facebook did a huge amount of effort for porting and converting. They actually run v6 only in their data centers.

Speaker 3: 41:05

Yeah, I was going to say their data centers are all v6. They just translate your v4 coming in their entire data centers are v6, native v6 only, so everything they're handing out or anything you're doing on Facebook.

Speaker 1: 41:16

It's using v6.

Speaker 3: 41:28

Yeah, it's native v6 because, like Ed said, you're staying end-to-end v6 native when you're on v6 to Facebook.

Speaker 4: 41:33

All the.

Speaker 1: 41:34

ISPs are handing out v6, we can probably assume right. Yeah, most of them are the large.

Speaker 3: 41:38

ISPs are Regional ISPs. They've definitely done more in the last few years if you're dealing with a regional carrier like an independent telco or a WISP or a FISP. There's been a big push in rural bandwidth, which is an area I've worked in a lot for the last decade, and they're starting to realize that hey, we've got to get IPv6 in and have been pushing for it. But I think one of the big holdouts that was a good marker for IPv6 adoption was Verizon Fios. Verizon Fios was not on IPv6 forever.

Speaker 1: 42:08

That's who I have and I don't see a v6 address.

Speaker 3: 42:11

Yeah, I don't know if Fios is 100%, but I know that most of Fios is now v6 capable. They finally kind of tipped the scales. And as much people bag on Comcast. Comcast was an early, early IPv6 adopter. They were running v6 for a long long time as much as people gripe about it.

Speaker 4: 42:31

Yeah, john, jason Borkowski did a lot of work early on around pushing v6 for Comcast and did a lot of work with helping to push standards, and then T-Mobile also. And on the international footprint basis, I mean, like Reliance Jio in India, that's an entirely rolled out mobile network. That's V6 only. They only provide translation services back in their data centers to get you to V4 internet and they pretty much tell everyone who wants to ride over their network and they're super cost-effective in India. So they have a huge, huge footprint there. So I think that was one of the reasons India's numbers shot up from like I think they were like 20 or less percent and then suddenly they were within like three years they were at like greater than 60% V6 adoption for the country. And one of the reasons why is because Reliance G rolled out across the country and provided super cost-effective mobile services. Everyone jumped over to them because they were more cost-effective. They were V6 only and guess what, everyone was suddenly on V6.

Speaker 4: 43:22

And it's stories like that that you realize how quickly impacts can happen with service providers who suddenly turn the switch on. That can change entire markets and that's true sort of across the board. And you're seeing this behavior happening over and over again. Because why are you going to invest in carrier grade NAT versus just providing a native transport? And then, hopefully, as more and more services go V6, your reliance on how much translation you have to provide with NAT64 or DNS64 starts diminishing, not increasing, and so therefore, you just leave your resources in place. You can burn down the capitalized costs. If you take the reverse approach, you're going to have to continue to pay more and more and more to do the carrier grade NAT just to get people to where they need to go, if you stay on V4. So there's a financial advantage of taking that model and sort of flipping it on its head. I don't know, kevin, you're way more in that space than I am, but I think that's true in terms of the long-term capitalized models, right.

Speaker 3: 44:17

Yeah, no, it absolutely is, and one of the things that I wanted to I think that I wanted to touch on that. You talked about Ed a little bit, because you were talking about the idea of an IP per transaction, which kind of one of the things that would probably be helpful to mention, that I think a lot of people don't realize, if they've not touched v6 is the concept of privacy extensions or temporary addressing, which is, you know, what you were mentioning was specific to the fintech space, but it's kind of an extension of that whole idea of I'm going to rotate my addresses. So, andy, we talked about let's talk about gaming and talk about your gaming. You've got kids, my kids game. I don't know if your kids game, but let's say you piss somebody off over somewhere and they are really mad and they're going to order up a DDoS attack on you, which happens all the time Working in the carrier space. I cannot tell you over the last decade the number of DDoS incidents that I've responded to, just because somebody got pissed in Call of Duty and they went and ordered a 10 gig DDoS attack on the dark web from one of their friends to go DDoS an IP address and they would throw a volumetric attack at another gamer just because they got mad. You can do that. You can literally put your credit card onto sites on the internet and order up a 10 gig volumetric DDoS attack. You know on demand and go pay. You know however many Bitcoin it is to go. Do that, you know, through your credit card, and that's so.

Speaker 3: 45:33

You have an IPv4 address. You may have a public at your house I guess you have a public that you NAT through that you're not going to get a whole bunch of those right, like you're going to get one. Maybe it'll shift every few months if you haven't ordered a static, but your IP is going to be your IP. Think about the math that Ed talked about, because this math gets talked about a lot. Slash 64 is in IPv6, you don't allocate addresses, you allocate a whole block, right Like you allocate a whole prefix, because no longer is the router responsible for managing the WAN connection. Like those addresses flow through to all your hosts. So you think about the way a router works in your home. You've got your RFC 1918 on the backside of a router. It's 192.168, whatever it is, and you've got your public IPv4. And you do NAT and DNACID. Call it a day, right, and maybe if you have two connections you fail over. But the router does that. It takes care of that. You don't have to worry about that In IPv6, because every host gets a public address.

Speaker 3: 46:28

It's incumbent upon the host to connect to the internet now it passes through the router. So the way that they thought about that and said, hey, we've got to come up with a way to make this a little bit more secure when they were working on IPv6 is the idea of temporary addressing. So you may not know it, you look at this, you can see this on your phone. But when your host gets an IPv6 address through Slack not the messaging platform, but stateless address auto-configuration, which is what we typically use in IPv6 instead of DHCP, and everything you use is already preset to grab one of these addresses via Slack. I don't care if it's Windows or Apple or Android or whatever it is, they will all grab an address via Slack, linux Well, I guess in Linux and Android there's a few quirks there, maybe, but most things will grab an address via Slack.

Speaker 3: 47:14

And so if your router grabs a slash 64, which, as Ed mentioned, is a stupidly large number, even though it's a single 64 that they hand out to you. Your computer will automatically rotate through IP addresses using the MAC address as a base, but then it randomizes the host portion of your IP address. So basically, when you browse the internet, the IP that you use the exact IP that you use may only be there for a few hours, maybe a couple days, and then it's going to rotate to a new one and you don't have to do anything. Your computer manages this for you automatically. But the likelihood that they can go find number one, they can even find that address. Because even if they do find out what your v6 address is, who says you're going to have it in two hours? And then they've got to figure out what subnet boundary you're using, because you could have a 64, you could have a 56, you could have a slash 48. And it's so hard to figure that out if you don't have a DNS entry to map to it when you're looking in on the LAN.

Speaker 3: 48:07

So I'll give you an example. I have an RDP server that is totally open to the internet, no firewall, on 3389, on IPv6. I've had it for four months, zero scanned hits. Nobody's ever found it. Because if you go and scan even just the 64 that they allocated to me in Vulture. Like Ed said, if you scan 10 million addresses, it would take 58,000 years just to scan that 164. And that 164 is like one grain of sand on all the beaches on the planet. The IPv6 space is so numerically large that it's impossible to scan. Now. That's not to say that you should abandon best security practices and that you'll never have to secure it, but the tools available to you to be able to secure your infrastructure at layer three are phenomenal compared to V4.

Speaker 4: 48:55

Yeah, there are ways of guessing, andy. We had a show that talks about there are operational procedures that a lot of folks do that make them more vulnerable to scanning detection, because of Right, like if you're manually assigning out of ranges and things.

Speaker 4: 49:10

yeah, Ranges, and there's things that you can do that are like oh, we're mapping our V4 back into our V6, which is like you're going to make yourself susceptible to scanning algorithms, because bright folks have figured out that these are the sorts of things that you can do. There's also things like pre-built addresses around EUI-64.

Speaker 4: 49:30

Well, those only have to exist within the 48-bit portion of it so you can do some mappings to figure out all the possible addresses that a EUI-64 could potentially use on a given manufacturer, so you can scan those range sets. Certain manufacturers don't exist anymore, you eliminate them. Like there's lots of things you can do to sort of reduce the space of or footprint of of size. So I don't want to, I don't want to play into too much of like it's so vast that you could never. You could never do it. But I think it's. I think it's it's unreasonable to do unless you have a targeted attack of really understanding what you want to do, and then it might be worth some effort and time. And to Kevin's point, you're not going to order up a 10 gig DDoS attack against an address that might just disappear in two minutes, cause it's not very financially cost-effective.

Speaker 3: 50:14

No, and I mean that's why I just do this. Out of curiosity. I throw away boxes that I monitor and just scan, like has it ever gotten hit? Because on a v4 address you're going to get hit 30 seconds within. Popping it up online, it's going to start getting scanned. But I haven't had a single hit on this and it just uses a Slack address. It doesn't use any, it's just using a randomized Slack that isn't based on the Mac, it's just randomized Slack and in four months, not one hit, not a single scan, not a single packet has come into that host from the internet. Can you?

Speaker 1: 50:41

I know I should know this. Can you quickly tell me what Slack is doing? It's like I know it's a protocol in v6 and it's somehow random.

Speaker 4: 50:50

Is it taking the?

Speaker 1: 50:50

v6 unique address and randomizing something.

Speaker 4: 50:53

Well, so Slack is actually an address allocation method. So we have DHCPv6 and we have stateless address auto-configuration and then we have manual or statically configuring something. Those are really the three principal address allocation methods and Slack is really designed to allow a host to come on and self-provision an address. And there are different methods of self-provisioning an address and that's what Kevin was was talking through.

Speaker 4: 51:15

so there's something called eui64, which is an older method that would basically use your mac address stuff's fffe in the middle and switches a global bit to a local bit so that you know that you're you're locally significant right and and that's utilized as as eui64 and many network gear. You still use as eui64 because it's perfectly legitimate and your router is not going to change its address Actually you'd have to look now, but Ubuntu and. Red Hat, don't do that anymore.

Speaker 4: 51:47

On the server side they may still do it, but for the general client side they don't do it anymore. Under Slack you also have the concept of a temporary address and you have the concept of a privacy address. And the privacy address is really that random generation of a 64-bit number that you utilize in order to build your address. And a temporary address is also a randomly built one, but you only have it for a short duration of time. And then there's some additional RFCs that just came out that are around the concept of what's called stable Slack, stable Slack being you're still randomizing or generating your address, but when you come back to the office and you're on the same SSID Wi-Fi, you're going to generate the same address that you used yesterday just to help with things like logging and other things that enterprises might want to have available to them and, just for consistency's sake, around things like DNS and a few other things. But those are the standard methods that you see. There's a couple other things that are we can throw in the loop there and there's some oddball things that impacted, like rotating Mac addresses and things of that nature that were put out there by certain hardware manufacturers to sort of introduce more entropy or more capabilities of providing randomness. But that's what Slack primarily is. It's an address provisioning method and it's designed to allow a computer to be able to come up, or an IoT device that has a very low footprint, doesn't have a DHCP client on it, to be able to come up, build itself an address and participate on the network. And the way it's able to do that is Slack is really an indication from the router. It's a router advertisement that says, like hey, go ahead and auto provision an address for yourself and we're going to provide you the prefix information. You go ahead and build your address portion and you know where the default gateway is, because I am the default gateway. I'm the router that's providing the router advertisement. Therefore, I am the default gateway and I'll also provide some other useful information. So there were two RFCs 6106 and 8106 that introduced the concept of basically, dns information in the router advertisement.

Speaker 4: 53:49

So that's the two things you need, right? You need an address. Well, three things you need an address, you need a default gateway, you need a DNS server. Once you have those three things, you can participate on the internet with zero issue. That's what Slack provides you. Can you do all the same things with DHCPv6? Yeah, absolutely. But you will get a singular address that was assigned to you from the DHCPv6 server in the traditional DHCPv6 realm from that server and you'll be provided a DNS server. That is provided in that message. But you will still use the default gateway from the router advertisement, unlike v4, where we provide a default gateway. We don't do that in v6 because the router advertisement is the default gateway.

Speaker 1: 54:26

Chris, you have a CCIE, so maybe this isn't happening to you, but is your brain exploding when they're talking about all the stuff that v6 does and can do?

Speaker 2: 54:36

No, I definitely knew a lot of this information, um, just with regards to how v6 functions and things like that. But one thing that I you know, calling back to your previous point, you were making kevin about security practices and what the implications are with v6, um, I would, I would be upset if I didn't take the opportunity to drop my favorite trope about v6. I like to just kind of drop this in the room like a grenade and run away, but I love saying that NAT, specifically within the concept of IPv4, nat, is security, because there is inherent things that you get out of obfuscating what the address is on your physical machine. From a philosophical perspective, we can definitely say like, oh, your security practice, your practices, should be up to par so that exposing your IP of your machine on the public Internet should not be a problem. Public internet should not be a problem. However, that that there's, you know, I think.

Speaker 4: 55:38

I think some of the V6 adoption that is scary is just what what that means for your entire endpoint security practice when you move to V6, right, that's a good point, chris I, I, I address that mainly as saying like bad operations is bad operations. Um, so what do you get out of NAT? Nat allows bad operations. That's the reverse. That's the reverse of the argument, which is really to say like, what does NAT enable you? Well, it allows you to be a lazy, not very studious, particularly refined security network operator, and if that's going to be your crutch, I'm okay with that. That's fine, because there's an inherent security posture that's associated with that.

Speaker 4: 56:15

Do I want you running my network? No, not particularly. I want people that have the training, the education and the understanding that they do all the right things, regardless of whether it's v4 or v6. This should be true regardless of the networking protocol that you're choosing to do, your implementations of your security practices, your network hygiene as it would be right In terms of how you choose to propagate your routes, your availability, how you handle failover. All those things should be designed in architecture decisions that were made intently, not by some default behavior of how an application or a protocol was built initially. And so that's sort of like saying, like yeah, I'm perfectly good firing up BGP and advertising default route Right. Like, okay, sure, but we got over that years ago back when we made those mistakes, and every single service provider is going to filter out your default route that you're providing upstream to them Are you a particularly good network engineer.

Speaker 4: 57:09

If you do that, I would say probably not. Can you do it Absolutely Right? So I see Nat falling in the same category. Is the default behavior and position of it a better security profile? Sure, is turning off your computer a better security profile? Absolutely, and I see that in the same category.

Speaker 3: 57:28

And this is where it's a bit of an apples and oranges to me, because, going back to the idea of if, nat, I have, like what? Maybe one address, five, six, maybe if I have a really big enterprise, maybe I have a whole slash 24. But you're talking about addresses that are shifting every few minutes that you're using. Like to Ed's point, what if you don't ever want to reuse an address, not just from an identification perspective but a security perspective, like, hey, I just bought something on Amazon and I used a one-time IPv6 address to make that purchase and clear that credit card transaction, and that IP address will never get used in the global internet, ever again. There is enough space to do that. So when you're having that kind of a conversation versus a oh, I got to make sure that nobody can get through my port forwards and my four or five IPs that I've got we're talking about entirely different worlds, operational worlds.

Speaker 2: 58:23

I totally agree with both of you guys. I think what we're getting at is there's obvious reasons why you should move to v6 and why it's great. I think what we're articulating here is why it's so fucking hard. There's so much more you've got to do. It's a lot of it's great. I think what we're articulating here is why it's so fucking hard, right? Oh yeah, there's so much more you got to do.

Speaker 3: 58:39

It's a mental. A lot of it's just mental. I'll use myself as an example. I remember I first touched IPv6 when I studied for the CCNP. I mean it was in the CCNP, the old CCNP, the route switch, t-shoot, ccnp, ipv6 in there and I mean, oh my God, I was like, you know, it was so foreign to me, it was so hard to work with. I think that was back when we had World IPv6 Day in 2011.

Speaker 3: 59:00

I remember that was going on and I tunneled in and I pinged stuff and felt awesome and you know, it was hard to learn and that was just learning it like in a textbook sense. Well then I, you know, spent another 10 years like actually learning how to implement it in a real network and all the things that go along with that of what are the operational challenges of integrating with software and in the ISP world, how do I make sure that I tie billing into IPv6 and make sure that the billing software is going to work with all this so that when you go buy a circuit, I know that it ties to your address and your physical street address and the router that you have? So there are all these challenges of how does IPv6 work in that world and I think for me the realization that I made and I was talking about this with Andy because we were talking in the CCNP Encore chat for people that were learning IPv6 for the modern CCNP and my advice was you just have to start using it and, whether that's tunneling in your house, use it in the cloud. The more you use it, the more comfortable you'll get with it, because the hex format seems scary but we use MAC addresses. That's always my oh my God. There's letters and numbers in there. But then you think we use MAC addresses every day as network engineers.

Speaker 3: 1:00:12

But I think it's the subnetting of it along. A MAC address is just a random string of numbers, right, like it's got to know UI and all that. But otherwise you don't have to think too hard about a Mac, it's just whatever it is. But you have to think about the subnetting of it and like I think that's what is scary to people is the very just the fundamental concept of you learn IPv4 subnetting and it's that's kind of hard when you first learn it, but you kind of figure it out and you understand the patterns. But then when you look at an IPv6 address you're like, well, when do I increment an A to a B, and when does a one?

Speaker 3: 1:00:40

become a seven and that's what's mentally hard as a network engineer to wrap your head around if you're doing it for the first time.

Speaker 4: 1:00:46

Well, and it's taking that V4 principle, thinking and thinking that it applies to V6, as opposed to just saying stay on the nibble boundaries and make your life simple right and don't take any of that baggage, going forward and understanding some of the baseline principles of how to build your address space, and also that you aren't operating in a world of scarcity anymore. I think that's really really, really hard. All of us are operating in a mode of scarcity in v4. Right, how much I gotta. I gotta design this subnet to absolutely fit whatever. I got eight hosts. I got to do exactly this in order to, or I got 17 hosts. I got to do like you're figuring out, you're doing all the bit level math to figure out what that subnet size needs to be. No, no, no, get rid of that.

Speaker 4: 1:01:29

V6, you've got a 64. You got a point to point link two addresses that you need, a and B right. Two. Link two addresses that you need A and B right. Two addresses out of a 64. You're like, oh my God, that's so wasteful. Okay, well, let's talk through some math again. You got 10 million hosts sitting on a LAN network. I don't even know how you get 10 million ethernet ports all connected together, but let's just assume we got 10 million biggest LAN party ever. You look at the rounding error of how many zeros you have. It's the same as using two addresses and using 10 million addresses, the number of zeros out to the right. It's ridiculous out of your two to 64.

Speaker 2: 1:02:04

Kubernetes is saying hold my beer, Hold my beer. Well, and so?

Speaker 4: 1:02:07

let's talk about that. Let's talk about the innovation and the opportunity that v6 provides. That is not there for v4. Scaling application services is a real-world problem for all of us, and doing this on a global basis with a global unique endpoint, where multiple Kubernetes, clusters, could be able to then spark services across clusters in different geographies and still have a unique identifier for each application that spins up. You can do that in v6. To do that in v4, we have to write three different layers of services abstractions to get that to function. That's the sort of brittleness that I'm talking about is that those state requirements are all in there, versus just a single update that says my service operates on this IP right, just route as routing and you're done. That's the opportunity that we have with v6. That does not exist in v4 and you can't solve with v4 as it exists today, and that's just the reality of just the laws of big numbers in terms of solving that particular problem.

Speaker 3: 1:03:11

Yeah, and I'll give you a good scale example, because everybody thinks scale. You think Amazon right, you think Amazon. You think Azure, you think Fortune 100. I worked on a regional utility company a few years back and they had 800,000 power meters that they had to network and that was. You know how many 10-dot subnets are you going to carve out for 800,000 hosts? Right, like that's a lot. And in IPv6, it was easy. It wasn't, I mean easy, it was still work, but it wasn't like it's not a big deal. I mean, 800 000 is a drop in the bucket of a 64, of a single 64 so you know when you're dealing.

Speaker 3: 1:03:44

And that's why I say scale is not just a big company problem. Scale is an everything problem, depending on what world you live in. And if you're a regional power company, you know that's um, that needs that, it needs to be able to, you know, to use that. And energy is big in this, like energy, not just power, but anywhere in energy and the industrial space, they're constantly have these scale problems, even if they're not, like you know, a Fortune 500 company. So I think that's you know, that's where IPv6 is the scale solution without a doubt in my mind.

Speaker 4: 1:04:15

Yeah, I mean there's some interesting side facts because of that exact portion of scale that Kevin mentioned. When we typically ask folks like hey, what's the most prolific routing protocol that exists out there period, Not just the internet but just as a routing protocol and I would say almost universally, everyone answers BGP. But that's actually not true. It's Ripple. It's probably the most widely used routing protocol, just because of how many end hosts are running it, because all these power meters are running Ripple in order to be able to provide mesh networks and v6 is a perfect alignment with being able to do that. But no one knows that these networks run or operate this way because the number of people that it takes to run and operate them is significantly smaller than the number of people that have to all agree on how the global internet routing table needs to work. So it's very interesting how some of this stuff plays out, because I think v6 in many ways is sort of a hidden adoption feature and enterprises don't necessarily see it in the same way. Adoption feature and enterprises don't necessarily see it in the same way. I would argue that many enterprises would probably be fine over the next decade not doing crazy accelerated work if their principal business is not to do work on the internet.

Speaker 4: 1:05:28

If your business is making money on the internet, not having a v6 presence will hurt you over the long run. Now, do you have to make everything v6? No, but the moment you turn up v6 presence will hurt you over the long run. Now do you have to make everything v6? No, but the moment you turn up v6 services, your app and qa and test team all and your dev team all need to have access to v6 to be able to test and validate and make sure it works. So then you start introducing more and more segments of v6 inside your, your network. Then your security team's like well, then we have to secure it and you might as well bring it into the fold and you might as well start using it. And so that's the natural progression that we see organizations take is really they're taking that first bite of the apple, saying we need to provide services. Some outsource it. They're like our CDN provider handles that for us, that's great.

Speaker 4: 1:06:07

And if their principal portion of your business is just to provide content, behind a CDN service for them to get to your website and that's the only way that you participate on the internet and everything else you sell doesn't have anything to do with that awesome. You're probably a really old company that isn't doing anything interesting and innovative. But if that's all you need in order to guarantee your revenue stream, I'm not going to tell you to use v6, except for to just go click the button on your CDN provider, Go, and Cloudflare probably already enabled it for you. Go, click the button in Akamai or whoever else you're using. That's fine, that's not a problem. But for those that want to innovate and those that want to grow their opportunity space on the internet, the internet is moving to v6 and leaving out 47% of the people on the internet as it exists today.

Speaker 4: 1:06:50

As Andy started off the top of the show with, If you don't want to address 47% of the market, okay, that's your choice right Now. If you want someone to proxy that session to you, INT Mobile and Verizon and AT&T are all going to proxy to get those folks over to you. You're going to deal with more latency. They own the customer at that point, not you anymore, because you're not providing a native service. You have to go through their service in order for them to get to you. If you're comfortable with that. If you don't want a direct relationship with your customer? Awesome, You're fine, no problem.

Speaker 4: 1:07:19

Go talk to your business folks and ask them if they would like to be in that position. We don't have a direct relationship with the customer. In other words, we only speak French, but they're all speaking English. But we're cool, we just have these companies are providing translators for us. Well, wait a second. We can't talk directly to our customer. No, we have to go through these other folks to do that. Are you comfortable with that? Do you not want to own it? Perhaps we should hire some people that are bilingual Suddenly. It's like will this impact our revenue? Well, I don't know. If suddenly we can't speak English anymore, is that going to impact us? I don't know. That's a business decision. It's not a technology decision anymore.

Speaker 3: 1:07:55

Not only that, but the other thing I'd say is that you know, going back to Andy's premise of well, I have NAT, isn't NAT and IPv4 good enough? Nat only works with IPv4 really well when you have a public endpoint to hit. Right Like it only works when both sides are NAT, like I don't know how much you've ever tried to like when both parties are not, like how many services it sucks right like there's not everything works when both parties are not. So imagine now I'm gonna, I'm gonna, uh, inject a thing that's happening. Now, you know, have you seen t-mobile selling their 30 internet? Have you seen verizon selling their 30 5g internet? Have you seen at&t selling their 30 or 40 5g internet?

Speaker 3: 1:08:32

You're not getting a public with those connections. That's the fastest growing. Fixed wireless access is the fastest growing segment of internet access around the world. Fiber's big, don't get me wrong. Fiber's out there. Fiber's a big thing, but selling the capacity of the 5G networks that they've built is a huge thing. And you've got Starlink in that equation. You've got the wireless ISPs I worked for in that equation, and so you're largely not getting public addresses. So NAT only works when you have a public address to go to, and that pool is shrinking right Like the pool of available addresses is just-.

Speaker 4: 1:09:05

Come on, kevin.

Speaker 3: 1:09:07

Upnp works great yeah yeah, I mean, and the people that are getting on these connections are starting to realize this the guy that's on Starlink that, like you, were talking about Ed gaming on CGNet.

Speaker 3: 1:09:19

If you're on Starlink and you're gaming with your buddy that's on the T-Mobile connection like you're not going to have the same experience unless you're native V6, which Starlink does have and you can be on V6. And most of the fixed wireless access have V6 native. You're going to have a better experience if you guys can ride across v6 and talk to each other than across the cgnat v4 gateways, because that's the fastest growing area of internet and so the landscape is changing as broadband changes and I think that's the other thing you're going to see out of the next five years is that people that say, hey, why am I paying verizon 100 bucks a month for fiber when I can go get this $30 a month 5G connection and it's great, and it is great until you got to talk to somebody else that's also NATed, and then your world's going to start to suck a little bit.

Speaker 4: 1:10:04

Well, and I think the other one for enterprises is now that the well. I guess it depends. I guess if you're working for Amazon right now, that's a work from home really isn't an option. But for those of us in the regular world, if you're working from home, you're more than likely your employee has both V4 and V6. Unless you're doing the work and the effort of really understanding how they're accessing resources, your sort of enterprise posture around zero trust may not be what you think it is right In terms of escape and just latent threat of the fact that V6 is operating.

Speaker 4: 1:10:37

If you're not putting into your security posture and your assessments and the work to actually control v6 for that endpoint in the same way that you're doing for v4, you actually don't have the same sort of secure posture that you think you do. And if you're only audit and logging and reporting on v4 related information, you're probably missing a ton of V6 related traffic that that you know end user is actually doing. And so my argument is you rather have it in the tent in the camp than out of, and so by putting it into something that you deploy, control and operate, you therefore are in the driver's seat of controlling what's going on. Do you have to be, you know, deploy everywhere? No, not necessarily, but it's nice to have those controls and understand how to how to operate it Right.

Speaker 1: 1:11:19

I feel like we could talk forever and still not like I. I've learned so much and this has been so great. And AJ is going to yell at me because we're over an hour, but I don't care. But I know Chris has a meeting coming up soon. He's on the other side of the world so unfortunately, I think we have to start wrapping up.

Speaker 1: 1:11:37

I learned a ton and there were like Ed, I've never heard anybody say through everything, a way that you know about V4 because it doesn't translate to V6, and just go in there clear-minded. I find that helpful because I think we do bring in our experience and our biases into things we're trying to learn. And if you come up learning one address scheme forever and then there's this new one, you are going to bring in what you know and they just don't. I'm going to try because I'm studying for the NP as well I'm going to flush out the V4 knowledge and go back to that V6 chapter, I think with a fresh mind, because there's just so much cool stuff and different stuff and unique things happening in V6. And my brain explodes because I'm like wait, this is so different than v4.

Speaker 4: 1:12:18

And Andy, since you just wrote your first Python for grabbing and doing config work out of a routing platform, it's the same thing. A CLI is going to be different in terms of how you interact and what you learn and how you think about interacting with that particular device, versus your Python code and how you think through code and how you think through libraries and how you're writing code sets to figure out how to gain that same information. You're doing the exact same thing. You're just showing a routing configuration, but the reality is the two processes are very, very different, and I think that's true for v4 and v6 in many ways, in that same sort of skill set and knowledge, it's different enough that you will make mistakes.

Speaker 4: 1:12:57

So don't do yourself a disservice and go back and learn the fundamentals. And to Chris's point well, you know we say OSI model, it's really TCP model, right? We get down to the real world of what's actually implemented versus this theory world of OSI. Osi is just a theory, TCP is an actual. It's a real. That's how code's actually written. Study that and really understand that. And when you understand the fundamentals of how neighbor discovery works and how the address protocol is put together and how extension headers are built and all of those fundamentals. It's the same knowledge and effort that you went in to learn before. You just forgot that you spent this much time and effort all those years ago putting into learning that and being an expert at it the way you are today. You have to do the same effort to become that expert in v6 so that you can make good architecture decisions and understand the impacts of the decisions you're making.

Speaker 1: 1:13:47

Well said, I almost quit my NA studies over v4 subnetting. So yeah, I just got to remember how hard it was and put half the effort into v6 and I'll be fine. Kev, you have any closing thoughts before we get out of here?

Speaker 3: 1:13:58

I think you know I would speak to the network engineers that are going to watch this and are going to say, hey, this is really hard, I just don't know how to wrap my head around it. There's so many resources out there that you can use to wrap your head around IPv6 and get into using it. Use the cloud. There's free resources, free training resources, things. There's certainly more today than there were a decade ago when we had to build tunnels into Hurricane Electric and all those things.

Speaker 4: 1:14:23

There's tons of stuff out there. Still use them, though Hurricane Electric's still great.

Speaker 3: 1:14:27

Hurricane Tunnels are great.

Speaker 3: 1:14:30

Yeah absolutely, but use your network modeling software, whether you're using Cisco Modeling Lab, eveng which is what I use GNS3. Use your modeling software, use your network operating system of choice and start playing around with it. Play around with it on the live internet. Play around with it and I promise you, if you spend even like an hour a week just playing around with v6, in a few weeks you're going to know so much more and feel so much comfortable about IPv6, just tinkering with it, than if you just keep staring at those, if you keep staring at the hacks and like, oh my God, I just don't want to even touch this with a 10-foot pole. So my advice is roll your sleeves up, dig in, don't be afraid to make mistakes and get into IPv6.

Speaker 1: 1:15:07

Great advice Kev Chris yeah.

Speaker 2: 1:15:10

I think, amazing points both from Ed and Kevin. And one thing I'll add to both of that is I think there's one thing that network engineers really value and that's predictability and control. And I think some of the things we talked about today can seem kind of scary, Like how are you going to predict when addresses are constantly changing and things like that? Right, so my feedback with the people that are just kind of maybe finding their feet and getting comfortable with the stuff, things that are not going to change and they're still going to be very important, no matter what, is routing and DNS so figure those things out, know them very well and and I think you'll be you'll be set for the long haul.

Speaker 1: 1:15:53

I love it. Thanks so much for coming on, guys. I learned so much. I don't think this will be our last V6 episode, because there's just way too much to cover and a lot to unearth there. Kevin, if folks want to find you, where are you at on the interwebs?

Speaker 3: 1:16:06

Yeah, so you can find me on Twitter X whatever you want to call it at StubArea51. Same as Drew on LinkedIn. I'm StubAA51 on LinkedIn and on my blog at StubberA51.net.

Speaker 1: 1:16:17

Awesome. Thanks, kev Ed. Where can folks find you besides the IPv6 Buzz podcast hosted by the Packet Pushers Network?

Speaker 4: 1:16:24

Yeah, you can look me up on LinkedIn. Ed Horley, I have a pretty unique last name. If you want, you can buy my incredibly old IPv6 book now because I wrote Practical IPv6 for Windows administrators. So if you feel like reading what my opinion was about IPv6 a decade or more ago, feel free.

Speaker 1: 1:16:42

Awesome. Thanks, ed and Chris, of the Cables to Cloud podcast, which you should definitely go check out. Very great show. Where can folks find you?

Speaker 2: 1:16:50

Yeah, definitely reach out to me on LinkedIn or on Twitter. I'm not on X, I'm only on Twitter, because that's the hill I'm dying on at BGP main. And, yeah, reach out.

Speaker 1: 1:17:03

Thanks guys. I'm at I don't know what the hell I'm at on Twitter Andy Laptev. I think it's Andy Laptev, I don't know. You can find me there. Look for Andy Laptev. I think I'm the only Andy Laptev on Twitter and probably on the internet Kind of a unique last name.

Speaker 1: 1:17:19

You can also find all the Art of Network engineering cool stuff on our link tree, link tree slash, art of NetEng. I like to remind folks that we have a Discord server called it's All About the Journey. I think we have about 3000-ish tech folks in there and I'm in there a lot currently studying for the CCMP. We have a great study group Kevin mentioned he's in there. There's experts in there that are helping the new folks. There's new folks coming in like what do I do? We have an NA study group, an MP study group and pretty much every other vendor just folks in there trying to learn and really helping each other out. It's all for free and everybody's just there to help each other, which is super sweet. So you can find that on our link tree as well as all the other cool stuff.

Speaker 1: 1:17:55

Great show, guys. Thank you so much and we'll see you next time on the Art of Network Engineering podcast. Hey everyone, this is Andy. If you like what you heard today, then please subscribe to our podcast and your favorite podcatcher. Click that bell icon to get notified of all of our future episodes. Also follow us on Twitter and Instagram. We are at Art of Net Eng, that's Art of N-E-T-E-N-G. You can also find us on the web at artofnetworkengineeringcom, where we post all of our show notes, blog articles and general networking nerdery. You can also see our faces on our YouTube channel named the Art of Network Engineering. Thanks for listening.