00:00
This is the art of network engineering  where technology meets the human side of IT.  Whether you're scaling networks, solving problems or shaping your career, we've got the insights, stories and tips to keep you ahead in the ever evolving world of networking. Welcome to the Art of Network Engineering podcast. My name is Andy Lapteff and in this episode, I am joined by Jeffrey Clark. Hello, Jeff.  Hello, Andy. How's it going? Good. It's good to see you. And rejoining us  is...

00:27
Probably one of the smartest guys I know, Kevin Myers. Hello, Kevin. Hey, Annie. What's going on, man? How you doing? Stubbarea51. What's it like being as smart as you are?  I don't know that I would say  that, but then people  like to stump the chump and ask you questions, and they're like, hey, what about this thing? like, I have no idea. I've never touched that.  Well, I know that you're very uh learned, educated in the topic that I wanted to cover in this episode. So in this episode, I would like to cover uh BGP, Border Gateway Protocol. Why?

00:57
I'll tell you why. I'm a career WAN guy. spent about, I don't know, 12, 13 years  managing gigantic  global WANs for huge companies. And  I consider myself pretty darn knowledgeable in BGP. We were talking on Discord, I don't know, a day, a week ago, and there are a few things in BGP I have never understood. I still don't understand. And no matter how many times  the brilliant Stub Area 51 Kevin Myers tries to explain these things to me, I still can't get my head around them. What would they be? IBGP?

01:25
Why? Why is that EBGP? one? Eh, IBGP communities. Communities are good. You tag stuff, right? But give me my RomEx a prefix list. Leave me alone. So like, I'm trying to understand IBGP communities and why. And the third topic, hopefully we get to by the end of the show is why in God's green earth did someone  I'm looking at you, Russ White's probably your fault. Why did we pick BGP to be our protocol of choice for routing and data centers? It's slow. It doesn't make any sense.

01:55
We have to do all this tweaking to make it not slow. I'm being half facetious here, but these are three things that I would love to uncover before we dive into those things that Andy doesn't understand for those living under a rock. Maybe somebody knew like when I have my CCNA and I got my first job in an ISP, BGP wasn't on the CCNA roadmap. I did not know it. The first question they asked me in my interview with Comcast was tell us what you know about BGP. I told them I can spell it. I know what it stands for, but unfortunately I didn't learn it.

02:25
So for those that don't know BGP, Kev, Jeff, what is BGP? What problem does it solve? And why did those two guys in 1989 draw it up on two napkins? Put it on the napkin. Well, so I think it's funny, first of all, that you were picking on Russ because Russ calls BGP the trash can of the internet because whenever we have a problem we can't solve, we stuff it into BGP  because Russ is an ISIS guy. He loves ISIS. And I'll let him come on here and defend himself. But we've had that conversation enough that I think I  can throw that out there. um

02:53
But yeah, so BGP was the  successor to EGP, if we're go way back into the dark history of the internet,  to connect autonomous systems. And I think you and I talked about this a little bit, Andy, because autonomous systems, think for modern networkers and networkers that are just getting into network engineering, they only think about autonomous systems as it relates to BGP. But other routing protocols talk about autonomous systems. If you read the RFCs,

03:20
because back when they were all being created, it just meant a routing domain that, you know, usually that you manage, maybe you didn't manage. It was just a collection of routers in an administrative domain. And so BGP was designed to connect those  autonomous systems together over  the internet or what, you know, became the internet. It was EGP at first. Then it became BGP. But that's where we started using BGP is we needed a way to connect all of these different networks together.

03:47
And that formed the early foundations of the internet. Now we've ended up using it for everything under the sun, not just the internet or not just connecting external autonomous systems together. But it  was a protocol that cared more about policy. And that's when you hear people talk about BGP, the Border Gateway Protocol, or the Bridging Gap Protocol. If you follow social media and that one outage we had a few years ago where they,  some news media outlet called it the Bridging Gap Protocol.  That was a BGP inside joke  forever.

04:15
Border Gateway Protocol fundamentally is a policy protocol that you can use to implement policy  and exchanging routes. And you contrast that to an IGP like OSPF or ISIS or EIGRP, they're very much topology protocols. They care about every link. They care about every path. They care about everything down to the most minute detail of how you get there and what links you get there. And BGP is the honey badger of topologies. It doesn't care. It just wants to get to its peer. It doesn't care how it gets there.

04:45
And so people can try to turn BGP into a topology protocol. There's designs like that. But at the end of the day, BGP is the policy that can best, the protocol that can best deal with implementing policy in exchanging routes between networks. That's its strength and that's what it does. That's what we use it for. When you say policy, I think route filtering, which IGPs can do well. So what's the difference between a policy and BGP or filtering routes in an IGP?

05:14
So there's a  fundamental problem when you get into trying to filter routes in an IGP.  Let's look at OSPF and ISIS. They both use the same base algorithm. They use Dijkstra's algorithm uh of shortest path tree. And when you run those protocols,  by default, if you're within an area, and we use OSPF because everybody knows OSPF. If you're within the backbone area of OSPF, all the routers need to have all the same routes in that area as every other router. So you can filter the routes inbound.

05:43
but you can't filter the routes outbound. So you're faced with all this problem of let's say you got 100 routers in an area and you have a route that you don't want to propagate. Well, you're going to go touch 99 other routers to go filter inbound to get that route filtered out if you want that route out of OSBF. Whereas in BGP,  can  build, I don't want to skip ahead to communities,  but we'll talk about that in a minute, but that's how I would solve that. That's how I'd have solved it in large enterprise  and in service provider.

06:10
is if I have routes that I don't want, like for a network merger, I did a network merger with several big publicly traded companies, I created communities that represented each company and where those routers met, we would filter those routes out. And I didn't have to go touch 100 other routers. I touched one router or two routers and implemented a policy that could be pushed out across all the routers. That's really hard to do in an IGP because you do have policy knobs. You're absolutely right.

06:36
There are things you can do to kind of approximate policy in OSPF or ISIS or EIGRP, but there's definitely limits and you have to start really rolling out the duct tape and your network design gets really kludgy and it gets really messy if you start trying to do heavy policy on an IGP. So you can't filter outbound routes in an IGP or you can't easily? this is where it gets tricky. In a lot of vendors, you cannot. I'm sure there's a vendor out there that makes  a thing that would break the protocol, but the problem is you're breaking the fundamental operation of the protocol.

07:04
So let's say you're in a multi-vendor environment and let's say you've got some flavor of router that can filter the routes outbound. There are definitely other routers that cannot filter it outbound. So now you've got a time bomb waiting to go off in your network because not everybody has the same routes and who knows what that behavior is gonna bring because you're breaking the standard of the protocol by doing that. So give me an example of  where you might wanna put that. You brought up companies merging  and.

07:30
Just kind of walk me through. So we had OK, so we were merging companies that had overlapped IP space. OK, so let's talk about it. Let's talk about IPv4,  RFC 1918. I won't even launch into IPv6 with you yet. Andy will wait on that for least three minutes.  so happy you said before  we'll talk before. So we had we had several large companies that had to get merged together.  And the engine, it's funny because the engineers that were working on it originally wanted to use just OSPF. They like.

07:59
Hey, let's just connect OSPF. Well, first of all, connecting two OSPF to IGP domains together that have never touched each other before is a really bad idea. You really need to know what you're doing because you have no idea what's gonna happen when all those different LSA types smash together. ah I've had outages, I've seen outages, I've seen meltdowns, I've seen flapping routes, because you may have a number of different things happening in the IGP because, again, IGPs maintain really complex databases of paths.

08:28
So if you have something weird going on and you're like, let's say you built a virtual link in an OSPF area that nobody knew about, and suddenly you have weird paths going on in OSPF, when you take those two companies and smash them together, really bad things can happen. And I would say statistically, when you take two big companies that have a lot of technical debt and you smash their two OSPF backbones together, more often than not, they melt down, then do they actually work cleanly. So BGP is an ideal protocol because BGP doesn't care about all that topology.

08:56
It just cares about routes and next hops, and you can filter inbound and outbound. So we say, okay, I'm gonna take my routers in the data center that I use to aggregate my WAN circuits, my private circuits, the L3 VPN circuits out to your MPLS cloud or whatever, and I'm gonna get a set of circuits  over to this other company that we're merging with, and we're gonna peer together, and we're gonna exchange some routes so that people without having to use a VPN can be in the company and get to resources in the other company.

09:25
But because we have IP overlap, let's say I've got 10.10.10.0 slash 24 in both companies. Well, obviously until that's renumbered, we can't share that network and that resource. So what I did with communities was I built a BGP community that tagged the routes and it specifically tagged routes that were in overlap. So I tagged the routes for each company, but then I added another community. Cause the cool thing about communities is you can add as many as you want. You can add a ton of them onto there for different reasons.

09:54
geography, hey, if you're in this community, this means that you're in Philadelphia, Pennsylvania is where this route originated from,  or you get this community because this route belongs to this company. But in my case, I said, hey, this community means that this is an overlapped route. And so anytime it would try to advertise outside that company and outside that ASN, it would automatically get filtered because it carried that community tag. And so that's where BGP was awesome because

10:20
I didn't have to touch 100 routers. I didn't have to touch a bunch of other things. I had two routers that talked to the other company through the circuits that went to the other company. And I was able to filter on that community. And you might say, hey, why don't I just build a route map and an ACL and I could just list all these subnets that I want. And you can absolutely do that. But by building it as a community, it was able to inform all the other routers that were sitting away from that in the entire autonomous system that these were overlapped routes as well.

10:48
because what ended up happening, like most mergers do, is other data centers got links between them, they had DCI get built between them, so all of these paths started existing where you had maybe three or four different ways that companies could get to each other. So now you've got to manage policy decisions at three or four points between two really big companies. And so when you build constructs that BGP is really, really good at, and OSPF and ISIS are not, to be able to carry the elements of that policy, like communities, which will dive into

11:17
you know what those are. But for now, I think it's safe to say just to, to articulate that a community is a piece of information that can be carried in a BGP route update that lets you take an action on a router, whatever that is, whether it's informational, filtering, permitting, whatever that is, it allows you to take an action or extract a piece of information from a BGP route. was going say, Kevin, it's a good point that you brought that up, that what is a community? We've been just, we jumped right into it. Sure.

11:44
without really explaining it. So, are communities a number? Are they named? How are communities built?  What is a community tag? So, community, there's three types of communities. There's standard communities, which is essentially like,  it's uh an ASN format. So, it's an ASN and then a colon, and another ASN is the way that it's supposed to be. But people will often not use the ASN. They'll often just come up with their own  integers. It doesn't necessarily have to match the ASN that you're using, although,

12:13
depending on what kind of routing you're trying to do, you might want to match the ASN that you're using. So a community is basically a, ah it's uh a two byte ASN format for standard communities. So you would have an ASN and then a colon and then another ASN. And then you decide what that means. Like you get to decide what it is that that means by building your filters and take action on that route. That's a standard. I have a question. Yeah, go ahead. What's up?  Are there pre like, so you said create your own rules  or like,

12:42
You're going to create a community, ASNs, and then you're going to define what that community is going to tag. Do they have, I don't know if I'm using the right terms, but like standard or predefined? are they just communities? There are some things in BGP that are called well-known communities, like no export. Yeah. So there are a few well-known communities. I can't remember all of them off the top of my head. No export is one where you can, like, if you set the no export community, then it's not going to advertise those to any other ASN. Then there's a few others out there. And just for context, before communities,

13:11
or at least before I knew what a community was, I'd create multiple prefix lists,  fancy ACLs. this was, know, IP prefix list, permit whatever, a different one, IP prefix list, deny whatever. And then in route maps,  you know, let these out, you know, deny these out. That's how we used to do it. And to your point, I think earlier, the prefix list would get really, really long. And invariably, if you'd have an A and a B side router, they'd get out of sync and then things would get to places they shouldn't.

13:38
Somebody did a change in the A router, not the B router. They forgot to put the prefix list in. So I guess communities are cleaner, right? If you look at the configs, you don't have long lists of prefix lists that you're trying to call on a route map. Is that one of the advantages? Yeah, you're still going to use things like prefix lists and route maps. But  the key point of communities are you'll often do that at the origin router that's advertising them. So you'll say, OK, I'm going to build a prefix list. And then I'm going to say, I'm going to build a prefix list and a route map. And I'm going to say, OK.

14:05
I'm gonna list these three or four RFC 1918 private IPv4 subnets in my prefix list. I'm gonna call it in a route map and say, you match these, set this community. And so those may be the subnets that are advertised on that router. By doing it at the point of origin, and let's say you got a bunch of other routers. You build those same community, you tag the same community. Let's say everybody gets the same community. Let's say it's a community for that data center. Let's say that you're gonna use ASN 65,000.

14:35
colon, let's do 11, let's use 11, because that's data center number 11, right? Whatever, whatever makes it work for you in the way that you wanna make it make sense. So then all the routers in that data center are going to apply 65,000 colon 11 to the routes they originate from within that data center. And then when you get to the routers that are at the border of that data center, whether that's the border of the internet, the internet has its own communities that allow you to do traffic engineering and some things with your upstreams.

15:03
But if you're gonna go within your network and you go to the WAN routers that take you to other data centers or other sites or things that are within your network, then you can build a filter or policy on those routers for those communities and say, hey, I wanna do something with all these routes that are tagged for this data center. Say, hey, I wanna let this go to every other data center, but I don't want it to go to Europe.  You can go to every data center and advertise out to every data center that's in the US, but not to Europe or not to this  place.

15:32
or not to this company we're merging with. So it gives you the ability to set actions and policies,  usually at points where routers are connecting, or data centers are connecting, major connection points in your network architecture, and then you build those policies of what you want it to do so that if you're setting it at the point of origin,  and you're setting it  as it's leaving the data center or going somewhere else or whatever kind of network you have, in the carrier world, we use it for  similar things, geography,

16:02
The you know, maybe some traffic engineering we might do some traffic engineering with a community and say if you carry this community then you're gonna go through, you know New York instead of instead of Philly right like on your path. It's a way to steer traffic So and enterprises do that too like data centers enterprises will use them for traffic engineering as well Depending on a few years back. We used it for a financial company They had a lower latency path for transactional stuff across the Atlantic to their stuff in London

16:30
And so we actually built more of a carrier style BGP network with EVPN.  And we put some communities on  those paths to basically say, hey,  if you're sourced from this router, these routes that were these services in the data center, then I'm going to steer you along this path because this is the lowest latency path for these transactional things to happen. And you're only going to use this other path  if  that path goes down.

16:56
So that was an easy way to steer that traffic without having to maintain this insanely long prefix list for every resource in the US for a financial company, which I know I think, I know you've got a FinTech background, Andy, I you've worked in financial enterprises before. So that allowed us to like simplify the policy  that was being applied to go  transatlantic.  And that's just it. mean, you could, the policy is whatever you make it. You can take that community and set what actually you can set local pref, you can set.

17:24
weight, you know, we can get into BGP and talk about, you know, BGP's attributes that it has, because that's, BGP has so many different ways that you can prefer a route. And then it has a list of  what we call the best path algorithm of, you know, so many things can tie or if this is greater than this, or if this is better than this, you can go through that BGP  best path algorithm and select a route. And these days, a lot of people just use local preference, which is one of the attributes of BGP that we use. I mean, you take a CCNA, you take a CCNP,

17:54
You know, people, you're gonna learn about weight and you're gonna learn about med and you're gonna learn about all these things that if you watch this show, you can go into Cisco's documents or Juniper's documents and go dig up all these. But at the end of the day, you'll  most often find in the practical real world that local pref is what is most commonly used. Not that you don't use the others, they get used, but local pref does most everything you wanna do for most ASNs. So let me try to summarize communities before we pivot off of them. So originating router, you might do a match set.

18:23
If maybe you have a prefix list like you said, and if you match this prefix list, set a community. That community is a tag basically, right? That somehow attaches itself to,  I guess those  routes. Is this a BGP attribute? It is. Yeah. So it's carried with the BGP update. Yeah. It's part of the BGP update. It's carried in the BGP route.

18:42
There's two other types of that's staged there, right? Does it get stripped off ever? It can, no, it absolutely can. that's a good point that you mentioned that you have to enable it in the peering to pass those communities because it's not going to do it by default. And if you're talking to an untrusted network, whether it's your ISP or maybe it's... gonna strip them, right? Yeah, or somebody that you're in a DMZ with that you're doing private peering with in a data center, a lot of times they won't accept them and they'll re-mark or re-impose their own communities on your route. So communities...

19:11
Sometimes they pass between entities, but they don't always. And so with all things BGP, it is almost always up to the administrator of the ASN as to how they wanna treat the routes. That's why BGP is so powerful. community seems so much, I'm sorry, community seems so much more elegant, because there's other ways to do it. Like we used to do a match set. So when we were trying to steer traffic in our way, and we would, I think it was a quick little ACL to match the next hop, like the PE.

19:39
So if you come from this address,  then set some metric tweak thing that we do so that we would go a different path. If you're coming in AT &T, go out the A side route or this side, Verizon B side route or that side. But it was,  you had these ACLs all over the place in the route maps and they would call this and then a different redistribution policy where the community, it seems more simple, it's elegant, it's just, if you're in this thing, here it is, it pretty much spans across your environment until it goes somewhere else and gets stripped. But communities seem like a much more

20:08
simple, elegant way to tag and treat traffic  as opposed to... They are, and I would also add they're safer.  The nice thing that I liked about communities when I was working as a network architect for a large enterprise is that if I were to use prefix lists and route maps for everything I had to go touch, well, I may have to go touch like eight or nine routers in the data center to go implement this policy or more or whatever it is I want to go implement. Well, only  engineers that have been around for a while are going to have any clue like what routers to go touch.

20:38
and what policies to go implement. But I could take a junior engineer that's like, you know, that's fresh off their certifications and say, hey, I want you to take these three lines of code or these three commands, and I want you to substitute this value. And you're to go in a maintenance window and you're to go paste these in.  And as long as it looks like this, it's going to be fine and it'll be safe. And then I can have confidence that the network is not going to get wrecked because the people that have the tribal knowledge of the network have already programmed what these things do.

21:06
You just need somebody to go in and put that value into the network. So they also make the network a lot more stable  and more predictable in the way that the network is going to perform because you don't have 10 network engineers just getting beer and a pizza one night and just go and have an ad it with route maps and ACLs. It's so relatable to my experience when  the same thing there was  two guys that had been around forever and they knew where all the bodies were buried and when we needed to change a policy or something, you'd have to talk to them.

21:35
you know, hey John, hey so and so. Yeah, where does this live? Why are we doing this? Yeah, and oh, by the way, this one has this weird thing we built for this BU one time, so you have to do it weird there too. And so you couldn't just hand a junior, like you said, just go do this policy with communities because it was very artisanal and it took a lot of time. know, hence all the maintenance windows I cried for years about on this show about, I was always working, because we just had to work all the time. Everything was so tedious because just...

22:02
the way we managed it. And then we merged. So  you mentioned earlier about a FinTech merger. We merged with another company. Guess what they were doing? Communities everywhere. It was much more elegant. They were also the ones automating everything. I'm like, oh hell, here we go.  What is this community crap? What are you doing with this Python thing? Like, oh no. uh Okay, so I think we beat up communities pretty well. Anything else we should touch on before we jump over to Yeah, I think the only thing that I would say, you know, I mean, this is, you know, we could talk for like 10 hours about this, but if you're wanting to learn about communities,

22:32
There's three types of communities that you want to learn about. There's the standard communities. Well, there's well-known communities, which there's a handful of those. You can go look those up. Those are published. There's the standard communities, which is the two-byte ASN format with a colon in the middle.  And there's large communities, which is for Andy's favorite, which is four-byte ASNs, which is a four-byte ASN colon four-byte ASN. um And then you have extended communities, which give you three, like basically four-byte ASNs delimited by two sets of colons.

23:01
And you can do all kinds of cool things with those. And again, it just gives you more options to dial  in and encode information that's helpful to you in a numerical format. Now, when you get in the ISP space, we stay more true to the ASNs because they matter, like they're public, they're registered. So we will often use ASNs, the real ASNs in there. But in an enterprise, you you can get away with a lot. You don't necessarily have to match the ASN that you're using in the communities that you build. So you have a lot of latitude to...

23:30
come up with something that makes sense to you and build a policy out of it. Well, in my experience with the extended AS, the extended communities, and maybe you can stop me if I'm completely talking garbage here, but we use them in SD-WAN. When like a hub and spoke topology, the spoke will, let's say there's two routes to my hub, maybe I'm going across two different IPsec tunnels, we'll tag those as essentially kind of in SLA or out of SLA so that if a link

23:59
from my perspective as the spoke goes out of SLA, may not be out of SLA from the hub's perspective, because really the  spoke is what really matters from the SD-WAN perspective. It's how's my connection to the stuff behind the hub. So we'll tag it with an  extended community telling it, hey, this is in SLA, use this link, you can send traffic back to me on this.  Or we'll send it with a tag that says, hey, this one's out of SLA, don't send anything across it. The link might be up, and from your perspective,

24:26
You don't care about jitter or latency, but I really do for this application I'm using. Yeah, mean, that's a great example of extended communities. Because you have so many places to put information,  you may have the same, the very first field may be the same ASN for everybody.  The second field may be, may be locational in nature or some other piece of information. Or in your case, the third piece of information, the third ASN  behind the last delimiter, that might be, like you said, your link preference.

24:54
or the second one may be your links and then your, like you said, your SLA value. They're used in so many different ways, far beyond what they were originally intended or designed to do because you can just encode whatever information is important to you and then build a policy around that. And so that's, I hadn't even heard of that for SD-WAN because I don't work in the SD-WAN space much, but I know SD-WAN relies on BGP a lot. Most, when SD-WAN was born a decade ago, I think most of the architectures that I saw were a lot of BGP and a lot of policy under the covers.

25:24
Yeah, it's policy route and a lot of IBGP. Yeah. All right. So really quick. Yeah. That's community stuff for us. Before we go to IBGP, we, kind of, Kevin is such a great guest because he just talks for like a minute or two and like 30 questions jump in my head. I'm like, Oh my God. we, we went from BGP and EBGP to communities to, to, so the question I wanted to ask earlier, but I didn't want to interrupt was which came first, the internet or BGP like

25:50
Was BGP a response to the internet and how are we going to manage all this? Yeah, the internet came first.  It was, it used to be something called EGP, which if you actually look in BGP's specification, uh there is actually a route type called EGP in there, although it only gets set by mistake anymore because I don't think, I don't mean the only routers that are even capable of talking EGP, you're like, you know, in somebody's, you know, they, have a lamp and they're using it as an end table, right? Like in their living room, like they're not.

26:16
The routers that talk to EGP are long, long, long gone. I we're talking, I don't know the exact timelines, but I want to say EGP was like mid to late  80s.  we started- So was a protocol predating BGP?  It was the predecessor to BGP.  so- the internet was running on EGP for a time? It was. I mean, the internet, like the internet had, you know, a number of different forms. You know, as we know it, it was more likely, uh well, it may have been EGP. I'd have to go back and look.

26:43
But you know, if you think about like, was just the government. Yeah, maybe go all the way back to like ARPANET and all that. I mean, we had public networks that were interconnected where they had, you know, they were they were doing in I think late seventies, right? Like mid to late seventies before any of these protocols even got built. I mean, they had proprietary protocols, things that they built to make this work. And then you had you had competing standards organizations in the 80s. So there was like this standards war. It's why we have ISIS and OSPF, because you had the IETF that was doing their standards.

27:12
And then you had the ISO that was doing their standards. And that's why ISIS and OSPF are so different, not to like, you know, diverge, but that's why things are so kind of, you know, sometimes you sit and ask like, why did they do it this way? And it's because you had two different companies that were both driving these different organizations for, you know, Cisco was heavily in the ITF. I think it was, who was it that was in ISO? I can't remember if it was DAC or Xerox or who it was, but one of the...

27:41
One of the heavy hitters for mainframes back in the day was heavily into ISO. So had all these routing protocols and these network standards that all competed against each other um for a while, just like IP competed against, you know, IPX, SPX and  SNA and all that stuff. So BGP came out of that era of that  80s era of all this competition and all this. We're going to make all the things and all the standards.  And EGP is what ran it for a while. And then it eventually became BGP. And I'd have to look up the dates. I don't remember exactly.

28:08
when that happened, but I want to say it was like late 80s, early 90s that we started transitioning from EGP to BGP. So BGP has been around for that because EGP couldn't handle the scaling that was happening? EGP was before my time. I never ran it. But my understanding is that as a protocol, like I've only ever seen it as a footnote on a page. I never ran it. I never looked at it. I never put a config on a router for EGP. My best understanding of EGP is that it wasn't designed for scale.

28:34
it didn't have any of the knobs or the, you know, it just did not have the  nerd knobs that we need and the capabilities that we needed to be an internet scale protocol as we know the internet today. I'm realizing as we're sitting here, all  of my questions are from my time in FinTech. So  right before we jump to IBGP, I have one more question. So you said you had two companies merging and they had overlapping IPs and you applied communities so that overlapping IP of company A wouldn't get advertised over to overlapping, you know,

29:03
company B, that semi correct? Yeah, no, it's a percent correct. overlapping IP. No, that's exactly. So how would they, so wouldn't that prefix or service be unreachable on the other side? Wasn't that a problem? And the reason I'm asking is how I saw it quote unquote solved, we put a gigantic honking net firewall between the two companies and that it tends to  11s. And I think it's still running like that because it's such a mess trying to deal with overlapping IPs. Now  converting to V6 would solve that, but I'm told that

29:31
that isn't happening yet. it does. Yeah, it does solve that. And I won't even go down that rabbit hole because I could talk for five hours. But that's what you should. I mean, that's you should do. Yeah, no. I mean, that's, in the and the nice thing about V6 that we use in the when you do mergers nowadays is you use it for management. So you use the IPV6 for management because if you have over if it's just management that you're overlapping, you build IPV6, they run in parallel. It doesn't matter if you know you make a mistake on V6.

29:58
because you still have your v4 and then you build a unique address space across the entire company. And then if you acquire another company, it's unique space. You'll never have to re-, like, it'll never overlap again. So that's the nice thing about IPv6 is if you're doing big company mergers and the big companies are already doing this and I'll get back around to your question here in a second, but since you asked, the big companies are starting to use IPv6 to consolidate their management. If you're a Fortune 50 or you're a Fortune 100, Ed Horley will tell you he works with those guys.

30:27
um when they have to, they  have IP overlap, they'll use IPv6 to reconcile it because why renumber something that you're gonna have to go back and keep touching and renumbering if you go through subsequent mergers and the cost of that merger in hours and all of that when you can just do one and done, because you're gonna get a unique address. But if you have hundreds of services sitting on v4  like subnets  and  customer routers,  it seems like such a big.

30:53
Well, yeah, so this is I'm talking about management when you're about service delivery. That's a well bad as I'm talking about just management like just manage because half is that a big problem to solve is mad. Yeah, because we had a lot of those. I'd say, OK, when I did that merger. And in fact, it wasn't just two companies we had. We had five companies merging. It started out as two companies merging. They bought three other companies. So we extended that design of the community overlap to three more communities. So we had five total communities.

31:20
on both sides of the Atlantic. had like two or three companies in the US and two or three companies that were in Europe.  And they eventually started getting connections to each other. And so we extended that community design. And that was where it really shone because if we kept if we kept up on the prefix list and ACL route, like we would have been there all day, every day, into the night, every weekend, writing prefix lists and ACLs to deal with these other three mergers. because we job security. Yeah, because we exactly because we took the time to build a more scalable, elegant solution.

31:49
we were able to apply it much more rapidly to the other three companies that were  coming in. So you rebuilt management on v6 and then just blocked the Well, if had done that, at the time, this was like a decade ago, but if I'd done that with v6, it would have been a lot faster. But to answer your original question, which was  those services become unreachable, right? If you're blocking them between the companies and they do what we did and every company, every enterprise probably handles this differently. But what we did was we said, look,

32:17
these services can't talk to each other yet because they're overlapped. We will have to renumber them or NAT them. And so what we did in the early stages of the merger is we allowed the routes, we blocked the routes that we were going to blow things up. And then we slowly worked through renumbering or NATing them so that they could, and if they needed to get to services, they had to hop on their respective companies VPN, you know, that was internal to that original company to get to those services. And then we slowly worked our way through having, you know, a hundred percent, you know, non-overlapped, everything was reachable, everything was good.

32:46
But the other cool thing it did for us, and this was kind of a side benefit, is when you went to go look at the routing table, if you weren't sure which routes were still in overlap, all you had to do was go look at BGP and go look at the routes that matched the community. So if you've got a big company and you had 100 routes that overlapped and you're, you know, every day sitting in there in the trenches working through that list, you know, renumbering stuff and doing that week in, week out, you can watch that list go down because you can just keep filtering on those BGP routes and say, hey, we...

33:14
We had 100 routes in this community, now we're down to 30. So these are the routes that we have left. And so it was a real time way for us to be able to say, this is how for merged we are in the network architecture, because we had that piece of information. Can you look at them like in the rib or you're like pipe matching like a community? Yeah, if you do like, you know, in Cisco, if you do  like show IPBGP, you can like look at various levels of detail. And same is true of, know, or Juniper or whatever.  You can, you can kind of pick the level of detail about a route that you want, you know, I mean, you can look at a very simple, you know,

33:44
Here's the route. Do you see the communities in the route table? Because I know you can see the ASNs. Yeah, so the routing table, depends on the vendor you're on. Some vendors will let you look at the community in the routing table by applying the right detail level to the show route command. But there's also a BGP database that you're holding these routes in. Sometimes you'll be looking at it there. And then sometimes you get really complicated in the ISP world where we're constantly stripping and re-adding communities.

34:09
Then you're looking at like, what communities did I have before this route got advertised out to, you know, Lumen or Hurricane Electric or Cogen or whoever.  And then what communities did it pick up as it was leaving the ASN? Because you have communities getting stripped off, you have communities getting added on, like when you get into the carrier space. So it gets a lot more complex because we're using communities  everywhere for everything, not just within like an ISP, but you're also using it to steer traffic through other ISPs or to filter routes.

34:37
or suppress routes by region or like whatever. So to your point in the question that you're asking,  most operating systems, Cisco, Juniper, Nokia, everybody,  usually has a way that you can kind of customize that output to show here's the community at this point in time or here's the community before I filter and here's what it is after you filter. So you were renumbering. That was your suggestion because you hate NAT, right?

35:00
I just want to get no I mean I had to use I had to use no you know it's funny I'm actually probably one of the IPV know why I'm messing with you right with no no  I the v6 they get it I actually or not forever Kate Nat I've actually done IPv6 Nat if you want to do that show I actually have done IPv6 now yeah I'll tell you why why did they build Nat into that so because if you have a if you have a subnet like let's say even mobile hotspot and they only hand you off like a slash 64 which is like one little size a small it's like one block of

35:29
you know, of land size space of IPv6 and you have to route it, there's no way to route it, right? Like you can't, you know, there's, if you're not, if you're not NATing, you can't route it because it's only, and this gets way in the weeds of IPv6, but if you have a mobile hotspot and you only have one subnet and they don't give you more than one subnet to like dish out other subnets out of that v6 pool, then you pretty much have to NAT it if you need to route it. It sucks, but that's why I can't you route a slash 64. That's a whole different show, right? It's too long. Well, because you

35:56
The 64 is the lowest that is recommended that you go for a host. I could break a 64 will break apart into like 70 gazillion more subnets. we don't do it. You advertise a 32. Well, we don't do it that way because generally all of the protocols around getting your host an IPv6 address are built around a 64 mask. OK. So while you could give it more, it's going to break a lot of stuff and you can have a real. It's a host mask and they It's the think of a 64 like your 192 168.1.0 slash 24. Right. Like that's what you're going to give.

36:26
you know, your Linksys or Netgear at your house.  64 is what you give in IPv6 for your,  you know, for most LAN segments that we use. And you really can't break that further down for hosts downstream easily. So you have to NAT it if you've got like a hotspot. That's one reason for it. There's a few others. If you want to do that show, I'll do the IPv6 NAT show with you.  You can get Ed back on. Ed and I will talk about that.  I'd love to have Ed back on. love Ed. So segue to IBGP. So we've been talking about EBGP.

36:55
up until this point, external BGP, which is out on the internet mostly, ASs to ASs talking to each other. I guess it doesn't have to be the internet. No, it doesn't. It used to be that Different ASs It doesn't have to be that way anymore, but it used to be that way. Yeah. So what I didn't understand, again, cause this is all about me and my, FinTech misunderstanding, but so we had two WAN edge routers running EBGP up to the same clouds. had the same routes in the ribs. They would send them down to a distribution layer. So that was handling the routing in and out.

37:25
We were running IBGP, I think, to share routes between those routers. Does that make sense so far? Is that something that normal people would do? Right. you're talking about border routers that are going out to the internet, which is I think what you're talking about.  They were MPLS clouds, but whatever. that's same thing. That's from a routing perspective. If you're going to an MPLS cloud, you're going to be usually doing EBGP to your carrier Verizon or Lumen or AT &T or whoever. So it's going to be a similar type of routing architecture. thing, all three VPN. Right. So in my simple brain,

37:55
If both of my routers have the same  routes, they do  because they're connected to the two ISP clouds up there, the L3 VPN that we bought and built. The routers have the same routes and then they're advertising those routes southbound to the distribution switch that's going to do all the routing in and out of the LAN to the WAN. I don't understand why we need IBGP to advertise those routes  across each other. That's my question. So why do we need IBGP?  in my specific use case, I don't understand why that

38:25
Is it a router to router connection? It's hard to say for sure without knowing whether routing protocols are in play and what redistribution is in play because everybody, the design that you just talked about... to EIGRP. Was that? BGP and EIGRP? Was EIGRP running between the routers?

38:41
The EIGRP was running on the inside of the distribution switch and BGP was running on the outside. an outside VRF with BGP, inside VRF with EIGRP.  internally the LAN was using EIGRP. I'm assuming you had a physical link between the routers that was your IBGP peering that you're talking We must have, right? You have to. Might have been a VLAN. I mean, it might have not been a physical link. It might have just been a VLAN. And maybe my use case is too weird and like not relevant to the people listening. No, here's where it gets really Why do we have IBGP? What's the purpose of

39:09
So what is the definition of IBGP? I'm looking at what you're talking about as a single use case for IBGP, but like to talk about IBGP and EBGP and more generalities in the old days, people would generally refer to EBGP as the protocol of the internet, right? Like people, weren't you running a of autonomous systems, private autonomous systems, they would generally run one, maybe two, you know, they'd have a handful of them. then, or if they had a public autonomous system, that's

39:36
registered through what we call an RIR, a regional internet registry. There's basically one for every continent uh in general. And that's who holds the IP addresses. If you need to get IPv4 space, public or IPv6 space or an autonomous system number and some other things, that's who you go to. They are the keeper of all the IPs and all the records. And so if you have something that is unique in a network that is uniquely numbered, they are the holders of it. so ASNs can be are numbered and registered. You can use that kind of an ASN.

40:05
And there's a couple different private ranges that we have. There's the, you know, 64,500 to 65,535 or whatever it is. Somebody will probably fact check me. I probably got that wrong, but it's like, it's about 1,023. right. I think that's right. Well, actually it's a thousand. Yeah. Yeah. I'm going to get nine. Well, actually is on that one, but it's a, yeah, I think it's 1,023 private ASNs under the two byte format.

40:30
And then you've got millions in the four byte format, which is at 4.2 billion. It goes from 4.2 billion to 4.2, 999, how many nines are in there, the range is, but it's millions. So if you want to build a private ASN that is not registered, which is just like private IPs, I can reuse an RFC 1918 IP all day long, and then I'll NAT it out if it needs to be public. ASNs are the same way. You can use them as much as you want.

40:59
But there are implications to doing that because IBGP is peering within an ASN, whether it's public or whether it's private. And EBGP is peering between ASNs. The biggest fundamental difference between them as the way BGP operates is the next hop behavior. So in EBGP, you're always going to rewrite the next hop as the route leaves the ASN. So if I advertise a route to you and you're in a different ASN, you're in Andy's ASN, and I'm in Kevin's ASN,

41:28
I may have a Next Hop that came from 10 routers away inside my network, but you'll never see that because I'm going to rewrite that Next Hop to the subnet that you and I peer on, and that's the only thing you're ever going to see. You're say,  in order to get to this route in Kevin's ASN, I'm going to go to the Next Hop of where I peer with Kevin, and that's all I know. That's all. But I will see the AS paths that... If I have AS paths that I leak out to you, you will. If it's over the internet and they're private ASes, they'll get stripped. uh

41:57
If you and I are running a private network and I have AS paths that I don't try to strip out in some way, if I'm doing a private EBGP and you learn that route from me, then yes, you will see  however many ASes I have in path because that's the way BGP works. Now, there's a bunch of knobs to influence that and you can do all kinds of magic tricks to change that path or filter that path and there's  all kinds of things that you can do. But in its most basic form, if you don't mess with it,

42:24
and you're not crossing a public ASN boundary, then yes, you'll see as many ASNs that are in path uh of a private EBGP peering.  it's basically just two different, EBGP is two different autonomous systems peering to each other and your next hop behavior by default, you can change it. You can actually change the next hop behavior to be unchanged and some designs that make sense, but by default, it's gonna change the next hop as it crosses that peering and advertises that route to you. That's.

42:51
EBGP and if a BGP router sees its own AS and the AS path that drops it that's by default You can change that you can do I think it's allow AS, know allow AS in on  a Cisco router But if you if you see your own AS and you want to allow it in you can absolutely do that Or you can why would you do that? That's isn't that breaking its loop? So  the the reason that you might do that is let's say for some weird reason Let's say that you have two different data centers in two different cities and you're advertising your public IP your public IP space out to the internet

43:21
Let's say for whatever reason that you have the same IP space advertised at each data center because you don't have much IPv4 space, right? It's gold. It's worth a ton of money. You don't have tons of it. So if you're going to advertise the same routes at the same two data centers, you might want to be able to get to a route that... no, sorry. Let me say it back. Let's say you advertise two different sets of routes out of those data centers. Two slash 24 is at one and two slash 24 is out the other, but you're using the same ASN. Now, let's say I want to build a VPN between those data centers.

43:51
If um I see that route coming in, in the global table, in the big, you know, we call this the global table, the route of all, it's all the routes on the internet. It's like a million, 1.1 million right now or something like that  for IPv4, 250,000 and  change for IPv6. It's what we call the DFZ, the default free zone, which is every route that exists on the internet for public space. But if you're advertising two different subnets out each data center from your same ASN, well, it's gonna advertise that out to your upstream.

44:21
your upstream will either keep it within their table or if you're doing using two different ISPs at the other data center, you'll learn that route back in your other border router and say, hey, here's this route. But oh oh, by the way, the originating ASN is the AS that I'm in right now. So it's gonna look at that and say, hey, that's probably a loop or something, a misconfig, I'm gonna drop that by default. So that's one of those fundamental rules of BGP. If you see your own ASN in the path, then by default, it's gonna reject that route.

44:46
But if I'm trying to run a VPN between two different sets of IP space, that is a legitimate route. I don't want it dropped. I want to be able to VPN out the internet of one data center and into the internet of the other data center. So that's a reason why you might decide to do that. need a whiteboard. So no, no, that's why I close my eyes when you're talking like I can kind of see it. We need We need the virtual. We need the virtual whiteboard. We can wave our hands as Greg Farrell used to say. We probably, we probably have one in here. So EBGP takes external routes.

45:15
Into our router and then  not  external routes I mean they're external in this they're external in the sense that they're coming from a different autonomous system But it doesn't necessarily mean that they're public right  it could be  it just means that you're gonna take in routes from another ASN Could be private coming in from your L3 VPN like through your your MPLS carrier for your private circuits  Or it could be from the internet. It's external only in the sense that it's coming from another ASN. So

45:42
We have to be careful not to conflate external with public. Yeah, that's a good call out. So where I was trying to go, and I don't even know if these notes I made are right, but EBGP brings in a route external to RAS and then IBGP distributes them internally. Correct. sound accurate? That's one of the things that it does. Yeah. So I guess in my brain, again, I'm old school. I would think I would redistribute that into whatever my IGP is. And then every distribution is bad and you lose.

46:12
things and metrics and all, but is that why IGP originally, like, oh, we got to take these externally, external to RIS routes and distribute them internally. That's the way we used to do it. That was the classic enterprise design. If you were to go crack a Cisco, when I first started doing like Cisco certifications and you know, these days I don't work with Cisco much anymore. So everything you're getting from me out of Cisco is probably, you know, dug around in my brain from, you know, a decade ago. But I work more on commodity and white box type stuff is, what, you know, I I've worked on.

46:42
in the last decade. if you go look at, everybody, you know, tends to use Cisco as when we talk about the history of how things came to be, they had a huge hand in how these protocols were shaped and deployed and the training of network engineers. So if you go back and look at the CCNA and CCNP guidelines of two decades ago, which is about when I was first starting, that is the classic enterprise design. You would have BGP out to the internet on a pair of border routers.

47:08
You'd have BGP out to some private circuits through one of your carriers that would connect you to other sites in the enterprise. And then it would get redistributed into OSPF or EIGRP or whatever it is on the inside of the corporate headquarters or the data center or whatever this site is that  you're dealing with. And that worked when networks were much simpler.

47:29
Like we didn't have the cloud, we didn't have  these, the larger enterprises had to deal with these problems. Like the really big enterprises were starting to have to deal with these problems uh when they  were building these big networks, but most enterprises did not. They kept things pretty simple and they didn't have like tons of data centers, right? Like maybe they had one data center or maybe they had like two data centers. Now,  your company may be split into a dozen colos and six different cloud instances.

47:58
So if you think, and then you may have corporate headquarters sites, you may have end sites, and I worked in enterprise retail, we had thousands of retail stores. As networks got bigger and more complicated and you had all these different paths that you had to traverse, you have more routers that are managing those paths. So let's say that I've got a route coming out, and let's say I've got a dozen or a half a dozen different paths that I could take to go get to this route.

48:26
You go back to the conversation we had about OSPF uh or ISIS or EIGRP. um Oddly enough, EIGRP is the best suited to deal with this. It actually had the most knobs for policy. OSPF and ISIS really didn't have as many knobs  for policy, but your policy starts to break down when you're dealing with IGPs. It gets really, really hard because when you're talking about link metrics like cost in OSPF, I could totally change the cost of one link and prefer it this way over this way.

48:55
But what happens when I say, you know what, I want these three subnets to come in from this path, but I want to filter out all the other subnets, and I want these 10 subnets to be preferred on this path, but I still want those routes to sit in backup on that path in case I need it so that they'll come active if this other path goes down.  That kind of policy decision is really, BGP is the only thing that can really get to that granular level of detail and say, hey, or maybe.

49:22
maybe I want to set four different priorities of routes. Say, this is my A path, my first choice, this is my second choice, this is my third choice, my fourth choice, my fifth choice. Like really only BGP can do that because you may take local preference, one of BGP's attributes that you can set, and you may set five different values for that route across the different paths. And so when you get down into, you look at where we are today, look at the complexity that we have in our networks now. Jeff works in SD-WAN.

49:51
SD-WAN has its own overlays, its own underlays, and then at some point SD-WAN is going to hook into a data center or back into the cloud. You're crisscrossing a zillion different paths and you need protocols  that can help to implement those policies. And at some point, you're not going to be all EBGP, right? At some point, it does not make sense to keep coming up with ASNs to use, which is your question about IBGP, which is why do we use this?  Because it's simpler.

50:18
IBGP, if I have a router, let's say I have a router that has 10 ways out, I have 10 different paths out of that router. If I  make everything out of that router a different autonomous system, that's 10 different peerings that I have to build out of that router and 10 new autonomous systems I have to manage. If I have an IBGB domain, I could have a pair of route reflectors sitting in that data center and then every router goes to the same two routers. So my config gets super copy and like cut and paste because

50:47
I can say, hey, these two core routers, the route reflectors for the data center, which is how we,  in IBGP,  one of the things that we kind of skipped over as we've raced through  BGP is that IBGP doesn't have the loop prevention mechanisms that eBGP does, which is the AS path.  So the way that IBGP prevents itself from looping is that by default, it will never advertise routes learned from one IBGP peer to another IBGP peer. So you have two choices to overcome that.

51:16
You can either full mesh peer everything together. So every router that's there becomes peered together, which means if I have three routers, then one router has two pairings to the other routers. But if I have 25 routers, then I've got, you know, what, 24 other pairings that I've got to deal with. So if you, the more routers you get, this problem grows exponentially and it becomes very, very hard to administer uh when you're doing full mesh. So we came up with this concept of a route reflector, which is a very, very special type of router.

51:45
that you put in the BGP config. It's just  a, it's a line of config you put into BGP on a router that says, hey,  I will let you take routes that learn from this peer and send them to this peer. And that's what a route reflector does. And so  because  the BGP next hop. Time out, out. Yeah, go ahead. what you got? Hold on. My brain exploded.  So I want to, I want to unexplode it before we move on. Yeah,  All  right. So the reason we have route reflectors,

52:14
is because  IBGP's route prevention mechanism is what again? won't accept a I think if memory serves, it's called BGP split horizon, which means that it will not, if  an  IBGP router,  imagine that you have three routers in a row, router one, router two, router three, okay? You've got a link from router one to router two and a link from router two to router three. If they're all in the same autonomous system and you only peer router one to router two,

52:42
and Router2 to Router3. Router2 is going to learn routes from Router1 to the left, but by default, it will never advertise those routes to Router3 to avoid a loop. Because it's a rule of IBTP. It's a fundamental rule of IBTP. was a cable connecting three back to one, there'd be a loop. Well, yeah, it doesn't always mean there will be a loop, the more you have of that, there's the potential. Because you don't have the... ASPath is not something that you can key in on.

53:11
So they said, hey, you either every router has to know about every other router so that BGP understands  all the routers and all the routes in the IABGP domain. Which is a nightmare and not scalable. Well, mean, oddly enough, I don't want to go down another rabbit hole, but some of the we thought that full mesh peering was a nightmare and would never be done. Some large transit carriers, which are like the people that are just all they do is sell Internet to each other. They're not not the kind of ISPs that are going to come out to your house and put fiber at your house.

53:38
They only exist to connect other ISPs and enterprises together. That's all they do is sell transit. Like Hurricane Electric's good example of that, or Cogent. They sell some other stuff, but mostly their  goals are,  or Telia,  or now, are they, Arleon now. There's carriers that their primary purpose  is to connect other carriers. Some of those carriers have decided that because of automation, because we've jumped into this world of automation, that I don't need to be scared about full mesh peering. I'm just gonna let Python and my automation engines

54:08
build all of those peerings  because route reflectors at scale have their own problems. They come with their own technical debt and their own baggage at scale. And so  now that we're in the world of automation and AI and all of that, people are looking  at problems in network engineering and design and saying, you know what, 10 years ago when I had to manage that as a human, I might not have done that because that would be a horrible choice. But if I'd no longer have to manage it and I can offload and abstract that complexity to something else that can manage it, I might consider that design.

54:38
So even though  in every Cisco book you'll read nobody does full mesh and full mesh is bad and you'll never see full mesh IBGP in the real world of large carrier operations there actually are a few ISPs that are doing full mesh because they don't want route reflectors. There's a tolling on the... Yeah. So it's... ...gotten good enough. This is where, yeah, this is where it's... ...and it's hard to know when you're learning what Cisco says in a book and what actually happens in the real world of BGP operations. Sure. Sure.

55:06
Okay, so I think we unexploded that part of my brain. Thank you. And then just to come back because it was after my brain exploded, route reflectors are advertising the prefixes learned in the AS out to  the other. Yeah, you're basically. NOSPF. So they advertised to other IBGP peers.

55:26
So the route, you're doing is, so imagine- So all the peers don't advertise to each other. Imagine a star, let's imagine a star topology. So let's take the network that I just talked about and let's imagine a star topology where you have router one in the middle and then you  have a spoke from router two to router one and then router three has a spoke into router one and you have this  basic hub and spoke  type star topology.

55:50
there where everything goes to router one. Well, if you set router one as the route reflector, which is just a line that you set in the BGP config, then the routes that it learns from router two  can then be sent over to router four because the route reflector command in the router allows you to override that behavior  because that router is managing all of those routes  and it, you by default, not to say you're never gonna get a routing loop because they do happen, but that is how they take the

56:19
problem of every router needs to peer to every other router and solve it in a little more elegant way is by implementing what's called route reflection, which is exactly like it sounds. It takes a route learned and it reflects it out to another router. And so those are your two options in IBGP is full mesh or route reflection. We're almost out of time and I don't know if we want to get into why the hell did we ever choose BGP in the data center, but I do want to note that the first two topics we touched on communities and IGP at some point in those conversations.

56:47
It was revealed to me that we use them because they're simpler. So I'm sensing a theme.  Well, it is simpler than the other stuff. I'm asked prefix and it's awful. Simpler is perception. Simpler is perception. A junior engineer would look at a community and say that is the most complex thing I've ever seen. A seasoned engineer will look at the holistic view of the entire network architecture and say, yes, I'm going to have to put a little more work into this on the front end. But when it gets up into operation, it is way simpler to operate.

57:16
So simpler is a matter of perspective. If you get a CCNA and you throw beach If that poor bastard can over...  Yeah, their head may explode. But if you've done this for a while, you can say, yeah, that's the way to go. Well, you have to overcome the bias too that like, I'm familiar with prefixes and route maps. I'm familiar with V4. I'm familiar with doing it in the CLA, like, know, automation, V6, everything you're talking about.  It's simpler, but for me, and I know I'm not everyone, but it's just,  there's that kind of...

57:44
familiarity and this inertia of like, I got to do it a different way. And like, yes, I know it's better.  But I know this over here. We're gonna say Jeff,  I was gonna say, I think it's it's similar to, you know, kind of that wall you've talked about your slowly overcoming with some of your programming stuff. Yeah.  When you looked at Python originally, or you looked at, know, expect or you looked at  whatever  scripting language bash,  they all seemed really complex.

58:11
But once you've solved what you could do if you automated a process, what you really realized was it was a lot easier to write that script once  and then  repeat this task a thousand times. Because the script doesn't care how many times you have to do it. The same thing is true with BGP in that because of the flexibility you have with BGP, it means that you can sometimes programmatically get around challenges that other routing protocols present.  Or scenarios like, know, like Kevin was saying, you know, when, you know,

58:41
two companies merge, the only way that most new engineers would know how to fix it is with NAT. But an engineer that understands BGP, there's, I don't know, least two or three different ways I can think about the top of my head within BGP alone to be able to solve for that same problem. I think it's a very programmatic language. don't know. I'm saying- I feel attacked. NAT forever. I'm kidding. Yeah.

59:07
I just mean that for me is why BGP has been a protocol that while I am no Kevin, that's for sure, I'm sitting here learning a ton. No one is. It's one of those oh programming languages that I'm just,  or programming languages, good grief,  the grounding protocols that I'm just really fascinated by because of the flexibility of it. You just reminded me, somebody said something,  isn't there a trope or a meme or something that BGP is an application? Yeah, so.

59:36
So here's the thing about BGP. If you want to get view, BGP is like,  is probably one of the biggest like, it's probably one of the one of the biggest like holy war religious debates you can get into in network engineering.  I mean, I'm not kidding. It's everybody has their view on,  you know, what it is. And it depends on the level that you, that you operate at as to how you view it. A carrier person is going to view how you should do it one way. And when you, when you,  when you start to get into the depths of

01:00:03
how is the protocol classified? I asked this question with ISIS and you guys are calling me the smart guy. I look at people that designed these protocols or had a hand in it or like Russ or Ivan Peplenac or there's a lot of other people out there that really know these protocols far better than I do. I've learned a little bit of everything in 30 or 40 something years of doing this stuff. There's people that really specialize in deep dive in this and I'll go ask them those questions. These, hey,

01:00:31
Back when the IETF was developing this protocol, why did we do it this way? And I asked that question of ISIS recently, and I said, hey, I'm reading this and this doesn't seem like everybody says it's this way, but it doesn't seem like it's this way. And my God, the smartest people on the planet on  these protocols, the people that helped like build those protocols weighed in on, was Twitter back then, they weighed in on Twitter. I mean, man, we had a full on holy war of perspectives  on what it was.

01:00:59
BGP is the same way. So your question about is BGP really a routing protocol or is it an application level protocol? Because somewhere, I read this 20 years ago, somewhere in a Cisco book, somebody classified BGP  as an application layer protocol and not really a routing protocol. And there were like two paragraphs of text. It was either the CCNA or the CCNP a couple of decades ago that had this like short little, you know, hey, BGP.

01:01:28
is, you know, functions more like an application than it does like a routing protocol. And man, you're going to get you could go get you could go get 20 of the best BGP engineers on the planet in the same room. And they're never going to agree on that. They're all going to view it and  not to draw it out. But the reason for that is, remember when I told you about the 80s about everything, all the standards competing against each other. The reason it's so hard to classify some of these things is we took some of the stuff from ISO. We took some of the stuff from the IETF. So like the seven layer OSI model.

01:01:57
You know, that comes from the ISO. It didn't come out of the IETF. So that's why it's sometimes hard to figure out, like, what layer does this operate at? Like, I was asking ISIS, what layer of the OSI model does this encapsulate ISIS routing at? They're like, well, it doesn't because it didn't come out of the IETF world. Like, that's why it doesn't fit cleanly. It came out of a different world and it came out of a different, you know, a different model. BGP is the same way. You're going to get people that are going to look at

01:02:25
the choices that were made when BGP was developed and the arguments that people had about how they viewed what BGP should be and how it operates. And you're going to always get two or three different views  on is BGP a routing protocol or does BGP function more towards the application layer as an application. I don't think you'll ever get 10 really smart engineers in the room to ever agree on that is what I found. It's part of why I love technology because it seems like you can go infinitely deep.

01:02:52
on any one of these things, BGP in particular, because I think I'm good at BGP. And then I talked to somebody like you or somebody like Ross or somebody like Ivan. Like it just keeps going and it seems to be a never ending uh journey. I'll tell you a funny quick story about that. Because I consider myself decent at BGP. There's a lot of people that protocol wise are probably better than I am, but I've been around it for a long time. I worked in the carrier space. I went to networking field day a few years back.

01:03:19
We went to one of those social events that you do after hours and they had all the vendors there presenting  and one of the guys comes up to me and said, hey, you asked a question about BGP in the presentation. I want to talk about that.  And this dude sitting here talking to me, having a great conversation and about five minutes in, I'm like, man, I've been doing BGP for 20 years and I'm not really sure I know what this guy is talking about. So I'm like sitting here, you know, a couple of beers in trying to keep up with what this guy is saying.  And he's like, you know, he's like, you get this, right? And you get this. And what about this? Well, after 20 minutes of that.

01:03:46
You know, I was like mentally exhausted and only later did I learn the next day he was one of the original inventors of BGP, like one of the original guys on the RFC  and like one of the original guys that helped code it at Cisco. And I'm like, well, damn, now I don't feel so bad because, you know, this guy like helped create the protocol. But like, that's the thing about network engineering. I've been using it a lot. I know a lot of stuff about it, but I'm certainly not the smartest guy on the planet as evidenced by, you know, Cocktail Hour in Silicon Valley with the guy that invented it.  But, you know, that's the thing about it is there's always somebody

01:04:16
we have our circle of knowledge, right? And you talk about imposter syndrome, you only know what you know. Somebody else may know a whole lot about this piece of it, but not about the piece that you know. So you always got to kind of try to keep that in perspective. Everybody knows their circle, right? I was going to say, okay, you saw my face. I was going to say something, Andy. You took a breath. Go ahead. Yeah, one of the things I was going to say, Kevin, this has been such an interesting topic, especially for us on the right of network engineering. And this is all about network engineering.

01:04:45
We're all network engineers, people that watch it, the people that are on it. think some, of the things that we  often have  missed maybe in this show  is some of these deeper conversations about things like BGP. So this has been really good. And I guess  I would ask to the larger A1 community uh as you're watching this, if this is the kind of content you want, this is the kind of content that's really fun, I think for Andy and I to get to have conversations with people like Kevin who,

01:05:13
blows our mind. Can you talk about the guy that created the BGP? other guy that created BGP blew my mind. Yeah, exactly. built a lab for this episode that we didn't even get to. Well, I to power the lab off. had back. to power the lab off because you were like, I got some Because I complained about the noise. the lab's dead, Yeah, I mean, I think directly going is, are we going to do more of this, Jeff? Is that kind of where we're headed here? Yeah, I mean, yeah, I'm interested in doing more of this. Kevin, think you've even said in our Discord chat,

01:05:43
previously that that was something that  you'd be up for. em Again, I would put it out to the A1 community as a whole  and really ask that because we do this stuff for you guys. em But if this is the kind of deep conversations you like, these are really fun to do. And they're a little outside of Yeah, we've had listener surveys in the past and it's usually a 50-50 split.  Half the respondents were like, we want more technical content. And half of the respondents were like, we want more career journey. So you try to do it both to  kind of serve.

01:06:12
Both of them and we try to do that in the episodes, but I  personally and especially having access to somebody like Kevin. uh I know that we could do a whole series on BGP. Kevin knows a ton about things like ISS and MPLS, so  I intend to bring Kevin back on here as much as we can.  If we didn't get to BGP in the data center, so that'll be a topic for the next BGP uh episode and we'll have to fire up the lab and look at some stuff Kevin created.

01:06:39
four-byte ASNs just to hurt my brain and make me feel like an imposter. I did get to see some of his configs and I'm like, oh my God, they're scary. But no, Kev, it's been a pleasure.  I've learned so much.  Where can people find you if they don't know who you are and where you're at? Yeah, so I'm pretty easy to find on  internet, on Twitter slash X, I'm at Stubbria51. You can find me on LinkedIn at  Kevin Myers. out there and I've got a blog that... um

01:07:06
I maintain at stubarea51.net. So those are the main places that you can find me. Jeff, where are at? How do people find you? Portageepp.com takes you to my LinkedIn page. Or I can just go to LinkedIn directly and look for me. Jeff Clark. Oh, look at you. Get a little redirect That's where I spend my time, yeah. For all things Art of Network Engineering, we have a Linktree.  Shout out to AJ Mary, who built that. It's amazing. It condenses all of our stuff in one little page. Linktree forward slash art of net eng. It has...

01:07:31
All the pod catchers on there, all of our socials. There's a merch store that we need to update, Jeff, one of these days very soon.  You can join our study group  called It's All About the Journey.  think you  named that. Was that you? Am I misremembering? Yeah. You are the reason it's called It's All About the Journey. So I think it was the same networking field day that like Mr. BGP inventor like blew my mind. AJ and I met at that networking field day.  And so, uh you know, he and I like connected on social media. And I think shortly after that, NFD

01:08:01
I think he was working on CCNP and I think he failed one of the CCNP exams and he was really down. He's like, look, I worked really hard on this. We were talking on Twitter at the time and I was like, look, man, it's great to get the pass, but ultimately it's about the journey that you're taking as a network engineer to learn the protocols and the knowledge. Because at the end of the day, mean, yeah, the certs get you in the door and they're great. I'm not trying to down the certs or anything like that, but...

01:08:30
the knowledge that you get and the journey that you take on  teaching yourself how to learn, because that's what network engineering is about. You have to learn how to learn, right? And you don't even know what you don't know  when you're first starting out and you  start to know what you don't know a little bit more as you get more senior. But that was what my comment to AJ was, you know, it's more about the journey than it is about the destination. And, you know, the further you get network engineering, you realize that. And I guess he kind of reflected on that as the comment that I made.

01:08:59
and put a tweet out about that, that, this is kind of like, you know, it a mindset shift for me and posted that out. And then All About the Journey became the tagline of Art of a Network Engineering. And not to take anything away from AJ, because AJ built this place. It's his baby. But yeah, I did play a very small, modest part in the All About the Journey part of it. So he took that insight from you.  It moved him internally to keep going, even though he kept failing the exams. Created the Discord study group that you can join in our Linktree. Tweeted about it.

01:09:29
which is where I had,  I saw it because I was also failing the CCMP exams  and it brought us and a few other people together. The podcast came out of it and here we are all these years later and 900,000 downloads later and just a wonderful community around us. So it just dawned on me as we're talking like, wait,  Kevin,  name that, that's amazing.  So  long way to go that the link tree for it slash art on that edge, all kinds of cool stuff. And we have,

01:09:56
A new addition in there, there's a support the show button. I've had people reach out and say, hey, I love the show, it's great, how can I support you? We used to have a Patreon, we got rid of it for reasons, so we just put up a support the show thing. You can find it on our link tree, it's in all of our show notes. If you want to support the show, you click on it, buy us a cup of coffee. It's been a great conversation. I hope to have many more, Kev, with you and Jeff. Thanks so much for being here, and we'll catch you next time on the Art of Network Engineering podcast.

01:10:22
Hey folks,  if you like what you heard today, please subscribe to our podcast and your favorite podcatcher.  You can find us on socials at Art of NetEng, and you can visit linktree.com slash artofneteng  for links to all of our content, including the A1 merch store and our virtual community on Discord called It's All About the Journey.  You can see our pretty faces on our YouTube channel named the Art of Network Engineering.  That's youtube.com forward slash artofneteng.  Thanks for listening.