A.J. Murray: 0:24

Welcome to the Art of Network Engineering. I am AJ Murray and we are back for another exciting episode. Joining me this evening is Andy. He is at Andy Laptef on Twitter and other social media platforms. Andy, how are you? It's good to see you.

Andy Lapteff: 0:38

Hey, aj, good to see you, buddy, I am well. What am I doing? I just published another piece of content for my employer, which always makes me happy that I get to create content and have that, is it anything public?

A.J. Murray: 0:50

It can be direct. It is public.

Andy Lapteff: 0:52

There will be something coming out around design that my employer is putting out. That's exciting. Hey man, I am getting to do some fun stuff at work. I won't talk about seasonal stuff because this God knows when this will come out it might be the summer and I will be talking about the impending fall. Things are good. I will give a free shout out to the USNUA. I went to the PA Network User Group first inaugural meeting and it was really nice. I got to meet Drew on Ray Murray from.

A.J. Murray: 1:21

Pack of Wishes. I saw that.

Andy Lapteff: 1:24

I was like fanboying. It was nice to get together in person with other network folks and just talk shop and what's going on in the industry. I'm grateful that I'm remote and have been remote for four or five years, but I also really, really enjoy being in person with people in tech. That was really nice. Man, how are you doing? I'm doing well.

A.J. Murray: 1:45

I want to extend the shout out to the USNUA For our listeners. If you don't know, the USNUA is the US Network User Association. They help form network user groups or nugs throughout the country. It's just a bunch of network engineering nerds getting together, talking, usually enjoying some free drinks and food. I myself have our next meeting coming up next week. It'll be the first week in November. We're looking forward to what we've already got just about 40 people registered. It should be a good size meeting. We got some great panelists lined up. We got some good topics that we're going to talk about. It's just fun to do these local events. It sounds like you had a great time there at the PA Nug. I'm looking forward to doing my second Nug, getting these people together. It's time to solve problems. Everyone's just like hey, I'm facing this, how do you deal with that? We have this problem. It's really cool to see and participate in.

Andy Lapteff: 2:43

One of the reasons I love the show is we got to interview Jason Ginter, founder of the USNUA, episode 116. Just cool people that I get to meet here, which then months later become this event that I'm at that just I don't know. I love what we do here. We have a very cool guest tonight.

A.J. Murray: 3:00

Yes, Annie why don't you introduce?

Andy Lapteff: 3:01

him, my good friend and colleague, Sean Liggan, also guest on the show previously, Don't remember the episode. I don't have it in front of me, Sean. Sean is an avid listener to our show. At one point he came to me and he's never done this before. That's telling. But he's like dude, I have the perfect guest for you. You have to have Doug Midori on the show. I'm like all right man, I trust Sean. He's a smart dude, he knows what he's talking about. I started looking up Doug and I'm like, oh my God, we have to have Doug Midori on the show. Joining us tonight is Doug Midori. He's been described as the man who sees the internet. What's up, Doug?

Doug Madory: 3:40

Hey guys, Thanks for having me.

A.J. Murray: 3:42

Thanks for joining us.

Andy Lapteff: 3:43

Thanks for coming on. Where do you work, doug?

A.J. Murray: 3:45

I work with Kentec Kentec yes, so you work with our friend Phil.

Doug Madory: 3:50

Yeah, yeah, we're on every day.

A.J. Murray: 3:53

So, doug, to kind of start out and kind of level set, can you give us the brief on your career? I mean, so today you see the internet and we're going to dive into what exactly that means, but you obviously have a relationship with networking. So what kind of got you into networking and how is it that you have come to see the internet the way that you do today?

Doug Madory: 4:15

Okay, let's see when to begin. So I got my undergrad at the University of Virginia. I got a degree in electrical engineering, computer engineering, and I attended school on an Air Force ROTC scholarship. To pay for school we didn't have money and my first job out of school and in the military was as a network engineer at the base of San Antonio, texas, and I would say that's probably where I really learned. I mean, I had a lot of computer science classes in school. I think I had one networking class. I don't think I knew it that much out of my undergrad.

Doug Madory: 4:57

But then, yeah, the group I was with we had a Cisco network. It was very, even though it was in the military, it was very similar to what a networking IT shop would look like and be doing. So we had a lot of money for training. So they would send me the one week Cisco classes, did my certifications and then after about a year of handling catalyst 6500s and stuff. Then there was another group that did UNIX administration and among the officers so I was an officer, a communications officer the officers that were there, nobody wanted to deal with UNIX. They were like, ooh, I don't want to, that's weird stuff.

Doug Madory: 5:41

And I was like I would love to do that and they're like really. And they're like, yeah, I'm like now you're in charge then. So, anyway, I got to do that. So, yeah, that was probably my first indirect introduction where I really liked the Cisco academic process. I thought it was really well written. I learned really really quickly. If you could apply yourself, you could pick it up pretty quick. I still probably draw on that knowledge Some of those things now. But yeah, and then I did that was in the military.

Andy Lapteff: 6:14

You were in the Air Force. I was in the Air Force, yeah. So I don't know if you can answer this without killing me, like, what kind of networks are you managing in the Air Force? I'm going to be just are you on a campus, kind of stuff.

Doug Madory: 6:27

Yeah, that one that one is more just a campus. So I was with the Information Warfare Center in San Antonio, although it's changed this name maybe six times between when I was there and now. I don't know what it's called now, but it was as a intelligence unit and, like the AFSERT, the Air Force assert was attached to it and yeah, so anyway, so they had a. There were people who were doing information operations and then we were actually just responsible for the networking itself that enabled that Again analogous to a networking group. But yeah, it was a campus and yeah, I was doing everything. I was pulling fiber through ceilings and installing equipment.

Doug Madory: 7:17

So it was a good, it was a great place to learn. There were some other smart people in our group and I also learned just the value of having a team. You know, when you're in that kind of environment where you've got a bunch of smart people you're working with, and how much faster you learn when you've got good, smart colleagues that are motivated. That's just a you can't put a price on that of just how like no class, there's no book you can read, there's no class you can take that's the same as having to do the work and having good colleagues that you can learn from.

Andy Lapteff: 7:49

I feel like outages would be super stressful in like a military environment. Like you said something about warfare, like I figure there's pretty critical services, or at least yeah, I mean communications happening.

Doug Madory: 7:58

I think I think you learned in that environment. I mean we had there was still let's see, this is beginning in 1999. So there were still operations in the Balkans going that were. There was, you know, some criticality to it of we had flights going over, mapping out the like radar jamming stuff that would come back They'd figure out the signature and change the way that we do our jamming of their jamming and stuff Anyway. So yeah, those yeah can't go down. Those kind of things went into, other things can and you just learn.

Doug Madory: 8:36

And then you know my second job was I did a good job in the first assignment the first two years in San Antonio, and then they kind of rewarded me with this command position that I was maybe too young to have, but you know I made it work. And then for the next three years I was a flight commander in Aviano Air Base in Italy on a unit that did deployable radar and air control. So basically we had guys sitting in front of scopes talking to pilots, talking to the military of pilots through their missions, and then we had radar, like fold up radar that you could like put on the back of a truck but still had. 300 mile radius Vantage is just like far greater than anything you get, like ATC, and so there's a lot of communications needs of the operators which are the guys in front of the scopes, talking to the pilots, talking to ATC, talking to other radar units, and yeah, in that case communications are pretty critical. We would have a satellite link out as our main connection to our deployed site. The whole thing had to be able to be packed up and deployed somewhere in 24 hours and that's like 100 people.

Doug Madory: 9:56

So this is like not a small thing, it's something we just have to practice all the time. We practice even to ourselves. We would give ourselves a no notice thing and just call everybody at 3am and be like everybody come in and we're packing up and you don't know if this is real or not. So come in and then midway through the day like this isn't real, but here's your grade on how well you came in the middle of the night to come, start packing up. So as an officer in that squadron, I think I got good at trying to anticipate how things are going to go wrong as we plan out either our exercises or ultimately, our deployment to Iraq, which I think is just Sean was in. You mentioned Sean. He was one of my airmen at the time and it's probably 20 years ago maybe today, I don't know what the day was that we went, that we ultimately went to Iraq in the first year of the war.

Andy Lapteff: 10:54

Was it still a networking job? Because it seems like it changed.

Doug Madory: 10:57

Yeah, so the second. The first job was very technical and it was again analogous. If I wasn't wearing a uniform, it would be really similar to what an Sysadmin, it or a network admin. Very little was, I think, different, to be honest. The second job was completely different. It was like being in the army and so, and as an officer in charge of 55 enlisted, I'm not on equipment anymore, so this is not a technical role anymore. Even though I was designated as the squadron engineer of our squadron of 260 people, I was the one person with an engineering background. Yeah, I mean, I didn't do a lot of engineering and, yeah, I needed to understand what all the guys under me, all the equipment, what it did and you know conceptually, but they were the ones, like Sean, would be the ones who'd be on going to the training beyond equipment, bringing up circuits and making sure we have all the calm we need so we can.

Andy Lapteff: 11:58

You feel like the military set you up for you know, some people I've talked to in the military they're like God, I got out and I just felt so. It was so hard to transition to civilian life and they didn't feel prepared. But to me, from where I'm standing, it seems like you had some really useful, awesome experiences in the military. That would have been a positive experience for me.

Doug Madory: 12:16

I mean there's certainly challenges. I mean honestly that transition from being a technical person sitting in a cubicle, you know, doing network admin, sys, admin stuff, and next thing I'm standing in front of 55 enlisted guys. That those folks mostly are people who were they get once, they get once, they get kind of coded that they're always going to be these deployable people. So these are like a lot of these guys had seen everything. These guys have been everywhere. And who is this? 24 year old? It's not those kid who's just like, yeah, he's got a CCNA who cares, like we've. You know, a bunch of us have been to war like who cares. So that was a tough transition.

Doug Madory: 13:00

I would say the first six months was pretty uncomfortable, to be very frank, and I wanted out. I wanted to transfer to join the comm squadron at the base, which the that's another place where an comm officer might be, and then they do the more. We call it fixed comm, the basic network admin, sys, admin stuff again very equivalent to civilian stuff. I tried many times to transfer out. I always got Nix and after a while I kind of accepted my fate, that this is where I belonged and in the end you know like I by the time Sean and I were going to Iraq in the fall of 2003, I was probably one of the most experienced officers for the type of role that I do, so it was probably fortuitous, it was probably good for everybody that I was in still in that role.

Doug Madory: 13:55

I had been doing it for a couple of years because that was going to be a tough job and we had a very tough assignment that we had to handle and we pulled it off flawlessly, which is pretty, which was pretty cool, but anyway, it all worked out. But yeah, you mentioned like transition. I yeah, I think I drawn it all the time and I I was. It was positive for me. That's, that's awesome.

Andy Lapteff: 14:22

I guess at this point in your career you still couldn't see the internet right.

Doug Madory: 14:27

No, I mean, I like I knew I knew BGP just from textbooks or getting into technical discussions with, like sometimes we would be in a remote site bringing up a circuit and yeah, occasionally there would be another like a BGP announcement, like we would have a route that they would have to enter into the table to handle our remote site. And so if you follow, like I don't know who does this besides me, but like, if you follow the routes are going through, like AS721, that's all the DoD stuff. You see rates, routes coming and going. You don't know which they are. Well, sometimes these are deployed sites coming online and coming offline, or ships. You know it shows up occasionally in BGP. That was, I had a little taste of it, but it wasn't until Renesys, where I was just doing this all day long, every day, and I don't know that there's a lot of people on planet Earth that have done that for as long as I have.

Andy Lapteff: 15:20

So did you just stumble into Renesys? I mean how? Because you know the rest of this conversation is probably going to be how you see the internet and all the insight and all this quick stuff so when you say you know the DoD, you know AS and stuff coming up on my quit. I've worked in BGP, I've never, even so I can't wait to dig into that stuff. But how, how did Renesys happen? Was that just a gig? And yeah.

Doug Madory: 15:41

So I, after after Aviano, I had to make a decision. I was doing very well in the Air Force, but I just I wanted to like I was going to be moving up management wise, and so they. So they had to make a decision, just career wise. So I want to be more technical or do want to continue this path, and I liked the idea. It wasn't really about the money. I kind of thought that I would rather be happier doing some more technical stuff. I guess, to be honest, though the so I. So we were in Iraq in the first year of the war. I wasn't thrilled about that, and more on a political perspective, that I just didn't think this is what we should be doing, and so I, that was a factor, anyway. So then I was like well, I'd like to go to graduate school somewhere we get back into having spent three years in a management position, try to get more into back into the technical space. So with the graduate school at Dartmouth College, and I've kind of I live in a town next to Dartmouth College, been there ever since. So I just wrote out emails to professors and tried to see if anybody would look at my background and find me interesting. I had gotten that advice from the US Air Force Academy, who, at one point, was courting me, courting me to be a lecturer there. There was a way that you could get picked up to be a lecturer at the Air Force Academy and then they would pay you to send you to civilian graduate school, which I was determined to go to. I started down that path. In the end, they kind of backed out unless I was going to commit to a PhD. And then, at the same time, I got an offer from Dartmouth to come and they were going to pay all my bills and giving me a stipend to be a student, and I was like gosh, like I think. I think I got to take that and move on.

Doug Madory: 17:36

So the way I got to Renesys, though, was, yeah, I worked a little while as a defense contractor. I wasn't super happy there. I worked healthcare, it security, and you know that didn't again. It didn't quite fit. And then my advisor at Dartmouth was socially a friend with one of the founders of Renesys, and I'd see this guy at parties and stuff, and when I was deciding I was going to leave the hospital, I was like, all right, well, let me. Who do I know Like let me write to this guy and I was like, hey, I hadn't really honestly, I hadn't really taken it very seriously. I think a couple of times, I think in the past, he had tried to recruit me and I didn't, I didn't. Yeah, I should have, I should have taken him up on it. But it all worked out.

Doug Madory: 18:23

And so Renesys for those you don't know, that was a company that did we sold a BGP alarm, bgp monitoring, probably one of the first companies. In fact I think that there's a patent out there that they had made the first patent. The patent is so broad I'm sure it's completely unenforceable. But you know, when there's a new technology, you don't know what's the, what's a, but the whole, the whole space. There's a patent that just covers the whole space, I'm sure. I guess, like I said, it's not enforceable anymore. It's probably on a big pile of patents owned by Oracle now.

Doug Madory: 18:57

But yeah, so I joined Renesys, happy to do something different, and they wanted somebody who would do all the things that the current crew didn't want to do, which was data QA. So trying to figure out like, where are our? We're doing all this BGP processing, classification of stuff, but it would make errors and and it was tedious to go look at them and figure out what's what's going wrong. How are we getting geolocation wrong? How are we doing relationship classification wrong? And and then also report writing. Occasionally we would have someone who's paying us a bunch of money want to go dig into some sort of problem then write up a report. And yeah, these were software guys and they're like I want to write a report. So I was like I'll do it, I'll do it. I'm not, I don't, I don't.

Doug Madory: 19:47

At the time I wasn't like super into writing, but I was like I'm happy to give it a shot and anyway. So I got it. But I got into it. I you know, like doing the data QA stuff. I still feel like that's a great way to learn a space is just live in the data and just really get into it. So I kept asking the guys that were there or like, teach me what's, how do I, how do I properly interpret this and where? Where are the errors? And what I ended up finding was that at the same time I'm finding errors, I'm also finding like really novel stuff that's interesting and actually the same. You know, techniques uncover the same thing, so you may make some discovery, that of something that's pretty cool, or you find the bug and you and you got to go fix it. Well, either way is success. So I started to kind of get figure that out.

Doug Madory: 20:36

And then Jim Cowey, who was one of the founders and he was probably the more the guy who had a more of a public profile. He would write blogs, he was on the PC of many of the Nogs, nanog, many E-Nog, e-nog, and he would do a lot of presentations. And I started to kind of become an understudy for him and help him write some of the stuff and some of the blogs and presentations you know, at a certain point were it's a big mix of my stuff and his stuff. We're kind of working together on that. And then at some point he became very like during the Arab Spring, when we were getting quoted for a lot of stuff.

Doug Madory: 21:21

He was very gracious to share some of the limelight with me and I always say that, you know, had he not done that, he had developed these contacts with the media, he had developed this process of wading into geopolitical stuff. If he kept that all to himself, I never would have held it against him. It was his company, he had done it all. So it's a credit to his character to say like we're the new guy, if you're smart, you're doing some good work. Let's have two people who can write and talk to the press and go to conferences. And at some point he wanted to go start another company. He's just an intensely intellectually curious person who can't sit still and you know he wanted to do, he wanted to go on and do something else, and so that all got transitioned over to me and then I had the benefit of you know, he was also the founder of the company you're running a company doing going to board meetings, doing all kinds of things that I didn't have to do. So when that got transferred to me, I could just do this full time.

Doug Madory: 22:20

So I'm doing fine, you know, diving into these outages you mentioned, like seeing the Internet. This is when I got that, you know, moniker from the Washington Post. We were just writing, I was, we could see so many things and there was nobody else reporting on these at the time, or not that much. And you know, once we kind of got the formula down, I kind of was like, well, I don't know how long this is going to last, so I want to just get everyone like. Every like a BGP leak, a BGP hijack, internet shutdown, every like we've got the data. I want to see if we can catch everything and capture the value as far as like telling the world about these things, anyway.

Andy Lapteff: 23:03

So customers asking for Internet analysis, because I want to frame this conversation like my relationship with BGP and my technical career has just been either look at a BGP table and see if our routes are there you can look in a looking glass, whatever or my company has this application we're going to host. I have to advertise this slash 24 out to the world so people can reach it right, and that's it. But yeah, you're talking about all this insight and geolocation. Like, I've looked in a BGP table and I wouldn't even know where. Like where's geolocation in there? So you know, what is this analysis you were? What are customers asking for? Like, what do they want to know?

Doug Madory: 23:38

Yeah, so I guess a couple of well on the geolocation thing. So we would sell products so that someone could go and pull up country X who are the transit providers operating in this country, who are their customers. It was basically competitive business intelligence and we would try to go around the world selling that as a subscription. But then that needed to have the geolocation correct so that if you down and geolocation is a notoriously difficult thing, then you would you know the product is wrong and we would get complaints. So but Having worked on that, that's what put us in a position.

Doug Madory: 24:15

When the Arab Spring started in Egypt, went offline. We had already ironed that out. We know who the providers are, we know all the routes we can pull up in a second. What is Egypt at any given second, based on our tracking of BGP. And then that helped enable us to tell the story, because in that case it was very much visible in BGP as far as customers. I think we would publish a lot of stuff about BGP.

Doug Madory: 24:42

So we attract questions of people. With yeah, I mean a question that I've gotten a couple times over the years that I think is a good use case for some of the techniques I've been developing is you have a BGP-based DDoS mitigation, prolexic, invented. This of another DDoS mitigation firm is going to announce your address space and there's ways this can go wrong and I don't think this is well understood. It would be something I would like to, if we can fully field some of these features I've been working on with Kentic. I'd like to really explore this topic, because what ends up happening is normally your company announcing routes to the internet, you have an upstream at least one, and so you want to have a DDoS mitigation vendor step in the way of an attack and so they're going to announce your address space. Okay, so there has to be some sort of transition between you announcing these routes and that guy announcing the routes, and how that gets done.

Doug Madory: 25:46

There's a few different techniques people use, but there's ways you can go wrong. You've got three parties you've got your network, the vendor, and then your upstream, maybe upstreams, and so that transition isn't always perfect, and so we would get people with big firms, with money, and they feel like they're paying a lot of money to a mitigation vendor and feel like they're not getting the protection in a timely manner and they feel like they can't get. They don't know themselves what's going wrong. They're not getting information from the vendor and they think maybe it's a BGP thing. Could we look at that? And sure enough, we'd look at it.

Doug Madory: 26:24

And you're like, yeah, if you look at the propagation through time, you can see, I can see when it begins, but when it fully, when that mitigation firm, when there are routes, fully propagate, you know, if it takes 20 minutes, that's not good. That means until it becomes 100%, you're not getting complete coverage, and so that transition time becomes important. And I don't think even today, I don't think that's so well, I don't know. I have a way I like to visualize this and most of them are fine, but occasionally you'll see a goof up where the upstreams continue announcing, the mitigation firm starts announcing, and now there's contention between the old way and the new way, and so the DDoS traffic is still coming through because the two are kind of fighting within BGP land. Anyway, so that's a that is a scenario that is worth digging into.

Andy Lapteff: 27:23

Where are you getting your analysis? Like is this magical software you wrote. Like again. I keep going back to like I've seen BGP route tables and looking glasses like I don't see any meaning there right Besides there.

Doug Madory: 27:36

So we used to. So Rannis has built their own BGP collection platform, I think around the same time that Ripe started and ours was even bigger, partly because we like actively, would recruit people to give us always said it'd be full tables and so and we would be selling services in like the Middle East or South Asia or places that are a lot off the beaten path, whereas some of the public resources were very heavy towards academic networks and which are maybe a little less interesting for the operational internet. So at those times we would collect our data. Now, when you say their, data.

Andy Lapteff: 28:19

I'm sorry to interrupt you. It's just prefix and the AS path no, no so yeah, let me explain that.

Doug Madory: 28:26

So the way it works for route collection is that you would configure yourself as if you were a customer, so we would have these. We have this custom software. This is Rannisys. Nowadays people use like Quagga or something to replicate the behavior of a router, but the purpose is you would peer with them, ask them to configure you as a as if you were a transit customer, meaning that now you're going to get a view of their full table, at least the selected routes of their full table, and so then you can just start recording as you get the announcements. You know, as you set up that session, you'll get a big push of data. You get a full, you get a. You'll now have the full table and then, as the announcements come in, you can just log those announcements, which can be a lot if you're pulling in all the announcements from the internet off that one session.

Doug Madory: 29:21

So if you look at like route views out of, the University of Oregon is a great public repository for BGP data. So they have collection points, basic software, same idea they connect, they will peer via BGP with an entity, get configured as a customer so they get a full table that they can. They can see and record, and so then you have all these different vantage points that each one has got a full table and with that, when you merge hundreds of these vantage points together, then you've got the chance to try to understand. Well, could I understand how does a route get propagated through the internet, writ large not just to one AS, but like how is this getting propagated through time? So that's kind of the data collection process, and so some of that happens now like route views.

Doug Madory: 30:15

Ripe, ris, are the two big public repositories of BGP data and then you have our own sessions to operate the Kantec platform. You have to peer with us because we use the BGP data to decorate or annotate the NetFlow as it gets pulled in. So we need a session there to annotate with the full AS path from the perspective of that route, or the AS path of the source and destination IPs of every flow record. So we have so you're not reflecting flows Kantec is yeah.

Doug Madory: 30:50

I mean, that's our main product, this flow data.

Andy Lapteff: 30:53

Yeah it's amazing.

Andy Lapteff: 30:55

So, I'm still trying to get my head around. I'm not the smartest tool in the shed, I guess. So you mentioned the Arab Springs and that was like an uprising in Egypt, right, and something awful happened and then Egypt eventually shut the internet down. I believe that's very Sure. That works somewhat accurate. I mean, we could talk for an hour about what happened, right. So how did you know? If I look at the BGP route table, how do I know that that's Egypt's AS and that their government shut it down? I still can't get my head around how you're extracting meaning from route tables.

Doug Madory: 31:31

Sure, I think I can walk through that maybe. So you have. Yeah, AJ, do you want to jump in?

A.J. Murray: 31:36

Yeah. So I wonder, Doug, you mentioned the Washington Post not too long ago that they had quoted you on something. I didn't read that article or see that. So what are some of the things that they have quoted you on? I know you saw internet events happen, so just give us a quick synopsis of what were some of those events. And then, how is it you're able to see those things? What technologies are you using to identify and see that stuff happening?

Doug Madory: 32:05

Oh, let's see. As far as getting quoted, I mean, I don't know how many times I've been in the Washington Post. I had a pretty good report there they had actually sent Once we got going back in dying days, they actually sent Craig Timberg as one of their lead tech reporters to spend a day with me in Hanover, new Hampshire, and then we've had a collaborative relationship since then. In fact, we had a this is a good BGP story I took to Craig and to that in 2021. So just a couple of years ago. You may guys may recall this. I'm sure it popped up at some point, but on the hand of the list, people started to notice that there was some sort of entity that had nobody knew what it was, and all of a sudden it starts announcing more DoD IP address space than anything that's ever existed on the internet and people were asking like, have you ever seen this? What is this, and are these guys hijacking the DoD, or is this who? We'd be concerned about this thing? And then there was someone piped up like, oh no, it's fine, don't worry about it, and I was like, what is this thing? And so then I started digging into it and I don't know. I guess the full story is still kind of out there, but what was intriguing to me was that it started in January 2021. And if you go back to the timing of the announcements, it was within three minutes of the statutory end of the Trump administration, which made it a very intriguing thing like what is going on here. And anyway, in the end it's a company that was contracted by the Department of Defense to do data collection stuff and this is still an ongoing thing.

Doug Madory: 33:55

But anyway, I got a lot of. We had a lot of play on that. I had contacted Craig at Washington Post. He had Paul Son, who's now with New York Times, but at the time he was there Pentagon guy and he was going inside the Pentagon to figure out what was going on. Right as we were about to publish our thing and we had a nice thing where I was going to write up a more technical thing for our, for my readers. They were going to write something that Washington Post reader would be able to understand and we were angling in towards a date to publish.

Doug Madory: 34:31

And then the AP also was working on the same story and then they reached out to get my take on this and I was like All right, well, I think I'm working on the same story with another outlet. And then I mentioned to the post guys, I was like All right, there's another outlet that's asking me and I'm like All right, we got to go, we got to publish. So then I became this race between the two organizations. That's why both stories came out on a Saturday morning, because I think I think the Washington Post wanted to put it out on a Friday. There they got bumped by something and going out on the Saturday morning and I let Frank at the AP know like the story is out with the Washington Post. He's like Shit and so that he gets his story out within a couple hours. He was almost done with it and then they had he had some unique stuff. So there's two, two competing outlets.

Doug Madory: 35:19

That was like a big story for like 24 hours. And I remember talking to Craig afterwards and he was like See, I've been doing this for 20 years, I've been a reporter at the post and that was the most read story I've ever done. I don't even understand why it was. It was the most read story on the Washington Post that weekend by according to our internal metrics. It was just had the nice formula of you know, there's some sort of intrigue maybe around Trump and then, like a, the PGP was very. There was a shadowy, you know company that's doing this had a lot of pieces that were interesting. But anyway, yeah, I don't know if I can. I've done a lot of different stories over the years, so I've done a lot of PGP stuff. I don't know, like just recently with with what's going on in Gaza, there was a lot of questions people have about like what's the state of telecommunications there? And so even before this, I was just in a chat with a reporter giving some updates and again now I've got NetFlow to play with, where I never had that before. You know, as a guy who was a PGP analyst, bgpr is basically so these are potential paths that traffic could take from one place to another, but I don't know that any packets went there, you know like, and so we would see, I might see a really suspicious BGB hijack. Was anything that anything actually followed that path? I don't know, but maybe it did.

Doug Madory: 36:55

Well now, so with the what? Now with NetFlow, I actually have records of traffic and the way I'm able to use that, so our customers. But we have some people for one reason or another, either because they're just so such a huge backbone carrier there's no way we can ingest the flow, kind of thing, or they're in a country where there's data sovereignty rules that prevent them from sending by law NetFlow out of the country, kind of thing. Most, everybody else, do a cloud based service and it's better for everybody, because then, you know, we can do this at mass. They don't have to maintain servers to do the data collection and processing. But then with that there's a data use agreement and I can use this according to that agreement. With some I've got some rules of the road of like what I can and can't do, and so it has to be done in aggregate. So I'm not outing any individual customer, and so you know, when I'm looking at Gaza, it doesn't really matter what which of our 350 customers this flow record came from. That's not the side that I'm looking at. I'm looking at the other side, and so if the country goes offline and we see the traffic dry up to zero, so that's, it works fine, because it's that's not the side of the record that I'm interested in.

Doug Madory: 38:22

But it is fascinating though, so we can dig into BGP even more nuanced things of like a leak, so someone's screwed up BGP. They're supposed to be announcing. So they announced all the routes they learned from one transit provider. They just announced out to another transit provider and now there's this loop going through their network and it's slowing down traffic or traffic is getting dropped or misdirected Again.

Doug Madory: 38:50

In the past we could either see that in BGP estimate or guess that some traffic was probably impacted by that, or we were running a lot of trace routes back then and some of them we get sucked in and we get followed.

Doug Madory: 39:02

Look at the trace route impact. But now I can just pull up the net flow again because we're annotating all the records with full, full as paths and source destination. Then I can just make a query and say all right, did I see any across our customers? Did we see any traffic that followed this? As subsequence? That captures the leak? And sure enough, I can see it. And then I can even see things like well, of the traffic that should have gone in that direction, how much like dropped off, like how much was lost because it's going in such a path that it's getting disconnected, and so yeah, that's another impact. Either traffic gets misdirected or traffic gets dropped because it's trying to go too far, something's congested or the latency is too high or something. And it's neat to be able to see a BGP event and pull it up in that flow and see those two impacts, because I never used to be able to see that. I just intellectually would know it should happen something along these lines, but anyway, why were you getting the net flow?

Andy Lapteff: 40:11

Because, like Gaza is an example of somebody asked you what's the internet look like as a result of what's happening there. I mean you have to be collecting everything all the time for a baseline, right? No yeah.

Doug Madory: 40:21

So I mean we have a lot of content providers. We have telecommunications companies, so gaming, whatever. There's all kinds of companies that are using our stuff, so they're your customers.

Andy Lapteff: 40:32

They're peered with you sending all their stuff. Okay that's the secret sauce.

Doug Madory: 40:37

That's missing right. So peer with you.

Andy Lapteff: 40:38

You see all their stuff.

Doug Madory: 40:40

Yeah, there's a. I guess I'm not the best person to explain the various options, but we've got a few ways. Either you send it over the internet or we have a direct connect with you somewhere, or there's a. There's a there's, depending on a bunch of conditions. We've got some way that we can, though and capture that, and then, yeah, we peer with you. We get ways we can pull in all the data we need.

Andy Lapteff: 41:02

So I guess you have different ways with unfriendly countries. I'm thinking like Russia, let's say with the Ukraine stuff, Like if you were trying to see it. Didn't something happen with BGP? Didn't Russia try to do something with Ukraine's?

Doug Madory: 41:15

Yeah, yeah, I mean that there could be a couple of answers to that, but I think I know the one you're talking about. So let's see, you know, when we're looking at a country like Russia or Cuba or a country we're just never, we're not going to have any customers in, so we're really just looking at the outside, in which, for a lot of these places, that's fine because that's most the traffic right, they're not the source of traffic, they're the destination traffic. So the fact that I've got that side of it, I'm happy. I they're not. There's not any meaningful amount of traffic coming, at least some of these places.

Doug Madory: 41:51

So, in the case of Russia, yeah, that there's been any number of BGP things, but I think, as far as you mentioned Ukraine, the story I did last year with the New York Times was around what to place in Kursan.

Doug Madory: 42:05

So it was a city in the south that now has been partially reclaimed based on a counter offensive by the Ukrainians about a year ago, during the during the course of 2022, as that city was taken over by by Russia. Russia military came in and, at gunpoint, had the network engineers reconnect their their lines down through Crimea, which they had taken over in 2014, and then on over to Russia over either the bridge or the submarine cable in the Kurs Strait, and so that meant that all the all the internet communications for the city of Kursan then had to go through Russian carriers would be subject to any kind of surveillance, manipulation of that traffic, and anyway, so that that took place. Where they cut over and then after the offensive, about half or more more than half of those providers then switch back to Ukrainian transit, depending on where they lie geographically. There's a river that splits the city, and the Russians have the south still the Ukrainians have the north.

Andy Lapteff: 43:17

So yeah, was there? Was there one place in Ukraine? Another thing I'm trying to get my head around is you know the way BGP works in the internet, right, like there's just so much, there's so many connections and so much resiliency to go around things. Like there's just one building, you can walk into Ukraine and just all the traffic in Ukraine goes through that building and they just send it to Russia.

Doug Madory: 43:36

I mean there's a no, oh. I guess there probably was a choke point in this case. There definitely was a network that was working on behalf of the Russian government in Crimea, and so that was back in 2014. I've been doing this for a while, so I was covering that stuff as well.

Doug Madory: 43:56

The fall of 2013, with the Maidan protests, and then it was months after that, in the spring, that the Russians had the little green men, guys without any insignia on, just show up with guns, and it was the Russian special forces taking over Crimea. And then they had a vote that they kind of ran at gunpoint to say where do you want to be? You want to be part of Russia, and now you're Russia. And then Medvedev which was Prime Minister at the time, I think came down and gave a speech All right, crimea is ours. And among his orders of the day were all telecommunications can no longer go through a foreign government, has to go through us. And so they created this outfit, miranda Media, which is an extension of the state telecom, of Ross Telecom. That would be the way the new transit carrier for that region, and all of everything would then get routed out through mainland Russia.

Andy Lapteff: 44:57

So then when Chris surveillance right like they wanted to see.

Doug Madory: 45:01

Yeah, I mean they want to, they want to have I think they want to have the ability to tap into any of the intercommunications they want all to go through their own. Stuff Shoulders both interception and then also you probably can do stuff of identifying. If you've got, you're trying to identify some dissidents or some sort of rebels and you know something about where they're connecting to, then you can try to trace back. Where are they, where are they in this, who are they getting service from? So you need to have all the traffic to be able to do that kind of thing, and so I think we can assume, when they take it over, that the risk gets high for people who are still using the internet and are trying to resist the Russian occupation.

Andy Lapteff: 45:44

So I know we've been going a while and I don't want to. I don't want to run out of time and not get to talk to this, but it didn't seem like it happened in the Russia scenario.

Andy Lapteff: 45:53

But you know, the government shutdowns are like you mentioned, arab Springs, and I think that was the first one you got pulled into. So I guess one of the things that I'm fascinated it's not the right word, but kind of amazed by is that a government can just shut down internet connectivity to all of its citizens. You know, I guess I'm in yeah, it's fortunate enough to live where I live, right and like.

Andy Lapteff: 46:13

That doesn't happen here and I don't even again. I have such a. I have such a, I guess a. I wouldn't think that our government could shut our internet. I know you. It just doesn't seem like within the realm of possibility because of the technical way. I know how BGP works it would have to be, like you said, a choke point, like there'd have to be one or two centralized places that you could well. Yes and no, yes and no.

Doug Madory: 46:37

I guess I think the United States is it would be, from a practical standpoint, really challenging to shut the internet. I think it'd be really hard to pull it off. But I would say that you know for a little while, I think you know so the the Arab Spring, the Egypt was January 2011,. A bunch of shutdowns, a subsequent to that and that became kind of a thing, and it was a lot of discussion of like, oh, how do we push back on this, how do we keep the internet on? And Jim Cowey I mentioned earlier, founders of Renesus had written a pretty good blog of like, oh, could it happen in your country? And he did a little a back of the envelope analysis of how many you know international connections does a country have. And if it just goes through one carrier's carrier, then they would be more at risk because they just have one place to turn it all off. There's a choke point. But if you had lots of connections out, then maybe you'd be more resilient. And unfortunately that has not held true.

Doug Madory: 47:37

And what ends up happening in other countries is that if, even if there isn't a technical choke point, there is a legal, authoritative choke point in the sovereign government of that country, and so for every country there is a telecom regulator that is the one that provides licenses for the telecoms to operate. You, me and we cannot go into country X and start providing telecommunication services. We have to have a license from the government to do that. And if we go awry of what the government wants us to do, then they can pull that. And so what we've seen in some of these places that have maybe some more diverse connectivity is that the government just issues, just contacts.

Doug Madory: 48:26

All the providers they have on the list, they have a list of who they've given licenses to. So you want to keep your license, you want to keep your investment in the country, then you're going to do what we say we want to turn things off. And so it no longer becomes a technical thing and it becomes more of a administrative thing. And in the case of me and Mara, I had some contacts there. One of the providers received this over fax, which I think is kind of I don't know why, that's kind of humorous, but they got their short shutdown orders over fax that they had to, by 10 pm that night, turn off the internet. So it doesn't.

Andy Lapteff: 49:03

I haven't got a licensing Like the FCC here licenses everything and could conceivably threaten to pull your license. I'm not trying to say it's going to happen here in here.

Doug Madory: 49:13

Right, it's still pretty hard to imagine, but in these other countries you can imagine. You've got a multinational like MTN out of South Africa, has got subsidiaries all over Africa. It takes some money to put those in. If the country, the government, says they have to shut it off or you lose your license, where they have to make a business decision, and at the end of the day they comply Because what are they supposed to do? So it's a very difficult situation. Sometimes they I guess the best thing that some of these companies have. The best way to. It's pretty weak, but the way to push back is just to publicize that and to make sure everyone knows. Hey, we were legally ordered to do this. We live in a Westphalian world, so every country has a sovereign government that has somewhat absolute power over its country and we don't have a way to override that. And if they do something bad, that is a reality legally.

Andy Lapteff: 50:18

What's your prediction that it'll happen here someday?

Doug Madory: 50:21

Oh, in the United States. Again, I think, for a practical standpoint, that's what's going to get us out, is they're going toif we approach that scenario, the engineers are going to be like man, I don't know how to do this. Yeah, I justwe have a veryI mean. One thing that saved Ukraine, that's been an ingredient to their resiliency, is the fact that the way they structured their telecommunications market is that a lot of the towns had little mom-and-pop ISPs and so there's all these little ISPs with all these different connections going, and so that ended up being this really wonderful tapestry of connectivity that, if you took something out, there wasn't that consolidation where you could just have two companies and if you just tell the two companies to turn it off, ended up being. So that was actually a really brilliant ingredient that I'm sure that that was notthat wasn't what they were thinking when they were making it.

Andy Lapteff: 51:15

That was Ukraine right.

Doug Madory: 51:16

Yeah, Ukraine.

Andy Lapteff: 51:18

I know there was a Starlink component too, wasn't there? Didn't they try to do some?

Doug Madory: 51:21

Yeah. So Starlink became a story there too that it got a little bit overstated as far as what was Starlink doing for Ukraine. So we would seeagain. This is like spring last year after Elon Musk said that I was going to send Starlink devices in and provide service there. Then there was headlines like Starlink is keeping the Ukraine connected to the internet. That's not true.

Doug Madory: 51:48

It's not to say that they weren't doing something super important and valuable. Certainly for somebody who did tactical comm in the military like, yeah, I get why a small movable satellite connection that you could take with you through the field, that would be incredibly valuable to the main tank communications. But the main lines that was keeping the connection up were the fiber lines going out to the west. Sometimes they got broke. They got blown up by airstrikes and, to the credit of the engineers there, our equivalents in Ukraine were driving into bombed out central offices. They were rewiring. They're still doing that. This is an ongoing thing. These guys are at great risk for themselves going into these areas to restore service, splice fiber and a smoking hole.

Doug Madory: 52:52

So that's what it takes, but it is super important. It is reallyagain for something thatthe three of us we haven't experienced. This I hope we never do. But if you are in a situation like that, a dangerous war type situation, you've got loved ones that you would like to either contact or let them know you're okay or find out they're okay. If you're cut off from all communications from an extended period of time, it is a traumatic thing even just that. I remember having this conversation withI've got a lot of contacts in Ukraine saying like oh, internet maybe isn't the biggest thing. Just because you can't surf the internet, you're like no, you don't understand. If you can't reach your family members and you don't know if they're alive or dead and it takes days to find out that that's not a very pleasant situation to be in.

Andy Lapteff: 53:46

That's awful. It seems nowhere near what they're experiencing there, but one or two small natural disasters that have happened around where I live A minor earthquake, for an example, happened 10 or 11 years ago, very minor. All the phone systems stopped working because everybody tried to call everybody at once. The switches get overwhelmed. It's not built to take that kind of. So my wife now a girlfriend at the time I'm a cable guy on poles she's at some corporate building and their whole building shakes and she's like, oh my God, my boyfriend works on telephone poles, I have to see if he's okay and we couldn't reach each other for about 20 minutes. Again, not a war zone, I'm not trying to draw that parallel but just that 20 minutes of not being able to reach a loved one, not sure what happened and if they're okay. I can't imagine that it's success.

Doug Madory: 54:35

Yeah, so that's the value of communications. I think sometimes we kind of trivialize it because we do a lot of trivial like cat videos and we do silly stuff. But it comes down to it, we all have loved ones that we need to reach out to in a tough situation and if you can't, it's very stressful. Those are huge.

Andy Lapteff: 54:52

I wrote down a quick list of types and causes of internet outages, like human error, like rat leaks, fat fingering stuff, physical damage, undersea cables, government shutdowns, BGP hijacking I mean, at everything you watch is there a number one cause of people breaking BGP on the internet because a human's doing dumb stuff, or is it just very insundery.

Doug Madory: 55:14

I'm not sure how I'd rank those In the category of submarine cable stuff. Like I've done a lot of that and that's definitely an anchor. It's a human related Someone either dropping an anchor and dragging it or dropping a. They do a lot of trolling. They drop a net for fishing and drag the net along to just scoop up all the life. It kind of kills everything. Then they pick out the fish they want, kill everything else. Those things snag cables. It's definitely human. This caused most submarine cable cuts.

Andy Lapteff: 55:45

Do they break often submarine cables?

Doug Madory: 55:46

Yeah, there's a couple of breaks every week, but it's a global infrastructure and there's a system for repair. I don't know. There's always a conversation of how much, if any, threat of submarine cable sabotage is there. Has this happened? There's been a couple cases where it has, but I know maybe 10 years ago when people started talking about like, oh, the Russians could cut all the cables and cut us off yeah, we take a lot of cables to really make it. Then, at least for US connectivity, Other countries, there are places that rely on a single cable.

Andy Lapteff: 56:35

You're looking at a map that's amazing. Just submarine cable.

Doug Madory: 56:38

It is an endlessly fascinating topic I had the pleasure of presenting at a handful of conferences, because that was actually I ran a system going back to that story.

Doug Madory: 56:50

When I started we were good increase from customers, kind of like okay, could you figure out some submarine cable stuff, like can you see it in your data and the BGP stuff.

Doug Madory: 57:02

And so that kind of got handed down to me like hey, work on this, see if you can figure out how do you find cable, submarine cable events, either activations or outages, in our data. So then I would go and just start reading the headlines and reading the you know what's, what are people saying is happening, and then try to see you can I find it in the data. And that led to you know a practice of spotting submarine cable activations and then within the submarine cable community, like people know that that's kind of something I do. So the last couple people have kind of given me a tip like hey, you know, like we know this cable is coming active. You're the guy who usually writes a story about this. So the latest one was last month in St Helena and the island in the South Atlantic just got a submarine cable activation. The cable had landed months earlier but we just started passing traffic and it's kind of amazing and so I don't like that.

Andy Lapteff: 58:02

How do you know, like you're just, it's ridiculous what you know? How do you know that a new submarine cable is laid, led up and passing traffic Like?

Doug Madory: 58:12

I mean, this is probably your secret sauce. You can't tell me, oh no, not at all, not at all. Like again, if you get to know the subject, you know there's a few places where cables like there isn't a cable and they're trying to get a cable there, like okay, where there's an island nation somewhere and the cable's coming someday, like all right, fine, like you'd want to set up something like I for St Helena. I knew this was coming at some point and so months and months ago I set up tests out of our like our performance monitoring. I started finding IPs that would respond to me, respond to our agents around the world located St Helena, run those tests, do a tracer, like okay, we're going over satellite link, all right, one day I'll come back and I'll see the transition and it's really dramatic From a just from a latency standpoint. I'm not doing anything to test, test capacity, but that capacity would probably be even a bigger change. But yeah, so then I kind of got to know just what are the, what are the stories? So then you know, when I was first getting into that, I stumbled upon a blog about the internet in Cuba and that at the time was still exclusively connected through Geostationary Satellite. They had a couple of different providers. There's one telecom, a state telecom, still this way is one Texas, a state telecom, for Cuba, and they had been due to the US embargo. They all the submarine cables that were laid in the Caribbean are they all go around Cuba. Then nobody lands there and nobody wants to mess with the embargo because it can be fairly punitive even for non US entities. And so it was the Venezuelan government, although I don't know, someone told me recently they thought the Chinese chipped in. I don't know whoever paid for it. They built a submarine cable between Cuba and Venezuela and then this was constructed and let's see and finished in 2011. And by 2013, the country was still on Geostationary Satellite.

Doug Madory: 1:00:17

And I came across this blog of this guy writing about this. This professor out in California who had an interesting internet developing places and you know, got to know this story. I was like, all right, well, I made some graphs, like I wrote him an email. I was like he was like maybe, maybe the cable's up and there's some we just don't know how to see it. Does anybody know? And he was kind of writing out to the world and I was like I've got data, we can see that it's it's all satellite providers. And I was like I'll do this for you, I'll set up a little thing and our thing and our stuff. And if we see a new BGP connection for a Texan, cuba, I'll get a little email to myself and and I'll know, so I'll let you know if I see anything new pop up. So I wrote something and it was like 18 months later that I got that email. So it took a long time.

Doug Madory: 1:01:04

And then, and then I got seeing like oh, telefonica is now, you know, there's now an adjacency between Telefonica and Retelsius now and attacks, attacks like all right, well, that alone doesn't mean anything, that that could still be a maybe they subcontract out for a satellite link. And so then we had trace routes running from servers around the world going to everything, every part of the world, so we could pull through time or with trace routes going into Cuba and you could see that transition from geostationary satellite to submarine cable and it's it's a pretty stark difference. And yeah, so we broke that story and I got a lot of press for that. And then I've done a bunch of cables since then and the most recent one was the one in St Lena, again, I used to present at submarine cable conferences about this.

Doug Madory: 1:01:55

At those conferences in the submarine cable industry, the people who attend them are people who are either the fabrication people to actually make the cable, fiber optics experts, you know presenting what's the latest, you know technology, they can push even more bits through a strand of glass. They're the installers and those are the people who drive the ships and feed the cable out the water. So these are ship captains or ship crew and financiers, but there's not really computer scientists that typically show up at these things. So I was a bit of a novelty for a while going to these and I would be like you know, I'd show like here's this cable cut and here's what it looked like, here's what happened. And they were like how do you know? Like cable operators don't publicize, this is fact, that's notoriously secretive. And they and I was like, well, I don't you know who I am?

Doug Madory: 1:02:46

This is before I got that title that I'm Neo, I see the.

Doug Madory: 1:02:52

Matrix. I had a very funny interaction for a little while I was. I was definitely trying to catch every submarine cable, this or that activation, break over the impacts and at one point there was a LinkedIn group. That's a really good thing. There's a lot of all the people in the submarine cable space are in this LinkedIn group and I used to publish this and I got a.

Doug Madory: 1:03:09

I got pushed back for I would get this from a couple of people. And one guy wrote to me and he's like you know, there's an NDA that signed between the cable operator and a telecom that's buying the bandwidth that they're not to publicize this because the cable operators don't want a lot of they don't want all the bad news about cable breaks and it happens to them all, but they just, since they're all secretive, they all have to stay secretive. And he's like so there's this NDA and you, by writing your public analysis, are violating the spirit of the NDA that they signed. I was like man. I was like I think that's amazing. I was like I'm not changing anything, but I'm just amazing that someone thinks I have to comply with the NDA. That's not even related. I didn't sign this. This is I look at our data, like the BGP stuff or this is essentially public data. This is a public event, so I'm not really going to change them, but I just found that so fascinating that that that mentality exists in that industry. Last question you ready?

Andy Lapteff: 1:04:10

Yeah, sure, so you can see the internet.

A.J. Murray: 1:04:15

Can't unsee it.

Andy Lapteff: 1:04:19

Some things I wish I could unsee. So you know I'd be remiss if I didn't ask you. You know what does the future of the internet look like from your perspective and, honestly, as we were talking, I looked up. So I'm going to make it simple for you, right, and go as complicated as you want. But I'm looking at there's 65% of the world's population has internet access. Now, you know whether that's accurate or not, I don't know. You know Google never lies. I Google like three or four times. So you know there's a hell of a lot of people that don't even have access to the internet on the planet today.

Andy Lapteff: 1:04:48

So I guess, in my utopian world, after we just spend an hour talking about how much the internet you know communications in general can like enable humans, I guess what I'd love to see, if you asked me, would be, you know, internet access for everybody. But you know is that I mean I know it's expensive, right, like I don't know who pays for all those under undersea cables and they're you know we're putting rockets in space for satellites. I mean it's expensive to get access. So I guess it's a it's a financial thing. But you know this is the first time I've ever asked a guest a question and then answered it myself, which is I'd like to see internet access for everybody. But let me get back to the question. To you, I mean, do you have any insight into what the future holds? You know, you're the guy who sees it all.

Andy Lapteff: 1:05:29

I don't know what would you like to see it Like? Do you have any idea where we're headed with all this?

Doug Madory: 1:05:34

Yeah, I guess I see a lot of progress. I mean, I write about routing security and I I just have been giving some talks. Recently. I wrote something about a little history of big BGP incidents. I've given this talk a couple times of just what, as someone who's covered it for the last 13 years. What I think are the the main incidents, and I would say that that space has gotten better, I think, and we've made progress there, and so, as an engineering community, we should take a moment and say, hey, like we're a lot of people who worked on this. It wasn't just people, there's a lot of people with a lot of companies, a lot of things, and we're in a better place now.

Doug Madory: 1:06:11

We don't have these fat finger origination leaks that take out the internet. They were like the last one was maybe 2019. I don't know. There's a lot of. There's a reason why they haven't happened for for a few years, and that's great, so that. But then in the BGP space, we still have this issue of determined adversary, of someone who understands the weaknesses, is sophisticated and can get around, like RPKI, rov and other security mechanisms we we built, and so it's progress that we can now now, having dealt with some of these bonehead errors. I made the internet more resilient to those that we can focus on, like how do we shore up, because there's a lot of, there are still a lot of weaknesses there that we don't we really haven't addressed very well. So, yeah, what's the future? I hope we continue to keep kind of moving the needle and making that, raising the bar on what a determined adversary has to do to pull off a attack that involves a BGP hijack or DNS hijack.

Doug Madory: 1:07:18

So, as far as your topic of the other three billion people that don't have internet access, yeah, I mean it's possible that these LEO low-earth orbit constellations can't help to democratize internet. So let's include Starlink and OneWeb and we'll see if Project Hyper gets off the ground. But you know there's it's unclear, I mean it's even unclear. Starlink has the huge benefit of having their own SpaceX. So the fact that they can launch their own that's the same company, or pretty much, can launch stuff into space, it's just a huge, huge advantage to that as a internet service provider, whereas these other companies have to contract out someone to launch and they don't control it Anyway. So, you know, will Starlink be transformational and that? Well, I hope so, and if it's not them, I hope it's somebody else. But yeah, it's very hard to participate in modern society with an internet connection, and there's still a lot of people who don't have it, so that's something we as a human race have to work on right now.

A.J. Murray: 1:08:38

Doug, this has been a fun conversation. The entire time you were talking, I was googling and adding links to notes, and so show notes on this one is going to be really, really good.

Doug Madory: 1:08:50

All right, oh man, for people who know me, you just don't want to get me started because I will talk for like six hours.

A.J. Murray: 1:08:56

They have so many good stories. Yeah, you do. Some of the stuff that you've written about is extremely fascinating.

Doug Madory: 1:09:02

I thought of writing a book because there's always like a background, like I've got sources in places. I have a source inside Syrian telecom giving me information up the back before he fled the country. I mean, just like some really good you just, you know, when you start writing this stuff, people reach out to you and they want to either correct the record or tell you know, fill out the story, or so then I'm it's. It ended up being kind of a snowball effect where the more you do, the more people come to you and then they give you, ask you better questions, or they give you more insight and then your product becomes better. And I get that both on the more technical BGP analysis side and then also the geopolitical stuff of Like I've got a contacts in Gaza, although one of them I haven't heard from from last week, so I hope he's okay but just folks trying to tell me from the inside what's going on.

Doug Madory: 1:09:57

And yeah, there's a lot of. You know, I look at that as those are. Those are our people, those are our network engineers doing kind of what we do but under completely different circumstances. And I find that, you know, I feel a little bit of a duty to tell the story of, especially like the we talked about Ukraine those guys going out and repairing lines that have recently been bombed to help their country. That's an act of patriotism, and there's people all over the world that do stuff like that, but just doing the same things that you and I you know are people in our audience do, but doing it in a much more dangerous situation, because communications is just that important.

A.J. Murray: 1:10:44

You know, when you go on social media and you look at some of the crap that you look at right that just posted out there, it's hard to think of how critical the internet I mean. There are days where I'm ready to walk away from the internet, but then something like this happens and you realize just how critical it has become, especially when you want to communicate with loved ones in situations like that. So, doug, thank you so much for spending time with us. Where can people find more about you? The stories that you post? You know LinkedIn, twitter. Where are you active?

Doug Madory: 1:11:14

Yeah, so let's see. I publish on our chirochemic blog. We have a. I publish most of the stuff. If I write something up, it's there. Otherwise, I'm on Twitter, and LinkedIn are part of the places to find me. Yeah, we'll see where the dust settles on the Twitter thing, but definitely, you know, we're technologists, we're professionals, so it's definitely going in. A lot of that stuff is showing up in LinkedIn, so shoot me a. If you can find me on LinkedIn, shoot me a invite, and I usually accept anybody who's in the industry, because I like to talk to people in different parts of our industry.

A.J. Murray: 1:11:56

Awesome. I have found your Twitter, your LinkedIn and some of the blog posts that you've written on Kentik, specifically ones that we were talking about here during the show. As few as some of the posts from, was it Washington Post? Yeah, a couple of Washington Post articles as well, even the one in 2014 that named you the man who can see the internet. That was a pretty cool article.

Doug Madory: 1:12:19

You know I didn't use that for a couple of years After that. I was super flattering. It's a little over the top. I was at an event in Europe where we were dying. At the time we had been, renes had been acquired, we were dying DNS and our head of sales for Europe was with me and he was like how are you not using that? And I was like that's too much man. I can't say that. I'm not a, it's just very not fitting for me. It's very uncomfortable for me to say something like that. And then he's like don't do it for you, do it for me. I'm trying to make sales.

Doug Madory: 1:12:54

It makes my job easier if you use that. So he's like look at the rest of the people who would benefit If you would just do that. I was like all right. So I always feel like I have to share it. I'm doing this for my team and we all benefit from that. I hope.

Andy Lapteff: 1:13:10

That's a hell of a moniker.

Doug Madory: 1:13:11

If somebody said something like that about me.

Andy Lapteff: 1:13:13

I'd have a cape with it on the back. I need to have our etched on a tombstone ready to go. Here lies.

A.J. Murray: 1:13:27

That's great. Well again, Doug, thank you so much for joining us tonight, and we'll see you next time on another episode of the Arts of Network Engineering Podcast.

A.J. Murray: 1:13:35

Hey everyone, this is AJ. If you like what you heard today, then make sure you subscribe to our podcast and your favorite pod catcher Smash that bell icon to get notified of all of our future episodes. Also, follow us on Twitter and Instagram. We are at Art of NetEng. That's Art of NetEng. You can also find us on the web at ArtOfNetworkEngineeringcom, where we post all of our show notes. You can read blog articles from the co-hosts and guests and also a lot more news and info from the networking world. Thanks for listening.