andy lapteff 0:16

This is the Art of Network Engineering, where technology meets the human side of IT. Whether you're scaling networks, solving problems, or shaping your career, we've got the insights, stories, and tips to keep you ahead in the ever-evolving world of networking. Welcome to the Art of Network Engineering podcast. My name is Andy Laptev. Where am I? I am in Munich, Germany, with my new friend James Bensley. How are you doing, James?

James Bensley 0:40

Hey, good to be here. Where are you from? Uh well, I guess some people can hear from the accent. I'm from the UK, but actually I live here in Munich. So nice that I didn't have to travel far to be here.

andy lapteff 0:48

I only had an eight and a half hour flight. How long was your did you drive a vehicle here? It took me about 40 minutes to get here. That's pretty nice. Yeah, it was tough. That's so nice. It's beautiful here. I haven't seen much of the area yet, but the architecture is gorgeous. Absolutely, yeah, yeah, yeah. And today's a beautiful sunny day. So I shall be going for a walk after this, making the most of it. Very nice. Um, so why am I in Munich? I am here at Network Automation Forums Autocon 5. I am recording podcasts. I will be in two days of Python dev boot camp stuff uh with packet coders. And then I am representing my employer at the booth. So I am really looking forward to this. This is my second autocon. I love Scott and Chris and the folks, and getting to meet people like you is like one of the best parts of it. Awesome. So, what are we going to talk about today? You, sir, are a fancy pants network architect. We were talking before the show that I always wanted to make it to network architect, and I didn't. I was like senior network engineer and then kind of pivoted to product and marketing. And but I still love the technical and I still yearn for it and I still like to get my hands on things. Um, I will mention that uh about 45 minutes before you arrived, I started a little fire in here. So I am that guy who's I gotta get my hands in, I gotta get so I needed another outlet, and I found another outlet back here they don't use. And when I plugged in my new stuff, poof, there was a puff of smoke. What is my point? Um, I always wanted to become an architect, and I didn't. So I think A, it would be compelling for me to hear. We don't want to go through every career fart you've had, but how do you go from because I believe you started out, you sent me a note like crimping cables. Yeah. So you started very much basic layer one, which they're my people. That's my jam. I'm all about the physical. I'd love to hear briefly kind of a quick career trajectory in two or three minutes of how you went from like the cable crimping guy to a network architect at a huge, I guess it's an ISP, right? Is that where you're working? And you also have an interesting you had two jobs, I think. You're a network architect and what else? Product developer. Right. I have never met anyone, let alone got to interview someone that was developing a product while simultaneously architecting a service provider network. So let's start with your quick career journey. How did you go from the cable crimper guy to fancy pants architect? It's very flattering that you say fancy pants. Yeah. It's a different kind of stress. You're just trading one kind of stress for a different kind of stress. But yeah, okay. So I started off doing the I worked for a company where I was the sole IT guy. So you do everything desktop servers, pulling cables through the walls, the whole shebang. But it was the network that part that really sparked my interest. This is nearly 20 years ago. So, like many people, I done done the training thing. Got and tried to train for certifications, go down the certification track. Doing that in my spare time. I spent five years studying in uh I think what we call night classes. Did you start with a certification that begins with a letter C? I did, yes. From a vendor that begins with C. Yeah, yeah. Yeah. So um nothing unusual there, but I worked my way through those tracks, ended up working in service provider networks, and like everyone, you start off doing support um for customers, working in the working in NOC or customer support, something like that. And I'm working on these issues, and after a while, I just come to the conclusion that the the issues that I'm working on, or the problems that my customers are having, uh because the network we've built is sort of designed poorly. I will keep having to troubleshoot this issue because there's a fundamental design flaw here. So if my customers they're a heavy VoIP user and they need a sort of HA internet connection, and they've just got two ADSL lines from the same provider, that isn't really helping, right? So eventually I get to the point where like, okay, I need to move out of the op side of things and get into the design because I need to change this. This is you know quick pause because this is compelling to me. Were you at the knock at this time that you were troubleshooting and you realized the design was not optimal? Yeah, I mean it was a much smaller company. We didn't exactly have separate knock and this kind of thing. But essentially, yes, yeah, I'm in a support role. The reason I'm troubleshooting this issue again and again is because this design is flawed. And how did you know that? And and I know that's a silly question to ask, but I wanted to make it to network architect, didn't. And my friends and folks like you that do seem to be very skilled at design. So I spent two years in Anach, and I don't think I ever had the thought that if our design was better, right? Now it's obvious in hindsight as you say that you you're introduced to design concepts and principles at every layer of uh certification journey. And so I get that, but for some reason in my mind, I always thought that, and maybe it's because the companies I worked in were so big, we had architects who designed all the things, and it was just up to monkey me to like make it work, fix it right, rack and stack, whatever. I guess I'm amazed that you knew like we could design this better because I like it's an obvious thing. If you spend enough money and you design a network right, you can get as many nines as you want, but then there's trade-offs, right? Like, hey guys, I need five million dollars for a perfect network. And usually the people with the budget are like, no, we can't do that. So then you start making trade-offs, right? So do you but two ADSL circuits in the same last mile? That that doesn't make sense. So I I didn't mean to interrupt you, but I find that interesting that you knew like we could fix this design. And what level were you in your studies at that point? Like, were you an NP? Yeah, I'm working through my NP when I'm at this company. I'm working my way through the NP. And I think actually it's a really good question of like, yeah, how I don't think I would have had that. I'm gonna say epiphany, perhaps an overuse of the word, but I'm gonna say I had that epiphany um through external means, let's say. So actually, I don't think I could have had that epiphany on my own. I went to try to look at what other people are doing, join mailing lists, go to conferences, be on forums, and just passively uh observe what other service providers are doing and how are they doing it, and then see, oh look, they actually do that in a different way to me. Why do they do it like that? And then kind of you know, chew on it for a bit and just I spent some years trying to get the lay of the land, learn as much as I could because also I'm doing this, uh working my way through the certificates in my spare time. So I've got an interest for understanding things. So I'm also just like I said, passively trying to observe the industry by going to conferences and being on mailing lists and forums and just kind of like other people are talking about they're having the same problem. Yeah, person A says we've got this problem, and person B says, well, if you redesigned it like this, and then I'm okay, yeah, yeah, okay, we could do that at my place too. So it's kind of it needed some external input. Um I'm not gonna pretend that I could have had all of my good ideas entirely on my own. Service provider is such an intense environment. Do you agree? Yes. I the two years I spent working at a service provider, NAC, I was responsible for last mile mostly, which was enough. Yeah, oh yeah. And you know, we also touch the core data center stuff occasionally and that, but the layers of protocol upon protocol, and I always found it so intensely technically deep and confusing. So just as you had an act for design, did service provider just kind of make sense to you? You didn't find it, you know, out of reach. Yeah, I'd say because I think people who get into networking, it's such a wide range. I could have gone into wireless or campus or data center or enterprise or really a whole number of things. But actually, that's just part of my personal uh interests. When I was very young, we got dial up, and that just you know, for want of a better phrase, it literally blew my mind, right? I was like, this is unbelievable. I love that you're of the age that dial-up existed. Yeah, because a lot of people I interviewed, they're like, wait, what? Yeah, yeah, we had dial up, it just it simply blew my mind. Then the whole internet thing. So from an early age, I was like, I need to learn about this internet thing and how it works. This is mind-blowing. So that's how I it never left me. So that's how I ended up getting into the service provider stuff. Was I want to know everything about how the internet works. Somehow that was always on the cards, let's say. But it is a very complex, complex environment. I think mainly because the special thing about service provider networks is that they're shared infrastructure. So we don't get to, if I run an enterprise, I might have some stuff over on the right-hand side, which is for my uh RD team, and some stuff over here on the left, which is for my field team. Service provider is about building one physical infrastructure and piling as many customers as possible onto that infrastructure to increase my revenue and TOI. And that makes it really complicated because you've got so many overlapping services that kind of overlap with each other and interact with each other all on the same infrastructure. All the overlays. Yeah. Which that's what I was thrilled to get out of service provider because I was so overwhelmed by all of the technologies and overlays. And then I got to enterprise, and I'm like, oh, you know, a branch with a router, like, okay, I get this. Like, oh, BGP, sure. Oh, we're redistributing, okay. Like, you know, it seemed much more route maps and prefix lists got intense, like, oh geez, and then like some traffic engineering. But the enterprise and then even data center, I mean, they're related, but there's so much in service provider. So I don't know. I'm just always I don't meet that many SP folks. Yeah. And when I do, I'm like, wow, like you're so smart. How did you do that? I also don't meet any SP folks, and I do it every day. So where we left off was you thought, hey, if we could design this better, we would have less outages. Yeah. So that was me. So I wanted to then move into design. And that's kind of eventually where I ended up. How does one move into design? Do you say to your manager, I would like to move into design? On a very small scale, it happened within the company I started in because we were a small team working together. So there's different ways. So in a very small company, I can just start to sit in on the meetings, be there, passively listen to what's going on. This was a small company? Because when I hear service provider, I think big company. Yeah. Well, so I started in a very small local service provider. Later on, I moved to some large national providers with thousands of pops and you know, thousands and thousands of routers and stuff and all that. That came later. But I started off at a very small company, just doing things I was sitting in on the meetings, right? Trying to hear what the customers are saying, what was our senior engineer saying in response, trying to understand what he's thinking about, quizzing him afterwards as well and saying, So why did you say that? You know, go through a period of this learning, and then uh it really culminated in some time later, me switching companies. In I ended up going to a company and actually having a full-time dedicated design role. So that was the role was full-time working on design. How was the culture at the company before you left? The reason I'm asking is when juniors, for lack of a better term, start asking a bunch of questions to seniors, which I've been the junior asking. Some people are open to that. Yeah and other people are like, dude, leave me alone. Like, why are you asking all these questions? Because you're almost questioning their decisions. If they're not secure and they see you as a threat, you know, well, why did you design it this way? Because I did, dude. Like you know what I mean? Leave me alone. Did you feel any like people didn't like you asking a lot of questions? Not at all. Super friendly team. We were a small but very close-knit, really good team of people, actually. We got on really well. We would hang out at the weekends as well. So that's a really good team. So do you so it was actually sad to leave to leave those guys? But I really wanted to get down the design rabbit hole more. And after a few years, there was not much more they could offer me. So that's why I ended up. Did you get a big bump? A pay bump when you left? Because usually when you leave, it's good, right? Yeah, well, there's there's two sides to that. I did get a big bump, but not just because I left, because it's also a much bigger company. Right. I left. Yeah. If you that's the trade-off, right? If you work for a smaller company, you probably get more freedom to do what you want, but you're also going to get paid perhaps less than market rate. You have to make the decision what uh that could be a whole episode unto itself. Yeah. My I've been working for gigantic companies for as long as I can remember. And my wife, who knows me very well, says, I think you'd be very happy at a smaller company. Yeah. Because you know, if you have an idea, it's not like three months of debates and calls and like uh big places can move slower, right? Oh, there's a large service provider I work at. I don't work there anymore, but everyone who ever worked there, we have a uh let's call it a survivors slack group. And uh survivors of what? Yeah, of this of this company and the kind of the and the uh the politics there, and exactly what you described. We all have the same joke, which is when we're at that company, um, even though we're there at different points in time, right? You asked to do a project, it's always the same response. Six months, half a million pounds. Everything takes six months and half a million pounds. So if it's not going to get us at least half a million in revenue, we're not interested. That's that's kind of the way that company worked. So I think there's there's something to be said about do you want to be a big fish in a small pond, or do you want to be a small fish in a big pond? Because you can you can learn a lot. If you're a small fish in a big pond, you can just go somewhere and be a sponge and like suck it all up, or you can take all that knowledge to somewhere much smaller and be a big fish in a small pond. And and I don't mean to disparage the business folks who are trying to make money for the shareholders and that you know, like I I the the more experience I get in my career, the more I see like, oh, right, this is we're servicing a business, we have to generate revenue. But it's disheartening when you have a great idea, you know you do, you've done a bunch of work, and they're like, there's not a return on this investment. You're like, yeah, but we could be more efficient, we can have less outages, you're not gonna make it 500,000 banks. Let's get to so now you're an architect, you're designing, you're at the big place. How did that go? Are you going from like you were working maintenance windows, doing changes? Yeah, architects to me seem like they sit on a golden throne and they just impart their wisdom on the dum-dums. Is that is that how the job was? Thankfully not. So it depends a bit on the company culture, but and I have experienced exactly that culture. So yeah, I know, I know what that's like. That company I switched to, they were not doing that, and I was really happy about that. So, first I was in design at this company, and then I actually moved into a separate team for architecture. And you need a feedback cycle. If you're going to design something and then you hand over to someone else to implement, and then another team is doing the the daily support, and you keep getting tickets about this thing coming up, that needs to get fed back to the design team because there's something wrong with a design if we keep getting tickets for the same thing coming in again. And they had this nice, yeah, this nice feedback process. So, yeah, it wasn't that kind of like these guys, these are design guys over here, and they're totally siloed from the operations guys over here. We had a nice cross-pollination of stuff and kind of feedback loop going on. I have a question. Yep. Two questions. How do you know the design is correct? Good, good question. And related to that, were you working on Brownfield or Greenfield? Because I'm guessing there's networks already in place that you're trying to improve the design. You know, it's probably easier to build a brand new thing. Like, yeah, right, here's the best practice, here's what we're gonna do, here's all the new latest and greatest. So you have an existing brownfield environment that has to operate, and then you're trying to move the design in a direction. I've spoken to other architects and designers who are like, well, you do it a bit at a time. When this stuff goes end of life, you swap that out, and but but then you have to interconnect all that somehow. So I guess back to the first question, it's probably obvious when you're in a knock and all the things are breaking and you see, oh, the design isn't good. So now you're an architect at the big place. When you look, do you immediately see the problems in the design, or does that surface through outages and pain points? And then how do you decide? Because I guess today there's this oversimplification of like, well, everything's spine and leaf and it's co-architecture, run EVPN and like, and then that somehow magically just waves this simplistic wand that like everything's solved. I've never managed a spine and leaf EVPN VX lane fabric, so I don't know if that's true. I understand why it's better for East West and more you know resilient and stuff, but how do you know the design is correct? And how do you know what direction to take it in? Because there isn't just one design decision to make, right? Absolutely, absolutely. Yeah, it's it's an open question. So I will first lay down a question that we can come back to, which is about requirements. So part of that is about requirements, and I ended up moving into product development to influence the requirements. That's a good pivot. I was trying to figure out how we were going to get there. So I'll come to that back into the requirements is one major. So coming back to that's how I ended up influencing the requirements. But at this point in my career, I'm working on design and I'm just getting the requirements handed to me. This is what we need to do. So, for example, new application? Is that what you mean? Or like new services? So it could be like, so I'm working in service provider networks, so it's usually some sort of connectivity um service or product. So one example, I'm gonna this is not a real example, but one example is like we've got broadband subscribers, you get a dynamic IP address. Yeah, you're from your broadband at home, that's very common. Static IPs are really painful from the technical side of things. It's a really every service provider guy will tell you doing static IP is a real pain in the backside. But at some point, you know, the requirement lands in our team, we need to do static IPs, and there's a bunch of different ways we can do that. Your question is basically how do we come up with a design and we say, yeah, this is the way to do it. Like static IPs out to the edge to the customer, or entirely? To my broadband subscribers, they want to have static IPs. Oh, for like a business service or something. Yes, for example, so business, they've got remote IT support. Got it. Yeah, they need a static IP. I mean, they're DHCP usually just a home, but yes, businesses need static for services on a yes. Let's say that I got it. We've got some some ISPs do do it for domestic customers as well. They charge you extra and you can use static IP. Everybody hates doing it. It's usually if you want to um like offer a service out to the internet, right? Like you need a static IP public place to hit. Yeah. Yeah. Okay. So you guys are gonna roll this out. So let's say we're gonna roll this out. That's that's the requirement that lands in our lap. So then how do we your question is like, how do we come up with a design and know that we got the requirement? We hit it. How is the design good? And it's a tricky question because there's a bunch of different ways we could technically do the thing. So when you get into design, you're kind of thinking about okay, well, not just technically, how do we achieve it, but for example, how do we monitor this? How will the uh operations guys troubleshoot it? I need to document what does it look like when it's working properly so we know what good looks like. Then when the when the when there's something goes wrong, the ops guys can go, hey, this doesn't look like what we've got documented as working, so there's a problem here. So how will we debug it and troubleshoot it and monitor it? And how will we, what alarms do we need to know when it's not working? You have all these kinds of things that you need to think about. Like it's the whole package around it and not just like how will I do static DHCP to my customers. You said there's a lot of ways to offer that service. Do you think that's part of the problem? The flexibility, I guess, is good. I don't know. If if we offer 15 ways to do a thing, yeah, but then we say we need to standardize our networks and stop building snowflakes, there are two opposing ideas, I think, right? Yes, yes, because maybe one of the things is vendor specific. So only there's there's a really nice way I can do it with my current vendor, but it's proprietary. So, okay, so I'm gonna probably scratch that off the list. There's another way we can do it. Um it's vendors. Yeah, it's the vendor's fault, right? Okay. So the thing we used to look at um was how do we strike these off the list? Yeah, kind of having a kind of pros and cons of each of these different ways and trying to compare them and trying to strike some off the list because it's perfectly normal to have a situation where A and B both just look like really good options. Yeah. You know, that's perfectly normal. And I th one of the lessons I learned back then was actually um a really important lesson, which is you can only work with the information you have. Right. And what I mean by that is if the information I've got is this is this this is the scale we need to get to, or this is how this is how quickly the failover needs to happen, or something like this, or we don't have that information, I can only work with the information that I've got. We can design something that as far as we know fulfills the requirements to the best of our knowledge. And at some point you do have to draw the line in the sand and say I have no idea what the subscriber growth will be for the next five years. So when will they all want static IPs? Will they not want static IPs? Will V6 get rid of this? Who knows? You know, at some point you do also just have to draw a line in the sand. And so we would have a process where we would hand a design around all the different teams. So the head of support, the head of um delivery, the head of design, each team would basically sign off the design and say, as far as I know, yeah, we can't really do any better than this. So we accept this. If in 12 months from now we need to review because we were wrong, we can review and and re-jig and redesign and do whatever. But that's also kind of a a way we decide that we got to the best design, which is that we can't think of any reason not to go ahead with a design. Everybody that looks at it says, This looks fine to me, but there definitely are some risks here because this we've added a box, the box has to carry a lot of state, we don't know how well that will scale. But we also accept that fine in 12 months maybe we add a second box where I said, Well, and I think working as a team is helpful. Yes. Because if you're the only, if if I were the only architect having to make all those decisions, I would be an analysis paralysis. Like I have 15 ways to do it. Yeah, I don't know what's gonna happen in 12 months. What if I do it wrong? Yeah, but if you can bounce it off people and come to a consensus and share that responsibility, make the best decision you can at the time. I forget her name, but there's a brilliant um woman I've seen in like different you know videos that she's like, you know, no one knows if the decision is exactly 100% the right decision. But the good news is you make a decision and you reevaluate, you can always make a new one. Yes, yes, you know. So same with design, right? Like you do your best, you figure it out. Before we run out of time, I want to pivot to your experience in product. So, right now you're a designer, you're an architect in this big place, you get requirements, you figure out how to do it, you have consensus with a team, you deal with the evil vendors and their proprietary nonsense and how they don't understand the meaning of. What standard is, and six vendors implement a standard six different ways. Don't get me started. Did you eventually wind up like how do you you developed a product, right? Like you got into like creating services. Yeah. So the the previous company I worked at and my current company have been working on product development. I actually just finished the product development stuff. So I handed it over to someone else because I was splitting my time between architecture and product development and kind of not doing a great job of either. So I kind of went to my manager and said, okay, I need to just do one. So hold on. You're an architect. And then someone comes to you and says, Hey, you want a second job? Yeah, well, so I was right to do the architecture thing. Yeah. And then we realized we needed help on the product development side. And I'd done it in the previous role. So I said, okay, I'll get involved. I'll help. So you were interested. It's like, oh, I can help with this. Yeah, I'd done it before, so I can help out with this. But then after a while, the role has grown. Now it needs a full-time person. So I had to turn around to a manager and say, I don't want to do a bad job, so please come and get someone else to come in and take it off my hands and I'll go back to architecture. I should have asked you up front, but I'm going to blame jet lag. Where do you work? Where are you doing this at? So we're for a company called Interlink. Yeah. We're a network as a service provider. So everything we do is kind of all automated and in a lovely little web UI. So customers just log in, order whatever they want through the web UI, and everything just poof happens. That's why I'm autocon, right? So you know we're huge users of automation. And that's also we do a lot of product development work because the company's four between four and five years old. So still quite young. So we're still you know filling out our portfolio, let's say, of all of all of all the products we'd like to have. And the pivot into so to answer your question, the pivot into product development came from in a previous role working in architecture. I said earlier we get something on our something that lands on our lap, those requirements already exist. So if I use my example from earlier, which is we want to offer static IPs, that requirement came from somewhere. Do they usually come from customers? Well, yes, we usually come from customers or sales. But if I go back one step further, I said um I started in operations. We're we're working on a on a network with a flawed design, and that's causing us a problem. So I wanted to move into design and architecture, but now I've got the same problem in design and architecture, which is the requirements that came in, they were also flawed. So I want to move one step along in the pipeline and actually have some input on those initial requirements. We need more time because everything you say, I I I love this conversation. How do you know a requirement is flawed? If you look at the requirement, so for example, in this example, this fictional example of we want to do static IPs, many people do follow this same rule, which is the rule of five whys. Why do we want to do static IPs? So we go back to sales or go back to the customer and say, why do you want that? And the customer says, Well, I run a small, like you said, I've got a small business and uh I've outsourced IT. So the IT guys need to like remote desktop in and do stuff in static. Okay. So why do you need a static IP for that? Well, if we don't have a static IP, you know, but they they won't know where to remote desktop too. Okay, okay. What so then the thing is you can push back on that and kind of say, okay, now that's the requirement. The requirement is not that you need a static IP, the requirement is somebody needs to connect into your network. That's the actually that's the requirement. You have to sort of push back on it with the five Y's. Someone wants to connect into okay. What about this? What about if we just give all our customers uh dynamic DNS? I can have the DNS record james.isp.net. Anytime my WAN address changes, the DNS record just automatically updates. How about that? Then we don't have to do static IPs, which no if it any service product guy would tell you is a real pain, like I said earlier, pain in the backside. So why don't we just do that? That's that solves your problem. You can have a DNS record you can give to everybody who wants to remote in. We don't have to do static IPs. Yeah. So that's kind of if you get into the product development, you can kind of then actually change the requirements. So you get one step earlier in the process. The five Y's are brilliant. Yes. I don't know who came up with it. And every time I hear it, I'm like, oh yeah, that's because at different places I've worked, yes, you know, a customer or a prospect will say, Well, if you would build such and such feature, then we would you know spend five million dollars with you. And the product uh managers that I would talk to that would be developing the products would say, you know what, everyone says that, and we could spend a bunch of limited development resources we have to build that. And it's for one customer and they might not even purchase it. Yeah. I thought it was compelling the good ones. They would start with the five whys, like, well, what problem are you trying to solve? Because they think it's just so fascinating to me how our minds work. And like, I have a problem and I think of the solution, and that's what I need from the thing, but then you can push back and say, okay, well, what what's driving this requirement? And then you can come up with something as easy as dynamic DNS instead of building this nightmare static route stuff that I don't need, and it might even cost me more. Like, I'm gonna pay for static routes. Yeah, oh, yeah, yeah. So it's dynamic DNS like if you pay for it, it's not much compared to like a static route. So you're saving me money. It's not like why don't these people build the features I want? It's like, no, they're trying to keep the network you know scalable and supportable, yeah. Right. And also maybe save you money. You're over-engineering something. You just need remote access, you don't need static IPs. For that, yeah, that's fascinating. Yeah, so there's a there's a a phrase that everyone will know, which is actually except me, because it's now escaped my mind. But it's something along the lines of um just enough knowledge to be dangerous. Yeah. So I've met many people, and actually I'm I'm guilty of it as well. You know, I'm no angel or miracle worker. There have been many times in my career where I've had just enough knowledge about something to be dangerous. And that's also kind of what happens. If we have people who work in pre-sales or solutions architectures, and they have just enough knowledge to be dangerous, they say, okay, what you need is static IPs. So that's what they'll throw from the commercial side, they throw into the technical side. We need to start doing static IPs because they know that is the way we could technically solve the customer's problem. Well, and here's an interesting wrinkle from the business side. If you can charge for static IPs and not dynamic DNS, then why wouldn't you right? Like lead with what you can. I was a cable guy back in the day, and we were compensated for a couple of years on what we could upsell the customers. So, of course, like if I'm gonna make you know 50 bucks on an internet upgrade, I'm gonna tell you you need more speed, do you? Yeah, I mean, what yeah, more speed is better, but then I get 25 bucks. I'm like, yay! So it's it's an interesting balance between you know the requirements coming in and then the architecture and the design decisions and then the business. And there's a lot of moving parts. Yo, yes, yeah. Yeah, product development is a bit different to architecture because yeah, we are so just what exactly what you just said, we have to go through all those questions of like, why would we want to offer this product? So maybe it's a value add, like you just said, we can actually add more value and sell it, and that's that's more revenue for us. Maybe it's because all our competitors do it, and so if we don't do it, we're not interested in the product, but our customers go somewhere else because they also need that product. They don't want to buy from two different vendors, they want to buy from one, so we have to do the product as well. You know, there's a bunch of kind of questions like that you have to kind of get into. But I think what's interesting is for people who are in tech and kind of stay in tech, there's this very um pervasive view on in the tech side of things that sales and marketing are evil, they create the problems. We solve, we solve the problems that all that might be partly true. But no, you I'm sorry, keep going. You make a good point. Well, the thing is not isolated, right? Yeah, yeah. That's it, they're isolated. If the salespeople knew better questions to ask, or if the like, you know, I would I even had that thought earlier, and I know better. When you said a bad requirement, my first thought was like, well, sales should surface why aren't they using the five Y's and surfacing the real requirement to hand it to the product folks? But to your point, they're just kind of isolated, compensated, different. Like I've been in sales, I want to service the customer as best I can, but I'm compensated on generating revenue. So the quicker I can do that, the better for the company, the better for me. And that may not be conducive to you know a three-hour long five Y conversation trying to figure out the most optimal way to do this. Like they said they want the thing, we have the thing, I can charge them for the thing, and I get compensated on the thing. Go. Yeah, why not do it? But then you receive it on your end and you're like, oh my God, why are they doing this? So it again, just all the moving parts. It's I think it's the isolation. Yeah, that's it. But you bridge those worlds, I guess, when you were working both of those roles. You can kind of work across that aisle. That's what I think is interesting about product development. If you come from a technical background, is yeah, you bridge the roles. So I can bring the technical experience and say, okay, there's another way we could do this, you know, that's better. You the customer gets what they want, the technical guys won't have a meltdown when we show them what we're gonna do next. You've got to try to bridge that gap somehow. Because the other thing is I worked at companies where you have this divide like you just described, but we're all on the same team. We're all on the same team, we're all in the same company. So we've got to find a way to kind of bridge this gap, work together, row in the same direction. Yeah, it's really hard at big companies. And I'm not trying to create excuses, but I know you've said you've worked at smaller companies. Like, I think you can move faster, it's easier to integrate. Like at big companies, there are different business units and different management structures and different incentivization. Like, what was the I'm tired so I can't remember, but I re I was somewhere and the the leader in sales said that it was important to get um bills of materials out faster. Like we were trying to speed up, you know, it shouldn't take three days. If a customer wants to buy things, it shouldn't take three days to create a bomb, a bill of material. So he wanted to go faster, but then there was a different thing I was working on with my team, where we were trying to I I forget what it was, but basically moving faster was the opposite of what we were trying to do. Like I think we were building a bomb tool, but he wanted to go faster and we needed time to develop this thing. And it again, it's the same thing of like you're working on a brownfield, but you're trying to, you know, you're you're trying to work on the plane in the air, basically. Yes, right. And there are compete, especially in large work large organizations, there's competing priorities, and somehow there has to be glue between them, which usually means seven hours of calls every day of your life to try to get everyone aligned, which is a nightmare because you can't get anything done when you're on calls, right? So it's hard in big organizations, I guess, is is my point. Yeah, fully agreed. Yes, yeah. So where do we go from here? You're out of product development, right? Are you back to architecture? Yes, yeah, yeah. And you said you were just spread too thin, it sounded like, right? Essentially, yeah, yeah. I'm I I'm very conscious of doing a bad job. So I didn't want to do a bad job, so I just put my hands up and said, I am making mistakes here and there. Before it gets any worse, let's get something on you for being transparent and letting them know like you've reached your yeah. Well, it could also, you know, backfire, right? So I'm like, Why are you doing such a terrible job all the time, James? Well, yeah. So have they backfilled your role yet? Or are they still good? So you're back to architecture. Yeah. Are you happy doing that? Do you miss the product stuff? Um, I mean, it's definitely interesting, but actually, no, I'm happy to be there's so much to do on the architecture side. I'm very happy there again. Yeah, it's nice. And also slightly different, much more long-term focus. You know, there's there's part of architecture, which is some business requirement comes in and we design something that meets the business requirement, you know. That's one part of architecture, but the other part of it is also thinking much more long term. So every six months for the last two years, you know, roughly every six months, we've had this problem come up. And it's two years, so it's happened about four times now. So there must be some more systemic issue here that we need to get addressed. So that's kind of coming from internal rather than external. But that's also kind of a very so then, okay, what do we need to do to resolve this issue? That may be a very long-term thing that takes three or four years to then get resolved. So that's also kind of a part of architecture, is thinking about these kind of more overarching long-term things. So actually, that means things like engaging with the vendor. It keeps happening that we haven't got this feature and we need this feature. So I'm gonna have to sit down, write the business case, send it to the vendor, have a few calls with them, discuss about how it would be good for us. Probably I think their other customers would use it too. So, for example, I'm also getting involved in the ITF now, so it's also that kind of thing of keeping an eye on what's happening there, trying to write drafts to put them in there, then go to the vendor and say, Hey, we put a draft into the IDF. Are you gonna add this to your roadmap? Those are like very long-term things that take three or four years to play out. It's a slightly different part of architecture. So we're getting toward the end, but you keep saying things that are awesome that I want to keep digging into. I'm gonna ask a question that I should probably know the answer to, but I know what the IETF acronym stands for, but now you say you're working for the IETF. So probably not for them, but just getting involved. Yeah, so what does that mean? The Internet Engineering Task Force, right? Is that what do they do and why are you involved and how do they push networking forward? Good great question. Yeah, should we probably wrap on this? Yeah, sure. The IETF are responsible for several things. The most, I would say the most famous thing that they're involved with that people know them for is standardization. So people write what we call drafts, they're draft standards. So you know, today we have IP version six, but it didn't used to exist. Someone had to write a draft for a new version of IP, submit that to the IETF, and it goes through this very long process of being you know interrogated and redesigned and refactored, and eventually, some years later, kind of popping out the other end of this giant sausage machine as a new RFC standard. Dude, we could have done like a three-hour show because my follow-up question was there, I think, are multiple standards bodies and how do they feed into each other? And I was going to ask you about RFCs next, but the IETF work generates RFCs, as they just so they generate RFCs for standards, they generate what we call BCOPs, best current operational practice documents. So they're not standards, they're advice on how you should do a thing. They generate also informational documents. So we've been running EVPN for the last five years, and we put out a document with all of our hard lessons learned for other people. So you've got these kind of different output tracks, if you like, that come from the IETF. So that's me. So as an architect, and my role is to think, okay, more long term, if there's some feature we're missing, um, we could go to the ITF, try and work on getting it developed as a new standard, going to a vendor, trying to get the vendor to also agree to implement the new standard and so on. How do you get involved with the IETF? Uh so actually, anyone that's listening, absolutely anybody can get involved with the IETF. That's one of the wonderful things about the IETF compared to some other standard bodies. Other standard bodies are you have to pay to play. Um, so which isn't which is not ideal. But IETF is completely open. The least you have to do is just go and sign up to the mailing list. There's all different areas. So for IP or for PGP or so for inter-domain routing or secure routing or um all different areas, right? So you go and find an area you're interested in, you can sign up to the mailing list, and just you'll start receiving the emails, the conversations going on, people talking about new standards they're working on. And you can just start by passively reading and learning. And then at some point, people will call out to the mailing list for feedback, and you can weigh in. Give your just respond. Give your we've been doing this for a couple of years ourselves, and I can tell you that you missed this thing off. That will bite you in the butt at some point if you don't factor this in. So you can just start as easy as that. Just be on the mailing list, see what's happening, learn, read, give some feedback if you've got something valuable to add. And you're contributing to moving the industry forward in a better direction. Yeah. Last question. How many times have I said that? Um, you said you can go to the IETF and get involved. And then you mentioned in passing you can also go to vendors and ask for them. So are the vendors involved in IETF, or is that a separate the reason I ask is I interviewed Radio Perlman and I don't know if it was the IETF, but there was some governing body that they said it was just a bunch of vendor people fighting each other trying to get their thing standardized. So are there vendors involved in the IETF? Yes, the IETF has got like a cross section of you've got people operating networks who are kind of in the trenches, and either they're either there is some existing technology that needs improving, so they want to kind of update an existing standard, or there's something that's missing, so they need a new standard. That's so that's kind of your network operators. Then you've got network vendors who then actually have to implement these things. So I'm missing a BGP knob that lets me do something, you know, some requirement. There's no there's currently no knob for this, so I'm going to go to the IETF the operators might go to the IETF, put a draft standard in for some new BGP knob. The vendors are there kind of picking up the drafts and trying to implement them, also submitting their own. So if you're a small company, you may not have time to get involved with the IETF, but you go to your vendor and say, I wish you had a knob for this, some configuration knob for this. They may submit that to the ITF themselves. And there's a third spoke. So you've got operators, you've got vendors, and there's kind of a third spoke, which is academia. People in academia thinking about new novel techniques and ideas for networking, and then putting that into the ITF as a draft to get it standardized. So we've come up with a new uh yeah, I don't know, someone's tweaked the way that the shortest path first algorithm works so we can converge the network even quicker. So someone in academia has done this, maybe they're going to put a new draft into the IETF to speed up OSPF and ISIS, for example. There's so many new ideas. I was because I'm thinking as you're talking, like, haven't we figured out how to solve the networking problems? But I guess there are smart people working on problems, seeing gaps, and then feeding it back into the system to improve it. So it's good that we're evolving. Yes. My bias is we've figured out the problems, spine and leaf and fabric it all and walk away. But that's not the case because there's people like you who are in there working on it, pushing the industry forward, trying to help, which is fantastic. Yeah, there's also this kind of every time we add a new thing, someone then comes up with a new use case based on the new thing. So then they run with that for a while and go, ah, but this is not as ideal as it could be. Back to the ITF, try and improve that new thing, you know, and kind of it's just like a never-ending feedback. And then there's interop. That's probably the wrong use of the term, but if you have a new thing, how does that work with other things? Yes, right. And does it work with the other vendor thing? Yeah. Uh oh, we have two vendors connecting and that thing is different than than the other thing. James, this has been fantastic. I could have talked to you forever. Do you have any parting advice for so because you've worked we because you work in service provider and you also worked in product, anybody looking to get into either, or if they're insane like you and want to do both at the same time, what would you say to someone who is interested in service provider? Because I think that's more interesting, probably, because there aren't that many of you. You're the second service provider person I know, you and Kevin Myers. Okay. So I'm guessing we need more. Yes. Oh, yeah, yeah. That's a real problem. What would you say to someone? I mean, obviously, from this conversation, if you are a level one or level two person and a knock working outages all the time and you see the design flaws, that might be a good sign that you work, that you belong in service provider and eventually architect. But what advice would you give folks for, you know, that that are interested in the type of work you do? The one thing I would say is there's no replacement for knowing the fundamentals. Interviewed many, many people over the years who just don't know the fundamentals. So some part of it is simply you just have to put the hours in labing. That's always, you know, you have there's kind of no escaping that. Like, so if you work exclusively with wireless, you know, you do like Wi-Fi stuff for big events and you want to get involved in doing broadband stuff, for example, there's that there'll always be some amount of you just have to put the hours in doing the labbing, learning the technology, reading the books, kind of that's one part of it. The easy to define part, let's say that you do have to have some technical knowledge about those technologies. The other part is then also, yeah, like like you asked me earlier, how do you get into design or this is a lot more tricky to define. The the thing that's opened my eyes up the most is actually just looking at what everyone else is doing, because everyone else is running networks in slightly different ways, they've got slightly different problems, slightly different customers. So look at what they're doing. Because uh so what we said earlier about the IETF and standardization, there is only one IP version 4 and one IP version 6, and there's only one EVPN and there's only one Ethernet. So we've all got the same tool sets, but all these different networks and companies are using the tools in different ways. So what opened my eyes the most was actually just trying to see how is everyone else doing it? They've got the same tools as me. How are they how is it they're managing to do something different with the same tool set? So go to conferences, or if you can't go like you know, watch they always end up on YouTube, right? So watch them on YouTube, listen to podcasts, be on mailing lists, be on forums, and just try to kind of learn from what other people are doing. You'll see people posting on a forum, we're doing this and this and this, and it's not working. Has anyone got any ideas? Just passively read the conversation and then you'll see okay, ah, that's how I could have done it as well. That's really smart. Widen your aperture, yeah. Get involved in the communities, join the conversations. I like that. Yeah, so I'm I'm I'm on many, many mailing lists and forums, and I listened to loads of podcasts, just just uh as a passive consumer of of the knowledge, right? To try to learn from other people. Speaking of mailing lists, we're launching a newsletter very soon, and I guess I'm gonna have to include service provider content to pull you in, sir. Um, James Benzley, this has been a fantastic conversation. Thank you so much for for being here. Um, where can folks find you? Uh best place is on LinkedIn. I'm on LinkedIn, so James Benzley on LinkedIn. I've got a sombrero on. I think I might be the only guy with a sombrero, so it should be easy to find. Uh, I'll put your uh link to your uh LinkedIn in the show notes. For all things Art of NetEng, you can check out our Link Tree at LinkTreeFord slash Art of NetEng. Shout out to the new website. Oh, yeah. At Art of Network Engineering.com. We have a new website, and there's a really fantastic resources section, mostly free resources. I know there's a bunch of RFCs in there, a bunch of different trainings that are um available, some communities. Uh I am gonna have to add some service provider, which I think I didn't do. So these are good conversations. They remind me of um of some gaps that we have to fill. But uh check out the website, bunch of cool stuff there, and uh, we're launching a newsletter very shortly. So keep your eye out for that. Thank you so much for watching and listening, and we'll catch you next time on the Art of Network Engineering podcast. Hey folks, if you like what you heard today, please subscribe to our podcast and your favorite podcatcher. You can find us on socials at Art of NetEng, and you can visit Linktree forward slash Art of NetEng for links to all of our content, including the A1 merch store and our virtual community on Discord called It's All About the Journey. You can see our pretty faces on our YouTube channel named the Art of Network Engineering. That's youtube.com forward slash ArtofnetEng. Thanks for listening.