00:48
This is the Art of Network Engineering podcast.

00:59
In this podcast, we'll explore tools, technologies, and technology people, re-enter for new information that will expand your skill sense and toolbox, and share the stories of fellow network engineers. Welcome to the art of network engineering. I am Tim Bertino. Well, I almost said I'm AJ Murray. That would have been awkward. Because, you know, clearly it's not awkward at all now that I pointed it out.

01:24
Anyway, I am at Tim Bertino on Twitter and joining me in this episode is Lexi aka track at Pacer. What's happening, Lexi? Howdy howdy Just winding down from a day of work Nothing too crazy this week. Luckily, I guess how about you Tim what's been going on? Uh, nothing kind of like you nothing too crazy. Um, I do have I appreciate you Lexi actually

01:52
jumped on a little bit earlier in this session because I've got a maintenance window tonight doing just some code upgrades, but with where it's at, I wanna go in for it just in case you just never know. We've had the scary stories episode and all of the oh shit moments, so I've learned to not trust anything. Never assume a maintenance will ever be simple.

02:20
Then you will not be disappointed when it isn't. Yeah. Or wait, yeah. Then you'll be happy. You'll be extra happy when it is simple. Yes. I am a professional now at setting the bar very low. Set those expectations. That's the name of the game. Especially when, yeah, anything to do with maintenance or break fix, that's the name of the game.

02:50
So I don't know if I'm just hypersensitive to this now that since you started working at Blue, but I seem to notice a lot more like rocket launches in the news and that kind of stuff now. So I don't know if like NASA's ramping things up. I saw they had something recently and Blue and some of the others, but.

03:12
Mm-hmm. Well, yeah, Blue had a launch not too long ago now and now but NASA is like the big news right now because they just launched Artemis SLS, which is a huge, huge, huge, huge, huge, huge achievement. It's it's a rocket that's had a lot of delays, but it's going to take humanity back to the moon. I think we're going to deliver the first female astronaut to the moon, which is insane. And I think maybe the first person of

03:41
or woman of color maybe. Okay. Like a lot of firsts with this and it took a lot of effort and I'm like, I admire NASA so much and the launch was yesterday, the day before? It was at midnight recently. And yeah, actually pretty funny story about that. They had apparently, supposedly, this is just hearsay slash news reporters, so who knows, but they had an ethernet switch or some kind of little switch die on the rocket just before.

04:11
I saw you put that on Twitter and I didn't I knew there was more context there Rust in peace. Yeah, apparently they had now. I don't know if that is actually people were calling it an ethernet switch which is You know, I don't know how I Don't know how like like they're trying to be specific and like make sure people know it's like a network switch I think is maybe what they meant by that sure cuz like an ethernet switches and

04:40
exactly a thing, I don't think, right? Like it's not really an Ethernet switch. It just knows how to work with Ethernet. You know, you and I have talked, we've talked about the different terminology and how people use it. And I personally, I think it's probably just people may think it's a fancy way of saying it's just a layer two switch. I don't know. Yeah. Yeah. But it's just funny because like,

05:04
people were like speculating on NASA's like network on the rocket because of that, which is always fun to do, right? Like how are they doing space network stuff? But I don't know if like either does, which is actually like what it was or like what is a network switch of some kind. Like was it layer two, was it layer three? We don't know, but it's funny. But A1 fans, I need to encourage you to check out our YouTube channel for this episode because seeing the genuine excitement

05:33
in Lexi's face when she was talking about that NASA launch. You don't see that, that genuine level of excitement when people are talking about their field. Yeah. So check that out. It's just really exciting on a lot of levels, right? Cause space is cool. It is my field and also like the funny part of the ethernet switch. And then, and then like, you know, we're delivering like the first woman to the moon. And that's really, really exciting. And just.

06:00
Humanity going back to the moon. I get all teary-eyed when we talk about Moving humanity forward and getting us out into space. So it is exciting. Yeah On that note, let's talk about Other stuff So jumping right into it this week. So in episode 61 we talked through some different options around home labs and Lexi had the idea to take that a step further

06:30
in this episode and talk about how can you translate the quote home lab to a work environment? How would you use a lab at work? What are the components? What are some of the different scenarios? So I guess I'll kind of kick it off right there at a high level. What makes a work lab different than a home lab? Well if you're asking me, the work lab is physically.

06:57
A lot bigger, probably, in a lot of circumstances. Well, maybe that, hmm, okay. I'll just say that's the difference between my work lab and home lab. Sure. I don't know, maybe some people have like 48 e-racks in their home. I don't know how common that is. Well, Taylor, if Taylor's watching this, AJ's buddy Taylor, he may. He's had some pretty hefty gear at his home. Okay, that's fair. But I agree with you. I think that...

07:27
you know, a company's budget, hopefully they budget to help their engineers plan things out and test things out in a lab environment. You would hope a company's budget for labs is going to be greater and give you more opportunities than you have to pour into it at home. Hopefully you're working with a higher budget for sure there. But I do think some of the similar opportunities and challenges apply. I think there are...

07:55
options for both hardware or virtualization in a work lab. And one of the things that I wanted to highlight is I do think there are some things that you really need to do justice with physical gear. I always talk a lot about Cisco because that's the most that I know, but there are certain technologies, and I'll call one out is there on their catalyst side, their multi-chassis ether channel.

08:24
either a virtual switching system or stack-wise virtual. To me, unless there's been changes, leaps and bounds since I've really messed with it, some of those are really specific to hardware and you really need the separate gear to be able to test that kind of stuff out in. But I do still think that there's a lot of benefits to virtualization and one of the biggest use cases that I have for...

08:51
doing labs, especially in a virtualized environment at work is planning out and testing maintenance windows. Up until like the last couple of years, we would get, me personally, I would get everything out on paper or digitally and go step by step and cross fingers and hope for the best. But in the last couple of years, we build out an even G environment.

09:19
and really been able to get virtualized switching in there to test things out. Now, of course, smart. I mean, you're not going to get like for like, I did have some labs where we were, I was planning out like an internet failover, we have multi multi home BGP that I was testing out and you learn things in that lab environment that you're very thankful that you learned them in the lab environment and not.

09:47
in production when you're trying to do this maintenance window. So now granted, I broke some rules. I didn't do exact like for like. We had some different models of switch OS in there and I kind of stuck to one, but I was really testing high level routing protocol failover and that kind of thing. So I wasn't too concerned about the intricacies between different hardware platforms, but you're able to build those.

10:16
maintenance window plans really based off of that lab and you can see the results and you can even put into your plan what the expected results are. And it's such a confidence boost to be able to just go down step by step, compare your results in production to what you saw in the lab. And it's good for you to learn. It's good for you to practice and it's good for the business because it's usually going to go smoother. So I would say that's the biggest use case. What's one of your best use cases?

10:46
Well, that's such a fantastic, like I wish I had in the past in past jobs, or had to do big maintenance is like, I really wish we had had a virtual, any kind of lab, any kind of lab to practice on even virtual or not. Because when I was brand new, uh, working break fix and a knock, and like, we had to fix things on the fly sometimes. Um, but they were part of sort of these standard.

11:12
sort of change processes that we knew about, we just like had to do them on the fly. I was very, very, very nervous for a long time. It took me a long time to like stop being nervous, like really nervous on maintenance windows and stuff like that. So if I had had a lab to practice in prior to that, like it would have been, like you said, such a confidence booster. Like even if it's not one for one, exactly what happens lab to actual maintenance.

11:38
It still gives you the practice of issuing the commands and looking for at least the basic symptoms of certain things, you know, like that's brilliant. I wish I had had that in past jobs. It was a game changer. Like I get like super nervous when I'm doing maintenance windows. It's going to take things down. Even, even if I know what it's going to take down, if I don't have a good feel, I haven't tried it, I haven't tested it. It's.

12:06
It's nerve wracking and those are the situations where you want to be at your best, you want to be focused and if you're not really sure or even somewhat close to sure of what's going to happen, if something goes wrong, you're going to be at least me, I get flustered. It's, it's easy to get off track. And again, just having that step-by-step plan that you've been able to build. From a lab environment, it just adds to that confidence and

12:35
and things just seem to go a lot smoother. A lot of people are doing maintenance windows, you know, at night, late for them, right? Like kind of like, I mean, it's not super, super late, I guess, for you, Tim, but it's still kind of late in the evening. And like if you're tired and you're doing a maintenance window at like 11 p.m. or to 2 a.m. or something like that, it can get, you know, you're not probably at 100%, even if you've tried to prepare for that. So if you've practiced it before in a lab environment.

13:04
you know, that really probably helps with that too. You're, you're less likely to make a mistake when you're tired if you've practiced it before. So that's awesome. Definitely. So I haven't had a lab at work until I started at Blue and I'm very, very grateful for the lab. Partly because I, I basically built it from the ground up.

13:30
And so I know how it works and how it should work and all the intricacies of it. Now, you said you build it from the ground up. Was that something that they already had there or they had the materials and you had to build it up or did you have to push for this and say, hey, I want to do this thing? A little bit of column A, a little bit of column B. I didn't have to push to get the project. It was already in existence, but it was this teeny tiny little rack.

13:58
And barely, you know, stuff really didn't fit very well in there. And it was like a rat's nest of cables. Someone before me had had, they weren't like a network person. It was just somebody from elsewhere who had just like briefly, um, been charged with like, at least getting it sort of working, um, in the meantime, before they like hired for my position. And so this person, you know, I think he did his best, but ultimately, like it was not working properly. It was in a tiny little rack that didn't. You know,

14:26
things were all crowded and cables were everywhere. It was like those horror photos you see, like data center nightmares. So it was a huge... I had never... So when my first job in networking, I never touched hardware. I've consulted into plenty of switches and things like that and done simple things. But when I worked there, 100% of my time was spent just directing.

14:53
other people to do the hardware work that I needed them to do. Plugging stuff in, switching out line cards, switching out SFPs, things like that. So I'd never done it myself and I definitely never built a rack. So I learned a lot building that thing. So it did technically exist but I ended up swapping out most of the hardware and redoing it and reconfiguring everything. So I did sort of have like a little bit of a template to go on, but I still say I built it.

15:21
mostly from the ground up because almost all the hardware in it is new now. So let's kind of start from the beginning there. I think when we talk about building labs, either on the show or in Discord or on Twitter, especially on the hardware side, or at least we can at least start on the hardware side, I think it can be kind of a nebulous topic. Like we just talk about quote the lab, like it's the cloud and people just...

15:48
are expected to understand what that means and exactly know what they need from ground up. So can you kind of talk about that ground up? And I'm gonna kind of push you a little in the direction of a great tweet thread that you had recently that talked about some tips and tricks for building a lab. So can you kind of walk us through that, what that process looked like from the ground up? So I'll try not to go too at length because I could talk.

16:14
for a very long time. Honestly, it was an extensive process. The first thing I'll say is I didn't build this overnight. I came on board and I had no idea what I was doing. And I didn't know the protocols that we were using in there, which I'll just preface this ahead of time. I can't talk specifics about the Rocket Network, but I can talk a lot about troubleshooting methods and some of the protocols and things that I use, the methods that I use in this lab, because they are not.

16:41
they don't have anything to do with the rocket network that I'm working on just before anyone asks. This has nothing to do with the rocket network. But when I first- It's okay, Andy's not here. Yeah, Andy's not here to like bother me too much about it. But yeah, I first came on board, didn't know what I was doing, so I had to learn a lot about what the purpose of the lab even was. And it turns out it's-

17:06
There are a few different purposes for it. Part of it has to do with making sure I know what I'm doing on the Rocket Network, but another part of it is just like allowing me to test certain things out, test out certain proprietary hardware that we've got and basically grow as a network engineer. My team is pretty small, the network team that I'm on, specifically in my business unit. And so a portion of my teammates are actually remote.

17:35
I am the person who's mainly in the office, local, and goes in and can physically be there in the lab. So I think of it mostly as my lab. It's our lab, but I, you know, because I'm there all the time working with it, it's really like my baby. You got the keys to the kingdom. Yeah. They have to ask your permission to get something out of the lab. Not permission, but I definitely, you know, there's parts of it where I definitely have to be present. It is not all. Sure.

18:04
remotely accessible by any means. Um, but yeah, I didn't build it overnight. It took a very long time. It's still actually in progress. It's just like, I'd say it's like 85, 90% done. And we're just waiting on some other stuff that's out of my control at this point to fill it out. But, um, yeah, it's, it's, it's done as far as what, you know, the major stuff. And so I wrote that thread about it because it took me, you know, I started at blue almost, almost a year ago, 11 months ago now, and, um,

18:34
But when I, I guess February would have been when I started. So however long, February to now, it's been like eight, nine months of work on this thing. And I'd say like six months of that was, was a lot of effort where I was just like every day in there figuring things out. And it's taken me so long, partly because, you know, lab building just takes that long. And also because I had no clue what I was doing. I made a lot of mistakes and I learned from them, but I made a ton of mistakes.

19:04
And so I had to often, I was going back over what I'd already done, like all three days prior and had to start over, right? To do it the right way. Um, so that was a lot of fun, but building, building a lab from the ground up at work when you've got bigger hardware, more complicated hardware and maybe protocols that people want you to use, but you're not familiar with is, is a big challenge. Or was it for me?

19:33
I guess where I'd say I started with this rack was because someone else had actually tried to build it before me. I couldn't communicate with this person about what was going through their head when they built it, but they did leave behind an Excel spreadsheet where they had actually, nice for me, they had documented where the physical cables were connected. We're using tap aggregation in this lab.

20:03
like which he sort of had some color coordination going on with the cables. So he had like certain color cables going into certain, like the monitor ports on the tap and then other color cables going into like the network ports on the tap. Um, so he had documented all of that, which was actually really helpful. And he had like documented the VLANs that he'd started assigning to things. So I started actually with an Excel spreadsheet and I went through that spreadsheet and.

20:28
You know, compared it to the rat's nest of cables and all the different things that were going on. And I corrected the spreadsheet because some of it was not totally accurate. And then from there, I actually built my own spreadsheet off of it. That was, um, a little more streamlined and had some more information in it. And I actually just started unplugging stuff and, you know, labeling everything. And then I just tore it all apart.

20:57
and ordered a whole new rack, a bigger rack. So this I think was maybe a 22U and I ordered a 48U. And I...

21:10
I ordered some of the hardware to like physically install these switches and everything in there. And one of the first mistakes I made was figuring out what size cage nuts to use. And if you're not familiar with cage nuts, everybody hates them. You don't have to use them on all racks. It's interesting. Like the 22U that I started with did not need cage nuts because it was literally just like a...

21:39
the steel or whatever the, you know, the... How do I describe it? I don't actually know the technical term for it. Like the sidebars or whatever with the holes in them were just like steel with holes. So you didn't need a cage nut because a cage nut provides, I guess, that it sticks into like the square part of your, where you're going to mount things and then it gives you, it has a hole in it that you can then screw things into.

22:05
Does that, am I explaining that okay, Tim? No, you are. I've actually, I don't know if this is the industry standard, if there is such a thing for the different racks. But what I've seen, and I'm not sure if it's just Panduit or some of the others, but where you have, I've seen them called where you have the square holes where you need cage nuts for, to use screws.

22:30
I've seen those referred to as server racks, and I've seen the ones that have the threaded rails that Eric mentioned in the chat here. I've seen those called network racks. I don't know if that's an industry thing, but I've seen that. Oh, that's interesting. Okay. Unless I dreamt that, I'm just completely making that up. But we needed Jordan in the chat. He used to work at Panduit. He could have helped me out.

22:55
So you mentioned something there that I think's really interesting because it's not the first thing that I think of in a lab environment. I know it's incredibly important in production, but you mentioned that you started taking notes, you started making documentation. How important do you think that is in especially a physical lab environment?

23:20
important. So I can't overstate it. It's so incredibly important. Documentation is always important. I'll never let that go. I hate when there's no documentation for stuff. You might think it's too trivial, but it is not too trivial, especially if there's a chance that someone will be in your role after you. I know we all hate to think of that when we're in our roles, but like, oh my God, please do the next person a favor out of the goodness of your heart and just document.

23:50
Even the guy that I followed who was not a network person and probably wasn't very happy with having to do rack stuff, he still documented in that spreadsheet and that was extremely helpful for me because I did not know what TAP aggregation was supposed to do in the beginning. So when you talk about documentation of the physical lab, is it really just...

24:15
I don't wanna say just, but is it really just layer one or are you talking documentation of the- One through three. One through three, okay. One through three, and by that I mean physical cable. Now, I'll say this, it will vary based on, excuse me, your use case. Maybe you really, see, I can't imagine the situation, but it's possible that you wouldn't necessarily have to document all of the layer one stuff, for example.

24:44
or layer two things, but like, I still think it's important. Right? So I'll just say that I documented one through three. And by that, I mean, where every physical cable is connected to every single VLAN. So I'll get into tap aggregation later a little bit, but like tap aggregation, it uses...

25:07
Basically 802.1Q tagging, so it's VLAN tags. It's not actually technically VLANs, but it's still basically VLAN tagging. So documented the VLANs for each individual connection, documented with tap aggregation, you have tap and tool ports. And the tool port is the one that aggregates everything and sends it out, basically, a trunk to your capture PC. So documented the VLANs allowed on those trunks and where they were, and physically

25:37
capture PC they connected to. Um, so that's like, I guess, layer one, layer two, technically. And then layer three, I documented every single device, um, what the hardware was. So like Arista 71 50 S or whatever, and like it's on this, you know, image and, um, uh, what its management IP was and anything else that I had. Like configured on there that could affect like remote access, you know, um,

26:08
And then like small note, you know, had like a note section of like, okay, well, this connection is going to change later on when this device gets into the lab, but I don't have it. So I'm doing this temporary thing now and things like that. Right. It was like my Bible basically, when I was in that lab working on things, like I had that open all the time and I was always changing it. Was anybody else looking at it? No, just me, but it was still a good habit to get into because you know, I'd go away for the weekend and like I needed to.

26:35
remember what I was doing and you like to think you can remember everything you were doing on Friday and Monday, but no, you can't. Or at least I can't, you know, so I always left little notes for myself like where I left off, what I was planning to do next and didn't have time to do what wasn't working that I didn't have time to fix or look into things like that. Color coding, so very helpful. I think that's really interesting at the. So when documenting it at layer two.

27:05
when you built this lab, and I'm not gonna get into specifics, but when you built this lab, did you have a specific use case in mind? And the reason I ask is, I mean, documenting down to the VLAN level, to me, I think if you're gonna do that, you're probably not gonna change things very often, you're not gonna add and remove VLANs, or are you, and you just make sure you document that.

27:31
every time. Yeah, correct. So the VLANs in particular, they actually did end up changing several times because I made mistakes. But in theory, no, once, you know, now that I have it like sort of settled, the VLAN shouldn't really change at all. And that's because with top aggregation, you tag your certain traffic with a specific VLAN number.

27:55
because once it comes in on the capture PC, all that traffic, you have to have a way to distinguish it. And you could look at things like, you know, an IP address or whatever, if you want to, or MAC address, but it's easier to just eyeball it with VLAN numbers, right? And so I could very easily look at my spreadsheet of like, oh, VLAN, like, you know, 4096 is assigned to like, you know, this port. And this port has to do with this specific thing that I am keeping track of.

28:24
And so I could easily like, you know, TCP dump on my cat PC and see, okay, am I, am I getting that VLAN or is there something wrong? And I'm being a little bit vague here because I can't explain too much, but like, basically if, if I expect traffic to be flowing on a certain link, when I sent, do a certain thing in this lab, um, and I don't see it coming into my cat PC, uh, I want to investigate that. Right. Typically when, when I've done labs, it's been.

28:54
I need to put something together quick so I can see if it's going to work the way I think it's going to work or I just want to get some practice and something to try to grasp a concept. What I think you're discussing is more of a long-term use case, something that you're building for a specific purpose. It's going to be labbed up for a decently long period of time. Yeah.

29:21
these labs are gonna evolve over time. I do think having that documentation is good because I like your weekend scenario because I'm very much that person. I put something down on a Thursday or Friday, I'm not gonna pick it up till Monday or Tuesday. I am gonna come back with nothing in my brain to be able to remember what I was even thinking. Because that's what weekends are for, right? Yeah, yeah. Empty your brain from work. So do you do...

29:50
So you talked about physical documentation, cabling, layer two, management IPs, that kind of thing. Are you drawing out like logical topologies as well? Yeah, yes. Sorry, I'm thinking, which I shouldn't have to think too hard about that. But honestly, it's a small lab. So the comp...

30:17
So it's a small lab, so the network I've created there is very, very simple. So the logical topology is not extensive. So mostly for me, in my use case at the moment, it's just mostly documenting the IP addresses and MAC addresses of things and that is basically it. But if I had like, you know, virtual.

30:46
route, you know, virtual routing going on and like, um, I don't know, VRFs and stuff. Like I would document that too. Right. Or like who is, you know, the master on this link? Am I using HSRP? What was supposed to be happening there? Like if there's any, any, any kind of like protocol specific like roles that your devices are taking, like I would definitely recommend.

31:11
documenting that too because working in breakfix in a knock definitely taught me like if you're if you have an emergency and Something's acting weird and you're looking at it and it's HSRP or something and you don't know who's supposed to be like the leader on that in that scenario like

31:28
then you have to look it up. You have to figure out who knows because what if this one is not supposed to be, and sorry, I'm using, it's been a long time since I've used like complicated protocols. So I'm probably using the wrong terminology, but like, is this router supposed to be the actual one that's receiving the traffic for the gateway or is it the other one? Because right now I'm seeing this one receive some of it and it's not the other one is, you know, like if you're in a failure scenario, you wanna know exactly what everything is supposed to be in that moment in time, assuming that it's not supposed to change dynamically.

31:59
So I would recommend documenting everything like that. Sounds tedious, but that's documentation for you. It's very helpful. I mean, that leads right back into the example I brought up earlier about planning for a maintenance window because I did exactly that in that internet failover testing example. I would pull that output that I would expect to see to say, hey, when you do this step after this step, this router should be.

32:27
the active for HSRP. So I can quickly see that, compare my notes when I'm in the maintenance window. And I know that if I hit this step and this isn't the result that I see out of that step, I know I need to stop and check and find out why. Rather than getting to the complete end of the maintenance window and knowing that I found out that I probably should have stopped 10 steps ago, but I went through the whole thing. It's really helpful.

32:57
Yeah, so document, document, document. And hopefully if your lab is meant to be like at least semi-permanent and configuration wise and physically, like it should, it'll be a lot of upfront work for a huge payoff in the end. So that's why I recommend documenting everything as tedious as it can be.

33:21
Um, a lot of other things I have to say are like, you know, learn about the sizes of cage nuts and the different metal alloys of cage nuts also matter. If you didn't know that some of them go into the square holes much, much easier and then fall out of them much easier too. If you have a certain metal alloy, um, if you have steel cage nuts, it's a lot harder to get them in, but easier to work with. Um.

33:46
There's all sorts of physical, just ground up things that before you even get to documenting your protocols or anything, which switch? Get a label maker, right? Like I'm talking about documentation, also get a label maker and label your report, label your links. That's also tedious as hell. You're gonna hate that, because I hated that so much. I labeled every single end of every single link. So every cable got two labels, right?

34:12
It's good, you know, especially because it's going to a tap So I had to like every tap had like four labeled little cables and it looks a mess but like you got to do it right Yeah, you've talked about a couple times how You use the lab to not only prove the concept that you're trying to do in production But use it as a way to practice how you're gonna you as an engineer are gonna behave in production and if you

34:42
practice that concept of doing the documentation and doing that labeling in the lab, you're more apt to do it in production. Whereas if you just say, nah, this is just the lab, I'm just practicing to try in production, you may end up picking up some of those habits. I don't know how much I wanna bring up sports references, but it's, I was always told growing up, you play in the game how you practice in practice, so.

35:12
That's a really good point. I don't think I'd even thought of it that specifically, but a hundred percent. Yeah, I never really have either till we started talking about it. It's like, you know, if you adapt good habits and when you're going through the lab environment, those may translate into how you handle things in production. Yeah, I can't tell you how.

35:33
Yeah, how nice it is to be able to look at a link and know exactly where it's going and not have to go look it up in your spreadsheet too. I mean, you should also have it in the spreadsheet, but like it's so much faster to be able to like to just organize yourself in that way. What else did I document? I think that's... But I documented every device, all the images they were running, their management IPs. I was documenting Mac addresses at one point.

36:03
and VLANs, tap aggregation, physical topology. I learned a lot about cable organization. That sucked as well. That was horrible. I started out thinking, oh, I'm going to be real cool and have color-coded cables. And then I couldn't make up my mind what the colors are going to mean and what color for which thing. And then I realized I have a ton of cables at my disposal already, but like,

36:30
they're not all the perfect right lengths for everything and like the same color. So I just gave up on that, right? Which it's a lab rack, right? Like it doesn't have to be pretty. So like my rack is, yeah, yeah. At the end of the day, who cares, right? Like you're the one using it. And so, so my rack is not like this nice, perfect, like everything is one color in a bundle thing. It's like a spaghetti mess looking colorful, like rainbow of, of cables, but.

36:54
They're all organized and they all go somewhere and they're bundled. So it's nice, but like you look at it and your eyes are like, Oh my God, but that's okay. Right? Like just whatever works for you organizationally, just make sure you don't have a rat's nest of cables. So that was another thing. Actually. I, the reason I changed, um, the reason I changed a lot of my, um, my mind a lot on certain things, specifically what I was done when I was like putting things in the spreadsheet of like, Oh, I'm going to make this tap port go to this

37:24
specific port on the Arista switch and I'm going to make the other tap port from the same tap go to the this other port on the same Arista switch like I had to I Don't know if people have like spent a lot of time looking at these pictures of like nice cable management You know But like you can kind of tell that some some div on some devices You'll have like the device split in half or something and then like on one half the cables go one direction

37:48
the other half the cables go the other direction. So I generally actually started working in that way, but I was like halfway through with assigning things to ports. So I ended up just scrapping a lot of it and starting over again, because I wanted to make sure that I wasn't crisscrossing cables constantly in the front of the rack. I wanted them all to go certain directions. Tedious, annoying, frustrating. I wanted to pull my hair out, but ultimately it was so, so, so, so worth it.

38:17
Plus it, I think doing it that way probably takes some stress off of the devices and the cable bundles themselves. I would also recommend if you're buying a new rack for your stuff, definitely get it with the, what do you call them, the cable management sort of combs all sticking out on the sides. Those are so nice because they'll just, you can just drape your cables over them and they'll hold the cables and it won't.

38:42
put a lot of strain on your cables and the devices they're connected to. Yeah, I think that's another thing too, that if you practice it in a lab, hopefully you'll do it in production because I've had to replace switches before. So like single switches and a larger switch stack and people had plugged cables in just straight down from a patch panel, just straight down to a switch without taking them to the sides like you mentioned.

39:12
And if cables are too short, I mean, you may not be able to get that switch out. So yeah, practice that stuff in a lab. I know we've talked about how some of these things sound tedious, but doing this in a lab environment will make you think about translating some of this into a production environment. Yeah, and if you're building, yeah, if it's a pretty big environment that you're gonna be going into, I would say,

39:42
Like with my little rack, I thought I had everything I needed. I had tons of kit. I had boxes and boxes of cables. And I was like, I will never need any other cables. But then I was halfway through it and I realized, Oh no, I'm going to run out of cables. Uh, and, and I also realized halfway through poor planning on my part, but this is me not knowing what I was doing. When I first started, I needed like two more switches and I did not have enough ports for what I was trying to do.

40:09
And with the current environment, I don't know how it is right at the moment, but it seemed pretty difficult to get the particular switch that I wanted within any good amount of time. So I ended up actually retrofitting one vendor switch for another one because I could not get in time the same exact switch that I was trying to swap out and then get multiples of. So I ended up just replacing

40:38
all of those switches with like a certain other type of switch with a different vendor. Okay. So then I ended up learning then about like, how do I translate like vendor syntax to vendor syntax for this specific technology? Also a huge headache. That was, it was a very big learning experience. That was fun. Did you have to do a lot of research for that? Or were both models like?

41:05
Was the feature parity there? Did you have to spend quite a bit of time making sure that you were going to get the right thing? I did have to research a lot. Um, and then I actually reached out to like Fender reps and stuff. And I was like, Hey, what do you recommend? Because like, I can't find actually the one I was trying to replace was end of life, end of sale. And they were like, not go early end of service or whatever. And they weren't going to like help me with getting another one and two extras or whatever.

41:30
So I was like, okay, what do you recommend? And they were like, these are the ones we recommend. And they were super, super, super expensive and they weren't going to come in for like six months. So then I ended up actually going to our, our sweet, wonderful, blessed, uh, enterprise networking department. And they happen to have, um, several spare switches of this other vendor, um, that are probably a little too, too powerful for what I need, but they definitely get the job done. So I just, so I had to go with what

41:58
what was available rather than like the perfect fit. Um, but luckily they more than made up for what that older switch, uh, needed to do. I got lucky there, but yeah, like it was, I had to look up, I actually created another spreadsheet of like, okay, here's vendor one syntax for the entire running config, here's vendor one. And then I actually translated by like manually just went through and every single line of config and the running config on the switch.

42:28
I translated it to the other vendor and then I tried it out on the other switch, the new one and a lot of it still didn't work even though I had done the research. So I had to do more research and I had to question mark my way through things. Yeah, it was great. Wait, so how far along were you in the configuration of the existing gear when you had the forklift and go to a different switch? 65% I'd say. Like I was... Wow. Yeah.

42:57
Once I figured it out, I figured it out, right? But it took a little bit. And also like a lot of like, please, can I have a switch? Please do you have a, please, I'm desperate. I'm desperate. Just one more. Yeah. So, but that was actually really, really wonderful because I learned a lot about vendor number two. Oh, you picked up a whole nother skill set.

43:19
Yeah, basically, it was really cool. I learned some new things. Like I thought I knew a lot about that vendor actually from having worked on their equipment before, but I learned some new things about them in that way. So it was actually really great. Yeah. Sorry, I'm rambling a lot. I have so much to say. I do want to take something since you're talking about a pretty significant physical lab.

43:49
below layer one. I think it's easy for, can be easy for us as network engineers to just kind of gloss over the need of power. And we just assume that power is a utility, it's gonna be there. Do you have any advice, any tips around planning for power, PDUs, power strips, any of that good stuff? Well, don't.

44:17
do what the person before me did, which is have three power strips. Sorry. I'm trying not to be too mean, but it was kind of funny. They plugged into each other? Yeah. Because I guess the ones like- It's okay, brother. I've been there. I'm sorry, man. You did your best. Yeah, he had three power strips. One of them was a managed PDU, but it was real small. So he had it-

44:43
plugged into another one that had the other devices plugged into it. And then it also had another power strip plugged into it that some other devices were connecting. It was very weird daisy-jading of power strips. So what I did is, I didn't know anything about power strips other than you should not plug them into each other. So I actually went to, I was lucky enough to,

45:11
Oh, by the way, at the very beginning of all of this, I actually had to migrate everything physically to a whole other building. So that was another, like, wrap. I wrapped it in Saran wrap as a side note, or whatever, like big plastic wrap. I just wrapped everything really tight in plastic wrap and prayed and it got there okay. But anyway, so I, where I ended up like building most of this was in like a larger lab environment and for like another thing. And the lab techs that were there, I ended up asking them like, hey,

45:41
I need a big power strip and they happen to have one that actually mounts into a rack and you can just like get zip ties or whatever, but it's got like holes in it and it's like, I don't know, 20 ports or something. It's really big. Okay.

45:58
at least 20 ports on it. But it's like the length of the rack that you can mount on the side of it inside. It's nice and sleek. So I got one of those and I left the managed PDU in there because that was important for some other reasons, but I did not plug that in. I plugged that into the ground instead of the other power strips. And then obviously that other big power strip is now providing power for everything else in there and that worked fine. So what I would say is don't.

46:28
just get just try your best to figure out how many devices roughly at least that you're gonna want and then add like five or ten more on to that and then like plan power for those. I never looked at like specific wattages I'll admit because this was just like one rack of stuff and none of it is like superpower intensive so I didn't have to plan super seriously for it other than like the basics.

46:54
I think that's something that I try to at least address when it comes to production. Like you said, not necessarily in a lab because, and a reason for that is I think a lot of vendor equipment when it comes to voltage, their power supplies will do like the auto ranging thing. They can do 120 or 210. So that might not be as big of a concern, but what do we always say? You don't wanna assume anything. So you wanna make sure.

47:25
before you just go plugging stuff in, especially if you're gonna have power strips plugged into power strips. That's fair. Another thing, yeah, like we were talking differences earlier about like between a home lab and a work lab. It's like your whole lab probably you need to think real hard about power. For sure. You know, you don't wanna like just trip, you know, trip the breaker every time you power things on. But at work, you know, depending on your situation,

47:52
I don't know, if you're in a smaller enterprise environment, it's a much more bigger concern. But if you're in a huge lab that's already planning, I was in the middle of an enormous, enormous lab that didn't have to worry about tripping any breakers there. But yeah, you definitely got to think about it. Yeah, basically try at least to plan out.

48:18
how many devices you're going to have and the general like I guess poll that they're going to take power wise before you actually install all the things. So one thing that we always harp pretty hard on in a production environment especially when somebody comes to us maybe in a different business unit is requesting something. One of the things we always ask for

48:46
time and time again is, hey, what are the requirements? I need to know exactly, or at least close to exactly what you really need. And I think in a lab environment, this is gonna vary to a use case, but with your use case, it seemed like this was something that you needed to be able to mimic something that was gonna be in production at some point. So how much time did you...

49:12
have to, before you even got into the lab, before you even started physically putting anything together, how much time did you really have to put into figuring out what exactly it was you were trying to accomplish and kind of what did that thought process look like? That's a good question, Tim. I feel like I'm on NPR now or something.

49:34
Where do I start? Uh, that was a compliment. I do listen to NPR. So it's that I will take me to, they're just like very thought provoking. So it was hard. I think in like the ideal world, this isn't how I did it, unfortunately, but like in the ideal world, you know exactly what it is. You're you're simulating.

49:59
with your lab, right? Like, is it a certain part of, like, I'm thinking about the big like global network I worked on break fix for like IBM, right? Like, are you simulating just like a part of the network that you often do maintenance is on and so it's very important for you to practice specifically there in that little chunk of topology? Or are you like, am I simulating like part of like a rocket or am I simulating like the entire office enterprise network?

50:29
And, and unfortunately, like I came into aerospace not knowing anything and like it was a huge, huge, huge learning curve for me. So, ideally, you know, I would have known everything about what I'm simulating ahead of time and all of the use cases or many of the use cases, at least ahead of time of what I would like need to be prepared for with this lab, but I didn't. So, and I did learn that over time, but

50:54
it was kind of I learned that in parallel with building this lab. And that's another reason for why it has taken so long. And I don't think it's necessarily a bad thing, because it's allowed me to learn so much about both like my actual job in production, like what that looks like, and also just networking in general and like building a lab rack and everything, right? So

51:16
Ideally, I would have known everything that I'm preparing for and that way I could have planned like, yes, I'll need exactly this many switch ports and I will need exactly this kind of switch because I will be using exactly this protocol and when I'm testing this thing in the lab, I'll need to be prepared with this. And when I'm testing this other thing, I'll need to be prepared with this. And I would have been able to order all the stuff and get it all in there, but I didn't do that right so as I was learning in parallel, it was a lot of like falling forward.

51:45
I think is I think that's the term falling forward or failing forward is something I've heard before where like basically, you know, you're walking up this mountain, you're making progress but falling on your face a lot, but you're falling forward on your face. So you're technically making progress. You just like, you know, you, you get up. I think that's a really good point. And at the end of the day, that's what the lab environment's for. Right? I mean, it's so you can.

52:11
figure these things out in a controlled environment so you know that when it comes time to do it, quote, for real, you've got the right idea, you're more seasoned, you know better what's gonna happen or what should happen so you can react. So yeah, I think originally when I was thinking about your use case, I'm like, well, you would have had to have everything figured out ahead of time, but that really wasn't the case. You'd think, well.

52:40
But you were able to figure it out as you go. And I think that's one thing that you've brought up a couple of times is making mistakes and learning from mistakes is again, that's what the lab environment's for. It's for you to be able to learn and prove that concept. So if you do fall on your face, you're not bringing down half of the entire company when you do it. Yep. And you know, one of the...

53:07
great things about working at a company doing some pretty innovative things is like, you know, you kind of have to roll with the punches. And that's pretty cool because while I'm learning in parallel and I'm building this lab that allows me to do that, other people are also learning and bringing things to me and I can actually, you know, I've been able to expand the lab over time to make room for some tests that are pretty important to the overall.

53:36
you know, mission and what I'm doing. And that's been so exciting to be able to do that on the fly. Yeah, it's really been a great learning experience. So, you know, when we're talking, but this doesn't just apply to like rocket networks, right? Or like my job, it's definitely something I think every network engineer at some point should have to do, even if it's just like a little lab. That's why a lot of us have home labs, right? Right.

54:04
I do think it's pretty cool what you're doing as far as making it look, I mean, to me, I'm trying to picture what you're doing in my head. And to me, I'm seeing this, this rack of gear that's been well thought out. It's been well designed. It's been well cabled so that you bring, you know, you could bring, it's kind of a recruiting tool in my head is, is blue could bring in other people.

54:31
other prospective network engineers and they can see this and see how cool it is and you're really kind of, you're setting that precedent for what that work can really look like. I think that's really cool. Oh, that's so kind of you to say. Give me a lot of credit there. That's all taken. All right, I'm gonna put you on the spot for a big question here. Oh, okay. In your role specifically,

55:00
Do you think you could do that role effectively if you didn't have this lab set up?

55:09
No. Okay. No, actually that's a solid no. If I didn't have this lab set up and if someone else had done the setup for me, I would not be nearly as effective in my job. I can't go into specifics as I've said, but there's a lot of trying new things, trial and error, testing stuff out, experimenting a bit.

55:38
I went through sort of like a phase where I was obsessed with like the O-scope and like our fast link pulses like, you know, through the ethernet cable, you know, like all sorts of things like that. And this lab has allowed me to branch out like that because even if it's not like mission critical or even has anything to do with the rocket network, I have this lab that lets me grow as an engineer because I can ask those deep questions if I want to and then I can go and...

56:06
do whatever experiments I want on it, you know? And that way I grow and I become a better engineer and I can do my job more effectively. And then more directly, I can, you know, people can bring things to me and ask, hey, I have this question about the network. Will it work this way? And I can say, you know, usually I have no idea. And so I just say like, I don't know, but I can test it in the lab. Let me get back to you. And I can do that. And it's so great to be able to like...

56:36
I don't know how to help better to put it. You don't have to guess. Yeah. You don't have to guess. I can just go find out. Yeah, exactly. The beauty of labs.

56:49
Well, we're getting kind of towards the end of the hour here. Is there any, uh, any other insight, any other advice you can give to somebody that, uh, may have a specific use case to, to put a lab together? You're going to fail a lot. So just like, it's okay, I guess, like it's, unless you're experienced and you've done it before, you're going to get so frustrated and there will be tedious things, so many tedious things about it, but, um,

57:17
they're all ultimately worth it. Even if you end up redoing the tedious thing you just did, you're still, I know this is sort of cliche sounding, but when it comes to building a lab rack or building a thing, really, engineering a thing, even steps backwards allow you to take the next steps faster, if that makes sense. No, it does. You're always-

57:44
Okay, yeah, you're just always learning something from it, even if you just like, assigned a billion villains to things and you realize, ah, shit, I was one number off for all of like, it's still it's still a learning experience. Not everything in life that where there's a mistake and you have to redo it is a great learning experience. But like, when it comes to engineering or like, you know, this kind of stuff, it is I promise. So just don't get discouraged and hang in there. And if you don't know something.

58:13
Google is your friend when it comes to building a lab. For sure. As in many scenarios, Google is your friend. Yeah, that's many tech workers' motto, I suppose. I think one of my key takeaways from this episode is that lab environments aren't just for learning to pass a certification exam or putting in that lab time so you can pass that test.

58:43
They can definitely translate to the work environment. In fact, and I think both of our opinions here, they should. So you can really make sure what you're putting out there in production is quality because you've tested it out. Now we understand that you may not be able to always do like for like, but to try to get as close as you can, especially when it comes to more high level concepts that aren't necessarily vendor proprietary when you're working with things like.

59:12
like VLANs and routing protocols and that kind of thing. It doesn't necessarily matter if you're working in a virtualized environment with virtual switches, virtual routers, if it's not quote the same model that it would be physically in production, still try it out, still work through that documentation, use it as a tool to build those maintenance window, step-by-step documents. It works, I've done it.

59:43
That's so smart. I can't get over how like, how smart that is. Every, every company that has like maintenance on their networks should always have their network engineers practice. I just, I think back to a couple of the, the pretty big changes that I've had to do in my environment, that I was able to leverage that lab environment before to make sure I had the configuration right, to make sure I knew what outputs I should expect.

01:00:12
I can't imagine trying to do those maintenance windows, essentially going blind. I mean, you can have the configurations that you think you need. You can have it all drawn out, but if you haven't practiced it, also if you practice it, you know you're gonna catch any typos because you can copy and paste from your lab environment. But it's just going through that multiple times will bring that stress level down when it comes to actually.

01:00:42
do the thing in production. So yeah, I highly recommend that. I think that's one of the biggest use cases I can come up with for doing labs in a production work environment. I shouldn't say production, in a work environment.

01:00:56
I want to y'all, whoever hears this, like tweet at us or whatnot and give us your, if you have any other use cases, when we post this, definitely comment and let us know. Cause, um, you know, we talked about this a little bit, you know, beforehand and had our own sort of experiences, but I think there's probably, you know, there's definitely more out there that we could all learn from that Tim and I maybe just haven't thought of today. So be interested to hear what y'all have to say.

01:01:25
Definitely. And that's a great segue. Time for me to put my AJ voice back on. Cause I practiced that earlier. You can tweet at us at art of net eng. We are online art of network engineering.com YouTube. Hopefully you're watching this on YouTube so you could see how excited Lexi got when we were talking about NASA. That was cool. And thanks for joining us for talking about labs.

01:01:53
See you all next time on the art of network engineering.

01:02:06
Hey y'all, this is Lexi. If you vibe with what you heard us talking about today, we'd love for you to subscribe to our podcast in your favorite pod capture. Also, go ahead and hit that bell icon to make sure you're notified of all our future episodes right when they come out.

01:02:20
If you want to hear what we're talking about when we're not on the podcast, you can totally follow us on Twitter and Instagram at Art of NetEng. That's Art of N-E-T-E-N-G. You can also find a bunch more info about us and the podcast at ArtOfNetworkEngineering.com. Thanks for listening.