Speaker 1: 0:00

This is the Art of Network Engineering podcast. In this podcast, we explore tools, technologies and talented people. We aim to bring you information that will expand your skill sets and toolbox and share the stories of fellow network engineers. Welcome to the Art of Network Engineering. I am AJ Murray Engineering. I am AJ Murray and I am joined this evening by Kevin Nance. Kevin, how you doing man?

Speaker 2: 0:30

I'm doing really well. I have to admit something that my mind has not been on this podcast at all. I am leaving for Europe in a few days. I'm kind of phoning everything in recently. So if I ask some dumb questions, just know that my mind is in Europe already. So I'm sorry.

Speaker 1: 0:47

Like where in Europe are you?

Speaker 2: 0:48

Flying into Paris, doing France for a week, and then going to Rome for a week.

Speaker 4: 0:53

Oh man, you already got the pasta on your mind, exactly.

Speaker 2: 0:58

I'm already there Mentally. I'm there Carb overload.

Speaker 1: 1:01

That's awesome. Well, congrats. I can't wait to hear about that. I am excited and my mind is laser focused on the podcast, because we're talking about something tonight that we don't normally talk about, right Like we're all network engineers, but we're pretty wired, wired up man. So wireless is the topic for tonight, and we are joined by Alex Marquodont. We met Alex at Cisco Live and boy he sold us on a show topic and we signed him right up. So, alex, thank you so much for joining us.

Speaker 4: 1:29

Yeah, thank you a lot, adrian. Kevin, it's my pleasure to be on here. I hope we can dive into some of the nitty gritty things about Wi-Fi and some of the cool deployments I've worked on in the past.

Speaker 1: 1:39

Yeah, absolutely. So to just kind of give some background on who are you, what do you do, how long you've been doing this thing, the cliff notes of your career, as it were.

Speaker 4: 1:50

Yeah, totally. So I'm pretty young in the whole realm of IT 23. But I mean working in the MSP space. I feel like you mature so fast. If there's any advice I have for anybody getting started it's like get your feet wet in an MSP and never look back. You learn so much. Started right out of college, kind of landed the job right out of college. My first job before out of college was a sysadmin role. Worked for a car automotive place, 20 locations. First week on the job he's like here's the admin and domain creds, we're running Windows 2008. We got to get out of this. So it was at college. We would be like labbing stuff up and I'm like sweet, I'm going to go do this in prod next week, transferring the FISMO roles, doing all the group policies and really learning by just lighting things on fire but not breaking production, which was kind of the best thing I could have asked for for. An entry-level IT job taught me a lot for sure.

Speaker 1: 2:49

Yeah, it's great when you land a role like that because you can go to class, learn something and then go to work and apply it. Or you might learn something at work and take it back to class and be like, oh, but I learned this at work and kind of share that knowledge. When you can immediately learn something and then put it to good use, I think it really helps it stick.

Speaker 4: 3:07

Yeah, agreed. And then after that I graduated college when I was 20 in the peak of COVID. There's really nothing you can do from a social aspect. So I put my head down in the books, went and got certified in a bunch of things, most notably my CCMP in Enterprise Wireless Design and Cisco Market Solution Specialist, most notably my CCMP in enterprise wireless design and Cisco Market Solutions specialist. I think those are the two key aspects. I drink blue all day but I'm very versed in a lot of vendors. And after that mixed hybrid network and systems admin role at an MSP called Marco, I migrated over to my current role right now is with Atomic Data. Number two my current role right now is with Atomic Data working on the Game Day Technologies portfolio that we offer, supporting our customers in Game Day Technologies when it comes to stadium deployments and other large venues, whether it's conference halls or your one-off other unique deployments. So I got to do a lot of flashy deployments and get to see a lot of cool things.

Speaker 1: 4:03

Awesome. To be an SME in anything at 23 is absolutely amazing. So kudos to you, man. That's incredible.

Speaker 4: 4:10

Appreciate it. I think the people on my team thankfully respect me a lot and I know I've definitely pinged a lot of my team and it's definitely cool, but I've earned their respect at my age.

Speaker 1: 4:22

Yeah, yeah, that's great Congratulations.

Speaker 2: 4:24

I barely remember my early 20s. That's crazy, let alone being responsible enough to be an admin somewhere and actually doing things I couldn't imagine. It's awesome. Kudos to you, man.

Speaker 1: 4:38

Awesome. So I want to get into wireless networking. We've talked a lot about network design on the show, but we've never talked about wireless design, and there's obviously some things that are very different when it comes to designing a wireless network versus a wired network. So let's start from the beginning. What's some of the high-level stuff that we're going to kind of cover or want to think about when we're doing a wireless network?

Speaker 4: 5:06

Yeah, when it comes to wireless, you kind of got to think of it like your speakers. You got to think about the proper placement. You got to think about all the things that could get in the way, whether it's the walls, the things that are going to attenuate, the things that are going to cause interference going from your projector that somebody's going to put up, the random device that someone's going to bring into the office, all the BYOD devices. How many devices are people going to have on average? What are the type of devices in the office? Is everyone using laptops, surfaces, iPads? And what are the commitments from a data perspective? Is your team built with a lot of CAD and engineers, high data traffic, or is your team just going to be sitting on Zoom calls all day, which is low latency dependent, but not a ton of throughput? There's a lot of things that go into building a wireless network and especially when you need to get things like analytics or data out of it. Placement is a huge thing and starting off and really taking a step back before you just jump in and propose a solution to a client, the one size doesn't really fit all. Sure, it works 90% of the time, but most of the time. If you take a step back, really work with the client and earn their trust, you're going to come out with a way, better solution in the end and hopefully a placement that's going to work for them from refresh to refresh and a lot less help desk tickets. If you invest in the front end of the design, you'll have way less noise over the years.

Speaker 4: 6:28

And I'm guessing that starts with a site survey Not necessarily a site survey. For the most part it's let's talk with the client and understand their business needs, look at their existing traffic use on the firewall side of things and the switching perspective and get a better understanding of the business needs. And then, of course, we go to the perspective of can we actually find a floor plan for this client? That's a big issue. Thankfully, with a lot of the stadiums that we work with, it's like the stadiums aren't built yet or it's in the process of being built. So we're getting the CAD files as they're hot off the press or we're able to get something.

Speaker 4: 7:04

But there's definitely cases where you're working with enterprises or medical or government facilities where they don't have floor plans because the building was built back in way before our times, like 1800s, 1900s and stuff like that. So a site survey is definitely going to be needed, and sometimes you have to create your own floor plans. Thankfully, like EchoHour comes out with this new fancy feature where it's just a ghost. It's called just go, so it's able to use the lidar sensor on the back of an ipad to just build a floor plan for you, which really does help in your in a pinch, but, uh, not the best option have you ever gotten a like a floor plan that's like on a napkin just written out like hey, there's some, some walls over here, it's walls over here.

Speaker 4: 7:44

Totally. And I think when you get wall like those, like really unique issues, it's kind of like all right, we're going to go out with a sidekick, we're going to try to mock up a floor. So you're sitting out there with the laser, you're calculating the distance between rooms to rooms and you're trying to create a somewhat accurate thing. But you spend more time on it on site and you're like calculating the distances and you're almost like, once you know Wi-Fi and you've done it for a while, you can kind of walk around a building and be like, okay, and AP should go here in the ceiling. And then you walked around a corner and it's like here's another one. And it becomes more natural and comfortable from that perspective of mapping things out.

Speaker 1: 8:22

So you talked about floor plans. Is there something that you can do ahead of time like a predictive survey? I've heard that tossed around before in my past careers professional services but I wasn't like a wireless network engineer, so I only heard the cool guys talking about it. I didn't actually get to do any of that stuff myself.

Speaker 4: 8:41

Yeah, it is very common and from a cost perspective, going on site is very costly, especially because most of the companies that end up using professional services they have to pay that engineer to everything you're. You're doing a predictive from the ground up and most of the time with a predictive you can get away with getting everything really accurate. And most of the time when you're doing just a predictive survey for a site and maybe we should take a step back quick and a predictive is where you're just simulating the wireless environment. I think we need to set that ground because we have the site survey, which is where you're on site and you're performing and you're taking a measurement of the RF environment. And then we have the passive survey, which is again on site and getting its current load, and then now we're on to the predictive, which is predicting the best possible outcome for the RF environment.

Speaker 4: 9:44

And most of the time you can get away with creating a predictive if you simulate the access points lower than their max output.

Speaker 4: 9:51

So most of the time if you want your radios and in wireless it's better to have smaller coverage cells, so most of the time if you're going to predict an office, you don't want your radios to be more than like 15 to 18 dBm for an office space and that's kind of high.

Speaker 4: 10:06

So most of the time what you're going to do in the simulation perspective is you're going to put all your radios to 12. So then if a wall gets stood up that is really thick or has some weird material types, you can kind of compensate from that by tweaking the RF radio, by increasing the power output, because most of the time when you're doing a predictive you don't have the in-depth CAD files that actually show like the wall thickness and stuff like that. That's going to cause those RF issues. So because a normal door versus a thick door is going to create a lot of issues. Or when it comes to glass, that's another big problem when it comes to just doing everything predictive. So yeah, I think predictive too is based off of a floor plan that you kind of feed of glass.

Speaker 1: 10:44

that's another big problem when it comes to just doing everything predictive. So yeah, I think predictive too is based off of a floor plan that you kind of feed into it and likely it's probably not an accurate floor plan, right, like it could be super old, uh, there could be walls in places. There's you know on the floor plan that there aren't. You know they could have completely remodeled over a couple of years. Like there's the of helpful, but is it really helpful?

Speaker 4: 11:05

Yeah, I suppose it's better than nothing, it's better than nothing. And then another good point is you don't really know what's there. You get really old files or you can't. Especially in this case, it's really important to work with those non-technical people or somewhat technical people on site to kind of have them walk through. I've done FaceTimes before with people and it's like let's just walk through the building quick, just like give me a casual stroll.

Speaker 4: 11:28

Like you walked in from the front door to your desk and like from that you're able to like look at things and like you kind of get an eye for certain materials and certain wall types and you're like, okay, that's a 40 DM wall type where you're sitting there, like walk awkwardly, walking up and down the hallways, like banging on the walls to see like what they're made of. This sounds weird. But like, um, I grew up on a farm and whatnot, so as a kid kind of just get used to like those little weird nuances of like material types and whatnot. You're like is that going to be strong enough to like create a pivot, to like lift this up or like a wedge, and you kind of get used to those material types. So it kind of get used to those material types.

Speaker 4: 12:02

So it's really common, like once you understand more of like the designing and building structure types that you kind of like you have like a little niche or like this eye, like pulling things out when you're kind of creating those predictives, because you never know, like on a floor plan it's like it's flat, it's 2d, you don't know if the ceiling's 10 feet tall, 20 feet tall or 8 feet tall. So like that plays a huge contributor to like what access point are we going to select? What are the antennas we're going to put on? If we have to use antennas, and like if we're trying to reuse existing network drops, like can we reuse any of them from a cost perspective, cause it's costly to run new cables, every new AP.

Speaker 2: 12:40

And how? How specific are you when it comes to that? So, like uh, I, when I we kind of did everything and one of those jobs was wireless and when we were doing a remodel, we tried to use existing infrastructure as much as possible. It was like some jacks, and if it was off by like a couple, you know a couple of yards, we're like, eh, this is probably good enough. How specific did it get to be?

Speaker 4: 13:05

That's the reality of things is like wireless always comes second and it's aesthetics is almost always more important than the placement of the access point. So, like, if it's within like 10 feet, you're probably not going to see a big difference. The biggest concern is like, let's say, you had an access point in like the center of a corridor where, like you have two hallways crisscrossing and it was supposed to be in the middle, and then you move it down the hallway 10 feet, like that's going to be a big issue because that's such a high traffic, like pivot point and like when you round that corner, if you don't have an ap there for, say, you're going to get dropped off the network most likely because in a lot of those tighter corridors the wall tights are concrete or like really thick material.

Speaker 4: 13:45

Um, so like ap placements like that really matter. But like if you're talking like a generic office space where it's just like pretty open and you're like 10 feet off, or like we said at campus, and it's like 10 feet off in the hallway or something like that, like that's not going to be a huge concern for total RF performance and most of the time if you, depending on the solution you offer or using those radios, are going to be able to communicate and like pull the receive signal strength clients and be like, okay, I need to increase my power a little bit to compensate for that and mitigate that hole and coverage.

Speaker 2: 14:16

So yeah, how, speaking of that, how? How much do you rely on that? Like, do you, because you know modern, you know cisco prime or whatever, whatever can you know on the fly, adjust power of the aps? But how much do you rely on that? How much do you actually go in and tweak the rf levels and power levels and all that kind of stuff so that you know you don't have to rely on, hopefully, the system automatically doing it?

Speaker 4: 14:40

Yeah, I've worked with a lot of vendors and it really comes down to the vendor. Now, if we're talking to like Cisco, I'm going to let that thing do what it needs to do. If it's Cisco or Juniper, mist like those things are set and forget it. They do such a good job, especially in the high-density venue spaces. Jump in, make sure it only uses 20 MHz channels. Let it go. It's going to do its proper coverage cells, dna Center and the wireless line controller and all the APs communicate so flawlessly. From that perspective, it will save you a ton of time.

Speaker 4: 15:20

Now, if you're working with a, a vendor like aruba, which loves fat 80 megahertz channels out of the box, and even if you go in, tweak it to 20, it's always going to try to use its max output. Something just doesn't sit right from the coverage cell perspective. You can enable dfs channels, but it won't use dfs channels. So, like when it comes to statically setting radio output power, it really depends on the vendor. I'm like I'm have to be completely honest, especially like, if you're doing the sub enterprise grades like a ubiquity, you're definitely have to do a lot of manual tweaking and, like when it comes to that manual tweaking, you're not always going to be able to go on site to do the validation survey.

Speaker 4: 15:45

Back to the cost perspective, so you're going to like simulate the overall output of the access points in a program like Ekahau or Hamina and you're going to sit here and be like, okay, so I need to be on this channel at this frequency and on this DBM range from an output power and DBM is like the overall output power that you can have on an AP. Just the measurement in milliwatts of the total output power that you can have and you can think of the higher output power leads to one the overall size of the coverage cell or how far away you can have. And you can think of. The higher output power leads to one um the overall size of the coverage dollar, how far away you can be, um.

Speaker 2: 16:18

So, and again, there's some more nuances, but so when you're, when you're deploying APs, is there a different technique? Um, you mentioned about like figuring out the customer's needs, but if you're doing like a high density, um, stadium verse, uh, office space, what are your strategies there? To, kind of, how do you approach each one, because they're completely different? Yeah, totally.

Speaker 4: 16:38

I think the large venue space is a lot of fun because before you really get started you're working with the architecture team to identify, like the building, and then from there you're working with a lot of sponsorships and you're reviewing RFPs or requests. You're reviewing this huge document that is consumed of hundreds of pages and most of them are all public and they're mostly put together by external vendors. So, like the NFL for example, they have their own commissions and their own standards. And the last time I read through an RFP for the NFL and this is public it was like must be able to support 85% connectivity rate and if you really think about it, some stadiums hold 60,000 people, 70,000 people. So when you're building a wireless network to support 85% of the connectivity of the clients, because most of the time people have multiple devices but you also have to think about the fact that there's a DAS system going in. So when you're building these high-density deployments, I'm going to pick on the clubs, because MLS is kind of our cup of tea here Over at Atomic Data. That's what we specialize in is working with MLS and we have a really good partnership. Minnesota United and MC Cincinnati are kind of our favorite two clubs.

Speaker 4: 17:51

But when it comes to designing, you're going to have additional access points near the gates because you're going to have access points that are set up for ticket scanners only and then you're going to have APs that are focused on just the client onboarding perspective, because all these guest networks are open, so there's so many devices trying to connect up even if you don't initiate the connection. So you have that. You're going to take up a lot of airtime with all that client onboarding, which is going to degrade the performance tremendously. Even if you try to set up all the quality of service profiles under the sun to like, say, I need the ticketing SSID to be the most important thing out there, it's not going to work that way, simply due to the fact that you have so many onboarding requests on the AP. That's going to take up airtime, and airtime is just how much time you get dedicated to communicate. So you're going to put multiple access points near the gates to really focus on one AP just has the tickets and one just has onboarding clients. So you're creating custom profiles and you're setting up niche AP profiles on certain access points to prioritize traffic and then also having redundant APs in some of those hired areas because sales again a lot, a lot of these clubs and the NFL like.

Speaker 4: 19:04

At the end of the day, it's a marketing gig and they're all about making money. And yes, we all love to go and see our favorite team lose on Sunday or Saturday, but if you can't get a beer at the stand, why are you there? For I feel like everyone goes there to have a good time, buy the merch and enjoy Again. It's a great opportunity and something that we can really enjoy. In the United States in general, we have some of the greatest facilities and some of the best sporting arenas, but if the most critical infrastructure doesn't work during the 90 minutes, what are you building it for? That's when thinking about high availability is such an important thing. Building redundant controllers and almost overbuilding everything. In the event of anything fails, the entire game can go off without a hitch or no impact to anybody on site.

Speaker 1: 19:56

Man, you've already like blown my mind in the first few minutes talking about some of this stuff, Because like client onboarding, like absolutely that makes a ton of sense. Never thought about it.

Speaker 2: 20:03

Yeah, we never thought of that Open.

Speaker 4: 20:06

SSIDs are horrendous for every network.

Speaker 4: 20:09

Like in no good space should you have an open network if you don't need it.

Speaker 4: 20:18

Like you should, even for your internal network, put a pre-shared key on it and put that guest password everywhere, because your performance is going to be way better just because you're not going to have devices trying to connect all the time.

Speaker 4: 20:24

You're not going to have everybody that walks outside your building trying to connect. Just the overall rf environment is going to be way better. You're not going to have like weird issues, like a pretty common issue that's noted with like onboarding. Onboarding, leading to issues, is like you'll notice, like you'll be connecting, everything's fine, and then all of a sudden you like randomly like somebody will walk by you and like you're like why did the wi-fi just like crap out on me and it's probably because of roaming, wasn't set up right or the access point is already under a lot of load from a CPU perspective and you're it just can't prioritize that traffic, because onboarding traffic is really like has a very high priority to an AP like that association to get on as a lot of lot higher priority in most cases than your traffic that's trying to load a webpage.

Speaker 1: 21:08

So um, man, where, where do we go from here? There's so much I want to cover. Uh, so so from a from a design architecture perspective, you know there, where do we go from here? There's so much I want to cover. So, from a design architecture perspective, you know, there's centralized management and decentralized management, right Like. So when we're talking about a large venue like that, there must be, you must be using both, right Like you want to, like you said, over-engineer it so that way, you've got multiple controllers.

Speaker 1: 21:32

You've got, you know, controllers on that way you've got multiple controllers. You've got controllers on site. You've got backup controllers in the cloud, maybe. So what does the management of all of this that we're about to dive into look like Management?

Speaker 4: 21:44

is pretty old school when it comes to venues. I would say we're still very heavy on controllers on site. There have been some cloud limitations. We've ran into working with vendors and they're like don't do this, please, putting that many access points. Also, you have to think of cloud latency and bandwidth utilization If you're going to have 1,000, 2,000, 3,000 access points. Uploading telemetry data to some dashboard at the same time during the game, that's low-priority traffic. Why are we wasting all that bandwidth? But at the same time, during the game, that's low-priority traffic. Why are we wasting all that bandwidth? But for the most part, we're using controllers still on-prem, simply due to the security fact A lot of controllers we can still cap-wap or encapsulate traffic from the AP back to the controller so we can create a DMZ network to egress all that traffic out of because guest network traffic, especially at the venue spaces.

Speaker 4: 22:33

Man, there's some crazy things that happen and like you're going to get DDoS during your first game, you're going to get a nice letter from the local ISP that says why are you trying to download X movie? And you're going to be like oh great, you know, it's like. Things like that You're never going to think about. It's like we have the DNS security in place, we have the firewall in place. How did this happen? It's like, oh, he's probably using some poor browser that's creating a VPN tunnel out. And most of the time when you think of guest networks for smaller enterprises, you're like it's just going to be internet access, don't care about anything. But with the venue space, you end up having to almost treat it as an internal network and create some of those security postures and to prevent, like, the things that can pop up. You know all right.

Speaker 1: 23:15

So you, you talked about numbers. Let's, let's throw some numbers around here. Um, let's say we're we're doing the stadium 60,000 people. I'm sure you know, obviously, how big that stadium is. There's probably a lot of factors, but you know the average stadium that you've dealt with around that size. How many aps are we, you know, thrown around here? Yeah, I would say for like nfl.

Speaker 4: 23:37

If you're talking like that many, you're going to be pushing like 13 to 1500 access points in the deployments. I've worked a part of um so I mean you can think about the fact that that's going to be cisco's biggest controller, wireless land controller, um. So like when you're tossing around that many access points, realistically for uh, mls club it's going to be about a to 400, depending on the size. But in MLS Club again, its capacity is around that 15,000 to 20,000 seating perspective and most of your APs are going to be in the bowl, not necessarily back of house and whatnot.

Speaker 2: 24:13

With that many people. How do you play? Because people are interference right.

Speaker 4: 24:17

We're giant sacks of water. You sit on top of that access point and you just destroy the rf.

Speaker 2: 24:21

Yeah so how do you, how do you plan for that? How do you plan to have that many people all jammed in space together all cutting the rf like I can't even I can't picture that yeah, also the fact that the rf has to work when no one is in the stadium is a bigger issue.

Speaker 4: 24:37

I think like wireless, yeah Cause when you think about it you can have two wireless frequencies. But like when the stadium's open, the powers are most likely going to be way too high when no one's in there. So it's like how can we still make the wifi work when no one's in there, along when people are in there? And it's like for me, example, I might cut five dbm on that ap by sitting on top of it, but by the end of the game I might drink a gallon of water and it might be six dbm um or a few years yeah, exactly so.

Speaker 4: 25:07

It's like that's a big thing and you almost you add a ton of access points in the bowl. Um, your coverage cells are super small and you create zones they're called within the bowl. So you have really small coverage cells that are going to cover like three to four rows in total up in some width. And again you're going to throw these into programs like Echo, how, and really simulate it. Because again, when you're putting APs too, it's really common to do under seat right now that's kind of the best thing. It's not the best for the RF, but it's the best for containing the coverage cell. Because if you're going to try to roof mount everything, it's an aesthetics perspective back at that point. And then, from a maintenance perspective, do you really want to get a lift every time you have to jump up and tweak an AP? It's a lot easier to put it under seat. And then also like the elements from that perspective, like the weather, but like when you're designing in the bowl, you're going to have an RF profile and it's going to have a certain range of RF and it's a lot better on the Cisco side of the world than like Aruba right now or some of the other vendors we work with, like Extreme, it's going to be pretty good at just being able to increase and decrease the overall power because most of the time you're not going to want to create static channels. But if you are using static channels it's planned for it to be a full venue, not necessarily an empty venue. From that perspective, and again, you're going to do a validation survey in most cases You're going to go on site and you're going to create an RF profile and you're going to validate when the stadium is empty and then you're going to do a validation survey when the stadium is full. You're not going to survey the entire stadium, but you're going to have portions Because for the most part the bowl's going to be the exact same all the way around.

Speaker 4: 26:48

I know everybody's seen that picture of some guy at an NFL game or some game sitting there with a laptop with a bunch of antennas on it. He's the front page of the means for a week and a half and that's really what it is. It's on site during the game and then not during the game. One to measure is that output power right. Is it too low? Is it too high? Did you set the bit rate too high from a connectivity perspective, where clients can't even get connected because the signal, the noise is so bad or it just doesn't have a clear path of communication, Like that's a big issue overall. The reality is is like you're never going to be able to get it perfect your first go and it's going to take a few matches or a few games on a new deployment to get everything I have a weird question for you.

Speaker 2: 27:26

Yeah, do you. This would go. This would go many ways um. Have you ever had anyone complain that their aps are underneath them?

Speaker 4: 27:34

no, I actually never. Never heard that. Um, I didn't know that they put.

Speaker 1: 27:40

APs under people in a stadium. It never crossed my mind. Now it's going to get out there. Now it's on a podcast.

Speaker 4: 27:48

I've had people be like you know, you sometimes see it in a railing too.

Speaker 2: 27:52

Yeah, I went to Cisco Live and I saw they had railing encasings and underneath the seat and all that kind of stuff. But I've been in office buildings where I've had people complain that they're too close to the AP, that's giving radiation and it's going to cause them to have whatever hair loss or something crazy. I was just curious if you had anyone be like no, you're going to fry my butt off during the game or something I don't know. My grandma tells me to kill all the bees with the 5g.

Speaker 4: 28:18

Every time I see her to be completely honest, but I've never heard a complaint from anybody. Um, especially a bee. Yes, yeah uh, we do have some like running jokes like in the office, like oh, we're gonna like decrease the like birth rate of some cities or something, but it's just complete jokes like you guys do joke about.

Speaker 4: 28:38

I knew it yeah um, there's nothing like from a perspective like you do have to be careful about it because there are rules that you have to have an ap. So far away from like people, um, like from a distance perspective, um, but for the most part, no, I've never had a complaint because most time people don't even know they're there because they're shoved underneath the seat and like they fold like most seats are like set where they fold down, so you sit down, you're sitting over it and like no one's really gonna know unless they're trying to shove a bunch of stuff underneath the seat. But then like what's this great box? And they don't really question as far as yeah, it's about as far as they go, I don't care, yeah, like nobody really knows what's in the box. It's just like what's the box for? They don't know it's an access point, because we purposely disguise them. We don't want people being like oh, we know there's an access point here, we're going to tamper with the box and try to steal it and stuff like that.

Speaker 2: 29:29

Some junk out of the screwdriver.

Speaker 1: 29:31

So that kind of answers a question we had in the chat. Dan asked if you ever had to deal with people tampering with it. Dan asked if you ever had to deal with people tampering with it but it sounds like you guys thought ahead in that regard.

Speaker 4: 29:41

Yeah, I don't think in my time we have not lost a single access point.

Speaker 2: 29:45

You sound like a cop or something I've never lost one on my watch.

Speaker 4: 29:52

I mean, they go offline all the time and I think weather is the biggest issue when it comes to access points for outdoor arenas.

Speaker 4: 29:58

It's like every year you're gonna have aps that go offline and you're just gonna like they're gonna have to wait until the next off season until you can get a low voltage vendor to go and run a new cable or figure something out like um and then ufc, like the first year. Like they, we went live with them. Like the running joke was there's people on my team like out there with turkey basters, like pulling water out of the enclosures because like they weren't sealed right. So like you have these outdoor aps. And like you open up the enclosure because they're covered with snow and ice in minnesota you're like scraping away at it and I've heard stories of people sitting out there with a turkey baster and like squeezing the water and getting it out during springtime.

Speaker 4: 30:37

Oh man, Some of the things like that we uncover and like run into and like, like getting shipped out to a client site and be like all right, you have 48 hours. How much can you get done in this entire deployment? Because you don't want to go back down there for another two months. If I can get everything online so I can configure remotely, Like so like you play games with yourself a lot of the times with these larger deployments and you try to automate everything if you can.

Speaker 2: 31:03

You sound like a typical network engineer who's just like get it remotely so I can go back to my office and do it all remote. I don't want to mess with it.

Speaker 4: 31:10

Or like who else in my team wants to travel, like the juniors, I don't know. Like five years of traveling took a toll on me, like two, three times a month. You know like you're gone and I'm like who's the junior, young bushy tail that wants to go and like sweat until they like are dripping in Houston and during the middle of summer and just like, yeah, and there's people that like raise their hand up because they've never seen it, they've never done it. Like, the first time you walk into a stadium, working it from a network perspective versus a fan perspective, your, your mindset changes.

Speaker 4: 31:42

I mean, when you go into a stadium, like everyone's like, oh, this is so cool. They're looking around, you know, you're like looking at all the knickknacks and then, once you work so many, it's like, oh, it's another venue. You know the layout of the stadium to stadium and it's like a lot of these, like a lot. Like I've looked at so many architect drawings that I probably can go crazy at this point. It's like you know, like, especially when it comes to stadiums, like they're pretty cut and paste, like the companies that work on them are they. They do, they're good at what they do for a reason, like they build a lot of stadiums, so like a lot of the floor plans become pretty like cookie cutter, almost like you kind of know, like oh, this is going to work here kind of thing in this bowl deployment. Um, you kind of assume material types and stuff like that too before things get built.

Speaker 2: 32:25

So so if you ever get bored of doing networking, you can become an architect.

Speaker 4: 32:29

Now you can just yeah I was in robotics back in high school and, like I would, we would run cad machines and like build 3d renderings.

Speaker 4: 32:38

I don't know if you guys have ever heard of SolidWorks oh yeah but yeah, I spent a ton of time with SolidWorks back in high school like caddying up and designing our robot FRC, and that's something that's pretty common these days but spent a lot of time in CAD, so it's kind of funny now that I work in Wi-Fi, it's like oh, I know what it takes to put one of these together and it sucks.

Speaker 3: 33:06

Unimist is a configuration management and network automation system built by network engineers for network engineers. The team behind Unimist believes network administrators should not have to learn programming languages or abstraction frameworks just to automate basic network configuration. With Unimist, you can rapidly and reliably automate common network workflows like pushing large-scale configuration changes or performing firmware upgrades on your routers and minutes. You can use the mass config push feature to send a custom set of commands to a group of devices to configure a new user, add a VLAN to a trunk or adjust firewall rules throughout your network, with variable support and macros for special handling. Device outputs are automatically grouped, which makes reading results easy. All functions are, of course, exposed via REST API endpoints, allowing integration with custom network automation, middleware or other systems if needed. Supporting 350 device types across 150 vendors, with disaster recovery, change tracking and config auditing features on top of automation, unimus is the most versatile NCM out there. Now back to the show.

Speaker 1: 34:06

All right, I want to compare and contrast a little bit. I think a lot of our audience is familiar with the enterprise. They've probably been to a few stadiums. What are some of the biggest differences when we talk about wireless networks, deployments between enterprise and a high-density location like a stadium? So you might ever get 60,000 people on a business campus, but that's going to be spread over a much larger area and several buildings compared to a stadium. So what's some of the biggest things that cause interference that maybe you, you know you wouldn't think.

Speaker 4: 34:43

Let's dive into that a little bit. Yeah, I think when it comes to deployments, it's a little bit different in venues and stadiums because a lot of the times you're not creating where your SLA is super high. You're going to have a lower service level agreement for all these clients, like you're going to be okay with certain levels of packet loss and your total data rate and your commitment to a client device is a lot lower. In most cases, like when you're thinking of an enterprise employment, you have to have perfect connectivity because you're expecting to be on a zoom call, but the reality is like when you're at a stadium, you're pretty engaged until someone scores or something, and that's when you're going to see the huge spikes in data.

Speaker 4: 35:19

And that's where it's really important to have high throughput on your firewall, because we don't like to limit clients, because we want to get client devices off the air done communicating as fast as possible.

Speaker 4: 35:31

That should go from enterprise and to the venue space, but the faster you can get a client to stop sending traffic, the better.

Speaker 4: 35:39

So, like making sure you have the underlying infrastructure built to ensure you can like really send all that traffic out to the internet right away, so like you can post to facebook right away, instagram right away, like post on twitter.

Speaker 4: 35:51

Like that is like the biggest thing you're building for is like those speak, those spikes of traffic and most of the time when people hit the upload button they put their phone back in their pocket. You know, like you're building a network that's not necessarily built for low latency and high throughput, but you're it's more like building something with higher capacity from a client perspective. So you're going to put aps in that are like have a ton of antennas, whether it's an eight by eight or a four by four, on the 5 gigahertz or 6 gigahertz frequency, and you're really going to focus on building APs that can handle a lot of connected devices not so much through perspective. So you're going to use, like a Cisco 9166, which is going to be their latest and greatest, or an Aruba 655 or some of their outdoor APs, some of the six, seven, nine models, or six, seven, seven. So it's it's really comes down to making sure that with the game day, like with the venue side, that you're focusing more on client connection than rather overall, like client quality and slas.

Speaker 2: 36:53

It's almost like a shotgun approach, instead of like you're just trying to get mass amount of people to get a little bit of data. Yep, yep, totally yep.

Speaker 4: 36:59

That's exactly what it is so and of like, you're just trying to get mass amount of people to get a little bit of data. Yep, yep, totally Yep, that's exactly what it is. So and I mean you're building for the 95, 98% of client devices and not if you can't support the 2% of devices, as long as they're not crucial to the business needs, you're not going to worry about them. So like, if someone comes in with their Samsung galaxy 3 and they can't get connected to the wi-fi, tough luck, we don't care like, that's just like that's just the reality of things we're.

Speaker 4: 37:27

We're building like a lot of the venues and like any campus you go into it's like build, put the latest and greatest in, because their technology like achievement per se and a lot of the times it's like a showcase for those vendors because they sponsor hardware, give discounts or like something from that perspective.

Speaker 4: 37:45

So like it's a showcase. So like just as important as an important deployment is, and like making sure everything looks good, it's a branding deal for a lot of these larger venue like companies. Um and again, like we help through the entire process with these clubs. Like because a lot of times like you'll have a larger club, they'll come to us and they're like we don't really know where to start. And like we'll offer like that c, that cto, that technology roadmap, to like work with the construction company, work with all the low voltage installers and like handle everything for me, for you, when it comes to it planning, like where should the idfs go in the stadium, stuff like that what a problem to have, huh kevin yeah, you're putting on rfp for for wireless uh in your uh in your area and and now you know, you got like all these vendors coming to give free stuff to the, to the nfl or you know whoever sounds terrible.

Speaker 2: 38:39

So you're saying that the latest and greatest, you always want the, the newest and best, and it's kind of a competition. Um, so what is your? What does your upgrade cycles look like then? What is your refresh? Look, because if you're always best, greatest, whatever, that's a crap ton of money that you're putting into these stadiums it's interesting.

Speaker 4: 38:56

It really depends on the club, like the audience, I would say, and the venue space like, and the overall like sports league, so like the nfl, has requirements that must be met and if you read their rfps, like that are hundreds of pages. Um, they set the refresh cycle, so it's not really controlled by us. The mls is a little bit different. Like it's managed by the club. There's no like real standard. You don't even need public Wi-Fi from that perspective. So the refresh cycle is mostly put a new deployment in right. When it's built, run it until its end of life, or the overall services are degrading drastically, or it makes more sense from a cost perspective to replace everything than to maintain it and deal with the possible cost of an outage.

Speaker 4: 39:40

So the refresh cycle is yeah, the refresh cycle is oh yeah, it's. It's pretty standard um it it yeah. It fluctuates from organization to organization. I mean, again, it's funding. At the end of the day, the winning this team's gonna have the nicest technology, most likely because they're gonna have the most sponsors and stuff like that. It is cool, though, being in, like in the venue, with the ecosystem. I'm in because I get to play with all the new flashy toys and I get to see all the old toys still like. I still get to work on the old, like cisco 2960xs one day, and I can be working on a cisco 9500 the next. So, like same with the access points. Like. We have clients that are in wave one access points from 2008, when I was in elementary school, to the access points that were not even released yet, and we're getting first models. It's a cool experience.

Speaker 1: 40:32

How about when you have wireless technologies that are emerging? So I'm not a wireless guru. I am aware that Wi-Fi 7 is out, about to be out I'm not sure exactly where it is, but when you have a scenario like that, is a venue more likely to go towards the new flashy, shiny stuff coming out or, you know, the tried and true known to be good? You know, because I have to imagine there's maybe certain benefits of the new architecture that in theory they sound better, but are they, you know, have they really been tested in a stadium type environment? Is that a risk that the you know, for example nfl, might be willing to take on a new technology? Uh, you know, can you speak to that at all?

Speaker 4: 41:15

yeah, I think for the most part. Uh, all the clients we work with, they come to us as the professionals and they really trust our opinions. Um, so, like, if we have a product we haven't embedded yet or we know it's not ready for main stage, we won't recommend it and we won't try to sell them the most flashy things. Um, it is a little bit different, like with the venues. Again, we're it's gonna be like try to get the latest and greatest thing in and some of the incentives we get on some of the new hardware versus the old hardware. It's like we'd be shooting ourselves in the foot not to take the new hardware. I know, like back in the day, like Cisco used to give out free DNA center appliances and free ice appliances and just be like put these things in, we need the data. It looks really good from a marketing perspective. But I mean, with enterprise, it's like let's do something that's trying, true, but also like analyze if the upgrade is worth it. I have a lot of clients like that have been pretty hesitant about Wi-Fi 6E as long as same with me me personally, it's like it's a marginal upgrade. We're going to see a lot of benefits when it comes to Wi-Fi 7. Like, if you're holding on to your Wi-Fi 5 access points or your Wi-Fi 6 access points, you can mark my word Wi-Fi 7 is actually worth it. There's going to be a ton of throughput improvements, especially like MLO it's a really popular terminology being thrown around right now. It allows a client device to connect to a 6 GHz radio and a 5 GHz radio simultaneously, so you can have a client basically doing full duplex communication. Because, like, when we think of wireless, it's half duplex, it's one way all the time. So, like, if we can do full duplex over six gigahertz and a five gigahertz frequency at the same time, you're going to see drastic throughput increases, way lower latency and, just a way, better user experience on the wi-fi from that perspective.

Speaker 4: 42:58

So, like with the refreshes and everything and emerging technologies, it's again it relies on us as a consulting firm to determine, like, is this the right choice for them and is it going to fit their budget? Because, like, we help them and assist with the budgeting of a lot of their refreshes and it's like like if we have to hold off a year or six months, like we can do that. Like, again, we have the best interest to make sure that the venue has the funds to continue going. Because tech debt is such a real thing and like how can we minimize that? Tech debt is like what we focus on, because it's like I don't know. I've worked for companies like in the past where it's like sell them the newest, latest, latest, greatest thing, get the biggest spiff and make the most amount of money from a sales perspective. But Atomic Data, we don't do that. That's why I like working here. It's never been a sales-focused mentality. It's always been what's right for the client and what's right what makes the most sense.

Speaker 1: 43:55

I love it. We've got a little bit of time left in this show. I want to pivot away a little bit because a lot of our audience is, you know, breaking in, uh, getting started earlier on in career. Um, if somebody wants to get more wireless focus, this question comes from digital bite in the audience. Um, when would somebody get started DigitalBite?

Speaker 4: 44:17

I think we all know her name. Oh, I know, I know who is she.

Speaker 1: 44:24

Where would somebody get started? Is it the CCNA? Is it the Network Plus? Wi-fi is a little bit different than wired networking, so where do we go, certification-wise, yeah.

Speaker 4: 44:37

Every thing is going to work different for people. I would say if you have a certain vendor that you already have in your organization, get certified in that vendor. But when it comes down to the best Wi-Fi certifications, I'm going to have to say the best starting is CWNA, the Certified Wireless Network Administrator. That is, they have an entire platform of certifications that can teach you very specific things. The CWNA is going to be like a high-level overview of everything where else and then from there you they have more specific certifications that you can get, whether it's the analyzer, analyzer side of things, the security side of things, the designing side of things, and going from there.

Speaker 4: 45:15

Like for me, like I was working super heavily in cisco with c Gear, my previous role and I felt comfortable with Cisco First company was Meraki. So go Meraki and then really Cisco heavy. So I'm like this makes complete sense. I'm supporting this vendor, I should go and we get the cert and a CCNP is a very well-respected certification within the industry. So, like for me, it was like let's get a CCNA to get started. And then I found out that I really like Wi-Fi from the CCNA and that was kind of like my niche and I went and pursued the CCNP and enterprise and wireless design, so tied into the Cisco side of things where you can only really use your CCMP for the implementation side. So the designing one made a lot more sense because I'm very vendor agnostic.

Speaker 4: 46:12

I know I've been talking about Cisco, but I'm partial to Cisco. Just at the end of the day, it's one of the best products out there, but again, cisco's expensive. We have to take that into reality. Just at the end of the day, it's like one of the best products. I'm out there, um, but again like cisco's expensive. Like we have to take that into reality. Like you can build an entire ubiquity stack and have extra switches in the corner and go hot swap to things when they die, like that that's just the reality of things. Like, yeah, but again, like with some of the larger venues, like downtime's not a thing, so hot swap doesn't save you during game day, like at the 90th minute of an mls. Like you can't do that, you know. Like could you be? Like imagining watching the premier league go and all of a sudden, like 90th minute they're doing like, like going into overtime and all of a sudden the core switch dies out or something and they only have one because like they didn't build the redundancy and stuff like that.

Speaker 4: 46:57

So yeah, wireless is fun, um, but I think before you get into wireless like super heavy and go get your ccn like cwna, I think you really have to have the fundamentals of, like the switching perspective, the firewall perspective, before you can really be a good wireless network engineer. Because you have to understand way more things. Because most of the time when it comes down to wireless issues, with the right deployment and the configuration set up and you have everything set to it perfectly, from the radio output power, the minimum bit rates, the SSID profiles and everything like that, most of the time it comes down to the client device being the issue or the underlying network infrastructure. So I think you have to know everything because you can't just be like I'm just a wireless engineer, I put APs in and set the RF radio. You have to understand what everything depends on, whether it's the DNS servers, the NATs, on the firewalls, the throughput limitations, the IPS and stuff like that. So there's a lot that goes into it.

Speaker 1: 47:56

So, in addition to all the wireless networking knowledge, you've got to be pretty good with your networking knowledge in general.

Speaker 4: 48:04

There's no cutting corners here. Yeah, no, no, quick way to wireless success. It sounds like, yeah, I think you can get really good. You can get really good at it. Like just the rf side of things, like going through some training I've not been through any of the account training and, uh, maybe matt needs to get me through it so I can, like vouch for him going forward. Um, but, um, I like the agao guys. They're good shit. Um, but that that's definitely something I've never done yet. So on the to-do list.

Speaker 1: 48:28

Awesome, alex. This has been a ton of fun. Uh, what? What should we have squeezed into this episode with a little bit of time that we have left?

Speaker 4: 48:36

um with networking, you'll probably never get over the reality of imposter syndrome.

Speaker 4: 48:40

I don't know about you guys, but I feel like every day you walk in and you're just as clueless as you were the day before, and you should never think that you're above anybody on your team. Again, it's a team effort and the moment you stop learning and stop slowing down, you're only falling behind. Where it's a constant evolving space and automation isn't going to steal our jobs because again, we're the plumbers of the internet. Only the good plumbers are going to stick around. That's just the reality of it.

Speaker 2: 49:08

So I like you, alex, I like you awesome so that's all I got.

Speaker 4: 49:16

I mean, feel free to reach out and connect with me on any socials. I'm pretty active on LinkedIn.

Speaker 1: 49:22

I was just going to ask where can people find you if they want to learn more?

Speaker 4: 49:26

LinkedIn is really all I do. I know I've definitely been convinced to jump into some of the social media stuff, but I don't know. I like to be an introvert end of day and I don't know, we'll see I'm starting a new role here pretty soon and work-life balance should be a lot better and maybe I'll have that time to jump into the social media realm of things. But just on LinkedIn right now is at Alex Marconaut. You can find me over there.

Speaker 1: 49:51

And do you have a blog or anything like that?

Speaker 4: 49:53

Oh yeah, I suppose I have a brief blog and it's just my initials A-O-M-I-Ttech. Again, it's just something I post some occasional articles on and things that I run into. It's something I'm starting to build out as one day I want to go and get a CWNE, and for a CWNE you kind of have to have some sort of personal record of either a blog or some sort of media presence. And for me, being more of an introvert at heart, I felt like blog was the easiest way to go. So, and also like I can go back and refer to those technical articles in the past and stuff like that. I know I've heard like other people be like oh, I just have a blog so I can always go back and look at things and that's just the reality of things. Like build documentation to share with everybody Because, again, like we're all young, like everyone comes from a starting ground and like you're gonna have mentors that are gonna help you a ton and it sucks having to learn everything on your own.

Speaker 4: 50:47

And like me coming up during COVID, I kind of felt like I was sheltered into, like just learn everything on your own and like learn through failing, and that's kind of like how I learned, um, but I mean throughout the way. I've had great mentors, like at my previous, and I mean some of those mentors have reached back out to me now and that's how I ended up getting my new role is through an old mentor. So I would say, have a mentor and really learn from them, and you never know where it's going to take you. So build your personal network. It's a big thing, great advice.

Speaker 1: 51:14

Alex, thank you so much for joining us. This has been such a fun conversation about wireless and stadiums and stuff. You blew my mind with some of the things I hadn't considered, but I don't do wireless every day, Kevin. Any closing thoughts?

Speaker 2: 51:28

No, this is really cool, thanks for joining us.

Speaker 4: 51:31

No, thank you, guys again for having me. Always feel free to reach out if you need anything.

Speaker 1: 51:35

Awesome. Well, we'll see you next time on another episode of the Art of Network Engineering Podcast.

Speaker 3: 51:43

Hey everyone, this is Andy. If you like what you heard today, then please subscribe to our podcast and your favorite podcatcher. Click that bell icon to get notified of all of our future episodes. Also follow us on Twitter and Instagram. We are at Art of Net Eng that's Art of N-E-T-E-N-G. Art of NetEng that's Art of N-E-T-E-N-G. You can also find us on the web at artofnetworkengineeringcom, where we post all of our show notes, blog articles and general networking nerdery. You can also see our pretty faces on our YouTube channel named the Art of Network Engineering. Thanks for listening. We'll see you next time.