this is the art of network engineering podcast in this podcast we'll explore tools technologies and talented people we aim to bring new information to expand your skill sets and toolbox and share the stories of fellow network engineers welcome to the art of network engineering podcast i'm one of your hosts my name is aaron weiler with me as per usual i have aj murray how are you sir very good aaron how are you i am excellente thank you sir uh daniel how art thou you have the church hat on i appreciate that thank you for getting dressed up for us oh thank you uh sunday best with his big cisco logo yeah yeah he's he's out here repping hard uh andrew how are you sir hey aaron i'm awesome man how you doing is that you on your t-shirt yes my wife got me a new t-shirt it says fast squash so that's awesome it's a bass guy whose looks like sasquatch therefore they combined the two words and made it basque funny and i get it yeah i get it funny enough it was a christmas present that arrived a couple days ago so shout out to the us postal service for my christmas present coming in february thank you folks thank you thank you for the happy surprise so good stuff it was nice uh so we normally would do wins here but we're trying to maximize time so we have a a very illustrious writer um i'm gonna make it i'm just making you blush on purpose you guys have probably heard me talk about this on the podcast more than a hundred times but it's my favorite book it's called make it stick uh they're gonna these guys are gonna make me look bad because i have the digital copy but there's a there's a reason i have the digital copy it's because i like to i like the highlights and when the highlights come to me you know like i'm just i'm just digital okay all right uh wait a minute you're you're highlighting yeah i'm highlighting in the i'm the guy that highlighting is useless hold on no no no no no no no no no no no no no no i'll explain later i'll explain later actually maybe i don't i don't want to say that in front of peter yeah he's going to yell at me so we have we have peter brown the author of make it stick thank you for joining us sir thank you thank you for having me let me start out by saying that i am one of three authors uh i i wrote this book with two cognitive psychologists uh henry roediger and mark mcdaniel who are experts in learning and memory and they are kind of psychologists at washington university in st louis washu yeah beautiful so yeah washu is like a very highly credited like medical school right and then obviously the mental sciences and things like that so it's a really really good school those two guys i have a feeling though uh relied maybe a little bit more on you in this case like in in a different way because i think it's one thing to have like the science like they do right because i've read other books that are that are very science-heavy but they aren't that good at explaining it to someone who isn't a scientist and i'm pointing directly at myself here yeah you can find it me i'm not either right right which is why i got involved so was it your idea to write the book or was it theirs no i should start by saying that henry roediger or he goes by her body has been my brother-in-law for 41 years oh yeah okay i've always known he was impressive but i didn't really understand how much until i started working with him on this project i was between writing projects uh we were chatting at a family get-together and he said you know i've been working with this team of cognitive psychologists at a bunch of different universities we've had a 10-year grant to do research into the question of what leads to learning and we're coming to the end of it and our findings are counter-intuitive and we're trying to figure out how we can get out to a general audience and so uh that was the germ that started our collaboration uh i wasn't at all sure i'd never written a science book before right i'd written non-fiction in fiction but not a science book and so i was a little nervous about it but i like a new challenge and we sat down and basically i felt that my role would be writing their roles was being smart yeah you know you can't like a normal you know book that scientists or professors write together you write three chapters and she writes three chapters and the other person writes three chapters well it didn't make sense in this case we were trying to write a book that someone like me who was a non-scientist a uh would be pulled through because it was interesting and b uh wouldn't come to understand the science in everyday language so that's how we came to to this project so before that then you had no previous knowledge as to like how people learn i mean were you interested in stuff like that before or he just said it you're like this is this sounds pretty groundbreaking because if somebody came up to me a family member and was like hey uh i just found out through a bunch of studies at my work that the entire public school system is a sham i'd be like yes sign me up do you know what i mean like that's earth-shattering to me i'm like oh yeah dude oh sure where do we go where do we go like i'll start writing right now i mean i'm not the worst writer in the world but like i i i would have been on the same page as you so was that the first kind of like of you hearing about anything like that well i didn't say yes right away i wanted to understand some of what their findings were we're talking about it and the fact of the matter is uh i mean i i have a bachelor's degree in english literature i don't have any high you know a higher degree beyond that uh and i've done a lot of different things in my life i've enjoyed encountering new things i spent a number of years 20 or 25 years as an independent management consultant working with companies to help them with their strategic planning and every time i come into a new client i'd say well how does your business work you know well why do you do it that way kind of thing so i found as as roddy described to me what they're learning i'm thinking ah trial and error i know that you know you try something doesn't work you change your strategy you reflect on it you know so in this process which involved a lot of reading of scientific papers and trying to understand them and asking questions of the of my co-authors and so forth i just felt hugely affirmed about how learning works and that notion of just you know working at figuring something out is a kind of bedrock and one of the things that we need to have school schools understand is you've got to create situations when you're asking students to figure it out right create their own understanding of the new material so anyway that's a long-winded answer known to how it was for me hey peter is this is this new science because the book came out in 2014 and i spent a scholastic career of doing it the wrong way so did is this new information like you know why wasn't i taught this as a young lad well uh yeah it is it's been put together a lot of the research is within the last decade or two i would say most of it i mean we've understood this notion of a forgetting curve thanks to a psychologist named herman ebbinghaus i think in the 1890s where he documented how after being exposed to something very quickly uh it your memory of it drops off until it begins to plateau down at a low level it's just how human beings are and so we've known that forgetting is a problem with learning uh and uh that what you need to find a way to do is interrupt the forgetting uh so a lot of this issue about long-term you know durable learning is how you can seed it into the mind in a way that it stays you know that it sticks and that that research really is is recent the power of what they call the testing effect or retrieval practice effect spacing and those kinds of strategies have really been researched and reported on over the last couple of decades well you you said like some of the like highlights from from me reading just even in chapter one is the chapter is learning is misunderstood right which uh sets the tone for the entire book clearly that's why it's for the first chapter thanks aaron for that insight but uh like just this there's just so many things that like that are just like gems to andy's point that that makes so much sense when i think about them now but but you're i'm kind of echoing what he said where it's like we spent so many years doing the exact opposite uh like you said learning is an acquired skill and the most effective strategies are counter-intuitive right so that's it you got it yeah yeah i wrote it down for a reason and we always picked the easy way right because aaron used to call me out on that yeah hold on a second well i'll get to that in a minute okay so so so so when you say that so like so like learning is an acquired skill so that begs the question then how does one acquire the skill like where do you go if school isn't teaching it yeah right yeah well i think you had to back up and say well we all have that skill in certain areas of our life i mean starting with going from crawling to walking uh you try it you you know you fail you work at it uh or you know when you get your first bicycle and you're trying to figure out how to stay upright on the dang thing and so i think naturally this the what i found so affirming was that the science basically says this is this is how you learn you have to figure it out you have to have setbacks you interpret that either as failure or give up or as information and you carry on in a little different way well you have to define learning too because for me i always thought learning was getting it short term so i could pass an exam and then i'd forget everything that's my history of learning you know yeah so that so if you take learning and go into the context of a school then we're sort of looking at a textbook we're looking down on the calendar that there's going to be an exam and i've got to remember these dates and i gotta put all the stuff in my head so that i can get the grade on the exam and continue on and uh the way i look at it like this a kid goes out for sports maybe it's a guy's eye playing football he understands his relationship to the coach coach is gonna give him some pointers he's gonna get out there and he has to do it and then he's gonna screw up a little bit he'll get some feedback and he's gonna fix it and he's gonna practice it he goes back into the classroom and sits at his desk and he expects the teachers gonna teach him fill him up with whatever it is he's supposed to know so he can pass the test it doesn't occur to him that the teacher if it's a good teacher is a coach and he's got to do the work of figuring this stuff out and trying it on your head yeah and getting some feedback that isn't quite right what is it about this that's a little different from what i showed you ah okay you know this iterative process is what learning is and we've never set up schools i don't think and maybe there are exceptions in which it's explicit from the instructor that i'm a coach you're the learner learning's a thing you do it's not something i do i will help you i'll give you feedback and so we all went to school all of us here with the expectation that we were going to be taught and that we had to get the grade if we got the grade we go on it doesn't necessarily matter if we remember the stuff that we got the grade on that's a bad system uh but it's been our system yeah so you tried it though a different way like and and before we started this like i kind of mentioned it because i went to high school like right by there but you you guys did some actual experiments in like an actual public school where you you put some of the theory kind of your point right you test it see if it works out and they put that theory to work in the middle school and specifically what it was i don't remember exactly i think i think you did uh periodic like pop quizzes instead of exams is that how it was yeah yeah yeah a very simple deal which is uh trying to compare the effect of retrieving something from memory with the effect of reviewing something and so this was a middle school the scientist roddy and mark and their colleagues got permission from the principal to do this research and for example in a social studies course the students were studying let's say geography and they're studying china and they're learning the rivers of china or something like that um one of the researchers uh would the instructor would step out because we didn't want the instructor they didn't want the instructor to know what was being tested and what wasn't uh would step out and we'd give the students a little pre test before our lesson uh then the instructor would convect they'd get a lesson and afterwards they'd get a little post-test and then a week before the exam they get another test now a third of the material was quizzed this way another third of the material they took just as much time with but it was reviewed they were told okay they weren't asked to retrieve from memory okay and another third of the material was uh neither reviewed nor quizzed okay and then and so then they did that they had their tests and this the stuff where they were reviewed it didn't do nearly as well and recalling it at the test time is the stuff where they had recalled it but not reviewed it so it just showed that reviewing something from memory retrieving it from memory as opposed to re-reading or re-hearing something is a far stronger way of interrupting the forgetting you know of of securing that in your memory like engraving yeah which was quizzing right they're pulling it back out of their memory via quizzing yeah quizzing so would you suggest that's the best way to go about it then sorry dan like uh because i he probably gonna ask what i'm gonna ask is that the best way to go about it then just keep quizzing yourself i mean yeah as a matter of fact uh that's very black and white there here it yep that's it yes that's it sir podcast over thanks for stopping it's a fact that uh retrieval from memory is uh far more effective at building a durable retention than review uh now there's many ways to retrieve something from memory you know you can go out and practice it on the field uh that's a kind of retrieval trying to remember and do something you can take a quiz uh you could try to explain it to a friend over lunch okay so lots of ways to try to pull from memory of something that you've learned why is this effective well you are in the networking business right i mean oh yeah the brain's a network definitely oh boy if you're going to make this make sense to a bunch of network engineers by by using network engineer analogies i might just leave because i have no business being here i have no business talking i have nothing to add but please go ahead sorry i don't i don't know anything about network engineering but i i i know uh that this thing's gotta connect to that thing and then that's gotta go off somewhere else that's absolutely right that's it dude that's it peter you're hired yeah toss in a couple hundred thousand dollars and there you go man so with memory uh when you are when you encounter new information it's it's recorded uh they call them traces maybe electrical signals in the hippocampus and if you're let's say you're going off to the grocery store and you're trying to remember six things you know you have your working memory and you and you can do that maybe or maybe not and uh or you can remember a phone number long enough to to dial it but if you want something to be long-term knowledge uh it has to be consolidated and moved into another part of the brain and consolidation is a process in which the brain kind of iterates this back and forth a little bit and tries to figure out what are the big ideas uh how does it relate to what i already know uh this happens largely in your sleep the presumably it will move into a part of the relevant part of the brain and the next day when you try to recall it uh that's a little hard you're thinking what were the main points you able to recall it it strengthens the connections uh between the neurons it strengthens the connections of the new knowledge to what you already know so this idea of retrieving from memory is uh causing the brain to to further win all that knowledge get rid of the irrelevant stuff really secure the most important ideas connect them to the other things you know so that you can find it again later and so you need you need to have the information in the brain you have to have good retrieval cues those two things help you get it i mean there's a lot of things we all know from childhood that we couldn't recall until maybe we catch a certain smell or someone says a certain thing and that's a cue when it comes rushing back so you need both it has to be anchored in there and then you have to have enough retrieval cues to find it and the more things in your network of your brain that the new information is connected to the more likely you are to find it again later if you have a mental image of it if you uh can associate it with a person or an event in your life it helps bring that back so is that what a retrieval cue is something that yep that's along the idea of the the memory palaces so when you're trying exactly memorize stuff you know you picture a place that you're familiar with like your home and then you can associate things that you're learning and place them in a certain room i i tried that when i was studying before and i i understand the concept but it's it's something that's new to me and and i don't think that i use it to it's it's you know best intent so it's still something i'm i'm trying to trying to use continuously and get better at but yeah it's an interesting concept i'm about where you are with that i i feel so bad i wrote about it i learned about it and i've used it a little bit uh and uh i have found that it works but uh i don't have occasion to make a whole lot of use of it for your listeners the notion is you take a familiar space let's say it's your house you walk into your house and you've got a coat hook and you've got you know different places in your house rooms that you might move through and if if you're trying to recall something or is it something you learned maybe you're going to be tested on it and you want to be able to retrieve the ideas you imagine walking into your house and you take each of the ideas and you somehow associate them with a space that you're familiar with in the house yeah so that later once someone asks you can you explain this theory to me or this whatever the the thing is you can walk into your house with that in your mind and remember these places that are familiar with you how you associated them with the ideas of this thing that you want to explain so it is no way of learning it's a way of of organizing the memory of being able to find it are they all retrieval cues those rooms in your house the drawers that yeah yeah right so if if i understand it correctly when i was studying for the route exam what i would do is um you know if i wanted to learn eigrp i put eigrp in my dining room because dining room starts with d and d is how it would show up in the router right nice nice and then i took ospf and i put it on my back deck because i took ospf and i turned it into outside padding outside outside patio furniture i'm going to remember that yeah yeah and then i had like the different lsa type sitting with me at the at the table uh on my back deck and yeah it kind of worked it didn't you know i don't think i practiced it enough to to have it work as as best as it could but you know reading in and make it stick the example that they gave was students at a coffee shop and then they would see a plant and picture somebody having a coffee and i forget the exact context of it but it was more of a it was more of a story than it was um you know a specific you know just trying to walk into a room and remember where things were placed so i thought using the storyline to help recall that information was a lot more helpful and i'm going to try that the next time i use that tactic yeah these were young people who are uh keen on some movies they really liked and they had the names of the characters and they could associate it with different places yeah that's right it worked very well for them hey peter you mentioned something i didn't want to to skip over so you said consolidation happens in your sleep so how important is sleep for getting this stuff in a long-term memory and i'm asking for a reason because i don't sleep as much as i should so am i shooting myself in the foot you know i'm studying and i'm doing flash cards and i'm trying to remember but then i get five hours of sleep at night am i doing myself a disservice because that's when consolidation happens if i'm understanding it correctly this is something that science is still coming to understand but both the neurologists and the cognitive psychologists are on the same wavelength that sleep plays an important part understanding how that works is still being learned but there's quite a bit of evidence that it is important through both the different forms of sleep you have over an extended a nighttime of sleep more than you get andy right and i mean i'll get more sleep if it helps me pass more of these exams you know this is good information to get i don't know if is it more about the quality of the sleep versus the quantity of this yeah do you have to hit that second level of like rapid eye movement or whatever hey i'm not authorized not authorized i'm not smart enough to really answer that question i will say this surely you've had the experience that i've had uh where you're working on writing something or figuring something out and you're kind of frustrated with it and you've done something with it and then you leave it and you go sleep and you you wake up in the morning and you say you know i think i got an idea or you know it work you know you've gotten something in the night that you didn't have when you went to bed because you kind of presented a problem i think this is something that that we don't give enough credit to our brains that the brain keeps working on stuff if you give an assignment and and then uh even i know in my life uh when i'm struggling with something i'm writing uh when and i get to a certain point my wife likes to go to the gym in the morning i said no i want to work and then go to the gym when i go to the gym in the afternoon and get on the elliptical or get on my bike something about moving and all of that oxygen i don't know what it is but i feel like the brain then is working when i said you're off duty yeah the brain works and then i got the idea i think that's part of this whole issue of consolidation that we're still learning about it's amazing now go ahead dan so um one thing i was wanting to kind of go back on though is you were talking about the students taking a quiz before they even got the lecture right oh yeah and then they got a quiz after they got the lecture and then they waited like i did you say a week and then they and they got a quiz again or was it right before the exam i think it was maybe a week before the exam yeah so one thing to our listeners so and i'm repping cisco and cisco is one of the bigger players in our field and the ocgs always have a do i know this for peter that's official guide so we have to take certification exams and they provide books there's there's several of them on the the shelf behind me that it contains everything we need to know in order to pass i mean they're like this thick they're theaters awful books peter written by people scientists right they're scientists like you were saying like you're trying to determine what all these scientists are trying to say it's like so what dan what dan's saying is that at the beginning of the chapter there's there's a basically a pretest before you even read the chapter right in in the way i was doing myself a disservice on just skipping that and going straight into reading the book or reading the chapter right and then afterwards i would go back and take that test and when aaron started talking about your book in like you know that that process of hey before you even try it quiz yourself on it and then take it so i was like you know what i'm gonna give that a try and i did it on the one of the next chapters and i took that that quiz before i read the chapter and then when i read the chapter as i was reading the chapter i remembered those questions and it like it stuck with me way better afterwards so like that that was that like blew my mind that hey maybe i should actually try this yeah because why would you take a test on material you haven't learned yet right like counter-intuitive it's more that counter-intuitive stuff and that was my that was my my whole problem with it that you know i basically just like i'm not gonna take that out how would i ever know that you know but then it's like oh i see what they're doing they're trying you know you're you're trying to prompt you yeah exactly and and it worked way better for me so i i can attest to that that your your study got much better after that yeah he didn't believe me we had to we had to get we had to get you the actual horse's mouth on this podcast to reinforce it no no no no you told me aaron and i did it so there you go i believe that's fair that's fair i mean to to be fair to peter uh you know you said earlier like i'm not qualified to give that advice boy oh boy uh i am certainly not qualified to give the advice but that never stops me one bit i'm not you know aaron's always slinging around yeah it's mainly stuff from your book but you know i and the my problem is i i uh this is awful to say like i forget like the details sometimes and i'm trying to explain stuff and like you know you remember those sort of things because you wrote the book but for me i'm like there was a test or this this study they did with these people in a high school you know what i mean so i i'm not very good at it which is why this is helpful to actually hear it from you so folks aren't just hearing it from me i'm because i'm just regurgitating what you said right so it's this is certainly helpful and i think one thing to add to that too and i think the book you did a really excellent job with this is every every piece of the book there's some kind of example of it being used right yeah between you know neurosurgeons baseball players you know the pilot all that stuff all that yeah i really like it yeah exactly yeah there's always a an example of of that being used or why we should be able to remember stuff you know like that and uh so i think you did an excellent job with that but you know i obviously recommend our uh listeners to read that because it's not a science it doesn't come off as a science book is what i'm trying to say well i don't yeah right that was one of my objectives was to write a book that i'd be interested in reading so i like a good story as long as it doesn't go on forever it has a point to it but to your point uh daniel yeah if you are asked to try to solve a problem before you've been taught how and you struggle with it and you say what is this like that i already know is it like this is it like that and you struggle with it and maybe you don't get it uh but then your instructor or whoever says okay this is how you solve this problem and here's why it's different from that or different from that that it sticks a whole lot better it's be somehow you've sort of primed your brain to be ready for that answer and uh it in this in the simple simplest and and also a much more complex test it's quite striking at how important it is to have tried to figure it out then uh get corrective feedback let's say you're on a quiz or whatever it is and be taught the correct answer and then you apply it again later to you know retrieve it and strengthen that so uh it's it's uh it's just kind of amazing when you go through these studies now the other thing is when you're reading something it helps if you start with a question and read for the answer so what i found reading these i went through a whole lot of research papers and trying to understand what's the question that the researchers were trying to answer what results did they get and how do they explain those results and what's the relevance of that to what i'm trying to write and i would have to read those things a couple of times and then mark is comes to underlining and highlighting you know uh that isn't a good way to learn but it is for me a way to organize the bits and pieces that i want to be able to put together and describe to somebody else and so it is not a study strategy where you figure you underline and highlight it and then you re-read that stuff later it doesn't really help out but if you're doing it in a way in which you're going to harvest those notes and put them in your own words or make it write an argument for it then it is and that's when you discover aaron to your point i know this is it but i don't remember why you go back and look to see what the detail was that you might wish you could have explained right right so that's the iterative process i'm sure in network engineering there's a lot of back and forth you know you're going to try something that works you know buddy yeah yeah yeah so don't you think at some point uh sorry don't you think at some point um either through the education system or just life experience that we interpreted failure as an end result rather than part of the process oh no doubt uh you know because that for for me until i started you know working with these guys in the podcast and aaron started espousing you know the great knowledge that comes out of your book and then i actually read the book for myself like failure was like the end game you know and it was you know usually a huge setback but once once i read the book and started realizing oh this is part of the process like it's okay to fail like you should almost be failing more right like like keep failing well there's a i guess you can make an argument that if you get it every time maybe you're not reaching far enough true yeah yeah cause like you said make it effortful right yeah yeah you want to reach somehow you gotta have a foot in the familiar and the other foot into the new yeah you can't learn anything new that you can't if you can't connect it to what you already know right right that makes sense so uh i think we have a problem with uh education system that's so dependent on getting a high grade in order to get admitted to the next level and all that kind of thing and as opposed to somehow emphasizing the taking the risk to figure it out and get it wrong and get the information from that and then finally create your own understanding and go on from there this is uh this the science in the book is having a huge impact uh there's a lot of material out now on this science oh okay and medical schools and law schools and places where there's a lot of highly complex learning required and a lot of these schools are restructuring their pedagogy around it really they're cutting uh lectures way down and they're uh moving to uh active learning exercises and team-based stuff ah that's awesome well it's phenomenal really i mean it's just amazing yeah yeah yeah i mean like i'm almost scared to you know like send my kids to public school after reading this book you know what i mean because look i'll just be the first i'll just i'll just admit it like i was awful at public school like i sucked so bad like i just can't retain information because i think i guess now learning that like kind of what andy was saying earlier is it just me is it just me i actually made the comment last week in our our group chat here with the podcast guys like like it made me feel stupid reading this book but in a good way right like it val it validated it validated that like andy said it's not just me but also that that there is a way to do it and it's okay for it to be effortful and to aj's point it's okay to fail in fact i i the lesson i got was like keep keep failing right because the more you do it like like he was saying it's like the more i was getting out of it to the point now where like i'm almost like i try to like torture myself like as much as possible so i'll i'll clarify a little bit so like i make flash cards you know when i'm trying to learn something that's usually what i do but but i don't make like uh you know a flash card that says it has a one winner to answer or something like is this true or false that just seems like pointless to me so what i do i mean yeah i make it as hard as i can and i'll say hey explain this and it just says explain this and so i'll have to think and try to explain that whole thing in my head you know it the first couple times you do it it just feels so weird you're like you know you're like i know that i know that like it's so weird how you talk to yourself you know what i mean like because in your head you're going you're going come on you know that anyway let's just move on to the next thing like yeah yeah and then you flip it over and you go yeah yeah i knew that i knew that right but but you didn't so but you didn't that's a really important point that if students are being taught now that you need to do self quizzing yeah but you need to write the answer down and you need to check it later you have to check it to make sure that you're correct your judgments of what you know and can do all right because the big issue is am i a good judge of what i know it can do and the best way to find that out is to do it to show it can you briefly i think it's what you're getting at can you briefly describe the illusion of competence i think it's called or i may be getting the term wrong but you think you know things that sure maybe you don't the illusions of knowing yeah a typical situation would be uh where you've read uh something text for an exam you've read it and then you've highlighted it and you've underlined it and then you yeah you had some notes from the lecturer and you you wrote the notes and then you read the notes yeah and you thought okay i've spent a lot of time on this i'm on it and you go into the exam and you don't do well in the exam you you feel like they cheated because they asked you all right what is the answer why do you think it's that or whatever you know and you don't have anything beyond the language that you read right you don't have you know but you have a fluency with that language and you mistake that for learning right so uh you just explained all of 2019 for me but there's another example i'll let me give you a difference you explained my entire scholastic career peter not just i'm serious that was my entire study method yeah i wish this book came along 20 years ago but thank god it's here you know but it's true in sports so uh i talk about the 20-foot putt i don't know where i got that but the idea is you know you're trying to improve your 20-foot putt so you do the 20-foot putt over and over again and you see yourself improving and you go home thinking damn that was pretty good you know uh and you you think you've learned it but the fact is it's all in short-term memory you know it's built you're leaning on what you just did i i was just reading someone recently who was writing about this uh who had read make it stick and and uh he said this thing that i wish i had thought of uh he said multiply uh 21 by 14 and uh tell me the answer so he waits for a while and someone will say it's 294. oh great how did you come to that well i multiplied 21 by 10 and then i multiply you know whatever he did is he went through this process and you got it the guy said great okay next problem multiply 21 by 14. oh i know the answer 394. well how'd you come to that he said well we just did it he said that's practicing your 20-foot putt over and over again to learn it you need to go through the process and then go through the process and so instead of doing the 20-foot putt in rapid succession do it a few times get a feel for it then do some other strokes then come back yeah yeah then go do some other things and then come back so this idea of mixing it up you're really trying to learn how do i judge distance lay i don't know the golf terminology you know but all the different things that come into play you want to learn how to be able to judge that for 20 feet 10 feet 30 feet i don't know and so to build that skill you have to mix it up yeah there was a story sorry yeah the kid yeah with the bean right that blew me away same they just practiced at the one distance and then the other group practiced it three or four different distances and the ones but not the one that but not the one they were tested on later right and the one they were tested on they were so much better than the people who only did the quote-unquote 20-foot putt so that really like like so much better like it's amazing crazy it's like breaking my brain though because i i'm so used to like i have to start a chapter and finish that chapter and then study over that and over that and over that and then once i'm done i put that chapter in the past and then i move to the next chapter and then i don't ever go back to that chapter and that's and it's like that's totally the wrong way to do it and then by the time you go take the exam you've forgotten everything you learned yeah the whole chapter yeah yeah well it's not hard so the interleaving thing is something that that's interesting to me too because you actually hit on i think one of the more important parts about interleaving which was the dissimilate differences and similarities right but different but somewhat the same topics in your analogy about the 20-foot putt you said yeah it actually works better if you are doing the 20-foot putt and then maybe you do a five foot putt and then maybe you go hit a driver right but the point is you didn't stop your 20-foot putt to go dunk a basketball and practice that right right yeah so so same same but different and like different enough which is why i think we have such an advantage like just these books that we get that are like gigantic they they are you know somewhat curated into a bunch of subjects that that make sense with each other right so like dan can show you right here i mean these things are ridiculous yeah i i know like why would anybody use that right all these little lines or chapters but i guess you could jump around right because they're all yeah that's the idea yeah exactly that's what i'm saying and i think that's why why this hits home for us or why we can put it into practice so easily is because we have that at our fingertips and although it's a huge book and there's like 20 some odd chapters of you know in each one of them is like 100 pages it's still somewhat similar right to the point where you know you can actually and i'll just use in a specific example and peter probably sorry if i'm just not making any sense to you at this point but but like some like two different routing protocols the thing is they're both routing protocols right so it's perfectly okay and it probably do you so much more good if you actually read them at the same time and and decided here's my goal there's a chapter on eigrp and a chapter on ospf i'm not even going to read the chapters i just want to know what is the difference like why why why do two exist right like why clearly there's two for a reason why is that and then work at it from that way because like what you were saying earlier is like uh and you even quoted the book like the einstein quote right where it says uh like basically like who cares about knowledge it's creativity like this is the most important thing and that's it it's the problem solving it's like figure it out what you know like you know you're teaching kids biology in fifth grade and it's like hey why do you think what how would we stop a disease just come up with some ideas like you know what do you think it is how would you stop it you know just just crazy stuff like that and like i just feel like the world would be so much different if we were just taught that early on to just think like like a problem solving you know what i mean right how to think how to think yeah how to think you know and i we always bitch about this like my wife and i are particularly fond of bitching about how school never teaches us how to do our taxes and stuff you know like we're like like we we actually had a conversation about this earlier which is why it's so topical like my wife is in healthcare like she has a master's degree in a healthcare field and she said to me like dude i didn't know that i could use my hsa account to do this and i'm like what okay how did you not know that she's like no one ever tells you any of this i'm like dude you went to school for like eight years for this and you didn't know that you could use it you know what i'm saying but that's like it's kind of the same idea it's like it's like well you constantly we're just like waiting for people to tell us stuff but like they should be more of just like a like let us go like you know make something up even if it's wrong just make something up right like that's how you that's how somebody else figured it out right i i'm well i enjoy that more in life than the other you know trying to figure it out and make it make it up and then see before i get wrong i find out i get it wrong because the thing breaks or i find out because i stop and ask someone is this thing going to work or how someone else has done it right but right but engaging with it for sure is i think more fulfilling and leads to better learning the whole thing about you know the different router methods it's sort of pausing to ask why and what if right and just asking those questions when you're learning something new what if it wasn't this way or what if i did it that way or why you know very simple questions that give you another dimension or what is this like that i already know can i think about parallel for this because those questions help make the connections in here and and give you uh a third dimension to the understanding of the topic well yeah you got it you got to be right too because certain analogies have stuck with me over the years just because they were so good right like a teacher we've all had that so so before we went out of time i just want to ask and obviously the overall message here is we've all read the book it's blown me away it's helping all of us retain more and we work in a field where we're constantly studying and being tested peter so this you know i think our audience is a very captive audience for this the so i hope before we're done maybe you can throw like i don't know top three or five tips out of like hey guys you know here's what i tell you to do and maybe we've been doing that throughout but the the real question i have is we generally read those awful books and then we do some labbing and different things and now i'm into the flashcards for you know to pull them out of memory and quiz myself but a lot of what we do as well is watch videos so i guess what i'm wondering is is that too passive am i wasting my time watching you know someone's telling me something but i'm not making flash cards let's say or is reading better that makes sense uh oh i can't answer that particularly i could tell you that uh when i dismantled the dashboard of my prius so i could get a new antenna plugged into the radio the video helped me hugely but if i hadn't actually been doing it also i probably wouldn't have learned nearly as much as the fact that i was using the video as a guide to help figure out when you can just jerk on something and it comes out yeah and when you got to find that screw and that can't screw it and that's my point is video is easier for me but i know a big undercurrent or theme is you know make it difficult and maybe the video can be used in as a complimentary thing to you know making it difficult but i'm right there with you when i need to do something at home i'm on youtube i watch a guy and it's much better i've always looked at the video as like the the pre-test right like the do do i already know this like watch the video first and that's that's the primer and then i go read something and then it's like oh that's what they were talking about in the video right right i think yeah that's right okay i almost think sorry i i was just saying i almost think andy like what you're saying i think that watching the video because here's where the disconnect is i i think and it just correct me from wrong andy um it's like okay when when i'm making things difficult and when i'm not making things difficult like at what point should there should those be right and here's here's where here's where i think this might help so when you're first learning something right like your first exposure to it i don't think it matters if it's a video or if you're reading something or someone's telling you necessarily it's what you do after that right so how you originally obtain the information isn't as relevant as how you retrieve it does that make more sense so if i'm making flash cards based on a video i'm watching it's probably the same thing as making them off of a because that's what i've been doing lately is i'm making flash cards as i'm watching videos because peter flash cards to me are the secret sauce right now so i hope i'm on the right path because oh yeah you definitely are it's it's in the retrieval from memory that the burning herb happens it isn't about getting it in it's about getting it out yeah it's about getting it in but it isn't it's about getting it out and if you're not able to get it out you got to go look it up and then start getting it out so i one one thing that worked for me one time uh uh after aaron went on his whole uh speech to us about all this um is i i watched several videos and and this is where i'm gonna go into a little bit of tech here but i i started watching several videos about hsrp right and and after i watched those episodes i started labbing them as i was watching them so i would watch a little bit and then i'd do it in my lab and then after i got done then i wrote about hsrp in the sense of like hey this is well basically i i was sharing my lab that i did and i tell you what that worked so well for me that i would be fantastic yeah i was like that i need to do that to every little piece of this book and i feel like at that point i really get it yeah that's exactly right so you're you're you're getting exposed to it and then you're doing it and then you're writing about it you're processing it at several levels and then you're coming back to it so i wanna i wanna point out something because i i kind of secretly like uh bursting everybody's bubble um oh boy no i love it i love it i love it can you just briefly touch on uh quote unquote learning styles well uh yeah you know learning styles uh the concept of learning styles is basically that some people learn better as listening others by reading some kinetically uh i don't know there's lots of different learning styles depending on who you uh read and the underlying premise is that some people are wired to learn better one way than another way i suppose if you have a sort of a condition like dyslexia that's true sure in some ways that people with dyslexia learn better than others absolutely but for most people the research into learning styles has been unable to demonstrate that in any cases where someone who considers themselves to be an auditory learner actually learns better when the material is taught in that matter than in a written way uh there's just not a body of research to support learning styles and uh if what we comparing that to uh retrieval practice and mixed practice and space practice and elaboration and those kinds of things that have a large body of research you're just better off going with what science shows works than trying to wrap yourself around the axle to translate some material into an auditory form right right yeah yeah people get hung up on that so much you know and and and that's like i've just been trying to help people because i feel like well i just feel like it's a i feel like it's a limiting behavior like if you if you brand yourself as someone who only learns from videos then you're just going to steer clear a text and that's just not you're just doing yourself a disservice there so right so the science is saying otherwise i just i wanted you to say that because i've been wanting to say that for a very long time on here i may have slipped and said it i don't actually know there's there's no live stream okay well aaron had a good point you had a good point aaron in that people's learning style i find this to be true once you called me out on it and we had a very heated debate on this i think but yeah you guys it was it was not pretty people when they pick a learning style that they prefer it's easier for them right like you're i'm picking the easy path so it's the opposite of what right the preferred preferred pattern which is humans are going to pick the easier path i think right so it that's the opposite of of making it hard last question peter i gotta ask before we run out of time i previously to being schooled to your book by aaron um i used to take copious notes i mean i would rewrite textbooks and i may or may not read them before before taking the test so i remember being taught that if you know if you read something you're taking it that way if you listen the more of your senses you can hit your brain with the information so is there any value in writing notes of the information you're learning not flashcards but you know actually just writing out things in note form because it i'm not doing that now i'm trying everything in your book new for me in the past couple months and i haven't tested yet to see you know how i'm going to do but i've stopped taking notes besides flash cards is that a mistake is there any value in rewriting sections of textbooks because it seems like a waste to me yeah i think it's a waste uh there's a little bit of research i love hey i need to hear that you know there's a little bit of research that shows handwritten notes are are are more effective for learning than than notes on a keypad because you have to synthesize because you can't keep up with the handwriting and the act of synthesizing the ideas that aids the learning but the uh the i think notes some people i've i've heard from say i've started writing down questions when i take notes yes so i go back and answer the questions and then i find that highly effective wow okay people say and i can see why that would be because that's that's what you're trying to know is can i can i answer those questions uh i find for me writing notes helps me if when i'm reading material like research papers i i can get an idea and say here's an idea i'm going to write a little note about that and i'm going to stitch ideas together and then i'm going to construct an argument that's a little different than what you're describing but there's a place for notes but i don't think rewriting a chapter or something is not really useful to you uh compared to saying what are the questions that i'm going to need to answer uh or um you know calling up aaron and saying i just read this thing can i describe to you what it is and yeah ask me some questions about it right so if i can make one change in my study habits do you think flash cards is the biggest bang for your buck because you're going to be quizzing yourself and pulling it down once you use them right yeah right you got to create decent ones i guess right i think that uh the big ideas for me are retrieval from memory uh and uh and elaborating on that with why or what if kinds of questions so it gives it her you know what is it like that i already know that kind of thing the second is to space it out so that you come back to it from time to time because when you get a little rusty at it retrieving it is hard and it strengthens those connections and it will it will update it to other stuff you've learned in the meantime so retrieving it with elaboration uh spacing out your practice like with flash cards the ones you know well put them further back and don't do them as often right yep is the spacing about fighting that forgetting curve yeah the spacing is is exactly that so you uh people say how long does space well it depends you want to space it far enough that it's hard to retrieve but you can do it yeah the goldilocks zone right yeah exactly exactly this takes some effort because that effort is firing up stuff and uh is useful more useful than doing it so the third thing so i've talked about retrieval and elaboration i've talked about spacing now the third one is mixing up similar problem types the practice of similar problem types so interleaving yeah in the book we call it interleaving yeah and uh so uh you're not spending a lot of time practicing a particular problem before you move on to another particular problem but mix them up so would that be the 20 foot putt the five foot putt is that exactly exactly it's mixing uh the fastball curve ball and change up right uh instead of doing 15 of one and 15. it's all baseball you're not going from baseball to basketball but yeah exactly yeah yeah exactly exactly right yeah makes sense yeah and i was chatting with a friend of mine who coaches basketball and he said well we do that because when we do drills they do every all these different things from different points on the court but the fact problem is they were doing the same thing at each point on the court so it was like an lp album you know what song is next right it doesn't help you so you almost have to like yeah because i i literally wrote down get rusty because that's that's exactly what you're saying like it should be effortful but like i said we just said the goldilocks zone should be effortful but you still should be able to yeah but you could do it man i think dude first of all thank you so much for watching one more question to ask questions so so the book was published in 2014 i'm curious to know has anything new come out since then uh that maybe you know obviously didn't make it into the book that you know we could be questions as we learn you know i asked that question of my colleagues uh and there's there's been some parsing of this question of uh um when you interleave or mix stuff up how far afield from one to another should you go and it seems not real far uh you want to stay within the domain uh it doesn't uh help you a lot to jump between subjects uh and mix it up i mean that yes in terms of spacing but not in terms of a practice session another question that people are working on is how do we get students early to understand these strategies and embrace them when they feel hard and uh so there's some research being done on that area but there hasn't been any uh anything momentous uh that has shifted uh the understanding it's just more fine-tuning i think that's well i mean as you mentioned and when uh you know this is still relatively new like only the last couple of decades which i i didn't realize and thought that was really interesting so uh thank you so much peter for for hanging with us this evening um i'd like for you to you know promote yourself a little bit you said you wrote some other books can you share with us uh some of the other titles that you've written for our listeners that might be curious to read more of your work where can we find your books that type of thing yeah one book i wrote was called jumping the job track uh security satisfaction and success as an independent consultant that might be something that would appeal to you oh yeah it's pretty old now it came out in 94 but it was very successful and it really is about um it's uh this is something i did i left a corporate job and it went out on my own and it's a lot about um that you have a lot more to offer than to think than you think you do you know and not getting yourself too too stuck as having to be an expert in this particular thing wow anyway yeah i know that's right up our app especially in the last year because we've had a lot of people coming from one industry into another because you know the industry they were in collapse because of covenant 19 and now they're finding their way into tech so i think that's more way more applicable than you think yeah well as i say it's a bit dated but but it's good um i don't know other stuff i don't know that your uh audience would be interested in the one possibility would be a historical novel i wrote in 2006 it was uh new york times uh editor's choice in washington post best books of 2006. this side and so no big deal that's a big deal to me yeah no i'm kidding i got yeah amazing yeah i said in the gold rush to normal alaska in 1900 i drew from my grandfather's diaries to give it a a certain historical accuracy and anyway wow it's uh it's that's a good story wow that's right i can be reached at authors at makeitstick.com okay if anybody wants to write i have my books are at peterc brown.com we'll put all this in the show notes as well so anybody that downloads it'll be able to get all your stuff so they don't have to remember us trying to spell things out because we're really not that good at it and depending on how good we did on this episode peter you might want to prepare for an influx of technical nerds emailing you with that address asking you questions i might have technical problems at my end oh dude this has been awesome man it's been really fun for me i've really enjoyed this no it's it's it's great and and it's been reinforcing like everything that you know we've been trying to help other folks out with here so to hear it straight from your mouth i think it means just a lot more than coming from any of ours because you're much more credible than us for one but your name is on the book so you're responsible whether you like it or not um hopefully the tides are changing here in the united states and elsewhere with with knowledge and and my hope is that we don't have to have these conversations in the future because everybody's like duh right and and if you change that that's huge man i really think that what you've done with this book and like i say all this stuff when you're not here so i don't think i'm just gassing you up because it's not my style okay it's not my style man but the the whole point is like like if if we get to this point where where we don't have to use this knowledge and this gets completely forgotten about like i you have changed so much about the world and just like you know just yeah your brother-in-law may have been the scientist and stuff like that but like without people to to feed that to us you know plebeians we wouldn't know we would know how to read it for one because trust me i've tried i mean we have a hard enough time with the stuff that we're supposed to be dealing with in our own subjects you know so uh we we we greatly appreciate greatly appreciate yeah and and to hear that you're human too so like hey that that to me that has a whole other dimension to it yeah and and so we appreciate you being on here um and certainly like taking time out here dave to to talk to all of us so also be be prepared for an influx of book sales just have a guess just have a guess i'll let you know thank you i appreciate that so thanks peter uh we appreciate you being on and for the rest of us uh the art of network engineering see ya hey everyone this is aj if you like what you heard today then make sure you subscribe to our podcast and your favorite podcatcher smash that bell icon to get notified of all of our future episodes also follow us on twitter and instagram we are at art of net edge that's art of n-e-t-e-n-g you can also find us on the web at art of network engineering.com where we post all of our show notes you can read blog articles from the co-hosts and guests and also a lot more news and info from the networking world thanks for listening you