Hey, it's Jason, host of the Code with Jason podcast.

You're a developer.

You like to listen to podcasts.

You're listening to one right now.

Maybe you like to read blogs and subscribe to email newsletters and stuff like that.

Keep in touch.

Email newsletters are a really nice way to keep on top of what's going on in the programming world.

Except they're actually not.

I don't know about you, but the last thing that I want to do after a long day of staring at the screen is sit there and stare at the screen some more.

That's why I started a different kind of newsletter.

It's a snail mail programming newsletter.

That's right, I send an actual envelope in the mail containing a paper newsletter that you can hold in your hands.

You can read it on your living room couch, at your kitchen table, in your bed, or in someone else's bed.

And when they say, What are you doing in my bed?

You can say, I'm reading Jason's newsletter.

What does it look like?

You might wonder what you might find in this snail mail programming newsletter.

You can read about all kinds of programming topics like object-oriented programming, testing, DevOps, AI.

Most of it's pretty technology agnostic.

You can also read about other non-programming topics like philosophy, evolutionary theory, business, marketing, economics, psychology, music, cooking, history, geology, language, culture, robotics, and farming.

The name of the newsletter is Nonsense Monthly.

Here's what some of my readers are saying about it.

Helmut Kobler from Los Angeles says thanks much for sending the newsletter.

I got it about a week ago and read it on my sofa.

It was a totally different experience than reading it on my computer or iPad.

It felt more relaxed, more meaningful, something special and out of the ordinary.

I'm sure that's what you were going for, so just wanted to let you know that you succeeded.

Looking forward to more.

Chris Sonnier from Dickinson, Texas says, just finished reading my first nonsense monthly snail mail newsletter and truly enjoyed it.

Something about holding a physical piece of paper that just feels good.

Thank you for this.

Can't wait for the next one.

Dear listener, if you would like to get letters in the mail from yours truly every month, you can go sign up at nonsensemonthly dot com.

That's nonsensemonthly.com.

I'll say it one more time.

NonsenseMonthly dot com.

And now without further ado, here is today's episode.

Welcome to the Code of Jason podcast.

We have double guest action today.

We have Austin Chadwick and Chris Lucian.

Uh welcome to the show, guys.

Hello, hello.

Thanks for having us.

Super excited.

Yeah, thanks for being here.

Um let's do some quick intros.

How about you go first, Austin?

Sure.

Uh so I've been in software about 16 years or so.

And uh I like to think of my career in three phases.

Uh one was like very waterfall, and the other phase was very like uh quote unquote scrum.

And then the last five or six years has been like, you know, extreme programming and mobbing or and pairing turned up all the way.

So like you know, continuous delivery, mobbing all the time, uh TDD all the time, and uh and I love it.

Um got a wife and four kids now.

We uh just had a new one, and so uh a busy household.

So if you hear crying in the background or some sort of chaos, that's that's that's what's happening.

So uh very understandable.

I have kids myself, although they're older now, they are uh 11 and 14.

So if if they start crying in the background, something's really wrong.

Um and I suppose it's it's germaine to mention that you guys have a podcast of your own, the mob mentality show, right?

Yeah, that's right.

Uh we yeah, we we we talk uh every week about m all things mob programming and extreme programming adjacent.

Yeah, and Chris, how about tell us about yourself?

Yeah, uh so um, you know, I I am uh a director of software development now, but uh kind of started on on at my current company uh as a software engineer originally.

Um we were kind of responsible for coining the term mob programming about, I don't know, 12 years ago, 13 years ago at this point, um, and talking about it.

Or at least we coined it for ourselves.

I think there's some some history there about it existing beforehand.

Um but uh yeah, so just did a lot of uh you know uh a lot of interesting stuff.

I I'm I'm very interested in like AI and machine learning.

So I have uh I also have my master's degree in in that space, and that was something that I was planning on doing before I kind of changed my plans.

Um and so I've been kind of in machine learning space since about uh 2014 or so as well.

And uh yeah, I just do all kinds of uh software-related stuff.

Okay, interesting.

Well, I'm very tempted to get into the AI and machine learning stuff, and I may well give in to my temptation.

Um first I'll I'll say a thing or two about myself.

Um, this is the first podcast I've recorded since a um kind of significant change in my career.

I was doing consulting for the last year and a half or so.

Um, and then recently I decided to get a job, and now I do have a job.

I started working for an organization called Cisco Meraki, um, which which had its history in a startup called Meraki.

Then they got bought by Cisco.

People ask sometimes like Cisco Meraki, is that the same Cisco as like the Cisco that I know?

Uh yes, it is.

Um, so I started working there as a tech lead on a new team, um, working on their build and deploy systems and and some other related stuff.

Um and I also want to mention I was on you guys' show um some weeks ago and we talked about testing, and that was a pretty fun discussion.

Um okay, so yeah, uh Chris, I have to get on get into this AI stuff.

Um I've been thinking about this a lot lately because I have come to a pretty firm conclusion that everybody's doing it all wrong.

Um but you know, I realize at the same time that I know very little.

Um so it's it's usually when I don't know very much about a topic, I hesitate to form very strong opinions.

But this is a rare case where the right answer just seems like abundantly clear to me, and I don't even feel like I need to know a lot more.

Um so what do I mean by that?

I mean this whole like statistical approach, um, that basically all the AI that I've seen is taking, I think it's on the wrong path, and like no matter how good the statistics get, we're we're not gonna get to AGI.

So like Chat GPT, the other LLMs, they're they're they're they're manipulating text, but they're kind of just rearranging symbols, you know.

The AI doesn't know what these symbols mean.

Um, and so there's always gonna be a pretty low ceiling on how smart it can be.

And what we need instead is uh I I'm I'm pretty I'm leaning increasingly in in this direction.

What we need is robots that experience physical reality and and start there, and then uh eventually uh through a long path, we get our way back to text, and then we can get an AGI that is truly intelligent and can work like a chatbot.

But in order to get to that intelligent text manipulation manipulation, you have to start with robots interacting in the real world.

Anyway, uh question for you, Chris.

Am I crazy?

Is there any merit to that, do you think?

Or do you have what's your take on all this?

Uh so so there there is a a faction of like the AGI space that basically says that we start as babies and and then interpret the world and learn and grow over time because that is part of the requirement.

Like you can't just be like born instantly with the smarts sort of thing.

So um there is there is a branch of research in this idea of you know learning from the state uh at which being a baby with like no knowledge and then gradually incorporating things.

And so and there's an interesting thing about the human brain, um, because you know, these large language models are a subset of neural networks, which is a subset of deep learning.

Deep learning can be you could do shallow or deep learning in um neural networks, but and then there's a subset of neural networks called transformers, and then that subset leads to these large language models that are very popular today, and all of that, you know, so this is like a very small slice of AI uh or machine learning in general.

Um and and so uh but but in that in the broader machine learning space, there is this idea of like teach it gradually over time, and you can simulate many years of learning to get up to a certain asymptote, right?

Uh or or like point of performance.

And then at some level, you do have to have like new experiences in general.

Um, and another thing that I think I've talked about a little bit before, and I don't I don't know how uh uh how boring this might be for Austin or not, but um the uh uh we um we've basically put all of human history into these these models, and this, like you said, the statistics can improve, but but it's it's not really creating things that are new.

But there is an area in in computer science and AI and machine learning that is not in the neural network, which there's actually two areas there's unsupervised learning.

So there's this idea that you can take in new data all the time and continue to learn.

And so um the K-nearest neighbors machine learning algorithm is one that does this.

Um, but it's also very um simplistic in nature and has not yielded the same results as something like uh the chatbots we see.

Um and and the other is evolutionary algorithms, but both of these can produce new experiences that then get learned further.

And uh and so eventually you end up creating or or uh moving far along farther along that path.

And so it's it's but like I'll back up for a second.

It's really funny that everybody's talking about AI now.

Like, so when I first got my master's and I was doing things in computer vision, it's like no one cared.

It was very hard to get anybody to talk about AI or anything like that.

Um, ever since Chad GBT and like you know, uh 22, 23, um, it was like it's all people wanted to talk about.

And but these transformer models and specifically these large language models, like it's all anyone's focusing about, but there's there's this like huge wealth of information in the other facets of the machine learning space that could help, but but they all the hype is in this concentrated heavily in this one area, yeah.

Um, so it's just really funny that it is really funny, and and I feel like there's I've I've talked about this before, I don't think it was on the air, but like there's almost these like two camps of of people, there's like this really overhyped uh group of people um that are like extrapolating from where we are now, and they're like, oh, AI is gonna take over the world in the next two years, because if you look at where we were a few years ago and where we are now, it's it's just gonna keep going in that direction.

So therefore it's gonna be super, super smart soon.

Um and they're kind of like, in my opinion, like detached from reality a bit.

And then there's another group that just kind of dismisses the whole thing.

Um, and I think both views are a mistake.

Um because I think like AI technology is like demonstrably it's huge.

Like it's happening now.

Like you don't have to guess about the future, like just look at the present and it's having huge impacts and it's extraordinarily useful.

Um and I think it's in a sense, it's like underhyped.

Um like I think it it's like it's it's not really right to say it's underhyped, I guess, but it's like people are taking these two fairly extreme views where they're either completely dismissing it or just like imagining based on very little information that it's gonna go crazy.

But there's like this like third path that says like, okay, let's take a sober look at what's going on and think about what might be possible based on that.

And I think that future is is really interesting.

Yeah, and well, absolutely, and uh, you know, I'll I'll just kind of say that like back in the 70s, um there was a paper that came out that that had basically said neural networks can't solve because back then there was only one layer of neurons that they were using.

And they said neural networks cannot solve uh the XOR problem.

And this paper got published and it was and it was provably true, and all the funding went away.

There was no funding for AI anymore.

And so um the next uh 30 years or so was like completely absent of research in this area, and no one was interested in it.

And so uh then um you know, I think I think then the multi-layer perception was was created, which is this you know, multiple layers of neurons.

And they showed, hey, we can do that, but still uh the momentum was like slowed down so considerably that um you know the investing was still very low.

Uh and then you know, I I think that there was a lot of stuff from like Google releasing TensorFlow and and other things to to like further democratize machine learning.

Um, and then things started to accelerate.

And but but even then it was still uh quite slow, uh, but the the research funding had increased considerably.

And and now we're we're kind of in this weird opposite space where you know there was a lot of funding in before in the 70s, then there was like no funding for 30 years, then there was like funding for the broad machine learning space, and now it's like funding to look at transformers doing LLMs with rags on top of them.

And and like we're so hyper focused as a uh as uh globally on these models now that they're going to be refined to a point where like we should be putting a lot of this investment in these other uh disciplines inside, you know, subdisciplines within machine learning to move the whole system forward.

And so um I think that yeah, so so investment in, you know, there's just further investment down this one path of research.

And while it will yield a lot of great things that we've been seeing in the last couple of years, um, we, you know, there are still other areas that need to be explored.

And I think it's going to be the combination of all of these AI systems um under the umbrella of AI that that will move us forward.

But while while the hype train is running on these large language models, a lot of people are just happy to get funding at all to do the machine learning research.

Um, because that has been exactly the opposite of like the prior 30, 40 years.

Um, so yeah, it's it's it's quite interesting in that.

And then and then I think if anybody's watched the AlphaGo documentary, I think, especially in in China and other areas, there, you know, there is a considerable amount of investment in research in AI because uh it it affected like the the cultural identity um you know uh in there.

And so uh there is this global sort of arms race almost with with AI that is going to produce a lot of great things um you know as the research evolves.

What is this Alpha Go documentary?

Uh so I think it's on Netflix, and um they talk about uh specifically when um the best uh or some of the best Go players in the world were struggling against like Google DeepMind's uh Go uh um computer.

And so um before that, machine learning had never really been able to defeat people uh well in the top rankings in chess and in Go, right?

Um some of the iconic worldwide games out there.

And uh this this computer was was, you know, the the deep mind was very good at Go, um, to the point where I think culturally um China had uh started investing considerably in machine learning a lot earlier than the US did.

Um and uh and it was it was primarily based off of this uh off of the matches that were played um in that.

So documentary is very well done.

Uh uh, you know, maybe there's obviously West Western bias in the documentary, um, but uh but it it kind of gives you a good insight to I think how we ended up where we are today, where um there's this like you know, there are these huge developments coming out of China, but but also um in China there there have been uh there's been a considerable investment um uh or many more years than the US has been investing because because really Chat GPT is what broke open the floodgates for the US.

Interesting.

I had no idea.

I had no idea that China had been investing in AI for so much longer than we had.

That's really interesting.

Um I would like to share what I think the path that I think AI should take, which you know, this is this is my woefully underinformed uh take on it.

Um but I read this book called A Thousand Brains, and it's by this guy, I forget his name, um Jeff Hawkins, actually, I think his name is.

Um and he is a researcher, um he's a neuroscientist, I believe.

He's a he's a brain researcher.

Um and he it's it's been his desire for the last uh few decades to figure out how the human brain works.

And he's actually the guy who invented the palm pilot, um, which I thought was pretty interesting.

It he kind of used that as a means to an end.

Um he invented that and sold the company for a bunch of money, and now he's spending the the balance of his life doing science, so kind of like a modern Benjamin Franklin kind of lifestyle.

Um because Ben Franklin did the same thing, he made his money in printing and then sp spent the rest of his life as a scientist and um statesman.

Anyway, what that book said was that the brain works by creating models of of the objects that we interact with.

Um if we have it it uses the example of a coffee mug a lot.

So it's like if you have a coffee mug, um your brain kind of overlays this uh reference frame over the coffee mug, where it's like this three-dimensional grid that moves with the orientation of the coffee mug and stuff like that.

Um and and so you have all these bazillions of models in your mind.

And to me that that idea seems just like so self-evidently correct.

Um and it's much different from um from this like statistical method of like, okay, if if if these prominent algorithms uh look at an image, for example, um, it can tell it's a coffee cup because it resembles all the images it's seen in the past of a coffee cup.

That seems much different for uh to me from having a model of a coffee cup in your mind or in the computer's mind or whatever, and recognizing it based on that.

Um my idea, which I'm kind of working on, um, I have this Lego robot here in my hand.

Um so far, all it does is uh it can it can grab stuff and pick it up and set it back down.

Um but the idea is that it'll have like uh senses, starting probably with binocular vision.

So you pick up an object and you kind of interrogate this object, uh turn it over in your hands, and you're like, what is this thing?

Um and you store some representation of all these various objects in your mind.

Um and I'm speaking again, like as the computer.

Um, and then you know, as I, the computer, develop these models of various things in the world, then maybe I can connect them uh to concepts and words and stuff like that, and be like, this is a cup, this is a squirrel, whatever.

Um, and that's about as far as I've gotten with my thinking.

Um because I I suspect that it's the the kind of thing where like you have to get into it and like actually work on it before you can really have some smart ideas.

So I'm planning, you know, I'm I'm working on my little robot and I'm gonna see how far I can get.

Anyway, any any reactions to my thought of like how we get there?

Yeah.

Well, go ahead, Austin.

Just to jump in here a little bit, uh, with completely different angle, uh, but just to hit on two things he said.

Uh so my uh master's level training is not in machine learning, but actually in philosophy and things like that, like uh theology and science, things like that.

And what's interesting to me about your approach, Jason, is I don't know where to go with this from a you know implementation standpoint, is where you're starting with senses and sensory and a robot.

Um if you go back to like Aristotle and Plato, what Aristotle did uh with uh Aquinas kind of jumping in along with him uh hundreds of years later, is that good thought starts with the senses, not that empiricism in the sense that only things from the five senses are true, but that good thought starts from the senses, and then from there you can build uh like the metaphysics to uh have reasoning and things like that.

Um so I have no idea how that works out in the in a robot implementation.

Uh and the other thing that jived with me with what you said is uh two, I've noticed the same thing that I think more often than it should be, people are too polarized.

Like AI is amazing, it's gonna conquer everything, you're not gonna have a job in two months, or it's terrible and you should never use it.

When in my experience, I don't really care if it understands what it's doing, but it saves me four hours of Googling.

You know.

Um but I'll I'll kick it to Chris to answer your original question with uh your your Lego bot starting point.

Yeah.

Um so yeah, I I do like the philosophy question because uh I think what comes to my mind is that in the brain there are models even for motivation, will, you know, other things like that.

So um a lot of these things are connecting and interacting neurons, and they, you know, uh electrical pulse will fire through the neurons.

Um, and then there are individual weights at which a electrical threshold will um you know will activate or not.

Uh and neural networks uh do the same thing.

So they have a um essentially a multi-dimensional array that contains a bunch of numbers that are weights, and then there's a dot product that happens for an input matrix that then creates some sort of activation and it simulates an axion firing, right?

Um anion to axion.

Um and so but I I think in in computation, um right now the neural networks are uh supervised learning models, meaning that you have to have a lot of data and and you have to have it beforehand, because you know, um if our neurons are currently the sum of all of our experiences throughout our entire lives, uh then our our training uh of the system has happened um you know uh throughout our whole lives, which is the whole like baby growing to um you know adult idea.

Um and uh and so when you when you take that approach of like interacting with the world, you either need unsupervised learning with some goal related, you know, and that that might be K nearest neighbors and other things like that.

Um or uh you need at some point to retrain and and recreate the current state of the brain given all of all of the observations made.

Um so uh you know, I I think my recommendation for your project would be to um to also look at the initial training and creation of the models to then um to then know where you're going to go next.

And and so like uh SK Learn is is not as hyper-focused uh as like something like Llama, right, from Facebook.

And so SK Learn is open source machine learning framework that has all kinds, you know, the whole I'd say it's it's a wide breadth of machine learning techniques um rather than a very focused, like depth uh machine learning model.

Um and so that can give you some ideas.

Um, and and there's a lot like when we're five years old, our brain stops creating new neural constructs and then it starts pruning instead.

And so um, you know, kids are like super crazy up until like five years old, and then they start to like become much more like focused and other things like that.

And that's because there are portions of the neural network that are just being chopped off.

Like there, you know, it's it's basically a um the that decision path is like no longer interesting to the human.

And so it like starts saying, okay, we're not gonna do that anymore, we're not gonna do that.

And so people's personalities become much more focused after five years old.

Um, and so and and before then, then things are just like growing, like more models, more models, more models, more models.

Um, and so uh there's like this really big inflection point there.

So uh, but I guess all of that to say that a lot of the advancements in machine learning have been not because some computer scientist was like, these statistics will be great, but especially in the neural networks space, it's really been some uh you know, some neurologist has made a big discovery about how brains work because they've been looking at you know CT scans and ultrasounds and other things like that.

And then they're finding this is how the brain works, it does back propagation.

And so it's gonna go you know, riv revisit old memories to then further learn and things like that.

And so your brain is doing all this stuff that's hidden in the background, and all the computer scientists have been trying to do is simulate that.

Um, and and this is where all this like hype is really funny to me, is because uh we're So we're we're hyper focused on this area of machine learning that is based off of neurology.

And so I think the number of people trying to advance this um this particular discipline uh is is not necessarily any bigger than it was.

Um but the but people trying to use these models in different ways has grown considerably.

Um and so uh I think you know if you want to pursue finding out how the the the model of the human mind works, then you or or even like animal minds, right?

Because there's a lot more, a lot of research around just animals and how they develop.

Um, you know, I think there are a lot of projects out there where um there's either electronics uh interacting directly with cockroach brains to move and control and things like that, um, and you know, all the way to uh kind of the idea that you have here where you experience the world in new ways and then you incorporate that and then do new things.

But but then the question of philosophy and motivation come into play because like what will your robot want to do with that information and and how does it um you know forward?

And so and then it becomes like large simulations, right?

So if you have multi-agent simulations, you could have you could basically say, you know, these things are gonna interact with each other and they're gonna both discover you know a tree.

Like, what does a tree mean to them?

And why, you know, and and so you get into this very like high-level philosophy as you as you go deeper.

Um, and so it depends on how close to practical application you want to go.

Yeah.

Um by the way, you guys have no idea what a treat it is for me to be talking with you guys um with with your background, Chris, in AI and your background, Austin, in philosophy.

I had no idea about that until just now.

Um, because I think all this is very inseparable from philosophy.

Um there's something super, super important to my whole like conception of all this that I haven't even mentioned yet.

Um there's I I've been reading and rereading these couple books lately.

Um, The Fabric of Reality by David Deutsch and The Beginning of Infinity by the same author.

Um maybe I maybe I even mentioned David Deutsch last time we talked because it's been like all I've been talking about lately.

Um by lately I mean for like the last two years or something like that.

Um but in his books, he talks about the difference between um inference and explanations.

Um sorry, inference and induction, that kind of idea.

Um so there's that famous problem in philosophy that uh David Hume's problem of induction.

Um and I I am not familiar enough with David Hume's problem of induction to give a faithful retelling of it.

Austin, are you familiar with this?

Yes, I would have to refresh a little bit uh to wax eloquent on it, but yes, I am familiar, uh roughly.

Yeah.

Yeah, so the idea as I understand it is like if you use logical deduction, um, which is going from um uh general uh known truths to um specific uh I don't know how to say it in a way that's not circular, specific deductions based on those general truths.

Um you can be sure that your answers are right.

You know, like the um oh let's see what's an example.

Um can we think of an example of of a of logical deduction?

Uh yeah, like the classic one Socratically for deduction is like all men are mortal, Socrates is a man, therefore he is mortal, uh kind of thing.

And then I think either Hume or someone inspired by Hume induction was kind of like the sun will rise tomorrow because it's risen in the past, like you know, previous events kind of thing.

Yeah, yeah, exactly.

He's saying that it will not based on induction or something like that.

He's more skeptical of lots of things, including that.

Yeah, yeah, and that's the exact example I've used before.

Um, because that's not a valid conclusion, you know.

Um the sun, it it won't always be true that the sun will rise tomorrow.

Um, there will come a day where that won't be true.

Um even if it is true that the sun's gonna rise tomorrow, you know, like we have every reason to believe that the sun will rise tomorrow, uh April 24th, 2025.

Um, but that's not because it has risen so many times in the past.

Um so induction is not a valid mode of reasoning.

Um, the reason that we know the sun will rise tomorrow is because we have an explanation of how the solar system works.

Um and so right now it seems to me that like LLMs, for example, um they're kind of using induction.

Um and and you can see evidence of that in the way that it um develops its opinions.

It's kind of like whatever the prevailing view is, is what it believes is true because it's encountered that more times.

What's that?

It's the most average viewpoint in the world, yeah.

Yeah, and there's so many cases where the the minority view is actually the correct one, um, and and the majority is mistaken.

Um, and that deeply, deeply bothers me.

Um, and so I think we need to have uh an AI that goes off of explanations rather than uh induction, and that um very much to me ties into this idea of models because to me a model it is I wouldn't say the model a model is the exact same concept as an explanation, but there's a lot of overlap.

You know, we have an explanation of how the solar system works, we have a model of how the solar system works, like there's a lot of overlap between those two things.

Um and so if we build an AI that can model a whole bunch of uh objects and phenomena and abstract concepts and stuff like that, um, those models can serve as explanations, and so that kind of AI could determine the truth based on um based on conjecture and criticism and arriving at explanations that way rather than relying on induction.

Yeah, yeah.

And to jump in real quick, I think I'll just I'm not a brain neurologist or anything like that, uh, but I guess I can my studies have more been the mind or psychology or you know, philosophy of mind and that kind of thing.

And I guess what I would just say is it seems properly basic or uh self-evident that uh our minds use induction and deduction and in different times and different places and they check each other, right?

So um, you know, maybe the prevailing view is A, but evidence and reasoning de deductively shows that that's false.

So that can't be true, right?

You know, so you use both in different cases.

And I think it's funny the the history of philosophy or the history of thought, how often it kind of circles back on itself, because Kant and Hume were skeptical of everything, which ends up turns out to be pretty self-refuting.

And so a lot of modern uh neurology and a lot of modern philosophy is founded on basically that you know nothing and that everything's just a perception.

Uh, but then that statement, you know, and then you run into all kinds of self-contradictions.

And so I don't know if I'm agreeing with you 100%, but I do like them the kind of classical move that like if we're gonna say anything and not just be silent and it's gonna be true, you have to start uh knowing things directly.

And I think that's kind of what Aristotle and Aquinas did, is I don't inductively see a hundred people and then derive the essence of a person.

It's just like I immediately grasp what you know man or personhood is, right?

And then I see another one, and then I know that it means that they're gonna die, right?

Or uh it's and so it's like this interplay of both worlds, but you need to start with some self-evident uh foundations, or else if you just start with induction, you'll never get anywhere.

And so um you have some self-evident foundations on which you can build useful cases of induction, uh, but it's not everything.

Yeah, if I understand part of what you said there, um you know, like a an image recognition algorithm would need to see bazillions of examples of something before it could recognize another one.

Um, but I only need to see one coffee cup before I can recognize another coffee cup, assuming it's similar enough.

Um but I only I only need one.

Um and I think that is a huge, huge difference.

Um and and if we are able to create an AI that only needs like one example of something, uh then that's like a whole different thing.

Well, so again, it's it's dangerous to anthropomorphize these large language models.

So like they are not like they don't have opinions, they don't, you know, it's so all it is is a matrix of of weights, right?

Like at its most at its most basic, and those weights get fine-tuned by uh reviewing um positive and negative cases and then and then reasserting um the weights.

And then and then what happens is you just you you end up with the most likely set of series of tokens that to appear.

Um so so again, at its basis, it is um it is a a model, and and you can think of it more along the lines of um, you know, like if you if you think about like systems one and system two thinking, or um just this idea that uh we have um we have these like conclusions that we jump to, and it's because our our brains are very good at being efficient with information, and so that's how our biases like form, is because it's we're looking to process the information as quickly as possible.

Um and in the coffee mug example, you have you have a lot of information about physical objects in the world, and so when you see a coffee mug, it's not that you're seeing only one example of it, you're you're seeing a uh additional uh specif specific example of uh what would be abstracted as an object, right?

And so there's a lot of inference that you can do there, and your brain has other models that are are also doing those processes in the background.

Wait, sorry, can you can you go deeper into that?

I want to make sure I understand exactly what you mean.

Um, like an abstracted version of a specific object or something like that.

Right.

So so your brain learns uh you know pretty early on, uh you know, when you're talking about financial uh fundamental like um truths that you have even as a baby.

So like when you're learning, uh when you know you're firstborn, you're just like developing your senses to the point where um you know, I can I see it all, can I make out objects?

Can I can I further refine that?

So your brain is refining the process for these raw inputs, which are um you know essentially nonsense at first.

And so um, you know, a baby's firstborn and it's just like oh senses, right?

But they have no meaning, right?

Um, and then gradually over time, uh those are refined to to then uh crew you know categorize, right?

And once those categories are uh categories are correct uh created, then um that's kind of like your first abstraction of like what an object is.

It's like, oh, I see something on the table and it's there, and because of gravity and physics and stuff, it's the same every time.

So now I've created a pattern.

That pattern then further uh reinforces as you see more and more examples.

And so um you you've you've actually seen uncountable examples of objects in the world.

Um, and and you've seen it from every angle.

And and by the time you first see a coffee mug, you you have this abstraction of this idea of what it means to have objects.

And yeah, if if you came out of the womb and the first thing the doctor handed you was a coffee cup, it'd be a much different story than you encountered at six.

It's not one shot, right?

Yeah.

Um, but and and in lot in large language models, they talk about few shot encoding, which is this idea of like, give it three examples and it'll create a fourth, right?

Because it's like the most likely thing to come afterwards, and the large language model can process that.

But it's inferring off of every piece of literature that it's seen in its in its history, and saying, because there's a pattern preceding this literature, I'm going to create something new that follows that continued pattern.

Um, and so so a lot of these these models, when you do one-shot encoding or few shot encoding, um you you end up with uh inferences created by the entirety of the model's existing um uh knowledge, right?

Um I wanna I wanna make something clear that I don't think is news to either of you guys, but I just want to, for the listener's sake, uh uh lay it out there.

Um you know, if you think of everything a human learns from um birth onward um as software, and if you think of our bodies and brains and and stuff like that, if you think of all that as hardware, um that's not the whole picture because we also have firmware, you know.

Um it's it's crazy like how much this is denied or or people just aren't aware of it.

Um but we're not born as blank slates, we're born, we're born knowing stuff.

Right.

Like for example, a a human baby is born knowing how to breastfeed.

Um we're born with knowledge hard-coded.

And it's even more evident in certain animals, you know, like a deer is born and it can just immediately start walking.

Um, like it has to know how to walk.

Um and so it must be born with that knowledge already in its firmware because it didn't have time to learn how to walk, it had to just be born knowing how to walk.

Um, and so I don't know exactly how this fits into the whole picture, but oh, oh, oh, there's yeah, yeah, yeah.

All right.

So there's this useful analog, I think.

Um an AI can learn stuff during its metaphorical lifetime.

And I think the um the code I'm going a little bit on a limb, so take this with a grain of salt.

I think like the knowledge that the programmer or the designer of the system puts into the AI, that is like the the instincts and such.

That's the firmware that we're born with.

The stuff that the um it it's like okay, quick quick tangent, hopefully quick.

Um how how did our brains come to be through the process of evolution?

Um and uh DNA is encoded knowledge.

Um so it's like these organisms have interacted with their environments and uh the process of evolution has learned things.

Um and those learnings have been saved, they've been encoded in our DNA.

Um and so our our DNA is obviously like uh the instructions for building an organism.

Um and so I don't think I can I can do a great job of articulating what I'm trying to say, because maybe I don't know exactly what I'm trying to say, but there's some analogy between um okay, yeah, this is what I'm trying to say.

Like the the designer of the system is kind of playing the role of the process of evolution and and creating that artifact, the the system that then goes out and learns.

It's really funny that you say that because uh so you might want to read my master's thesis.

It's like 50 pages of stuff, but um it in the so my my whole master's thesis was uh improving a computer vision model using evolutionary algorithms.

And so it creates a genome that evolves and then creates uh um you know thousands of shallow models and then scores them and then creates multiple generations and it evolves over time.

And you can see the process of evolution in uh and so rather than having the programmer improving that process, uh I I use the process of genetic evolution, um also using some uh algorithms to mimic the swarming of birds looking for food to avoid uh local optima in that process.

But um kind of exact that is what I did back in 2014, is that exact process that you described, where it's evolution guiding the improvement of those models, which is kind of what I was getting at before.

So it's so expensive to make these large language models that they have humans configuring them, like you say, and acting as the role of evolution.

Um but uh but for shallow models, the computational complexity is much smaller than these deep models.

And so when you it's actually computationally feasible to enact evolution on shallow models.

Um so maybe more primordial, I guess.

So yeah, yeah, it's interesting.

I've done some genetic programming myself because I've tried to create um virtual life forms.

Um, I guess I shouldn't just say tried.

I've I've been successful with it to a very rudimentary, rudimentary degree, but then I realized at some point, like um this virtual environment, I create there are no built-in laws of physics, there's no gravity, electromagnetism, light, all that stuff.

And so you have to like be God to some extent and and create the laws of physics, and it's a real struggle to desi decide like how much should I play the role of God in this situation and and arbitrarily build in um laws of physics and stuff like that, and when should I stop?

But anyway, my my hunch based on not much, but my hunch is when we finally crack the code to AGI, it'll be something uh relatively simple.

Um like I I wouldn't be surprised at all if I'm totally wrong about that, but I also wouldn't be surprised if we if we figure it out, and it turns out that um from that point on we don't need to like spend millions of dollars training models and stuff like that, because we can just uh create a simple system and let it do its own thing and and it'll learn on its own.

Yeah, and well, and that's a little bit of where um and and that's kind of back to my initial thoughts on where the investment is right now.

So so like the the investment is not in that, and so the number of people I think exploring those concepts today is the same as it was a year, you know, a couple years ago before large language models, maybe a little bit more because the interest in the the field is a little higher.

Um but uh yeah, I I think it will take some level of recognition of that idea to to um to activate where the funding's coming from.

And so um, you know, I I it and like I said, for um for China it was like AlphaGo.

And so it's actually a different model, and not necessarily these transformer models.

Um and and then and then Chat GPT became um you know a big thing, and that's what created the interest.

So so I I think a lot of that stuff gets accelerated when when the general masses are excited about it.

Um and so, you know, I think there will be more moments like that in the in the future, but it's very hard to predict you know, I I could not have uh predicted that you know the adoption of large language models would have gone business.

I I would have never expected that because it was going so slowly before then.

Um because I was uh you know, with even within our own own organization, I was trying to promote um using machine learning of different types.

It was very challenging to get traction.

Yeah, yeah, yeah, yeah.

And to jump in from the the third lens here, the philosophical lens, uh, just to throw in a few more points here before we're out of time is to take your uh DNA analogy and then add another one and then come back to it.

It may I wonder if it will turn out to be like that, right?

Where once they find out the initial conditions, uh what's the acronym you guys keep using?

A something?

Oh, artificial general intelligence.

Oh, okay, yeah.

That I wonder if it'll be kind of like uh the fine-tuning uh conditions of the universe.

So like uh I don't know if you've heard of uh the anthropic principle or things like that.

But basically, the laws of physics, and then there's like they keep finding more of them.

There's hundreds of these initial conditions.

And if you let them, you know, like the laws of physics and all that kind of stuff, but even even within the laws, there are constants and things that are initial conditions that are just kind of built into the way the universe is.

Um so but if you tinker with those numbers, um, because you know, uh there is like an infinite, like 99.9% amount of universes that could have been.

Um, and none of those are life permitting.

It's only a very thin, small number of initial condition numbers that allow a universe where life can exist, where the stars aren't too close to planets or they're not too big, or we don't expand too fast or expand too slow.

So that's like the uh yeah, one of the constants is uh like expansion rate or that that kind of thing.

And I think it's the same thing with uh cells, like the you know, even following that theory, the initial cell already has built into it a huge amount of DNA and instructions and initial conditions, right?

And so um yeah, so yeah, it will be fascinating if it turns out that uh AI is that way.

Once they find the right mix of initial conditions, maybe it's easy at that point and it doesn't take a million quantum computers, but uh I have no idea how that's something that concerns me a little bit.

You know, like uh you you could say that reality is a computer in the sense that like um uh um how how do you put it?

Um like if you wanted to simulate with perfect fidelity the workings of a single cell, um, like modeling all the an atoms and stuff like that in a computer, that's like insane.

Like you'd need so much computational power.

Um, but reality has that much computational power.

Um and so sometimes I'm concerned like okay, we can never have like enough computational power to model reality right down to to the tiniest uh parts of it.

Um so is that is that gonna be like an insurmountable limitation?

Um but I don't know that that's not really a I haven't reached that bottleneck yet.

And eventually, you know, when you you can cross that bridge when you get there, right?

Right, right.

Every model, you know, so what's the quote?

Every model is is uh wrong, but some are useful, right?

Um and so you know, I I do think there's a lot of learning that a system can do in a simulated reality to then, you know, and and that's like very similar.

Like you can learn from your own imagination, right, as a person, right?

You can imagine a number of scenarios.

This is uh like especially evident when people are about to have a very difficult conversation, they might run through that conversation and how it might go in hundreds of different ways.

And and in preparing for that, you might then when you you know it might be useful to you in the sense that you might anticipate some things happening, but something will always be different when you actually get into that conversation.

Um, I I think that's that's true for uh your project there too.

You can have a simulation of what that uh what your Lego robot might experience, um, and that could give it an initial basis basic understanding of the worlds, and you can do that you know uh at super speed compared to having it interact manually with the worlds initially, right?

Yeah.

Um okay, well, it's probably time for us to start wrapping up.

Um, but this has been really incredible for me.

Um I I never I didn't start this conversation realizing that that we had an AI guy and a philosophy guy.

Um, so again, that's that's quite a treat for me.

Um anything that you guys want to share, anything for links and stuff like that before we go?

I mean, just the podcast, mom mentality show, you know, check it out.

Um I don't know that we get into these topics so much.

Every once in a while we do, but uh you know, maybe that's good feedback.

It's kind of interesting.

I've I've enjoyed this very much.

So yeah.

Yeah.

Yeah, I would say uh mommentality.com for tech stuff and then uh philosophy stuff.

You can go to my LinkedIn.

Mommentality Show.com.

Mommentality Show.com.

Thank you, Chris.

Thank you.

All right, we'll put that stuff in the show notes and thanks so much for coming on the show.

Thanks, Jason.

Good time.

Thanks, Jason.

Yeah.