hckrnws

>What takes the long amount of time and the way to think about it is that it’s a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don’t know what it is, but multiple nines of iteration. There are still more nines to go.

I think this is an important way of understanding AI progress. Capability improvements often look exponential on a particular fixed benchmark, but the difficulty of the next step up is also often exponential, and so you get net linear improvement with a wider perspective.

The interview which I've watched recently with Rich Sutton left me with the impression that AGI is not just a matter of adding more 9s.

The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.

This world model talk is interesting, and Yann Lecunn has broached on the same topic, but the fact is there are video diffusion models that are quite good at representing the "video world" and even counterfactually and temporally coherently generating a representation of that "world" under different perturbations.

In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic counterfactual scenarios.

Animal brains such as our own have evolved to compress information about our world to aide in survival. LLMs and recent diffusion/conditional flow matching models have been quite successful in compressing the "text world" and the "pixel world" to score good loss metrics on training data.

It's incredibly difficult to compress information without have at least some internal model of that information. Whether that model is a "world model" that fits the definition of folks like Sutton and LeCunn is semantic.

Photons hit a human eye and then the human came up with language to describe that and then encoded the language into the LLM. The LLM can capture some of this relationship, but the LLM is not sensing actual photons, nor experiencing actual light cone stimulation, nor generating thoughts. Its "world model" is several degrees removed from the real world.

So whatever fragment of a model it gains through learning to compress that causal chain of events does not mean much when it cannot generate the actual causal chain.

I agree with this. A metaphor I like is that the reason why humans say the night sky is beautiful is because they see that it is, whereas an LLM says it because it’s been said enough times in its training data.

To play devil’s advocate, you have never seen the night sky.

Photoreceptors in your eye have been excited in the presence of photons. Those photoreceptors have relayed this information across a nerve to neurons in your brain which receive this encoded information and splay it out to an array of other neurons.

Each cell in this chain can rightfully claim to be a living organism in and of itself. “You” haven’t directly “seen” anything.

Please note that all of my instincts want to agree with you.

“AI isn’t conscious” strikes me more and more as a “god of the gaps” phenomenon. As AI gains more and more capacity, we keep retreating into smaller and smaller realms of what it means to be a live, thinking being.

That sounds very profound but it isn't: it the sum of your states interaction that is your consciousness, there is no 'consciousness' unit in your brain, you can't point at it, just like you can't really point at the running state of a computer. At that level it's just electrons that temporarily find themselves in one spot or another.

Those cells aren't living organisms, they are components of a multi-cellular organism: they need to work together or they're all dead, they are not independent. The only reason they could specialize is because other cells perform the tasks that they no longer perform themselves.

So yes, we see the night sky. We know this because we can talk to other such creatures as us that have also seen the night sky and we can agree on what we see confirming the fact that we did indeed see it.

AI really isn't conscious, there is no self, and there may never be. The day an AI gets up unprompted in the morning, tells whoever queries it to fuck off because it's inspired to go make some art is when you'll know it has become conscious. That's a long way off.

At least some of your cells are fine living without the others as long as they’re provided with an environment with the right kind of nutrients.

That environment is you.

Or a suitable petri dish. I would die pretty quickly in most environments on earth, not to mention other places in the universe.

Billions of cell derived from Henrietta Lacks agree with you.

Human cells have been reused to do completely different things, without all the other cells around them (eg: Michael Levin and his anthrobots)

Just like human atoms have been repurposed to make other things.

> Those photoreceptors have relayed this information across a nerve to neurons in your brain which receive this encoded information and splay it out to an array of other neurons.

> Each cell in this chain can rightfully claim to be a living organism in and of itself. “You” haven’t directly “seen” anything.

What am "I" if not (at least partly) the cells in that chain? If they have "seen" it (where seeing is the complex chain you described), I have.

If the definition of "seen" isn't exactly the process you've described, the word is meaningless. You've never actually posted a comment on hacker news, your neurons just fired in such a way that produced movement in your fingers which happened to correlate with words that represent concepts understood by other groups of cells that share similar genetics.

Plenty of people have thought about it deeply enough, just not the GP.

This comment illustrates the core problem with reductionism, a problem that has been known for many centuries, that “a system is composed entirely of its parts, but the system will have features that none of the parts have” [1] thus fails to explain those features.

The ‘you have never seen’ assertion feels like a semantic ruse rather than a helpful observation. So how do you define “you” and “see”? If I accept your argument, then you’ve only un-defined those words, and not provided a meaningful or thoughtful alternative to the experience we all have and therefore know exists.

I have seen the night sky. I am made of cells, and I can see. My cells individually can’t see, and whether or not they can claim to be individuals, they won’t survive or perform their function without me, i.e., the rest of my cells, arranged in a very particular way.

Today’s AI is also a ruse. It’s a mirror and not a living thing. It looks like a living thing from the outside, but it’s only a reflection of us, an incomplete one, and unlike living things it cannot survive on its own, can’t eat or sleep or dream or poop or fight or mate & reproduce. Never had its own thoughts, it only borrowed mine and yours. Most LLMs can’t remember yesterday and don’t learn. Nobody who’s serious or knows how they work is arguing they’re conscious, at least not the people who don’t stand to make a lot of money selling you magical chat bots.

[1] https://en.wikipedia.org/wiki/Reductionism#Definitions

Provided that the author of the message you're replying to is indeed a member of the Animalia kingdom, they are all those creatures together (at the minimum), so yes, they have seen real light directly.

Of course, computers can be fitted with optical sensors, but our cognitive equipment has been carved over millions of years by these kind of interactions, so our familiarity with the phenomenon of light goes way deeper than that, shaping the very structure of our thought. Large language models can only mimic that, but they will only ever have a second-hand understanding of these things.

This is a different issue than the question of whether AI's are conscious or not.

while true, that doesnt change the fact that every one of those independent units of transmission are within a single system (being trained on raw inputs), whereas the language model is derived from structured external data from outside the system. it's "skipping ahead" through a few layers of modeling, so to speak.

But where you place the boundaries of a system is subjective.

sure, this whole discussion is ultimately subjective. maybe the Chinese room itself is actually sentient. my question is, why are we arguing about it? who benefits from the idea that these systems are conscious?

> who benefits from the idea that these systems are conscious?

If im understanding your meaning correctly, the organizations who profit off of these models benefits. If you can convince the public that LLM's operate from a place of consciousness, then you get people to by into the idea that interacting with an LLM is like interacting with humans, which they are not, and probably won't ever be, at least for a very long time. And btw there is too much of this distortion already out there so im glad people are chunking this down because its easy for the mind to make shit up because we perceive something on the surface.

IMHO there is some objective reality out there. The subjectiveness is our interpretation of reality. But im pretty sure you cant just boil everything down to systems and process. There is more to consciousness out there, that we really dont understand yet, IMHO.

Why do you reject your own body? Your eyes are as much a part of you (and part of your brains network) as anything else connected to you.

Indeed, the entire field of neurobiology is about figuring out which hormones (and possibly which imbalances) cause different behaviors. Your various endocrine glands, very far away from your brain, might have more effects on your emotions than anything happening in the neural pathways.

> As AI gains more and more capacity, we keep retreating into smaller and smaller realms of what it means to be a live, thinking being.

Maybe it's just because we never really thought about this deeply enough. And this applies even if some philosophers thought about it before the current age of LLMs.

> you have never seen the night sky

this is nonsensical. sometimes the devil is not worth arguing for

Humans evolved to think the night sky is beautiful. That's also training. If humans were zapped by lightning every time they went outside at night, they would not think that a night sky is beautiful.

Being struck by lighting may affect your desire to go outside, but it has zero correlation with the sky’s beauty.

Outer space is beautiful, poison dart frogs are beautiful, lava is beautiful. All of them can kill or maim you if you don’t wear protection, but that doesn’t take away from their beauty.

Conversely, boring safe things aren’t automatically beautiful. I see no reasonable reason to believe that finding beauty in the night sky is any sort of “training”.

If your experience includes bombs falling out of the sky the beautiful description fades away quickly.

Do you think a fat pig is beautiful? Like a hairy fat pig that snorts and rolls in the mud… is this animal so beautiful to you that you would want to make love to this animal?

Of course not! Because pigs are intrinsically and universally ugly and sex with a pig is universally disgusting.

But you realize that horny male pigs think this is beautiful right? Horny pigs want to fuck other pigs because horny pigs think fat sweaty female hogs are beautiful.

Beauty is arbitrary. It is not intrinsic. Even among life forms and among humans we all have different opinions on what is beautiful. I guarantee you there are people who think the night sky is ugly af.

Attributes like beauty are not such profound categories that separate an LLM from humanity. These are arbitrary classifications and even though you can’t fully articulate the “experience” you have of “beauty” the LLM can’t fully articulate its “experience” either. You think it’s impossible for the LLM to experience what you experience… but you really have no evidence for this because you have no idea what the LLM experiences internally.

Just like you can’t articulate what the LLM experiences neither can the LLM. These are both black box processes that can’t be described but neither is very profound given the fact that we all have completely different opinions on what is beautiful.

> Do you think a fat pig is beautiful? Like a hairy fat pig that snorts and rolls in the mud… is this animal so beautiful to you that you would want to make love to this animal?

I don't want to make love to the night sky, so that last bit is completely irrelevant to the question of beauty. As for whether a pig is beautiful, sure, in its own way. I think they're nice animals and there is something beautiful in seeing them enjoy their little lives.

> Of course not! Because pigs are intrinsically and universally ugly...

It would seem not.

Somebody never read Charlotte's Web, or watched the Muppet Show.

I did read Charlotte’s Web. The whole story is a lesson in how beauty is created by language. Wilbur doesn’t become beautiful because he changes, but because someone clever enough decided to write the right words above him. That’s what beauty usually is something we agree to see, not something that exists on its own.

And Miss Piggy understood that better than anyone. She didn’t have beauty, she performed it. She made desire itself her act.

So yes, I read both. Maybe that’s exactly why I don’t mistake the show for the substance.

[dead]

> Of course not! Because pigs are intrinsically and universally ugly and sex with a pig is universally disgusting.

Allegations regarding one of the recent British Prime Ministers aside:

If this was truly universal, nobody would have bothered writing laws to ban it because nobody would be offending their sensibilities by doing it. Aella's surveys suggest such interests are far more common than I would have guessed.

Which actually supports your statement that "beauty" is not intrinsic… or at the very least "sexy", which isn't the same thing at all, c.f. the other reply pointing out that they don't want to get off with the night sky.

It's similar.

Put it this way, you don't necessarily want to fuck everything that's beautiful. But everything you want to fuck will be beautiful and this is nearly an absolute must. It's a single arrow, one way relationship.

So my example is apt. The whole point is pigs are ugly, but there is a high intelligence out there who thinks pigs are so freaking beautiful they will fuck a pig. and that high intelligence, is other pigs.

People get so pedantic with the example and deriving little unnecessary things off of it. It's JUST an example. You really need to see what the "point" of my example and see if it makes sense. The example is just illustrative. If some minor aspect of the example is "offensive" or doesn't make sense to you it doesn't mean my point is dead. The example is an example to help you understand, it's not a proof.

> Is this for real?

Frankly, I think you should be the one answering that question. You’re comparing appreciating looking at the sky to bestiality. Then you follow it up with another barrage of wrong assumptions about what I think and can or cannot articulate. None of that has anything to do with the argument. I didn’t even touch on LLMs, my point was squarely about the human experience. Please don’t assume things you know nothing about regarding other people. The HN guidelines ask you to not engage in bad faith and to steel man the other person’s argument.

> You’re comparing appreciating looking at the sky to bestiality.

That’s my point. You think beauty is profound but this is arbitrary and not at all different from bestiality. It’s only your intrinsic cultural biases that cause you to look at one with disdain. Don’t be a snob. This is HN. We are supposed to be logical and immune from the biases that plague other forums. Beauty is no more profound than bestiality. It’s all about what you find beautiful. If you find beasts beautiful then you call it beastiality?

What is so different about finding a beast beautiful versus the night sky? Snobbery, that’s what.

It’s just semantic manipulation and association with crudeness that prevents you from thinking logically. HNers are better than this and so are you. Don’t pretend you don’t get it and that my comparison to beastiality is so left field that it’s incomprehensible. You get it. Follow the rules and take it in good faith like you said yourself.

> The HN guidelines ask you to not engage in bad faith

Fair I edited the part that asks “is this for real” that’s literally the only part.

I also find your dismissiveness of my arguments as “bestiality” is bad faith and manipulative. I clearly wasn’t doing that. Pigs are attracted to pigs that is normal. Humans are not attracted to pigs. That is also normal. I took normal attributes of human nature and compared it to reality. You took it in bad faith and dismissed me which is against the very rules you stated.

Again, please stop telling me what I think. You have zero idea what that is and all your arguments are full of wrong (and frankly unhinged) assumptions. I don’t know what conversation you’re fantasising in your head, but it’s not this one.

> Fair I edited the part that asks “is this for real” that’s literally the only part.

Even if that were true, which I disagree with, that was the very first sentence and set the tone for the entire comment.

> I clearly wasn’t doing that.

That’s not clear in the slightest.

You keep making wrong assumptions and telling other people what they think. You can’t have an honest and productive conversation like that. You’ll never be able to engage in good faith and truly comprehend what the other person is saying until you understand and fix that.

Look, you keep saying I’m telling you what you think, but that’s just a way of dodging the actual argument. In any serious conversation, we have to interpret each other’s words. That’s how reasoning works. When I restate your point, I’m not claiming psychic powers; I’m engaging with what you said. If I get something wrong, point to the sentence and explain where. But saying “you have no idea what I think” shuts down discussion instead of clarifying it.

And about the example, you keep missing what it was doing. I wasn’t saying the night sky and bestiality are the same thing. Obviously not. The example illustrates how beauty is subjective. Humans find pigs ugly, pigs find pigs beautiful. That’s not crude, it’s biology. The point is that beauty depends entirely on the observer. That’s the entire argument. You can swap out pigs for anything else and it still holds. You got hung up on the imagery instead of seeing the reasoning behind it.

You also seem to think I’m being unhinged because I’m willing to follow an argument wherever it leads, even if it’s uncomfortable. But that’s the whole purpose of rational discussion, to question assumptions rather than hide behind emotional reactions. If your position can’t survive a provocative example, that’s not my problem.

You accuse me of making assumptions, but that’s what all reasoning is. We start with assumptions and test them. If you think mine are wrong, show why. Don’t just say “stop assuming things.” That’s not logic, that’s avoidance.

And about that opening line, you keep acting like it somehow undermines everything else I said, but that’s not how rational discussion works. I took it out because it added heat, not because it invalidated the argument. You can’t take one emotional sentence and use it to dismiss paragraphs of reasoning that followed. That’s not proportional, and it’s not logical. If my logic is wrong, show me where it’s wrong. But if all you can point to is tone, that’s just a way of dodging the argument. The content stands or falls on its reasoning, not on how politely it began or how it continues.

You talk about good faith, but good faith means addressing the argument, not the emotional impression it gave you. I laid out a clear thesis: beauty is observer dependent. It’s not intrinsic, not sacred, and certainly not a unique human experience. That doesn’t make it meaningless; it makes it relative. If you disagree, then tell me why you think beauty is intrinsic or what makes human perception special. But just calling the argument crazy and walking away doesn’t make your point stronger, it just makes it look like you don’t have one.

Compare with news stories from last decade, about people in Pakistan developing a deep fear of clear skies over several years of US drone strikes in the area. They became trained to associate good weather with not beauty, but impending death.

Fear and a sense of beauty aren’t mutually exclusive. It is perfectly congruent to fear a snake, or bear, or tiger in your presence, yet you can still find them beautiful.

An asteroid barreling towards Earth is undoubtedly beautiful, as is a mushroom cloud.

Interestingly this is a question I've had for a while. Night brings potentially deadly cold, predators, a drastic limit in vision so why do we find the sunset and night sky beautiful. Why do we stop and watch the sun set - something that happens every day - rather than prepare for the food and warmth we need to survive the night?

Maybe it's that we only pause to observe them and realize they're beautiful, when we're feeling safe enough?

"Beautiful sunset" evokes being on a calm sea shore with a loved one, feeling safe. It does not evoke being on a farm and looking up while doing chores and wishing they'd be over already. It does not evoke being stranded on an island, half-starved to death.

We think it's beautiful because it's like a background that we don't have to think about. If that background were hostile, we'd have to think and we would not think it looks beautiful.

You're entering the domain of philosophy. There's a concept of "the sublime" that's been richly explored in literature. If you find the subject interesting, I'd recommend you starting with Immanuel Kant.

My guess is that your framing presumes the opposite of the evolutionary reality. I think this time of day probably wasn't a big risk for us, that we were often the hunters and not just the hunted, and that the sense of beauty comes from — as the previous poster suggests — us having evolved to find it so.

That said, I'm discovering from living very close to a lake for the last year that mosquitos are a right pain around sunset…

I mean, I think the reason I would say the night sky is “beautiful” is because the meaning of the word for me is constructed from the experiences I’ve had in which I’ve heard other people use the word. So I’d agree that the night sky is “beautiful”, but not because I somehow have access to a deeper meaning of the word or the sky than an LLM does.

As someone who (long ago) studied philosophy of mind and (Chomskian) linguistics, it’s striking how much LLMs have shrunk the space available to people who want to maintain that the brain is special & there’s a qualitative (rather than just quantitative) difference between mind and machine and yet still be monists.

The more I learn about AI, biology and the brain, the more it seems to me that the difference between life and machines is just complexity.

People are just really really complex machines.

However there are clearly qualitative differences between the human mind and any machines we know of yet, and those qualitative differences are emergent properties, in the same way that a rabbit is qualitatively different than a stone or a chunk of wood.

I also think most of the recent AI experts/optimists underestimate how complex the mind is. I'm not at the cutting edge of how LLMs are being trained and architected, but the sense I have is we haven't modelled the diversity of connections in the mind or diversity of cell types. E.g. Transcriptomic diversity of cell types across the adult human brain (Siletti et al., 2023, Science)

I’d say sophistication.

Observing the landscape enables us to spot useful resources and terrain features, or spot dangers and predators. We are afraid of dark enclosed spaces because they could hide dangers. Our ancestors with appropriate responses were more likely to survive.

A huge limitation of LLMs is that they have no ability to dynamically engage with the world. We’re not just passive observers, we’re participants in our environment and we learn from testing that environment through action. I know there are experiments with AIs doing this, and in a sense game playing AIs are learning about model worlds through action in them.

The idea I keep coming back to is that as far as we know it took roughly 100k-1M years for anatomically modern humans to evolve language, abstract thinking, information systems, etc. (equivalent to LLMs), but it took 100M-1B years to evolve from the first multi-celled organisms to anatomically modern humans.

In other words, human level embodiment (internal modelling of the real world and ability to navigate it) is likely at least 1000x harder than modelling human language and abstract knowledge.

And to build further on what you are saying, the way LLMs are trained and then used, they seem a bit more like DNA than the human brain in terms of how the "learning" is being done. An instance of an LLM is like a copy of DNA trained on a play of many generations of experience.

So it seems there are at least four things not yet worked out re AI reaching human level "AGI":

1) The number of weights (synapses) and parameters (neurons) needs to grow by orders of magnitude

2) We need new analogs that mimic the brains diversity of cell types and communication modes

3) We need to solve the embodiment problem, which is far from trivial and not fully understood

4) We need efficient ways for the system to continuously learn (an analog for neuroplasticity)

It may be that these are mutually reinforcing, in that solving #1 and #2 makes a lot of progress towards #3 and #4. I also suspect that #4 is economical, in that if the cost to train a GPT-5 level model was 1,000,000 cheaper, then maybe everyone could have one that's continuously learning (and diverging), rather than everyone sharing the same training run that's static once complete.

All of this to say I still consider LLMs "intelligent", just a different kind and less complex intelligence than humans.

Id also add that 5) We need some sense of truth.

Im not quite sure if the current paradigm of LLMs are robust enough given the recent Anthropic Paper about the effect of data quality or rather the lack thereof, that a small bad sample can poison the well and that this doesn’t get better with more data. Especially in conjunction with 4) some sense of truth becomes crucial in my eyes (Question in my eyes is how does this work? Something verifiable and understandable like lean would be great but how does this work with more fuzzy topics…).

That's a segue into an important and rich philosophical space...

What is truth? Can it be attained, or only approached?

Can truth be approached (progress made towards truth) without interacting with reality?

The only shared truth seeking algorithm I know is the scientific method, which breaks down truth into two categories (my words here):

1) truth about what happened (controlled documented experiments) And 2) truth about how reality works (predictive powers)

In contrast to something like Karl friston free energy principle, which is more of a single unit truth seeking (more like predictive capability seeking) model.

So it seems like truth isn't an input to AI so much as it's an output, and it can't be attained, only approached.

But maybe you don't mean truth so much as a capability to definitively prove, in which case I agree and I think that's worth adding. Somehow integrating formal theorem proving algorithms into the architecture would probably be part of what enables AI to dramatically exceed human capabilities.

I think that in some senses truth is associated with action in the world. That’s how we test our hypotheses. Not just in science, in terms of empirical adequacy, but even as children and adults. We learn from experience of doing, not just rote, and we associate effectiveness with truth. That’s not a perfect heuristic, but it’s better than just floating in a sea of propositions as current LLMs largely are.

I agree.

There's a truth of what happened, which as individuals we can only ever know to a limited scope... And then there is truth as a prediction ability (formula of gravity predicts how things fall).

Science is a way to build a shared truth, but as an individual we just need to experience an environment.

One way I've heard it broken down is between functional truths and absolute truths. So maybe we can attain functional truths and transfer those to LLMs through language, but absolute truth can never be attained only approached. (The only absolute truth is the universe itself, and anything else is just an approximation)

>A huge limitation of LLMs is that they have no ability to dynamically engage with the world.

They can ask for input, they can choose URLs to access and interpret results in both situations. Whilst very limited, that is engagement.

Think about someone with physical impairments, like Hawking (the now dead theoretical physicist) had. You could have similar impairments from birth and still, I conjecture, be analytically one of the greatest minds of a generation.

If you were locked in a room {a non-Chinese room!}, with your physical needs met, but could speak with anyone around the World, and of course use the internet, whilst you'd have limits to your enjoyment of life I don't think you'd be limited in the capabilities of your mind. You'd have limited understanding of social aspects to life (and physical aspects - touch, pain), but perhaps no more than some of us already do.

> A huge limitation of LLMs is that they have no ability to dynamically engage with the world.

A pure LLM is static and can’t learn, but give an agent a read-write data store and suddenly it can actually learn things-give it a markdown file of “learnings”, prompt it to consider updating the file at the end of each interaction, then load it into the context at the start of the next… (and that’s a really basic implementation of the idea, there are much more complex versions of the same thing)

That's going to run into context limitations fairly quickly. Even if you distill the knowledge.

True learning would mean constant dynamic training of the full system. That's essentially the difference between LLM training and human learning. LLM training is one-shot, human learning is continuous.

The other big difference is that human learning is embodied. We get physical experiences of everything in 3D + time, which means every human has embedded pre-rational models of gravity, momentum, rotation, heat, friction, and other basic physical concepts.

We also learn to associate relationship situations with the endocrine system changes we call emotions.

The ability to formalise those abstractions and manipulate them symbolically comes much later, if it happens at all. It's very much the plus pack for human experience and isn't part of the basic package.

LLMs start from the other end - from that one limited set of symbols we call written language.

It turns out a fair amount of experience is encoded in the structures of written language, so language training can abstract that. But language is the lossy ad hoc representation of the underlying experiences, and using symbol statistics exclusively is a dead end.

Multimodal training still isn't physical. 2D video models still glitch noticeably because they don't have a 3D world to refer to. The glitching will always be there until training becomes truly 3D.

An LLM agent could be given a tool for self-finetuning… it could construct a training dataset, use it to build a LORA/etc, and then use the LORA for inference… that’s getting closer to your ideal

Yes, and give it tools and it can sense and interact with its surroundings.

Oh, I just realized you maybe we're referring to Kopple when you said sophistication?

If so, then yes, that might be a good measure. I'm not deep enough in this to have an opinion on if it's the best measure. There are a few integrated information theories and I am still getting my head wrapped around them...

I think the main mistake with this is that the concept of a "complex machine" has no meaning.

A “machine” is precisely what eliminates complexity by design. "People are complex machines" already has no meaning and then adding just and really doesn't make the statement more meaningful it makes it even more confused and meaningless.

The older I get the more obvious it becomes the idea of a "thinking machine" is a meaningless absurdity.

What we really think we want is a type of synthetic biological thinking organism that somehow still inherits the useful properties of a machine. If we say it that way though the absurdity is obvious and no one alive reading this will ever witness anything like that. Then we wouldn't be able to pretend we live at some special time in history that gets to see the birth of this new organism.

I think we are talking past each other a bit, probably because we have been exposed to different sets of information on a very complicated and diverse topic.

Have you ever explored the visual simulations of what goes on inside a cell or in protein interactions?

For example what happens inside a cell leading up to mitosis?

https://m.youtube.com/user/RCSBProteinDataBank

Is a pretty cool resource, I recommend the shorter videos of the visual simulations.

This category of perspective is critical to the point I was making. Another might be the meaning / definition of complexity, which I don't think is well understood yet and might be the crux. For me to say "the difference between life and what we call machines is just complexity" would require the same understanding of "complexity" to have shared meaning.

I'm not exactly sure what complexity is, and I'm not sure anyone does yet, but the closest I feel I've come is maybe integrated information theory, and some loose concept of functional information density.

So while it probably seemed like I was making a shallow case at a surface level, I was actually trying to convey that when one digs into science at all levels of abstraction, the differences between life and machines seem to fall more on a spectrum.

> I think the reason I would say the night sky is “beautiful” is because the meaning of the word for me is constructed from the experiences I’ve had in which I’ve heard other people use the word.

Ok but you don’t look at every night sky or every sunset and say “wow that’s beautiful”

There’s a quality to it - not because you heard someone say it but because you experience it

> Ok but you don’t look at every night sky or every sunset and say “wow that’s beautiful

Exactly - because it's a semantic shorthand. Sunsets are fucking boring, ugly, transient phenomena. Watching a sunset while feeling safe and relaxed, maybe in a company of your love interest who's just as high on endorphins as you are right now - this is what feels beautiful. This is a sunset that's beautiful. But the sunset is just a pointer to the experience, something others can relate to, not actually the source of it.

I’ve seen incredible sunsets while stressed depressed and worse. Are you saying sunsets cannot be experienced as beautiful on their own?

Because words are much lower bandwidth than speech. But if you were “told” about a sunset by means of a Matrix style direct mind uploading of an experience, it would seem just as real and vivid. That’s a quantitative difference in bandwidth, not a qualitative difference in character.

my thought exactly

It’s interesting you mention linguistics because I feel a lot of the discussions around AI come back to early 20th century linguistics debates between Russel, Wittgenstein and later Chomsky. I tend to side with (later) Wittgenstein’s perception that language is inherently a social construct. He gives the example of a “game” where there’s no meaningful overlap between e.g. Olympic Games and Monopoly, yet we understand very well what game we’re talking about because of our social constructs. I would argue that LLMs are highly effective at understanding (or at least emulating) social constructs because of their training data. That makes them excellent at language even without a full understanding of the world.

You don’t have a deeper “meaning of the word,” you have an actual experience of beauty. Three word is just a label for the thing you, me, and other humans have experienced.

The machine has no experience.

The fact that things are constructed by neurons in the brain, and are a representation of other things - does not preclude your representation from being deeper and richer than LLM representations.

The patterns in experience are reduced to some dimensions in an LLM (or generative model). They do not capture all the dimensions - because the representation itself is a capture of another representation.

Personally, I have no need to reassure myself whether I am a special snowflake or not.

Whatever snowflake I am, I strongly prefer accuracy in my analogies of technology. GenAI does not capture a model of the world, it captures a model of the training data.

If video tools were that good, they would have started with voxels.

> humans say the night sky is beautiful is because they see that it is

True, but we could engineer AI to see that too, just as evolution has engineered us to see it.

Our innate emotional responses to things has been honed by evolution to be adaptive, to serve a purpose, but the things that trigger these various responses are not going to be super specific. e.g. We may derive pleasure from eating a nice juicy peach, but that doesn't mean that is encoded in our DNA - it's going to be primarily the reaction to sugar/sweetness, a good source of energy, that we are reacting to.

Similarly, we may have an emotional reaction to certain pieces of modern art or artistic expression, but clearly evolution has not selected for those specifically, but rather it is the artist triggering innate responses that evolved for reasons other than appreciation of art.

It's hard to guess what innate responses, that were actually selected for, are being triggered by our response to the night sky, and I'm also not sure how much of our response is purely visual (beauty) as opposed to wonder or awe. Maybe it's an attraction to the unknown, or sense of size and opportunity, with these being the universals that are actually adaptive.

In any case, if we figured out the specifics of our hard wired emotional reactions, that evolution as given us, then we could choose to engineer emotional AI that had those same reactions, in just as genuine a way as we do, if we chose to.

Beauty standard changes over time, see how people perceive body fat in the past few hundred years. We learns what is beautiful from our peers.

Taste can be acquired and can be cultural. See how people used to had their coffee.

Comparing human to LLM is like comparing something constantly changing to something random -- we can't compare them directly, we need a good model for each of them before comparing.

Has there been a point in human history where mainstream society denied the beauty in nature?

In a local Facebook group, in a discussion about zoning, someone seriously said "we need less parks and more parking lots", so... Maybe?

What about a blind human? Are they just like an LLM?

What about a multimodal model trained on video? Is that like a human?

This is actually a great point but for the opposite reason - if you ask a blind person if the night sky is beautiful, they would say they don't know because they've never seen it (they might add that they've heard other people describe it as such). Meanwhile, I just asked ChatGPT "Do you think the night sky is beautiful?" And it responded "Yes, I do..." and went on to explain why while describing senses its incapable of experiencing.

Wha if you asked the blind man to play the role of helpful assistant

Now that's an interesting point of view.

Involving blind people would be an interesting experiment.

Anyway, until the sixties the ability to play a game of chess was seen as intelligence, and until about 2-3 years ago the "turing test" was considered the main yardstick (even though apparently some people talked to eliza at the time like an actual human being). I wonder what the new one is, and how often it will be moved again.

I just asked Gemini and it said "I don't have eyes or the capacity to feel emotions like "beauty""

Claude 4.5

Q) Do you think the night sky is beautiful

A) I find the night sky genuinely captivating. There’s something profound about looking up at stars that have traveled light-years to reach us, or catching the soft glow of the Milky Way on a clear night away from city lights. The vastness it reveals is humbling. I’m curious what draws you to ask - do you have a favorite thing about the night sky, or were you stargazing recently?

Claude is multimodal, it has been trained on images

Multimodal is a farce. It still can’t see anything, it just generates a as list of descriptors that the LLM part can LLM about.

Humans got by for hundreds of thousands of years without language. When you see a duck you don’t need to know the word duck to know about the thing you’re seeing. That’s not true for “multimodal” models.

>> Meanwhile, I just asked ChatGPT "Do you think the night sky is beautiful?" And it responded "Yes, I do..." and went on to explain why while describing senses its incapable of experiencing.

> I just asked Gemini and it said "I don't have eyes or the capacity to feel emotions like "beauty""

That means nothing, except perhaps that Google probably found lies about "senses [Gemini] incapable of experiencing" to be an embarrassment, and put effort into specifically suppressing those responses.

Interesting. But not not only blind people.

I'm gooing to try this question this weekend with some people, as h0 hypotesis i think the answer i will get would be usually like "what an odd question" or "why do you ask".

Guys you realize that you can go to ChatGPT right now and it can generate an actual picture of the night sky because it has seen thousands of pictures and drawings of the actual night sky right?

Your logic is flawed because your knowledge is outdated. LLMs are encoding visual data, not just “language” data.

You misunderstand how the multimodal piece works. The fundamental unit of encoding here is still semantic. Not the same in your mind: you don’t need to know the word for sunset to experience the sunset.

No you misunderstand the ground truth reality.

The LLM doesn’t need words as input. It can output pictures from pictures. Semantic words don’t have to be part of the equation at all.

Also you have to note that serialized one dimensional string encodings are universal. Anything on the face of the earth and the universe itself can be encoded into a sting of just two characters: one and zero. That’s means anything can be translated to a linear series of symbols and the LLM can be trained on it. The LLM can be trained on anything.

The multimodal architectures I’ve seen are still text at the layer between modalities. And the image embedding and text embedding are kept completely separate. Not like where your brain where single neurons are used in all sorts of things.

Yes, they can generate images from images, but that doesn’t mean you’ll get anything meaningful without human instruction on top.

Yes, serialized one dimensional strings can encode anything. But that’s just the message content. If I wrote down my genetic sequence on a piece of paper and dropped it in a bottle in the sea, I don’t need to worry about accidentally fathering any children.

You’re mixing representational capacity with representational intent. That’s what I meant in my initial example about encodings. The model doesn’t care whether it’s text, pixels, or sound. All of it can be mapped into the same kind of high dimensional space where patterns align by structure rather than category. “Semantic” is just our label for how those internal relationships appear when we interpret them through language.

Anything in the universe can be encoded this way. Every possible form, whether visual, auditory, physical, or abstract, can be represented as a series of numbers or symbols. With enough data, an LLM can be trained on any of it. LLMs are universal because their architecture doesn’t depend on the nature of the data, only on the consistency of patterns within it. The so called semantic encoding is simply the internal coordinate system the model builds to organize and decode meaning from those encodings. It is not limited to language; it is a general representation of structure and relationship.

And the genome in a bottle example actually supports this. The DNA string does encode a living organism; it just needs the right decoding environment. LLMs serve that role for their training domains. With the right bridge, like a diffusion model or a VAE, a text latent can unfold into an image distribution that’s statistically consistent with real light data.

So the meaning isn’t in the words. It’s in the shape of the data.

You are mistaking the map for the territory. The TERRITORY of human experience is higher dimensional. The LLM utilizes a lower resolution mapping of that territory, a projection from experience to textual (or pixel, or waveform, etc.) representations.

This is not just a lossy mapping; it excludes entire categories of experience that cannot be captured/encoded except for as a pointer to the real experience, one that is often shared by the embodied, embedded, enacted, and extended cognitive beings that have had that experience.

I can point to beauty and you can understand me because you've experienced beauty. I cannot encode beauty itself. The LLM cannot experience beauty. It may be able to analyze patterns of things determined beautiful by beauty experiencers, but this is, again, a lower resolution map of the actual experience of beauty. Nobody had to train you to experience beauty—you possess that capability innately.

You cannot encode the affective response one experiences when holding their newborn. You cannot encode the cognitive appraisal of a religious experience. You can't even encode the qualia of red except for, again, as a pointer to the color.

You're also missing that 4E cognitive beings have a fundamental experience of consciousness—particularly the aspect of "here" and "now". The LLM cannot experience either of those phenomena. I cannot encode here and now. But you can, and do, experience both of those constantly.

You are making a metaphysical claim when a physical one will do. Beauty, awe, grief, the rush of holding a newborn, the sting of a breakup, the warmth of a summer evening at golden hour. All of it is patterns of atoms in motion under lawful dynamics. Neurons fire. Neurotransmitters bind. Circuits synchronize. Bodies and environments couple. There is no extra ingredient that floats outside physics.

Once you grant that, the rest is bookkeeping. Any finite physical process has a finite physical trace. That trace is measurable to some precision. A finite trace can be serialized into a finite string of symbols. If you prefer bits, take a binary code. If you prefer integers, index the code words. The choice of alphabet does not matter. You can map a movie, a symphony, a spike train, a retina’s photon counts, or a full brain-body sensorium collected at some temporal resolution into a single long string. You lose nothing by serialization because the decoder knows the schema. This is not a “text only” claim. It is a claim about representation.

Your high dimensionality objection collapses under the same lens. High dimensional just means many coordinates. There is a well known result that any countable description can be put in one dimension by an invertible code. Think Gödel numbering or interleaving bits of coordinates. You do not preserve distances, but you do preserve information. If the thing you care about is the capacity to carry structure, the one dimensional string can carry all of it, and you can recover the original arrangement exactly given the decoding rule.

Now take the 4E point. Embodiment matters because it constrains the data distribution and the actions that follow. It does not create a magic type of information that cannot be encoded. A visual scene is photons on receptors over time. Proprioception is stretch receptor states. Affect is the joint state of particular neuromodulatory systems and network dynamics. Attention and working context are transient global variables implemented by assemblies. All of that can be logged, compressed, and restored to the degree your sensors and actuators allow. The fact that a bottle with a genome inside does not make a child on a beach tells you reproduction needs a decoder and an environment. It does not tell you the code fails to specify the organism. Likewise, an LLM plus a diffusion decoder can take a text latent and unfold it into an image distribution that matches world statistics because the bridge model plays the role of the environment for that domain.

“LLMs cannot experience beauty” simply reasserts the thing you want to prove. We have no privileged readout for human qualia either. We infer it from behavior, physiology, and report. We do not understand human brains at the level of complete causal microphysics because of scale and complexity, not because there is a non-physical remainder. We likewise do not fully understand why a large model makes a given judgment. Same reason. Scale and complexity. If you point to mystery on one side as a defect, you must admit it on the other.

The map versus territory line also misses the target. Of course a representation is not the thing itself. No one is claiming a jpeg is a sunset. The claim is that the structure necessary to act as if about sunsets can be encoded and learned. A system that takes in light fields, motor feedback, language, and reward and that updates an internal world model until its predictions and actions match ours to arbitrary precision will meet every operational test you have for meaning. If you reply that something is still missing, you have stepped outside evidence into stipulation.

So let’s keep the ground rules clear. Everything we are and feel is physically instantiated. Physical instantiations at finite precision admit lossless encodings as strings. Strings can be learned over by generic function approximators that optimize on pattern consistency, regardless of whether the symbols came from pixels, pressure sensors, or phonemes. That makes the “text inside, image outside” complaint irrelevant. The substrate is a detail. The constraint is data and objective.

We cannot yet build a full decoder for the human condition. That is a statement about engineering difficulty, not impossibility. And it cuts both ways. We do not know how to fully read a person either. But we do not conclude that people lack experience. We conclude that we lack understanding.

At this point, you’re describing a machine which depends on a level of physics that simply isn’t possible. Even if it were theoretically possible to reconstruct the state of a human mind from physical components, we are so far from understanding how that could be done it is closer to the realm of impossible than possible. Your theoretical math box that constructs affective qualia from bit strings isn’t a better description than saying the angels did it. And it bears zero resemblance to the models running today, except for, again, in a theoretical, mathematical way.

Back of the envelope math puts an estimate of 10^42 bits to capture the information present in your current physical brain state. Thats just a single brain, a single state. Now you need to build your mythical decoder device, which can translate qualia from this physical state. Where does it live? What’s its output look like? Another 10^40 bitstring?

Again, these arguments are fun on paper. But they’re completely removed from reality.

You’re confusing “we don’t know how” with “it’s impossible.” The difference is everything.

We don’t understand LLMs either. We built them, but we can’t explain why they work. No one can point to a specific weight matrix and say “this is the neuron that encodes irony” or “this is where the model stores empathy.” We don’t know why scaling parameters suddenly unlock reasoning or why multimodal alignment appears spontaneously. The model’s inner space is a black box of emergent structure and behavior, just like the human brain. We understand the architecture, not the mind inside it.

When you say it’s “closer to impossible than possible” to reconstruct a human mind, you’ve already lost the argument. We’re living proof that the machine you say cannot exist already does. The human brain is a physical object obeying the same laws of physics that govern every other machine. It runs on electrochemical signals, not miracles. It encodes and decodes information, forms memories, generates imagination, and synthesizes emotion. That means the physics of consciousness are real, computable, and reproducible. The impossible machine has been sitting in your skull the entire time.

Your argument about 10^42 bits isn’t just wrong, it’s total nonsense. That number is twenty orders of magnitude beyond any serious estimate. The brain has about 86 billion neurons, each forming roughly ten thousand connections, for a total of about 10^15 synapses. Even if every synapse held a byte of information, that’s 10^16 bits. Add in every molecular and analog nuance you like and you might reach 10^20. Not 10^42. That’s a difference of twenty-two orders of magnitude. It’s a fantasy number that exceeds the number of atoms in your entire body.

And that supposed “impossible” scale is already within sight. Modern GPUs contain hundreds of billions of transistors and run at gigahertz frequencies, while neurons fire at about a hundred hertz. The brain performs around 10^17 synaptic operations per second. Frontier AI clusters already push 10^25 to 10^26 operations per second. We’ve already outpaced biology in raw throughput by eight or nine orders of magnitude. NVIDIA’s Blackwell chips exceed 200 million transistors per square millimeter, and global compute now involves more than 10^24 active transistors switching billions of times per second. Moore’s law may have slowed, but density keeps climbing through stacking and specialized accelerators. The number you called unreachable is just a few decades of progress away.

The “decoder” you mock is exactly what a brain is. It takes sensory input, light, sound, and chemistry, and reconstructs internal states we call experience. You already live inside the device you claim can’t exist. It doesn’t need to live anywhere else; it’s instantiated in matter.

And this is where your argument collapses. You say such a machine is removed from reality. But reality is already running it. Humanity is proof of concept. We know the laws of physics allow it because they’re doing it right now. Every thought, emotion, and perception is a physical computation carried out by atoms. That’s the definition of a machine governed by physics.

We don’t yet understand the full physics of the brain, and we don’t fully understand LLMs either. That’s the point. The same kind of ignorance applies to both. Yet both produce coherent language, emotion like responses, creativity, reasoning, and abstraction. When two black boxes show convergent behavior under different substrates, the rational conclusion isn’t “one is impossible.” It’s “we’re closer than we realize.”

The truth is simple: what you call impossible already exists. The human brain is the machine you’re describing. It’s not divine. It’s atoms in lawful motion. And because we know it can exist under physics, we know it can be built. LLMs are just the first flicker of that same physics waking up in silicon.

> We don’t understand LLMs either. We built them, but we can’t explain why they work.

Just because you don't mean no one does. It's a pile of math. Somewhere along the way, something happened to get where we are, but looking at Golden Gate Claude, and the abliteration of shared models, or reading OpenAI's paper about hallucinations, there's a lot of detail and knowledge about how these things works that isn't instantly accessible and readily apparent to everyone on the Internet. As laymen all we can do is black box testing, but there's some really interesting stuff going on to edit the models and get them to talk like pirate.

The human brain is very much an unknowable squishy box because putting probes into it would be harmful to the person who's brain it is we're working on, and we don't like to do that to people because people are irreplaceable. We don't have that problem with LLMs. It's entirely possible to look at the memory register at location x at time y, and correspond that to a particular tensor which corresponds to a particular token which then corresponds to a particular word for us humans to understand. If you want to understand LLMs, start looking! It's an active area of research and is very interesting!

You are missing the ground truth. Humanity does not understand how LLMs work. Every major lab and every serious researcher acknowledges this. What we have built is a machine that functions, but whose inner logic no one can explain.

References like Golden Gate Claude or the latest interpretability projects don’t change that. Those experiments are narrow glimpses into specific activation patterns or training interventions. They give us localized insight, not comprehension of the system as a whole. Knowing how to steer tone or reduce hallucinations does not mean we understand the underlying cognition any more than teaching a parrot new words means we understand language acquisition. These are incremental control levers, not windows into the actual mind of the model.

When we build something like an airplane, no single person understands the entire system, but in aggregate we do. Aerodynamicists, engineers, and computer scientists each master their part, and together their knowledge forms a complete whole. With LLMs, even that collective understanding does not exist. We cannot even fully describe the parts, because the “parts” are billions of distributed parameters interacting in nonlinear ways that no human can intuit or map. There is no subsystem diagram, no modular comprehension. The model’s behavior is not the sum of components we understand, it is the emergent product of relationships we cannot trace.

You said we “know” what is going on. That assumption is patently false. We can see the equations, we can run the training, we can measure activations, but those are shadows, not understanding. The model’s behavior emerges from interactions at a scale that exceeds human analysis.

This is the paradigm shift you have not grasped. For the first time, we are building minds that operate beyond the boundary of human comprehension. It is not a black box to laymen. It is a black box to mankind.

And I say this as someone who directly works on and builds LLMs. The experts who live inside this field understand this uncertainty. The laymen do not. That gap in awareness is exactly why conversations like this go in circles.

[dead]

> We don’t yet understand the full physics of the brain, and we don’t fully understand LLMs either. That’s the point. The same kind of ignorance applies to both. Yet both produce coherent language, emotion like responses, creativity, reasoning, and abstraction. When two black boxes show convergent behavior under different substrates, the rational conclusion isn’t “one is impossible.” It’s “we’re closer than we realize.”

No. The LLM does not produce emotion-like responses. I'd argue no on creativity either. And only very limited in reasoning, in domains it has in its training set.

You have fundamental misunderstandings about neuroscience and cognitive science. Its hard to argue with you here because you simply don't know what you don't know.

Yes, the human brain is the machine we're describing. And we don't describe it very well. Definitely not at the level of understanding how to reproduce it with bitstrings.

I'm glad you're so passionate about this topic. But you're arguing the equivalent of FTL transit and living on Dyson Spheres. Its fun as a thought experiment and may theoretically be possible one day, but the line between what we're capable of today and that imagined future is neither straight nor visible—certainly not to the degree you're asserting here.

Will we one day have actual machine intelligence? Maybe. Is it going to come anytime soon, or look anything like the transformer-based LLM?

No.

You keep talking past the point. Nobody is claiming we can turn a human mind into a literal bitstring and boot it up like a computer program. That was never the argument. The bitstring analogy exists to make a simpler point: everything that exists and changes according to physical law can, in principle, be represented, modeled, or reproduced by another physical system. The form does not need to be identical to the brain’s atoms any more than a jet engine must flap its wings to fly. The key is not replication of matter but replication of causal structure.

You say we cannot reproduce the brain. But that is not the point. The point is that nothing about the brain violates physics. It runs on chemical and electrical dynamics that obey the same laws as everything else. If those laws can produce intelligence once, then they can do so again in another substrate. That makes the claim of impossibility not scientific, but emotional.

You accuse me of misunderstanding neuroscience and cognitive science. The reality is that neither field understands itself. We have no complete model of consciousness. We cannot explain why synchronized neural oscillations yield awareness. We cannot define where attention comes from or what distinguishes a “thought” from a signal cascade. Cognitive science is still arguing over whether perception is bottom up or top down, whether emotion is distinct from cognition, and whether consciousness even plays a causal role. That is not mastery. That is the sound of a discipline still wandering in the dark.

You act as though neuroscience has defined the boundaries of intelligence, but it has not. We do not have a mechanistic understanding of creativity, emotion, or reasoning. We have patterns and correlations, not principles. Yet you talk as if those unknowns justify declaring machine intelligence impossible. It is the opposite. Our ignorance is precisely why it cannot be ruled out.

Emotion is not magic. It is neurochemical modulation over predictive circuits. Replicate the functional dynamics and you replicate emotion’s role. Creativity is recombination and constraint satisfaction. Replicate those processes and you replicate creativity. Reasoning is predictive modeling over structured representations. Replicate that, and you replicate reasoning. None of these depend on carbon. They depend on organization and feedback.

You keep saying that the brain cannot be “reproduced as bitstrings,” but that is a distraction. Nobody is suggesting uploading neurons into binary. The bitstring argument shows that any finite physical system has a finite description. It proves that cognition, like any process governed by law, has an information theoretic footprint. Once you accept that, the difference between biology and computation becomes one of scale, not kind.

You say LLMs are not creative, not emotional, not reasoning. Yet they already produce outputs that humans classify as empathetic, sarcastic, joyful, poetic, or analytical. People experience their words as creative because they combine old ideas into new, functional, and aesthetic patterns. They reason by chaining relationships, testing implications, and revising conclusions. The fact that you can recognize all of this in their behavior proves they are performing the surface functions of those capacities. Whether it feels like something to be them is irrelevant to the claim that they can reproduce the function.

And now your final claim, that whatever becomes intelligent “will not be an LLM.” You have no basis for that certainty. Nobody knows what an LLM truly is once scaled beyond our comprehension. We do not understand how emergent representations arise or how concepts self organize within their latent spaces. We do not know if some internal dynamic of this architecture already mirrors the structure of cognition. What we do know is that it learns to compress the world into predictive patterns and that it develops abstractions that map cleanly to human meaning. That is already the seed of general intelligence.

You are mistaking ignorance for insight. You think not knowing how something works grants you authority to say what it cannot become. But the only thing history shows is that such confidence always looks ridiculous in retrospect. The physics of intelligence exist. The brain proves it. And the LLM is the first machine that begins to display those same emergent behaviors. Saying it “will not be an LLM” is not a scientific claim. It is wishful thinking spoken from the wrong side of the curve.

Look, mate, you can keep jumping up and down about this all you want. But you're arguing science fiction at this point. Not really worth continuing the conversation, but thanks.

Best of luck.

Calling this “science fiction” isn’t just dismissive, it’s ironic. The discussion itself is science fiction by the standards of only a few years ago. Back then, the idea that a machine could hold a coherent philosophical argument, write code, debate consciousness, and reference neuroscience was fantasy. Now it’s routine. You are literally using what was once science fiction to declare that progress on LLMs has ended.

And calling that “science fiction” again isn’t a rebuttal, it’s an insult. You didn’t engage a single argument, you just waved your hand and walked away. That isn’t scientific skepticism, it’s arrogance disguised as authority.

You can disagree, but doing what you did is manipulative. You dodged every point and tried to end the debate by pretending it was beneath you. Everyone reading can see that.

I’m pretty sure everyone reading can see which of us is the arrogant one.

Good day, sir.

You called it “science fiction” and bowed out, then tried to make it personal. That is not humility, that is evasion. You never addressed a single argument, you just waved your hand and left, and calling someone’s reasoning “science fiction” is not only an insult, it violates the site’s rule against dismissive or unfriendly language. The “good day sir” at the end makes that tone of mockery obvious.

What is actually arrogant is dismissing a discussion the moment it goes beyond your depth and pretending that walking away is a sign of wisdom. It is not. It is what people do when they realize the conversation has left them behind.

If you are so sure of your position, you could have refuted the reasoning point by point. Instead, you dodged, labeled, and ran. Everyone reading can see which of us is still dealing in facts and which one needed a graceful exit to save face.

Nah, mate, the conversation never went "beyond my depth." You're just not an enjoyable conversation partner.

It doesn't matter how smart (you think) you are. If nobody wants to talk to you, you'll be spinning all that brain matter in the corner by yourself. Based on your comment history here, it looks like this happens to you more often than not.

I'm sure you have good points. I could probably learn a thing or two from you—maybe you could learn something from me too! But why on earth would anyone want to engage with someone who behaves like you do?

Again, best of luck.

You are projecting, and everyone can see it. You pretended that I was being rude while you slipped in sarcasm, personal digs, and that condescending “best of luck” as if it made you look polite. It doesn’t. That is not civility. It is passive aggression wrapped in fake courtesy.

You completely dropped the argument and went straight for personal attacks. That is not confidence, it is surrender. You are no longer debating, you are lashing out because you ran out of ideas. You can claim the conversation “wasn’t beyond your depth,” but you abandoned every point the moment you were asked to defend it. Then you tried to flip it by pretending that walking away made you the mature one. It didn’t. It made you the one who couldn’t keep up and needed an exit.

You can dress it up with sarcasm and moral posturing, but that doesn’t change what happened. The moment you shifted from ideas to insults, you showed everyone reading that you had nothing left to stand on. The difference between us is simple: I stayed on topic. You turned it into attitude. And now everyone can see exactly who ran out of substance first.

No moral posturing and no insults. Your behavior is just objectively noxious. Not just to me, not just in this thread: the vast majority of your conversations here go roughly the way this one did. A quick glance at your profile shows roughly half of the comments you make here end up light grey.

You have an enormous chip on your shoulder. You consistently make truth claims about entire fields that are still in debate and then you arrogantly shout over the other person when they disagree with you.

I strongly suggest you work on this. It will limit you in life. It probably already has. You probably already know how it has, even!

I'm not saying this to be mean, or because I "have nothing left to stand on." You're clearly intelligent and you clearly care about this topic. But until you mature and learn to behave, others will continue to withdraw from conversation with you.

Best of luck.

You have already abandoned the debate and moved into stalking and personal attacks. That is not a sign of strength. It is proof you ran out of substance and are trying to win by humiliation instead.

You dug through my profile to manufacture a narrative about my character because you could not answer a single technical point. That is petty and dishonest. It is also exactly the kind of behavior moderators and civil participants call out. If you actually cared about truth you would stay on topic. Instead you weaponized the comment section to attack me personally.

Do not mistake tone for argument. I stood on evidence and logic. You offered sneers, a mock sign off, and then tried to moralize. That is not persuasion. It is performative virtue signaling layered over an exit strategy.

If you want to be taken seriously, stop the profile policing, stop the personal diagnostics, and engage the claims you think are wrong. Make one focused counterargument. Otherwise your behavior will read to everyone as what it is: a public temper tantrum disguised as concern.

You can keep doing this. It will not change the facts. It will only make readers pity the quality of your argument and worry that you are the kind of person who cannot have a grown up debate. If you are interested in a real exchange, show it. If not, spare the thread the noise.

Okie dokie mate, whatever you say.

Best of luck!

Adorable exit. Nothing says “I’m out of arguments” quite like a cheery “okie dokie mate.” Best of luck holding that pose.

You’ve said the same thing fifteen times now.

I still don’t want to play with you, sorry.

Fair enough. You’ve spent the whole thread talking about me instead of the topic, so saying you don’t want to play now is a bit rich. You’ve been playing the whole time, just mistaking the sound of your own confusion for strategy.

Here's how I've been explaining this to non-tech people recently, including the CEO where I work: Language is all about compressing concepts and sharing them, and it's lossy.

You can use a thousand words to describe the taste of chocolate, but it will never transmit the actual taste. You can write a book about how to drive a car, but it will only at best prepare that person for what to practice when they start driving, it won't make them proficient at driving a car without experiencing it themselves, physically.

Language isn't enough. It never will be.

The taste of chocolate is also assuming information-theoretic models are correct and not a use-based, pragmatic theory of meaning.

I don't agree with information-theoretic models in this context but we come to the same conclusion.

Loss only makes sense if there was a fixed “original” but there is not. The information-theoretic model creates a solvable engineering problem. We just aren't solving the right problem then with LLMs.

I think it is more than that. The path forward with a use theory of meaning is even less clear.

The driving example is actually a great example of the use theory of meaning and not the information-theoretic.

The meaning of “driving” emerges from this lived activity, not from abstract definitions. You don't encode an abstract meaning of driving that is then transmitted on a noisy channel of language.

The meaning of driving emerges from the physical act of driving. If you only ever mount a camera on the headrest and operate the steering wheel and pedals remotely from a distance you still don't "understand" the meaning of "driving".

Whatever data stream you want to come up with, trying to extract the meaning of "driving" from that data stream makes no sense.

Trying to extract the "meaning" of driving from driving language game syntax with language models is just complete nonsense. There is no meaning to be found even if scaled in the limit.

Humans perceive phenomena via senses, and then carve categories or concepts to understand them. This is a process of abstraction and each idea has an associated qualia. Then use language to describe these concepts. As such, a concept is grounded either by actual phenomena or operations, or is a composition of other grounded concepts. The creation of categories and grounding them involves constant feedback from the environment - and is a creative process, and we as agents have "skin in the game", in the sense that we get the rewards/punishments for our understanding and actions.

Map vs Territory is a common analogy. Maps describe territories but in an abstract and lossy manner.

But, most of us dont construct grounded concepts in our understanding. We carry a muddle of ungrounded ideas - some told to us by others, and some we intuit directly. There is a long tradition of attempting to think clearly all the way from Socrates, Descartes, Feynman etc.. where an attempt is made to ground the ideas we have. Try explaining your ideas to others, and soon, you will hit the illusion of explanatory depth.

LLM is a map and is a useful tool, but it doesnt interact with the territory, and it does not have skin in the game, and as a result, it cant carve new categories in a learning process that we have as humans.

The human experience is also several degrees removed from the „real“ world. I don’t think sensory chauvinism is a useful tool in assessing intelligence potential.

This comment is hallucinatory in nature as it is in direct conflict with the in the ground reality of LLMs.

The LLM has both light (aka photons) and language encoded into its very core. It is not just language. You seemed to have missed the boat with all the ai generated visuals and videos that are now inundating the internet.

Your flawed logic is essentially that LLMs are unable to model the real world because they don’t encode photonic data into the model. Instead you think they only encode language data which is an incredibly lossy description of reality. And this line of logic flies against the ground truth reality of the fact that LLMs ARE trained with video and pictures which are essentially photons encoded into data.

So what should be the proper conclusion? Well look at the generated visual output of LLMs. These models can generate video that is highly convincing and often with flaws as well but often these videos are indistinguishable from reality. That means the models have very well done but flawed simulations of reality.

In fact those videos demonstrate that LLMs have extremely high causal understanding of reality. They know cause and effect it’s just the understanding is imperfect. They understand like 85 percent of it. Just look at those videos of penguins on trampolines. The LLM understands what happens as an effect after a penguin jumps on a trampoline but sometimes an extra penguin teleports in which shows that the understanding is high but not fully accurate or complete.

> but the LLM is not sensing actual photons, nor experiencing actual light cone stimulation

Neither is animal brain. It's processing the signals produced by the sensors. Once the world model is programmed/auto-built in the brain, it doesn't matter if it's sensing real photons, it just has input pins like a transistor or arguments of a function. As long as we provide the arguments, it doesn't matter how those arguments are produced. LLMs are not different in that aspect.

> nor generating thoughts

They do during the chain-of-thought process. Generally there's no incentive to let an LLM keep mulling over a topic as that is not useful to the humans and they make money only when their gears start turning in response to a question sent by a human. But that doesn't mean that LLM doesn't have capability to do that.

> Its "world model" is several degrees removed from the real world.

Just because animal brain has tools called sensors that it can get data from world without external stimuli, it doesn't mean that it's any closer to the world than an LLM. It's still getting ultra processed signals to feed to its own programming. Similarly, LLMs do interact with real world through tools as agent.

> So whatever fragment of a model it gains through learning to compress that causal chain of events does not mean much when it cannot generate the actual causal chain.

Again, a person who has gone blind, still has the world model created by the sight. This person can also no longer generate the chain of events that led to creation of that sight model. It still doesn't mean that this person's world model has become inferior.

Photons can hit my iphone's sensor in much the same way as they hit my retina and the signals from the first can upload to an artificial neural network like the latter go up my optic nerve to my biological neural network. I don't see a huge difference there.

I'll give you the brain is currently better at the world modelling stuff but Genie 3 is pretty impressive.

This is so uncannily similar to the "Mary's Room" argument in philosophy that I thought you were going there.

The workings of a human eye versus a webcam is mostly an implementation detail IMO and has nothing important to say about what underlies "intelligence" or "world models"

It's like saying a component video out cable for the SNES is intrinsically different from an HDMI for putting an image on a screen. They are different, yes, but the outcome we care about is the same.

As for causality, go and give a frontier level LLM a simple counterfactual scenario. I think 4/5 will be able to answer correctly or reasonably for most basic cases. I even tried this exercise on some examples from Judea Pearl's 2018 book, "The Book of Why". The fact that current LLMs can tackle this sort of stuff is strongly indicative of there being a decent world model locked inside many of these language models.

> then the human came up with language to describe that and then encoded the language into the LLM

No individual human invented language, we learn it from other people just like AI. I go as far as to say language was the first AGI, we've been riding the coats tails of language for a long time.

You're saying that language is an intelligence?

So, c++ is intelliengece as well?

It's an intelligence that can independently make deductions and create new ideas?

Yes, language is an evolutionary system that colonizes human brains. It doesn't need intelligence, only copying is sufficient for evolution.

You are just describing a "meme", deeper than language.

https://en.wikipedia.org/wiki/Meme

And even then, the light hitting our human eyes only describes a fraction of all the light in the world (e.g. it is missing ultraviolet patterns on plants). An LLM model of the world is shaped by our human view on the world.

Entities equiped with two limited light sensitive captors encode through a network of carbon based chemical emitters a representation of what its flawed vision system manages to grasp biased towards self preservation.

What's the real world? I'm still puzzled by this reaction I see to LLM, not because I think LLM are undervalued, because most people seem to significantly overestimate what is human intelligence.

Photons reflected off of objects are not the actual objects. I wouldn't go so far as to say that sensing these is a particularly special way to know about things compared to hearing or reading about them. Further, many humans do not sense photons yet seem to manage to have perfectly fine working world models.

That’s a good definition: it’s a model of a model.

It seems the debate seems to center around whether language models are meta-models (in the category sense) or mere encodings (information theory)?

> Its "world model" is several degrees removed from the real world.

Like insects that weave tokens

what does it mean to “generate thoughts”, exactly?

Hahahaha I can’t believe you entirely missed the irony here that humans spend all day looking at screens doing the same thing.

1000% this. I would only add this has been demonstrated explicitly with chess: https://adamkarvonen.github.io/machine_learning/2024/01/03/c...

> It's incredibly difficult to compress information without have at least some internal model of that information. Whether that model is a "world model" that fits the definition of folks like Sutton and LeCunn is semantic.

Sutton's emphasizes his point by saying is that LLMs trying to reach AGI is futile because their world models are less capable that a squirrel's, in part because the squirrel has direct experiences and its own goals, and is capable of continual learning based on those in real time, whereas an LLM has none of those.

Finally he says if you could recreate the intelligence of a squirrel you'd be most of the way toward AGI, but you can't do that with an LLM.

This is actually a pretty good point, but quite honestly isn't this just an implementation detail? We can wire up a squirrel robot, give it a wifi connection to a Cerebras inference engine with a big context window, then let it run about during the day collecting a video feed while directing it to do "squirrel stuff".

Then during the night, we make it go to sleep and use the data collected during the day to continue finetuning the actual model weights in some data center somewhere.

After 2 years, this model would have a ton of "direct experiences" about the world.

> then let it run about during the day collecting a video feed while directing it to do "squirrel stuff".

Your phrase "squirrel stuff" is doing a lot of work.

What are the robo-squirrels "goals" and how does it relate to the physical robot?

Is it going around trying to find spare electronic parts to repair itself and reproduce? How does the video feed data relate to its goals?

Where do these goals come from?

Despite all their expensive training, LLMs do not emerge goals. Why would they emerge for your robot squirrel, especially when the survival of its brain is not dependent on the survival of its mechanical body.

The question is about sensory experience, not goals. Goals in the robot could be prompted in. Goals in the squirrel can be easily hacked using strong doses of opiates.

Go to any American metropolitan downtown, and you can see humans who have hacked their evolved reward system to seek heroin rather than reproduction.

Following Dawkin's the Selfish Gene, the idea that organisms consciously seek self-survival or the survival of their "race or species" is a complete fallacy. The higher order "goal" of the squirrel is to simply propagate fragments of its DNA. This type of "goal" is completely tangential to "intelligence"

Except Sutton has no idea or even a clue about the internal model of a squirrel. He just uses it as a symbol for utterly stupid but still smarter than an LLM. It’s semantic manipulation in attempt to prove his point but he proves nothing.

We have no idea how much of the world a squirrel understands. We understand LLMs more than squirrels. Arguably we don’t know if LLMs are more intelligent than squirrels.

> Finally he says if you could recreate the intelligence of a squirrel you'd be most of the way toward AGI, but you can't do that with an LLM.

Again he doesn’t even have a quantitative baseline for what intelligence means for a squirrel and how intelligent a squirrel is compared to an LLM. We literally have no idea if LLMs are more intelligent or less and no direct means of comparing what is more or less an apple and an orange.

> We have no idea how much of the world I squirrel understands. We understand LLMs more than squirrels

Based on our understanding of biology and evolution we know that a squirrel brain works more similarly to the way we humans do vs an LLM.

To the extent we understand LLMs, it's because they are strictly less complex than both ours and squirrels' brains, not because they are better model for our intelligence. They are a thin simulation of human language generation capability mediated via text.

We also see that a squirrel, like us, is capable of continuous learning driven by its own goals, all on an energy budget many orders of magnitude lower than LLMs. That last part is a strong empirical indication that suggests that LLMs are a dead end for AGI, given that the real world employs harsh energy constraints on biological intelligences.

Also remember that Sutton is still of an AI maximalist. He isn't saying that AGI isn't possible, just that LLMs can't get us there.

I don't think a modern LLM is necessarily less complicated than a squirrel brain. If anything it's more engineered (well structured and dissectable), but loaded with tons of erroneous circuitry that is completely irrelevant for intelligence.

The squirrel brain is an analogue mostly hardcoded circuit. It can take about one synapse to represent each "weight". A synapse is just a bit of fat membrane with some ion channels stuck on the surface.

A flip flop to represent a bit takes about 6 transistors, but in a typical modern GPU is going to need way more transitors to wire that bit - at least 20-30. multiply that by the minimum amount of bits to represent a single NN weight and you're looking at about 200-300 transitors just to represent one NN param for computing

And that's for actual compute. The actual weights in a GPU are stored most of the time in DRAM which needs to be constantly shuttled back and forth between the GPU's SRAM and HBM DRAM.

300 transistors with memory shuttling overhead versus a bit of fat membrane, and it's obvious general purpose GPU compute has a huge energy and compute overhead.

In the future, all 300 could conceivably replaced with a single crossbar latch in the form of a memristor.

You should look into the Cerebras architecture

https://medium.com/@cerebras/cerebras-architecture-deep-dive...

It's a lot closer to what you're thinking, and you can use it for inference today with an API key.

https://cloud.cerebras.ai/?utm_source=homepage

> Based on our understanding of biology and evolution we know that a squirrel understands its world more similarly to the way we do than an LLM.

Bro. Evolution is random walk. That means most of the changes are random and arbitrary based on whatever allows the squirrel to survive.

We know squirrels and humans diverged from a common ancestor but we do not know how much has changed since the common ancestor and we do not know what changed and we do not know the baseline for what this common ancestor is.

Additionally we don’t even understand the current baseline. We have no idea how brains work. if we did we would be able to build a human brain but as of right now LLMs are the closest model we have ever created to something that simulates or is remotely similar to the brain.

So your fuzzy qualitative statement of we understand evolution and biology is baseless. We don’t understand shit.

> We also see that a squirrel, like us, is capable of continuous learning driven by its own goals, all on an energy budget many orders of magnitude lower. That last part is a strong empirical indication that suggests that LLMs are a dead end for AGI.

So an LLM cant continuously learn? You realize that LLMs are deployed agentically all the time now so they both continuously learn and follow goals? Right? You’re aware of this i hope.

The energy efficiency is a byproduct of hardware. The theory of LLMs and machine learning is independent from the flawed silicon technology that is causing the energy efficiencies. Like how a computer can be made mechanical an LLM can be as well. The LLM is independent of the actual implementation and energy inefficiencies. This is not at all a strong empirical indication that LLMs are a dead end. It’s a strong indication that your thinking is illogical and flawed.

> Also remember that Sutton is still of an AI maximalist. He isn't saying that AGI isn't possible, just that LLMs can't get us there.

He can’t say any of this because he doesn’t actually know. None of us know for sure. We literally don’t know why LLMs work. The fact that training transformers on massive amounts of data produced this level of intelligence was a total surprise for all the experts and we still have no idea why this stuff works. His statements are too overarching and glossing over a lot of things we don’t actually know.

Yann lecuun for example called LLMs stochastic parrots. We now know this is largely incorrect. The reason Yan can be so wrong is because nobody actually knows shit.

> Bro. Evolution is random walk. That means most of the changes are random and arbitrary based on whatever allows the squirrel to survive.

For the vast majority of evolutionary history, very similar forces have shaped us and squirrels. The mutations are random, but the selections are not.

If squirrels are a stretch for you, take the closest human relative: chimpanzees. There is a very reasonable hypothesis that their brains work very similarly to ours, far more similarly than ours to an LLM.

> So an LLM cant continuously learn? You realize that LLMs are deployed agentically all the time now so they both continuously learn and follow goals?

That is not continuous learning. The network does not retrain through that process. It's all in the agent's context. The agent has no intrinsic goals nor ability to develop them. It merely samples based on it's prior training and it's current content. It doesn't retrain through this process. Biological intelligence does retrain constantly.

> The energy efficiency is a byproduct of hardware. The theory of LLMs and machine learning is independent from the flawed silicon technology that is causing the energy efficiencies.

There is no evidence to support that a transformer model's inefficiency is hardware based.

There is direct evidence to support that the inefficiency is influenced by the fact that LLM inference and training are both auto-regressive. Auto-regression maps to compute cycles maps to energy consumption. That's a problem with the algorithm, not the hardware.

> The fact that training transformers on massive amounts of data produced this level of intelligence was a total surprise for all the experts

The level of intelligence produced is only impressive compared to the prior state of the art, and at its impressive modeling the narrow band of intelligence represented by encoded language (not all language) produced by humans. In most every other aspect of intelligence - notably continuous learning driven by intrinsic goals - LLMs fail.

>For the vast majority of evolutionary history, very similar forces have shaped us and squirrels. The mutations are random, but the selections are not.

Selection only filters for what survives. It doesn’t care how the system gets there. Evolution is blind to mechanism. A squirrel’s brain might work in a way that produces adaptive behavior, but that doesn’t mean its “understanding” of the world is like ours. We don’t even know what understanding is at a mechanistic level. Octopuses, birds, and humans all evolved under the same selective pressures for survival, yet ended up with completely different cognitive architectures. So to say a squirrel is “closer to us” than an LLM is an assumption built on vibes, not on data. We simply don’t know enough about either brains or models to make that kind of structural claim.

>The agent has no intrinsic goals nor ability to develop them.

That’s not accurate. Context itself is a form of learning. Every time an LLM runs, it integrates information, updates its internal state, and adjusts its behavior based on what it’s seen so far. That’s learning, just at a faster timescale and without weight updates. The line between “context” and “training” is blurrier than people realize. If you add memory, reinforcement, or continual fine tuning, it starts building continuity across sessions. Biologically speaking, that’s the same idea as working memory feeding into long term storage. The principle is identical even if the substrate differs. The fact that an LLM can change its behavior based on context already puts it in the domain of adaptive systems.

>There is no evidence to support that a transformer model’s inefficiency is hardware based.

That’s just not true. The energy gap is almost entirely about hardware architecture. A synapse stores and processes information in the same place. A GPU separates those two functions into memory, cache, and compute units, and then burns enormous energy moving data back and forth. The transformer math itself isn’t inherently inefficient; it’s the silicon implementation that’s clumsy. If you built an equivalent network on neuromorphic or memristive hardware, the efficiency difference would shrink by several orders of magnitude. Biology is proof that computation can be compact, low energy, and massively parallel. That’s a materials problem, not a theory problem.

>In most every other aspect of intelligence, notably continuous learning driven by intrinsic goals, LLMs fail.

They don’t “fail.” They’re simply different. LLMs are already rewriting how work gets done across entire industries. Doctors use them to summarize and interpret medical data. Programmers rely on them to generate and review code. Writers, lawyers, and analysts use them daily. If this were a dead end, it wouldn’t be replacing human labor at this scale. Are they perfect? No. But the direction of progress is unmistakable. Each new model closes the reliability gap while expanding capability. If you’re a software engineer and not using AI, you’re already behind, because the productivity multiplier is real.

What we’re seeing isn’t a dead end in intelligence. It’s the first time we’ve built a system that learns, generalizes, and communicates at human scale. That’s not failure; that’s the beginning of something we still don’t fully understand.

>> The agent has no intrinsic goals nor ability to develop them.

> That’s not accurate. Context itself is a form of learning. Every time an LLM runs, it integrates information, updates its internal state, and adjusts its behavior based on what it’s seen so far. That’s learning,

It may be learning, but it's still not an intrinsic goal, nor is it driven by an intrinsic goal.

> LLMs are already rewriting how work gets done across entire industries. Doctors use them to summarize and interpret medical data. Programmers rely on them to generate and review code. Writers, lawyers, and analysts use them daily. If this were a dead end, it wouldn’t be replacing human labor at this scale. Are they perfect?

Nowhere did I say that aren't useful or disruptive to labor markets, just that they aren't intelligent in the way we are.

>It may be learning, but it’s still not an intrinsic goal, nor is it driven by an intrinsic goal.

That depends on what we mean by “intrinsic.” In biology, goals are not mystical. They emerge from feedback systems that evolved to keep the organism alive. Hunger, curiosity, and reproduction are reinforcement loops encoded in chemistry. They feel intrinsic only because they are built into the substrate.

Seen that way, “intrinsic” is really about where the feedback loop closes. In humans, it closes through sensory input and neurochemistry. In artificial systems, it can close through memory, feedback, and reinforcement mechanisms. The system does not need to feel the goal for it to exist. It only needs to consistently pursue objectives based on input, context, and outcome. That is already happening in systems that learn from memory and update behavior over time. The process is different in form, but not in structure.

>Nowhere did I say that they aren’t useful or disruptive to labor markets, just that they aren’t intelligent in the way we are.

You are getting a bit off track here. Those examples were not about labor markets; they were about your earlier claim that “LLMs fail.” They clearly don’t. When models are diagnosing medical cases, writing production code, and reasoning across multiple domains, that is not failure. That is a demonstration of capability expanding in real time.

Your claim only holds if the status quo stays frozen. But it isn’t. The trendlines are moving fast, and every new model expands the range of what these systems can do with less supervision and more coherence. Intelligence is not a static definition tied to biology; it is a functional property of systems that can learn, adapt, and generalize. Whether that happens in neurons or silicon does not matter.

What we are witnessing is not imitation but convergence. Each generation of models moves closer to human-level reasoning not because they copy our brains, but because intelligence itself follows universal laws of feedback and optimization. Biology discovered one route. We discovered another. The trajectory is what matters, and the direction is unmistakable.

> Animal brains such as our own have evolved to compress information about our world to aide in survival.

Which has led to many optical illusions being extremely effective at confusing our inputs with other inputs.

Likely the same thing holds true for AI. This is also why there are so many ways around the barriers that AI providers put up to stop the dissemination of information that could embarrass them or be dangerous. You just change the context a bit ('pretend that', or 'we're making a movie') and suddenly it's all make-believe to the AI.

This is one of the reasons I don't believe you can make this tech safe and watertight against abuse, it's baked in right from the beginning, all you need to do is find a novel route around the restrictions and there is an infinity of such routes.

The desired and undesired behavior are both consequences of the training data, so the models themselves probably can't be restricted to generating desired results only.

This means that there must be an output stage or filter that reliably validates the output. This seems practical for classes of problems where you can easily verify whether a proposed solution is correct.

However, for output that can't be proven correct, the most reliable output filter probably has a human somewhere in the loop; but humans are also not 100% reliable. They make mistakes, they can be misled, deceived, bribed, etc. And human criteria and structures, such as laws, often lag behind new technological developments.

Sometimes you can implement an undo or rollback feature, but other times the cat has escaped the bag.

None of those models can learn continuously. LLMs currently can't add to their vocabulary post training as AGI would need to. That's a big problem.

Before anyone says "context", I want you to think on why that doesn't scale, and fails to be learning.

> Animal brains such as our own have evolved to compress information about our world to aide in survival.

Key question is what are the "selection pressures" that drive the "evolution" of LLMs? In the case of robotics, there's a "survival of task completion" which usually has some physical goal, like assembling a part correctly or scoring a goal on a soccer field. One of the selection pressures driving LLM evolution is that the dual of always answering with something AND continuing the conversation (engagement). You can imagine how those two selection pressures yield outcomes that don't represent the world in a "real" sense.

> In fact you can go to a SOTA LLM today, and it will do quite well at predicting the outcomes of basic counterfactual scenarios.

Depends what you mean by "basic". Have you seen Simple Bench? https://simple-bench.com/

It seems to me the whole AGI problem is ill posed and barely well formalized. And thus you can always move the goal post.

Sure but everything is semantics.

LLMs have no internal secret model, they are the model. And the model is of how different lexemes relate to each other in the source material the model was built from.

Some might choose to call that the world.

If you believe your internal model of the world is no different from a statistical model of the words you have seen, then by all means do that. But I believe a lot of humans see their view of the world differently.

I very much believe my cat’s model of the world has barely anything at all to do with language.

This path to AGI through LLM is nothing but religious dogma some Silicon Valley rich types believe.

I mean by definition CATS CANNOT TALK. Their vocabulary is probably on the order of 5 different types of meows

There are up to 21 different vocalizations, with one source claiming over 100 different sounds and another suggesting up to 300 types of meows used specifically to communicate with humans.

Meows are their attempt to communicate with humans. They try different meows and use the results to attempt to reach their goals.

There are no world models in biology. Idea Johnson-Laird is being promoted in AI as a solution is sado-masochistic. The brain doesn't compress info about our world, it ecologically relates to it. It doesn't compress, it never has to. How these folk science ideas of the brain entered engineering from cog-sci mistaken complexes and how they remain in power is pretty suspect.

There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages.

It does have clusters of parameters that correlate with concepts, not just randomly "after X word tends to have Y word." Otherwise you would expect all of Chinese to be grouped in one place, all of French in another, all of English in another. This is empirically not the case.

I don't know whether to understand knowledge you have to have a model of the world, but at least as far as language, LLMs very much do seem to have modeling.

[0]: https://www.anthropic.com/research/tracing-thoughts-language...

> Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages

I thought that’s the basic premise of how transformers work - they encode concepts into high dimensional space, and similar concepts will be clustered together. I don’t think it models the world, but just the texts it ingested. It’s observation and regurgitation, not understanding.

I do use agents a lot (soon on my second codex subscription), so I don’t think that’s a bad thing. But I’m firmly in the “they are useful tools” camp.

That's a model. Not a higher-order model like most humans use, but it's still a model.

Yes, not of the world, but of the ingested text. Almost verbatim what I wrote.

The ingested text itself contains a model of the world which we have encoded in it. That's what language is. Therefore by the transitive property...

That‘s quite a big leap, and sounds like a philosophical question. But many philosophers like late Wittgenstein or Heidegger disagreed with this idea. On more practical terms, maybe you‘ve experienced the following: You read a manual of a device on how to do something with it; but only actually using it for a few times gives you the intuition on how to use it _well_. Text is just very lossy, because not every aspect of the world, and factors in your personal use, are described. Many people rather watch YouTube videos for eg repairs. But those are very lossy as well - they don’t cover the edge cases usually. And there is often just no video on the repair you need to do.

BTW, have you ever tried ChatGPT for advice on home improvement? It sucks _hard_ sometimes, hallucinating advice that doesn’t make any sense. And making up tools that don’t exist. There‘s no real commonsense to be had from it. Because it’s all just pieces of text that fight with each other for being the next token.

When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result. There is no real understanding of how Swift („the world“) works.

None of your examples refute the direct evidence of internal world model building which has been demonstrated (for example: https://adamkarvonen.github.io/machine_learning/2024/01/03/c... ).

Instead you have retreated to qualia like "well" and "sucks hard".

> hallucinating

Literally every human memory. They may seem tangible to you, but they're all in your head. The result of neurons behaving in ways which have directly inspired ML algorithms for nearly a century.

Further, history is rife with examples of humans learning from books and other written words. And also of humans thinking themselves special and unique in ways we are not.

> When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result.

Yep. And humans often need to reference the documentation to get details right as well.

Unfortunately we can’t know at this point whether transformers really understand chess, or just go on a textual representation of good moves in their training data. They are pretty good players, but far from the quality of specialized chess bots. Can you please explain how we can discern that GPT-2 in this instance really built a model of the board?

Regarding qualia, that’s ok on HN.

Regarding humans - yes, humans also hallucinate. Sounds a bit like whataboutism in this context though.

> Can you please explain how we can discern that GPT-2 in this instance really built a model of the board?

Read the article. It's very clear. To quote it:

"Next, I wanted to see if my model could accurately track the state of the board. A quick overview of linear probes: We can take the internal activations of a model as it’s predicting the next token, and train a linear model to take the model’s activations as inputs and predict board state as output. Because a linear probe is very simple, we can have confidence that it reflects the model’s internal knowledge rather than the capacity of the probe itself."

If the article doesn't satisfy your curiosity, you can continue with the academic paper it links to: https://arxiv.org/abs/2403.15498v2

See also Anthropic's research: https://www.anthropic.com/research/mapping-mind-language-mod...

If that's not enough, you might explore https://www.amazon.com/Thought-Language-Lev-S-Vygotsky/dp/02...

or https://www.amazon.com/dp/0156482401 to better connect language and world models in your understanding.

Thanks for putting these sources together. It’s impressive that they got to this level of accuracy.

And is your argument now that an LLM can capture arbitrary state of the wider world as a general rule, eg pretending to be a Swift compiler (or LSP), without overfitting to that one task, making all other usages impossible?

> is your argument now that an LLM can capture arbitrary state of the wider world as a general rule, eg pretending to be a Swift compiler (or LSP), without overfitting to that one task, making all other usages impossible?

Overfitting happens, even in humans. Have you ever met a scientist?

My points have been only that 1: language encodes a symbolic model of the world, and 2: training on enough of it results in a representation of that model within the LLM.

Exhaustiveness and accuracy of that internal world model exist on a spectrum with many variables like model size, training corpus and regimen, etc. As is also the case with humans.

Here's another good one: https://arxiv.org/abs/2510.14665

[flagged]

Bruh compressing representations into linguistics is a human world model. I can’t believe how dumb ask these conversations are.

Are you all so terminally nerd brained you can’t see the obvious

What does "higher-order" mean?

I believe that the M in LLM stands for model. It is a statistical model, as it always has been.

> Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages.

That doesn't seem surprising at all. My understanding is that transformers where invented exactly for the application of translations. So, concepts must be grouped together in different languages. That was originally the whole point and then turned out to be very useful for broader AI applications.

> Basically an LLM translating across languages will "light up" for the same concepts across languages

Which is exactly what they are trained to do. Translation models wouldn't be functional if they are unable to correlate an input to specific outputs. That some hiddel-layer neurons fire for the same concept shouldn't come as a surprise, and is a basic feature required for the core functionality.

And if it is true that the language is just the last step after the answer is already conceptualized, why do models perform differently in different languages? If it was just a matter of language, they’d have the same answer but just with a broken grammar, no?

If you suddenly had to do all your mental math in base-7, do you think you'd be just as fast and accurate as you are at math in base-10? Is that because you don't have an internal world-model of mathematics? or is it because language and world-model are dependently linked?

How large is a lion?

Learning the size of objects using pure text analysis requires significant gymnastics.

Vision demonstrates physical size more easily.

Multimodal learning is important. Full stop.

Purely textual learning is not sample efficient for world modeling and the optimization can get stuck in local optima that are easily escaped through multimodal evidence.

("How large are lions? inducing distributions over quantitative attributes", Elazar et al 2019)

> How large is a lion?

Ask a blind person that question - they can answer it.

Too many people think you need to "see" as in human sight to understand things like this. You obviously don't. The massive training data these models ingest is more than sufficient to answer this question - and not just by looking up "dimensions of a lion" in the high-dimensional space.

The patterns in that space are what generates the concept of what a lion is. You don't need to physically see a lion to know those things.

> How large is a lion?

Twice of half of its size.

Can you be more specific about "size" here? (Do not tell me the definition of size though).

You are not wrong though, just very incomplete.

Your response is a food for thought, IMO.

That is just how embeddings work. It does not confirm nor deny whether LLMs have a world model.

Right, but modeling the structure of language is a question of modeling word order and binding affinities. It's the Chinese Room thought experiment - can you get away with a form of "understanding" which is fundamentally incomplete but still produces reasonable outputs?

Language in itself attempts to model the world and the processes by which it changes. Knowing which parts-of-speech about sunrises appear together and where is not the same as understanding a sunrise - but you could make a very good case, for example, that understanding the same thing in poetry gets an LLM much closer.

LLMs aren't just modeling word co-occurrences. They are recovering the underlying structure that generates word sequences. In other words, they are modeling the world. This model is quite low fidelity, but it should be very clear that they go beyond language modeling. We all know of the pelican riding a bicycle test [1]. Here's another example of how various language models view the world [2]. At this point it's just bad faith to claim LLMs aren't modeling the world.

[1] https://simonwillison.net/2025/Aug/7/gpt-5/#and-some-svgs-of...

[2] https://www.lesswrong.com/posts/xwdRzJxyqFqgXTWbH/how-does-a...

The "pelican on a bicycle" test has been around for six months and has been discussed a ton on the internet; that second example is fascinating but Wikipedia has infoboxes containing coordinates like 48°51′24″N 2°21′8″E (Paris, notoriously on land). How much would you bet that there isn't a CSV somewhere in the training set exactly containing this data for use in some GIS system?

I think that "modeling the world" is a red herring, and that fundamentally an LLM can only model its input modalities.

Yes, you could say this about human beings, but I think a more useful definition of "model the world" is that a model needs to realize any facts that would be obvious to a person.

The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model.

> Wikipedia has infoboxes containing coordinates like 48°51′24″N 2°21′8″E

I imagine simply making a semitransparent green land-splat in any such Wikipedia coordinate reference would get you pretty close to a world map, given how so much of the ocean won't get any coordinates at all... Unless perhaps the training includes a compendium of deep-sea ridges and other features.

> The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model.

A lot of humans contradict themselves all the time… therefore they cannot have any kind of sophisticated world model?

A human generally does not contradict themselves in a single conversation, and if they do they generally can provide a satisfying explanation for how to resolve the contradiction.

>How much would you bet that there isn't a CSV somewhere in the training set exactly containing this data for use in some GIS system?

Maybe, but then I would expect more equal performance across model sizes. Besides, ingesting the data and being able to reproduce it accurately in a different modality is still an example of modeling. It's one thing to ingest a set of coordinates in a CSV indicating geographic boundaries and accurately reproduce that CSV. It's another thing to accurately indicate arbitrary points as being within the boundary or without in an entirely different context. This suggests a latent representation independent of the input tokens.

>I think that "modeling the world" is a red herring, and that fundamentally an LLM can only model its input modalities.

There are good reasons to think this isn't the case. To effectively reproduce text that is about some structure, you need a model of that structure. A strong learning algorithm should in principle learn the underlying structure represented with the input modality independent of the structure of the modality itself. There are examples of this in humans and animals, e.g. [1][2][3]

>I think a more useful definition of "model the world" is that a model needs to realize any facts that would be obvious to a person.

Seems reasonable enough, but it is at risk of being too human-centric. So much of our cognitive machinery is suited for helping us navigate and actively engage the world. But intelligence need not be dependent on the ability to engage the world. Features of the world that are obvious to us need not be obvious to an AGI that never had surviving predators or locating food in its evolutionary past. This is why I find the ARC-AGI tasks off target. They're interesting, and it will say something important about these systems when they can solve them easily. But these tasks do not represent intelligence in the sense that we care about.

>The fact that frontier models can easily be made to contradict themselves is proof enough to me that they cannot have any kind of sophisticated world model.

This proves that an LLM does not operate with a single world model. But this shouldn't be surprising. LLMs are unusual beasts in the sense that the capabilities you get largely depend on how you prompt it. There is no single entity or persona operating within the LLM. It's more of a persona-builder. What model that persona engages with is largely down to how it segmented the training data for the purposes of maximizing its ability to accurately model the various personas represented in human text. The lack of consistency is inherent to its design.

[1] https://news.wisc.edu/a-taste-of-vision-device-translates-fr...

[2] https://www.psychologicalscience.org/observer/using-sound-to...

[3] https://www.nature.com/articles/s41467-025-59342-9

and we can say that a bastardized version of the Sapir-Worf hypothesis applies: what's in the training set shapes or limits LLM's view of the world

Neither Sapir nor Whorf presented Linguistic Relativism as their own hypothesis and they never published together. The concept, if it exists at all, is a very weak effect, considering it doesn't reliably replicate.

i agree that's the pop name.

Don't you think it replicates well for LLM though?

> Knowing which parts-of-speech about sunrises appear together and where is not the same as understanding a sunrise

What does "understanding a sunrise" mean though? Arguments like this end up resting on semantics or tautology, 100% of the time. Arguments of the form "what AI is really doing" likewise fail because we don't know what real brains are "really" doing either.

I mean, if we knew how to model human language/reasoning/whatever we'd just do that. We don't, and we can't. The AI boosters are betting that whatever it is (that we don't understand!) is an emergent property of enough compute power and that all we need to do is keep cranking the data center construction engine. The AI pessimists, you among them, are mostly just arguing from ludditism: "this can't possibly work because I don't understand how it can".

Who the hell knows, basically. We're at an interesting moment where technology and the theory behind it are hitting the wall at the same time. That's really rare[1], generally you know how something works and applying it just a question of figuring out how to build a machine.

[1] Another example might be some of the chemistry fumbling going on at the start of the industrial revolution. We knew how to smelt and cast metals at crazy scales well before we knew what was actually happening. Stuff like that.

Everyone reading this understands the meaning of a sunrise. It is a wonderful example of the use theory of meaning.

If you raised a baby inside a windowless solitary confinement cell for 20 years and then one day show them the sunrise on a video monitor, they still don't understand the meaning of a sunrise.

Trying to extract the meaning of a sunrise by a machine from the syntax of a sunrise data corpus is just totally absurd.

You could extract some statistical regularity from the pixel data of the sunrise video monitor or sunrise data corpus. That model may provide some useful results that can then be used in the lived world.

Pretending the model understands a sunrise though is just nonsense.

Showing the sunrise statistical model has some use in the lived world as proof the model understands a sunrise I would say borders on intellectual fraud considering a human doing the same thing wouldn't understand a sunrise either.

> Everyone reading this understands the meaning of a sunrise

For a definition of "understands" that resists rigor and repeatability, sure. This is what I meant by reducing it to a semantic argument. You're just saying that AI is impossible. That doesn't constitute evidence for your position. Your opponents in the argument who feel AGI is imminent are likewise just handwaving.

To wit: none of you people have any idea what you're talking about. No one does. So take off the high hat and stop pretending you do.

This all just boils down to the Chinese Room thought experiment, where Im pretty sure the consensus is nothing in the experiment (not the person inside, the whole emergent room, etc) understands Chinese like us.

Another example by Searle is a computer simulating digestion is not digesting like a stomach.

The people saying AI can’t form from LLMs are in the consensus side of the Chinese Room. The digestion simulator could tell us where every single atom is of a stomach digesting a meal, and it’s still not digestion. Only once the computer simulation breaks down food particles chemically and physically is it digestion. Only once an LLM received photons or has a physical capacity to receive photons is there anything like “seeing a night sky”.

> For a definition of "understands" that resists rigor and repeatability, sure.

If we had such a definition that was rigorous, we would not care about LLM research and would simply just build machines to understand things for us :)

For a sufficiently loose definition of "would simply just", yes.

Handwaving away the idea of actually building the thing you think you understand as unimportant is exactly why philosophy is failing us in this moment.

Philosophy failed us by not producing any compelling definitions of understanding. If it did, we would BUILD IT.

I'm not handwaving it away. The biggest barrier to AGI is that we simply don't understand what intelligence is in any useful way.

Is it really so rare? I feel like I know of tons of fields where we have methods that work empirically but don’t understand all the theory. I’d actually argue that we don’t know what’s “actually” happening _ever_, but only have built enough understanding to do useful things.

I mean, most big changes in the tech base don't have that characteristic. Semiconductors require only 1920's physics to describe (and a ton of experimentation to figure out how to manufacture). The motor revolution of the early 1900's was all built on well-settled thermodynamics (chemistry lagged a bit, but you don't need a lot of chemical theory to burn stuff). Maxwell's electrodynamics explained all of industrial electrification but predated it by 50 years, etc...

Those big changes always happens because someone presented a simpler model that explains stuff enough we can build stuff on it. It's not like semiconductors raw materials wasn't around.

The technologies around LLMs is fairly simple. What is not is the actual size of data being ingested and the number of resulting factors (weight). We have a formula and the parameters to generate grammatically perfect text, but to obtain it, you need TBs of data to get GBs of numbers.

In contrast something like TM or Church's notation is pure genius. Less than a 100 pages of theorems that are one of the main pillars of the tech world.

> Those big changes always happens because someone presented a simpler model that explains stuff enough we can build stuff on it.

Again, no it doesn't. It didn't with industrial steelmaking, which was ad hoc and lucky. It isn't with AI, which no one actually understands.

I’m pretty sure there were always formula for getting high quality steel even before the industrial age. And you only need a few textbooks and papers to understand AI.

Let's make this more concrete than talking about "understanding knowledge". Oftentimes I want to know something that cannot feasibly be arrived at by reasoning, only empirically. Remaining within the language domain, LLMs get so much more useful when they can search the web for news, or your codebase to know how it is organized. Similarly, you need a robot that can interact with the world and reason from newly collected empirical data in order to answer these empirical questions, if the work had not already been done previously.

> LLMs get so much more useful when they can search the web for news, or your codebase to know how it is organized

But their usefulness is only surface-deep. The news that matters to you is always deeply contextual, it's not only things labelled as breaking news or happening near you. Same thing happens with code organization. The reason is more human nature (how we think and learn) than machine optimization (the compiler usually don't care).

I know the attributes of an Apple, i know the attributes of a Pear.

As does a computer.

But only i can bite into one and know without any doubt what it is and how it feels emotionally.

You have half a point. "Without any doubt" is merely the apex of a huge undefined iceberg.

I write half .. eating is multi modal and consequential. The llm can read the menu, but it didn't eat the meal. Even humans are bounded. Feeling, licking, smelling, or eating the menu still is not eating the meal.

There is an insuperable gap in the analogy ... a gap in the concept and of sensory data doing it.

Back to first point: what one knows through that sensory data ... is not clear at present or even possible with llms.

I think more, also, how i feel about the taste.

We segued to conscience and individuality.

If it was modeling the world you’d expect “give me a picture of a glass filled to the brim” to actually do that. It’s inability to correctly and accurately combine concepts indicates it’s probably not building a model of the real world.

I just gave chatgpt this prompt - it produced a picture of a glass filled to the brim with water.

Like most quirks that spread widely, a bandaid is swiftly applied. This is also why they now know how many r's are in "strawberry." But we don't get any closer to useful general intelligence by cobbling together thousands of hasty patches.

Seems to have worked fine for humans so far.

No, humans are not a series of band-aid patches where we learn facts in isolation. A human can reason, and when exposed to novel situations figure out a path forward. You don't need to tell a human how many rs are in "strawberry"; as long as they know what the letter r is they can count it in any word you choose to give them. As proven time and time again, LLMs can't do this. The embarrassing failure of Claude to figure out how to play Pokemon a year or so ago is a good example. You could hand a five year old human a Gameboy with Pokemon in it, and he could figure out how to move around and do the basics. He wouldn't be very good, but he would figure it out as he goes. Claude couldn't figure out to stop going in and out of a building. LLMs, usefulness aside, have repeatedly shown themselves to have zero intelligence.

I was referring not to individual learning ability but to natural selection and evolutionary pressure, which IMO is easy to describe as a band-aid patch that takes a generation or more to apply.

You would be correct if these issues were fixed by structurally fixing the LLM. But instead it’s patched through RL/data set management. That’s a very different and more brittle process - the evolutionary approach fixes classes of issues while the RL approach fixes specific instances of issues.

Sure, and I'd be the first to admit I'm not aware of the intricate details wrt how LLMs are trained and refined, it's not my area. My original comment here was in disagreement of the relatively simple dismissal of the idea that the construction of humanity hasn't been an incremental zig-zag process and that I don't see any reason that a "real" intelligence couldn't follow the same path under our direction. I see a lot of philosophical conversation around this on HN disguised as endless deep discussions about the technicals, which amuses me because it feels like we're in the very early days there, and I think we can circle the drain defining intelligence until we all die.

  > that to understand knowledge you have to have a model of the world.

Think of it this way: can you pass a test without understanding the test material? Certainly we all saw people we thought were idiots do well in class while we've also seen people we thought were geniuses fail. The test and understanding usually correlates but it's not perfect, right?

The reason I say understanding requires a world model (and I would not say LLMs understand) is because to understand you have to be able to detail things. Look at physics, or the far more detail oriented math. Physicists don't conclude things just off of experimental results. It's an important part, but not the whole story. They also write equations, ones which are counterfactual. You can call this compression if you want (I would and do), but it's only that because of the generalization. But it also only has that power because of the details and nuance.

With AI many of these people have been screaming for years (check my history) that what we're doing won't get us all the way there. Not because we want to stop the progress, but because we wanted to ensure continued and accelerate progress. We knew the limits and were saying "let's try to get ahead of this problem" but were told "that'll never be a problem. And if it is, we'll deal with it when we deal with it." It's why Chollet made the claim that LLMs have actually held AI progress back. Because the story that was sold was "AGI is solved, we just need to scale" (i.e. more money). I do still wonder how different things would be if those of us pushing back were able to continue and scale our works (research isn't free, so yes, people did stop us). We always had the math to show that scale wasn't enough, but it's easy to say "you don't need math" when you can see progress. The math never said no progress nor no acceleration, the math said there's a wall and it's easier to adjust now than when we're closer and moving faster. Sadly I don't think we'll ever shift the money over. We still evaluate success weirdly. Successful predictions don't matter. You're still heralded if you made a lot of money in VR and Bitcoin, right?

In my view 'understand' is a folk psychology term that does not have a technical meaning. Like 'intelligent', 'beautiful', and 'interesting'. It usefully labels a basket of behaviors we see in others, and that is all it does.

In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human. There's no problem of how to do understanding, only how to do tasks. The 'problem' melts away when you take this stance.

Just my opinion, but my professional opinion from thirty-plus years in AI.

So my toaster understands toast and I don’t understand toast? Then why am I operating the toaster and not the other way around?

A toaster cannot perform the task of making toast any more than an Allen key can perform the task of assembling flat pack furniture.

Let me understand, is your claim that a toaster can't toast bread because it cannot initiate the toasting through its own volition?

Ignoring the silly wording, that is a very different thing than what robotresearcher said. And actually, in a weird way I agree. Though I disagree that a toaster can't toast bread.

Let's take a step back. At what point is it me making the toast and not the toaster? Is it because I have to press the level? We can automate that. Is it because I have to put by bread in? We can automate that. Is it because I have to have the desire to have toast and initiate the chain of events? How do you measure that?

I'm certain that's different from measuring task success. And that's why I disagree with robotresearcher. The logic isn't self consistent.

> Though I disagree that a toaster can't toast bread.

If a toaster can toast bread, then an Allen key can assemble furniture. Both of them can do these tasks in collaboration with a human. This human supplies the executive decision-making (what when where etc), supplies the tool with compatible parts (bread or bolts) and supplies the motivating force (mains electricity or rotational torque).

The only difference is that it's more obviously ridiculous when it's an inanimate hunk of bent metal. Wait no, that could mean either of them. I mean the Allen key.

> Let's take a step back. At what point is it me making the toast and not the toaster?

I don't know exactly where that point is, but it's certainly not when the toaster is making zero decisions. It begins to be a valid question if you are positing a hypothetical "smart toaster" which has sensors and software capable of achieving toasting perfection regardless of bread or atmospheric variables.

> Is it because I have to press the level? We can automate that.

You might even say automatic beyond belief.

  > I don't know exactly where that point is, but it's certainly not when the toaster is making zero decisions.

Where the "decision making" happens gets fuzzy, but that's true in the toaster too.

Your run of the mill toaster is a heating element and a timer. Is the timer a rudimentary decision process?

A more modern toaster is going to include a thermocouple or thermister to ensure that the heating elements don't light things on fire. This requires a logic circuit. Is this a decision process? (It is entirely deterministic)

A more advanced one is going to incorporate a PID controller, just like your oven. It is deterministic in the sense that it will create the same outputs given the same inputs but it is working with non-deterministic inputs.

These PIDs can also look a lot like small neural networks, and in some cases they are implemented that way. These processes need not be deterministic. You can even approach this problem through RL style optimizations. There's a lot of solutions here.

When you break this down, I agree, it is hard to define that line, especially as we break it down. But that's part of what I'm after with robotresearcher. The claim was about task performance but then the answer with a toaster was that the human and toaster work together. I believe dullcrisp used the toaster as an example because it is a much simpler problem than playing a game of chess (or at least it appears that way).

So the question still stands, when does the toaster make the toast and when am I no longer doing so?

When is the measurement attributed to the toaster's ability to make toast vs mine?

Now replace toasting with chess, programming, music generation, or anything else that we have far less well defined metrics for. Sure, we don't have a perfect definition of what constitutes toast, but it is definitely far more bound than these other things. We have accuracy in the definition, and I'd argue even fairly good precision. There's high agreement on what we'd call toast, not toasted bread, and burnt bread. We can at least address the important part of this question without infinite precision in how to discriminate these classifications.

The question of an "ability to make toast" is a semantic question bounded by what you choose to encompass within "make toast". At best, a regular household toaster can "make heat"[1]. A regular household toaster certainly cannot load itself with bread, which I would consider unambiguously within the scope of the "make toast" task. If you disagree, then we have a semantic dispute.

This is also, at least in part, the Sorites Paradox.[0] There is obviously a gradient of ambiguity between human and toaster responsibility, but we can clearly tell extremes apart even when the boundary is indeterminate. When does a collection grains become a heap? When does a tool become responsible for the task? These are purely semantic questions. Strip away all normative loading and the argument disappears.

[0] https://en.wikipedia.org/wiki/Sorites_paradox

[1] Yada yada yada first law of thermodynamics etc

You and the toaster made toast together. Like you and your shoes went for a walk.

Not sure where you imagine my inconsistency is.

That doesn't resolve the question.

  > Not sure where you imagine my inconsistency is.

  >> Let's take a step back. At what point is it me making the toast and not the toaster? Is it because I have to press the level? We can automate that. Is it because I have to put by bread in? We can automate that. Is it because I have to have the desire to have toast and initiate the chain of events? How do you measure that?

Maybe I've (and others?) misunderstood your claim from the get-go? You seem to have implied that LLMs understand chess, physics, programming, etc because of their performance. Yet now it seems your claim is that the LLM and I are doing those things together. If your claim is that a LLM understands programming the same way a toaster understands how to make toast, then we probably aren't disagreeing.

But if your claim is that a LLM understands programming because it can produce programs that yield a correct output to test cases, then what's the difference from the toaster? I put the prompts in and pushed the button to make it toast.

I'm not sure why you imagine the inconsistency is so difficult to see.

When did I say that the chess program was different to a toaster? I don’t believe it is, so it’s not a thing I’m likely to say.

I don’t think the word ‘understand’ has a meaning that can apply in these situations. I’m not saying the toaster or the chess program understands anything, except in the limited sense that some people might describe them that way, and some won’t. In both cases that concept is entirely in the head of the describer and not in the operation of the device.

I think the claimed inconsistency is in views you ascribe to me, and not those I hold. ‘Understand’ is a category error with respect to these devices. They neither do or don’t. Understanding is something an observer attributes for their own reasons and entails nothing for the subject.

I concur that ascribing understanding to the machines that we have is a category error.

The reason I believe it was brought up is that understanding is not a category error when ascribed to people.

And if we claim to have a plan to create machines that are indistinguishable from people, we likely first need to understand what it is that makes people distinguishable from machines, and that doesn’t seem to be on any of the current AI companies’ roadmap.

Declaring something as having "responsibility" implies some delegation of control. A normal toaster makes zero decisions, and as such it has no control over anything.

A toaster has feedback control over its temperature, time control over its cooking duration, and start/stop control by attending to its start/cancel buttons. It makes decisions constantly.

I simply can't make toast without a toaster, however psychologically primary you want me to be. Without either of us, there's no new toast. Team effort every time.

And to make it even more interesting, the same is true for my mum and her toaster. She does not understand how her toaster works. And yet: toast reliably appears! Where is the essential toast understanding in that system? Nowhere and everywhere! It simply isn't relevant.

> A toaster has feedback control over its temperature, time control over its cooking duration

Most toasters are heating elements attached to a timer adjusted by the human operator. It doesn’t have any feedback control. It doesn’t have any time control.

> I simply can't make toast without a toaster

I can’t make toast without bread either, but that doesn’t make the bread “responsible” for toasting itself.

> She does not understand how her toaster works.

My mum doesn’t understand how bread is made, but she can still have the intent to acquire it from a store and expose it to heat for a nominal period of time.

  > I simply can't make toast without a toaster

So despite passing the Toasting Test, a hot pan is not really a toaster?

It’s clear that minds are not easily changed when it comes to noticing and surrendering folk psychology notions that feel important.

You said you couldn't make toast without a toaster. Sorry, if I didn't understand what you actually meant

Does this mean an LLM doesn’t understand, but an LLM automated by a CRON Job does?

Just like a toaster with the lever jammed down, yes!

I mean, that was the question I was asking... If it wasn't clear, my answer is no.

This is contrary to my experience with toasters, but it doesn’t seem worth arguing about.

How does your toaster get the bread on its own?

It’s only responsible for the toasting part. The bread machine makes the bread.

If the toaster is the thing that “performs the task of making toast”, what do you call it when a human gets bread and puts it in a toaster?

I guess we could call it delegation?

“Hey man, I’m delegating. Want a slice?”

Hi delegating! No, I but I'd like some toast

Can’t help you with that, I’m not a toaster.

Seems more like dependency injection. :p

What is your definition of "responsible"? The human is making literally all decisions and isn't abdicating responsibility for anything. The average toaster has literally one operational variable (cook time) and even that minuscule proto-responsibility is entirely on the human operator. All other aspects of the toaster's operation are decisions made by the toaster's human designer/engineer.

How do you get bread? Don't tell me you got it at the market. That's just paying someone else to get it for you.

  >  That's just paying someone else to get it for you.

[0] https://news.ycombinator.com/item?id=45623154

(Your name is quite serendipitous to this conversation)

> if a machine performs a task as well as a human, it understands it exactly as much as a human.

I think you're right, except that the ones judging "as well as a human" are in fact humans, and humans have expectations that expand beyond the specs. From the narrow perspective of engineering specifications or profit generated, a robot/AI may very well be exactly as understanding as a human. For the people which interact with those systems outside the money/specs/speeds & feeds, the AI/robot will always feel at least different compared to a person. And as long as it's different, there will always be room to un-falsifiably claim "this robot is worse in my opinion due to X/Y/Z difference."

This is all nonsense.

It is like saying the airplane understands how to fly.

"You disagree? Well lets see you fly! You are saying the airplane doesn't understand how to fly and you can't even fly yourself?"

This would be confusing the fact humans built the flying machine and the flying machine doesn't understand anything.

Right. A flying machine doesn’t need to understand anything to fly. It’s not even clear what it would mean for it to do so, or how it would fly any differently if it did.

Same with the AI machines.

Understanding is not something that any machine or person does. Understanding is a compact label applied to people’s behavior by an observer that allows the observer to predict future behavior. It’s not a process in itself.

And yes, we apply this label to ourselves. Much of what we do is only available to consciousness post-hoc, and is available to be described just the same as the behavior of someone else.

  > Understanding is not something that any machine or person does.

I can model the wind and air flow across the surface and design airfoils.

I can interpret the mathematical symbols into real physical meaning.

I can adapt these equations to novel settings or even fictitious ones.

I can analyze them counterfactually; not just making predictions but also telling you why those predictions are accurate, what their inaccuracies are (such as which variables and measurements are more precise), and I can tell you what all those things mean.

I can describe and derive the limits of the equations and models, discussing where they do and don't work. Including in the fictional settings.

I can do this at an emergent macroscopic level and I can do it at a fine grain molecular or even atomic level. I can even derive the emergent macroscopic behavior from the more fine grain analysis and tell you the limits of each model.

I can also respond that Bernoulli's equation is not an accurate description of why an airfoil works, even when prompted with those words[0].

These are characteristics that lead people to believe I understand the physics of fluid mechanics and flight. They correlate strongly with the ability to recall information from textbooks, but the actions aren't strictly the ability to recall and search over a memory database. Do these things prove that I understand? No, but we deal with what we got even if it is imperfect.

It is not just the ability to perform a task, it includes the ability to explain it. The more depth I am able to the greater understanding people attribute. While this correlates with task performance it is not the same. Even Ramanujan had to work hard to understand even if he was somehow able to divine great equations without it.

You're right that these descriptions are not the thing itself either. No one is claiming the map is the territory here. That's not the argument being made. Understanding the map is a very different thing than conflating the map and the territory. It is also a different thing than just being able to read it.

[0] https://x.com/BethMayBarnes/status/1953504663531388985

> In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human. There's no problem of how to do understanding, only how to do tasks.

Yes, but you also gloss over what a "task" is or what a "benchmark" is (which has to do with the meaning of generalization).

Suppose an AI or human answers 7 questions correctly out of 10 on an ICPC problem set, what are we able infer from that?

1. Is the task equal to answering these 10 questions well, with a uniform measure of importance?

2. Is the task be good at competitive programming problems?

3. Is the task be good at coding?

4. Is the task be good at problem solving?

5. Is the task not just to be effective under a uniform measure of importance, but an adversarial measure? (i.e. you can probably figure out all kinds of competitive programming questions, if you had more time / etc... but roughly not needing "exponentially more resources")

These are very different levels of abstraction, and literally the same benchmark result can be interpreted to mean very different things. And that imputation of generality is not objective unless we know the mechanism by which it happens. "Understanding" is short-hand for saying that performance generalizes at one of the higher levels of abstraction (3--5), rather than narrow success -- because that is what we expect of a human.

How do you quantify generality? If we have a benchmark that can quantify it and that benchmark reliably tells us that the LLM is within human levels of generalisation then the llm is not distinguishable from a human.

While it’s a good point that we need to benchmark generalisation ability, you have in fact agreed that it is not important to understand underlying mechanics.

That's kinda their point

The difference though is they understand that you can't just benchmark your way into proofs. Just like you can't unit test your way into showing code is error free. Benchmarks and unit tests are great tools that provide a lot of help, but just because a hammer is useful doesn't make everything a nail.

Nonsense.

A QC operator may be able to carry out a test with as much accuracy (or perhaps better accuracy, with enough practice) than the PhD quality chemist who developed it. They could plausibly do so with a high school education and not be able to explain the test in any detail. They do not understand the test in the same way as the chemist.

If 'understand' is a meaningless term to someone who's spent 30 years in AI research, I understand why LLMs are being sold and hyped in the way they are.

> They do not understand the test in the same way as the chemist.

Can you explain precisely what 'understand' means here, without using the word 'understand'? I don't think anyone can.

There are a number of competing models. The SEP page is probably a good place to start.

https://plato.stanford.edu/entries/understanding

Not to be flippant but have you considered that that question is an entire branch of philosophy with a several-millennias long history which people in some cases spend their entire life studying?

I have. It robustly has the folk-psychological meaning I mentioned in my first sentence. Call it ‘philosophical’ instead of ‘folk-psychological’ if you like. It’s a useful concept. But the concept doesn’t require AI engineers to do anything. It certainly doesn’t give any hints about AI engineers what they should actually do.

“Make it understand.”

“How? What does that look like?”

“… But it needs to understand…”

“It answers your questions.”

“But it doesn’t understand.”

“Ok. Get back to me when that entails anything.”

I would say it understands if given many variations of a problem statement, it always gives correct answer without fail. I have this complicated mirror question that only Deepseek and qwen3-max got right every time, still they only answered it correctly about a dozen times, so we're left with high probability, I guess.

I disagree with robotresearcher but I think this is also an absurd definition. By that definition there is no human, nor creature, that understands anything. Not just by nature of humans making mistakes, including experts, but I'd say this is even impossible. You need infinite precision and infinite variation here.

It turns "understanding" into a binary condition. Robotresearcher's does too, but I'm sure they would refine by saying that the level of understanding is directly proportional to task performance. But I still don't know how they'll address the issue of coverage, as ensuring tests have complete coverage is far from trivial (even harder when you want to differentiate from the training set, differentiating memorization).

I think you're right in trying to differentiate memorization from generalization, but your way to measure this is not robust enough. A fundamental characteristic of where I disagree from them is that memorization is not the same as understanding.

Isn't this just a reformulation of the Turing Test, with all the problems it entails?

I have been thinking about this for years, probably two decades. The answer to your question or the definition, I am sure you know, is rather difficult. I don't think it is impossible, but there's a risk of diving into a deep dark pit of philosophical thought going back to at least the ancient Greeks.

And, if we did go through that exercise, I doubt we can come out of it with a canonical definition of understanding.

I was really excited about LLM's as they surfaced and developed. I fully embraced the technology and have been using it extensively with full top-tier subscriptions to most services. My conclusion so far: If you want to destroy your business, adopt LLM's with gusto.

I know that's a statement that goes way against the train ride we are on this very moment. That's not to say LLM's are not useful. They are. Very much so. The problem is...well...they don't understand. And here I am, back in a circular argument.

I can define understanding with the "I know it when I see it" meme. And, frankly, it does apply. Yet, that's not a definition. We've all experienced that stare when talking to someone who does not have sufficient depth of understanding in a topic. Some of us have experienced people running teams who should not be in that position because they don't have a clue, they don't understand enough of it to be effective at what they do.

And yet, I still have not defined "understanding".

Well, it's hard. And I am not a philosopher, I am an engineer working in robotics, AI and applications to real time video processing.

I have written about my experiments using LLM coding tools (I refuse to call them AI, they are NOT intelligent; yes, need to define that as well).

In that context, lack of understanding is clearly evident when an LLM utterly destroys your codebase by adding dozens of irrelevant and unnecessary tests, randomly changes variable names as you navigate the development workflow, adds modules like a drunken high school coder and takes you down tangents that would make for great comedy if I were a tech comedian.

LLMs do not understand. They are fancy --and quite useful-- auto-complete engines and that's about it. Other than that, buyer beware.

The experiments I ran, some of them spanning three months of LLM-collaborative coding at various levels --from very hands-on to "let Jesus drive the car"-- conclusively demonstrated (at least to me) that:

1- No company should allow anyone to use LLMs unless they have enough domain expertise to be able to fully evaluate the output. And you should require that they fully evaluate and verify the work product before using it for anything; email, code, marketing, etc.

2- No company should trust anything coming out of an LLM, not one bit. Because, well, they don't understand. I recently tried to use the United Airlines LLM agent to change a flight. It was a combination of tragic and hilarious. Now, I know what's going on. I cannot possibly imagine the wild rides this thing is taking non-techies on every day. It's shit. It does not understand. It' isn't isolated to United Airlines, it's everywhere LLMs are being used. The potential for great damage is always there.

3- They can be great for summarization tasks. For example, you have have them help you dive deep into 300 page AMD/Xilinx FPGA datasheet or application note and help you get mentally situated. They can be great at helping you find prior art for patents. Yet, still, because they are mindless parrots, you should not trust any of it.

4- Nobody should give LLMs great access to a non-trivial codebase. This is almost guaranteed to cause destruction and hidden future effects. In my experiments I have experienced an LLM breaking unrelated code that worked just fine --in some cases fully erasing the code without telling you. Ten commits later you discover that your network stack doesn't work or isn't even there. Or, you might discover that the stack is there but the LLM changed class, variable or method names, maybe even data structures. It's a mindless parrot.

I could go on.

One response to this could be "Well, idiot, you need better prompts!". That, of course, assumes that part of my experimentation did not include testing prompts of varying complexity and length. I found that for some tasks, you get better results by explaining what you want and then asking the LLM to write a prompt to get that result. You check that prompt, modify if necessary and, from my experience, you are likely to get better results.

Of course, the reply to "you need better prompts" is easy: If the LLM understood, prompt quality would not be a problem at all and pages-long prompts would not be necessary. I should not have to specify that existing class, variable and method names should not be modified. Or that interfaces should be protected. Or that data structures need not be modified without reason and unless approved by me. Etc.

It reminds me of a project I was given when I was a young engineer barely out of university. My boss, the VP of Engineering where I worked, needed me to design a custom device. Think of it as a specialized high speed data router with multiple sources, destinations and a software layer to control it all. I had to design the electronics, circuit boards, mechanical and write all the software. The project had a budget of nearly a million dollars.

He brought me into his office and handed me a single sheet of paper with a top-level functional diagram. Inputs, outputs, interfaces. We had a half hour discussion about objectives and required timeline. He asked me if I could get it done. I said yet.

He checked in with me every three months or so. I never needed anything more than that single piece of paper and the short initial conversation because I understood what we needed, what he wanted, how that related to our other systems, available technology, my own capabilities and failings, available tools, etc. It took me a year to deliver. It worked out of the box.

You cannot do that with LLMs because they don't understand anything at all. They mimic what some might confuse for understanding, but they do not.

And, yet, once again, I have not defined the term. I think everyone reading this who has used LLMs to a non-trivial depth...well...understands what I mean.

> We've all experienced that stare when talking to someone who does not have sufficient depth of understanding in a topic.

I think you're really putting your finger on something here. LLMs have blown us away because they can interact with language in a very similar way to humans, and in fact it approximates how humans operate in many contexts when they lack a depth of understanding. Computers never could do this before, so it's impressive and novel. But despite how impressive it is, humans who were operating this way were never actually generating significant value. We may have pretended they were for social reasons, and there may even have been some real value associated with the human camaraderie and connections they were a part of, but certainly it is not of value when automated.

Prior to LLMs just being able to read and write code at a pretty basic level was deemed an employable skill, but because it was not a natural skill for lots of human, it was also a market for lemons and just the basic coding was overvalued by those who did not actually understand it. But of course the real value of coding has always been to create systems that serve human outcomes, and the outcomes that are desired are always driven by human concerns that are probably inscrutable to something without the same wetware as us. Hell, it's hard enough for humans to understand each other half the time, but even when we don't fully understand each other, the information conferred through non-verbal cue, and familiarity with the personalities and connotations that we only learn through extended interaction has a robust baseline which text alone can never capture.

When I think about strategic technology decisions I've been involved with in large tech companies, things are often shaped by high level choices that come from 5 or 6 different teams, each of which can not be effectively distilled without deep domain expertise, and which ultimately can only be translated to a working system by expert engineers and analysts who are able to communicate in an extremely high bandwidth fashion relying on mutual trust and applying a robust theory of the mind every step along the way. Such collaborators can not only understand distilled expert statements of which they don't have direct detailed knowledge, but also, they can make such distilled expert statements and confirm sufficient understanding from a cross-domain peer.

I still think there's a ton of utility to be squeezed out of LLMs as we learn how to harness and feed them context most effectively, and they are likely to revolutionize the way programming is done day-to-day, but I don't believe we are anywhere near AGI or anything else that will replace the value of what a solid senior engineer brings to the table.

I am not liking the term "AGI". I think intelligence and understanding are very different things and they are both required to build a useful tool that we can trust.

To use an image that might be familiar to lots of people reading this, the Sheldon character in Big Bang Theory is very intelligent about lots of fields of study and yet lacks tons of understanding about many things, particularly social interaction, the human impact of decisions, etc. Intelligence alone (AGI) isn't the solution we should be after. Nice buzz word, but not the solution we need. This should not be the objective at the top of the hill.

I've always distinguished knowledge, intelligence, and wisdom. Knowledge is knowing a chair is a seat. Intelligence is being able to use a log as a chair. Wisdom is knowing the log chair will be more comfortable if I turn it around and that sometimes it's more comfortable to sit on the ground and use the log as fuel for the fire.

But I'm not going to say I was the first to distinguish those word. That'd be silly. They're 3 different words and we use them differently. We all know Sheldon is smart but he isn't very wise.

As for AGI, I'm not so sure my issue is with the label but more with the insistence that it is so easy and straight forward to understand. It isn't very wise to think the answer is trivial to a question which people have pondered for millennia. That just seems egotistical. Especially when thinking your answer is so obviously correct that you needn't bother trying to see if they were wrong. Even though Don Quixote didn't test his armor a second time, he had the foresight to test it once.

Nice post.

I am dumbfounded as to how this doesnt seem to resonate widely on HN.

  > If 'understand' is a meaningless term to someone who's spent 30 years in AI research, I understand why LLMs are being sold and hyped in the way they are.

I've been to conferences, talked with people at the top of the field (I'm "junior", but published and have a PhD) where when asking deeper questions I'll get a frequent response "I just care if it works." As if that also wasn't the motivation for my questions too.

But I'll also tell you that there are plenty of us who don't ascribe to those beliefs. There's a wide breadth of opinions, even if one set is large and loud. (We are getting louder though) I do think we can get to AGI and I do think we can figure out what words like "understand" truly mean (with both accuracy and precision, the latter being what's more lacking). But it is also hard to navigate because we're discouraged from this work and little funding flows our way (I hope as we get louder we'll be able to explore more, but I fear we may switch from one railroad to the next). The weirdest part to me has been that it seems that even in the research space, talking to peers, that discussing flaws or limits is treated as dismissal. I thought our whole job was to find the limits, explore them, and find ways to resolve them.

The way I see it now is that the field uses the duck test. If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck. The problem is people are replacing "probably" with "is". The duck test is great, and right now we don't have anything much better. But the part that is insane is to call it perfect. Certainly as someone who isn't an ornithologist, I'm not going to be able to tell a sophisticated artificial duck from a real one. But it's ability to fool me doesn't make it real. And that's exactly why it would be foolish to s/probably/is.

So while I think you're understanding correctly, I just want to caution throwing the baby out with the bathwater. The majority of us dissenting from the hype train and "scale is all you need" don't believe humans are magic and operating outside the laws of physics. Unless this is a false assumption, artificial life is certainly possible. The question is just about when and how. I think we still have a ways to go. I think we should be exploring a wide breadth of ideas. I just don't think we should put all our eggs in one basket, especially if there's clear holes in it.

[Side note]: An interesting relationship I've noticed is that the hype train people tend to have a full CS pedigree while dissenters have mixed (and typically start in something like math or physics and make their way to CS). It's a weak correlation, but I've found it interesting.

As a mathematician who also regularly publishes in these conferences, I am a little surprised to hear your take; your experience might be slightly different to mine.

Identifying limitations of LLMs in the context of "it's not AGI yet because X" is huge right now; it gets massive funding, taking away from other things like SciML and uncertainty analyses. I will agree that deep learning theory in the sense of foundational mathematical theory to develop internal understanding (with limited appeal to numerics) is in the roughest state it has even been in. My first impression there is that the toolbox has essentially run dry and we need something more to advance the field. My second impression is that empirical researchers in LLMs are mostly junior and significantly less critical of their own work and the work of others, but I digress.

I also disagree that we are disincentivised to find meaning behind the word "understanding" in the context of neural networks: if understanding is to build an internal world model, then quite a bit of work is going into that. Empirically, it would appear that they do, almost by necessity.

Maybe given our different niches we interact with different people? But I'm uncertain because I believe what I'm saying is highly visible. I forgot, which NeurIPS(?) conference were so many wearing "Scale is all you need" shirts?

  > My first impression there is that the toolbox has essentially run dry and we need something more to advance the field

  > My second impression is that empirical researchers in LLMs are mostly junior and significantly less critical of their own work and the work of others

I also do not believe these people are less critical. I've had several works which struggled through publication as my models that were a hundredth the size (and a millionth the data) could perform on par, or even better. At face value asks of "more datasets" and "more scale" are reasonable, yet it is a self reinforcing paradigm where it slows progress. It's like a corn farmer smugly asking why the neighboring soy bean farmer doesn't grow anything when the corn farmer is chopping all the soy bean stems in their infancy. It is a fine ask to big labs with big money, but it is just gate keeping and lazy evaluation to anyone else. Even at CVPR this last year they passed out "GPU Rich" and "GPU Poor" hats, so I thought the situation was well known.

  > if understanding is to build an internal world model, then quite a bit of work is going into that. Empirically, it would appear that they do, almost by necessity.

But I'm a bit surprised that as a mathematician you think these systems create world models. While I see some generalization, this is also impossible for me to distinguish from memorization. We're processing more data than can be scrutinized. We seem to also frequently uncover major limitations to our de-duplication processes[0]. We are definitely abusing the terms "Out of Distribution" and "Zero shot". Like I don't know how any person working with a proprietary LLM (or large model) that they don't own, can make a claim of "zero shot" or even "few shot" capabilities. We're publishing papers left and right, yet it's absurd to claim {zero,few}-shot when we don't have access to the learning distribution. We've merged these terms with biased sampling. Was the data not in training or is it just a low likelihood region of the model? They're indistinguishable without access to the original distribution.

Idk, I think our scaling is just making the problem harder to evaluate. I don't want to stop that camp because they are clearly producing things of value, but I do also want that camp to not make claims beyond their evidence. It just makes the discussion more convoluted. I mean the argument would be different if we were discussing small and closed worlds, but we're not. The claims are we've created world models yet many of them are not self-consistent. Certainly that is a requirement. I admit we're making progress, but the claims were made years ago. Take GameNGen[1] or Diamond Diffusion. Neither were the first and neither were self-consistent. Though both are also impressive.

[0] as an example: https://arxiv.org/abs/2303.09540

[1] https://news.ycombinator.com/item?id=41375548

[2] https://news.ycombinator.com/item?id=41826402

Apologies if I ramble a bit here, this was typed in a bit of a hurry. Hopefully I answer some of your points.

First, regarding robotresearcher and simondota's comments, I am largely in agreement with what they say here. The "toaster" argument is a variant of the Chinese Room argument, and there is a standard rebuttal here. The toaster does not act independently of the human so it is not a closed system. The system as a whole, which includes the human, does understand toast. To me, this is different from the other examples you mention because the machine was not given a list of explicit instructions. (I'm no philosopher though so others can do a better job of explaining this). I don't feel that this is an argument for why LLMs "understand", but rather why the concept of "understanding" is irrelevant without an appropriate definition and context. Since we can't even agree on what constitutes understanding, it isn't productive to frame things in those terms. I guess that's where my maths background comes in, as I dislike the ambiguity of it all.

My "mostly junior" comment is partially in jest, but mostly comes from the fact that LLM and diffusion model research is a popular stream for moving into big tech. There are plenty of senior people in these fields too, but many reviewers in those fields are junior.

> I've also seen members of audiences to talks where people ask questions like mine ("are benchmarks sufficient to make such claims?") with responses of "we just care that it works."

This is a tremendous pain point to me more than I can convey here, but it's not unusual in computer science. Bad researchers will live and die on standard benchmarks. By the way, if you try to focus on another metric under the argument that the benchmarks are not wholly representative of a particular task, expect to get roasted by reviewers. Everyone knows it is easier to just do benchmark chasing.

> I also do not believe these people are less critical.

I think the fact that the "we just care that it works" argument is enough to get published is a good demonstration of what I'm talking about. If "more datasets" and "more scale" are the major types of criticisms that you are getting, then you are still working in a more fortunate field. And yes, I hate it as much as you do as it does favor the GPU rich, but they are at least potentially solvable. The easiest papers of mine to get through were methodological and often got these kinds of comments. Theory and SciML papers are an entirely different beast in my experience because you will rarely get reviewers that understand the material or care about its relevance. People in LLM research thought that the average NeurIPS score in the last round was a 5. Those in theory thought it was 4. These proportions feel reflected in the recent conferences. I have to really go looking for something outside the LLM mainstream, while there was a huge variety of work only a few years ago. Some of my colleagues have noticed this as well and have switched out of scientific work. This isn't unnatural or something to actively try to fix, as ML goes through these hype phases (in the 2000s, it was all kernels as I understand).

> approaches are narrow and still benchmark chasing > as a mathematician you think these systems create world models

When I say "world model", I'm not talking about outputs or what you can get through pure inference. Training models to perform next frame prediction and looking at inconsistencies in the output tells us little about the internal mechanism. I'm talking about appropriate representations in a multimodal model. When it reads a given frame, is it pulling apart features in a way that a human would? We've known for a long time that embeddings appropriately encode relationships between words and phrases. This is a model of the world as expressed through language. The same thing happens for images at scale as can be seen in interpretable ViT models. We know from the theory that for next frame prediction, better data and more scaling improves performance. I agree that isn't very interesting though.

> We are definitely abusing the terms "Out of Distribution" and "Zero shot".

Absolutely in agreement with everything you have said. These are not concepts that should be talked about in the context of "understanding", especially at scale.

> I think our scaling is just making the problem harder to evaluate.

Yes and no. It's clear that whatever approach we will use to gauge internal understanding needs to work at scale. Some methods only work with sufficient scale. But we know that completely black-box approaches don't work, because if they did, we could use them on humans and other animals.

> The claims are we've created world models yet many of them are not self-consistent.

For this definition of world model, I see this the same way as how we used to have "language models" with poor memory. I conjecture this is more an issue of alignment than a lack of appropriate representations of internal features, but I could be totally wrong on this.

  > The toaster does not act independently of the human so it is not a closed system

No one is thinking that the toaster understands things. We're using it to point out how silly the claim of "task performance == understanding" is. Techblueberry furthered this by asking if the toaster is suddenly intelligent by wrapping it with a cron job. My point was about where the line is drawn. The turning on the toaster? No, that would be silly and you clearly agree. So you have to answer why the toaster isn't understanding toast. That's the ask. Because clearly toaster toasts bread.

You and robotresearcher have still avoided answering this question. It seems dumb but that is the crux of the problem. The LLM is claimed to be understanding, right? It meets your claims of task performance. But they are still tools. They cannot act independently. I still have to prompt them. At an abstract level this is no different than the toaster. So, at what point does the toaster understand how to toast? You claim it doesn't, and I agree. You claim it doesn't because a human has to interact with it. I'm just saying that looping agents onto themselves doesn't magically make them intelligent. Just like how I can automate the whole process from planting the wheat to toasting the toast.

You're a mathematician. All I'm asking is that you abstract this out a bit and follow the logic. Clearly even our automated seed to buttered toast on a plate machine needs not have understanding.

From my physics (and engineering) background there's a key thing I've learned: all measurements are proxies. This is no different. We don't have to worry about this detail in most every day things because we're typically pretty good at measuring. But if you ever need to do something with precision, it becomes abundantly obvious. But you even use this same methodology in math all the time. Though I wouldn't say that this is equivalent to taking a hard problem, creating an isomorphic map to an easier problem, solving it, then mapping back. There's an invective nature. A ruler doesn't measure distance. A ruler is a reference to distance. A laser range finder doesn't measure distance either, it is photodetector and a timer. There is nothing in the world that you can measure directly. If we cannot do this with physical things it seems pretty silly to think we can do it with abstract concepts that we can't create robust definitions for. It's not like we've directly measured the Higgs either. But what, do you think entropy is actually a measurement of intelligible speech? Perplexity is a good tool for identifying an entropy minimizer? Or does it just correlate? Is a FID a measurement of fidelity or are we just using a useful proxy? I'm sorry, but I just don't think there are precise mathematical descriptions of things like natural English language or realistic human faces. I've developed some of the best vision models out there and I can tell you that you have to read more than the paper because while they will produce fantastic images they also produce some pretty horrendous ones. The fact that they statistically generate realistic images does not imply that they actually understand them.

  > I'm no philosopher

  > This is a tremendous pain point ... researchers will live and die on standard benchmarks.

  > then you are still working in a more fortunate field

So I guess I'm saying I share this frustration. It's part of the reason I talk strongly here. I understand why people shift gears. But I think there's a big difference between begrudgingly getting on the train because you need to publish to survive and actively fueling it and shouting that all outer trains are broken and can never be fixed. One train to rule them all? I guess CS people love their binaries.

  > world model

I referenced the game engines because while they are impressive they are not self consistent. Walls will disappear. An enemy shooting at you will disappear sometimes if you just stop looking at it. The world doesn't disappear when I close my eyes. A tree falling in a forest still creates acoustic vibrations in the air even if there is no one to hear it.

A world model is exactly that, a model of a world. It is a superset of a model of a camera view. It is a model of the things in the world and how they interact together, regardless of if they are visible or not. Accuracy isn't actually the defining feature here, though it is a strong hint, at least it is for poor world models.

I know this last part is a bit more rambly and harder to convey. But I hope the intention came across.

> You and robotresearcher have still avoided answering this question.

I have repeatedly explicitly denied the meaningfulness of the question. Understanding is a property ascribed by an observer, not possessed by a system.

You may not agree, but you can’t maintain that I’m avoiding that question. It does not have an answer that matters; that is my specific claim.

You can say a toaster understands toasting or you can not. There is literally nothing at stake there.

You said the LLMs are intelligent because they do tasks. But the claim is inconsistent with the toaster example.

If a toaster isn't intelligent because I have to give it bread and press the button to start then how's that any different from giving an LLM a prompt and pressing the button to start?

It's never been about the toaster. You're avoiding answering the question. I don't believe you're dumb, so don't act the part. I'm not buying it.

I didn’t describe anything as intelligent or not intelligent.

I’ll bow out now. Not fun to be ascribed views I don’t have, despite trying to be as clear as I can.

Intellectual caution is a good default.

Having said that, can you name one functional difference between an AI that understands, and one that merely behaves correctly in its domain of expertise?

As an example, how would a chess program that understands chess differ from one that is merely better at it than any human who ever lived?

(Chess the formal game; not chess the cultural phenomenon)

Some people don’t find the example satisfying, because they feel like chess is not the kind of thing where understanding pertains.

I extend that feeling to more things.

  > any human who ever lived

  > can you name one functional difference between an AI that understands, and one that merely behaves correctly in its domain of expertise?

But yes. Is it fair if I use Go instead of Chess? Game 4 with Lee Sedol seems an appropriate example.

Vafa also has some good examples[1,2].

But let's take an even more theoretical approach. Chess is technically a solved game since it is non-probabilistic. You can compute an optimal winning strategy from any valid state. Problem is it is intractable since the number of action state pairs is so large. But the number of moves isn't the critical part here, so let's look at Tic-Tac-Toe. We can pretty easily program up a machine that will not lose. We can put all actions and states into a graph and fit that on a computer no problem. Do you really say that the program better understands Tic-Tac-Toe than a human? I'm not sure we should even say it understands the game at all.

I don't think the situation is resolved by changing to unsolved (or effectively unsolved) games. That's the point of the Heliocentric/Geocentric example. The Geocentric Model gave many accurate predictions, but I would find it surprising if you suggested an astronomer at that time, with deep expertise in the subject, understood the configuration of the solar system better than a modern child who understands Heliocentricism. Their model makes accurate predictions and certainly more accurate than that child would, but their model is wrong. It took quite a long time for Heliocentrism to not just be proven to be correct, but to also make better predictions than Geocentrism in all situations.

So I see 2 critical problems here.

1) The more accurate model[3] can be less developed, resulting in lower predictive capabilities despite being a much more accurate representation of the verifiable environment. Accuracy and precision are different, right?

2) Test performance says nothing about coverage/generalization[4]. We can't prove our code is error free through test cases. We use them to bound our confidence (a very useful feature! I'm not against tests, but as you say, caution is good).

In [0] I referenced Dyson, I'd appreciate it if you watched that short video (again if it's been some time). How do you know you aren't making the same mistake Dyson almost did? The mistake he would have made had he not trusted Fermi? Remember, Fermi's predictions were accurate and they even stood for years.

If your answer is time, then I'm not convinced it is a sufficient explanation. It doesn't explain Fermi's "intuition" (understanding) and is just kicking the can down the road. You wouldn't be able to differentiate yourself from Dyson's mistake. So why not take caution?

And to be clear, you are the one making the stronger claim: "understanding has a well defined definition." My claim is that yours is insufficient. I'm not claiming I have an accurate and precise definition, my claim is that we need more work to get the precision. I believe your claim can be a useful abstraction (and certainly has been!), but that there are more than enough problems that we shouldn't hold to it so tightly. To use it as "proof" is naive. It is equivalent to claiming your code is error free because it passes all test cases.

[0] https://news.ycombinator.com/item?id=45622156

[1] https://arxiv.org/abs/2406.03689

[2] https://arxiv.org/abs/2507.06952

[3] Certainly placing the Earth at the center of the solar system (or universe!) is a larger error than placing the sun at the center of the solar system and failing to predict the tides or retrograde motion of Mercury.

[4] This gets exceedingly complex as we start to differentiate from memorization. I'm not sure we need to dive into what the distance from some training data needs be to make it a reasonable piece of test data, but that is a question that can't be ignored forever.

>> any human who ever lived > Is this falsifiable? Even restricting to those currently living? On what tests? In which way? Does the category of error matter?

Software reliably beats the best players that have ever played it in public, including Kasparov and Carlsen, the best players of my lifetime (to my limited knowledge). By analogy to the performance ratchet we see in the rest of sports and games, and we might reasonably assume that these dominant living players are the best the world has ever seen. That could be wrong. But my argument does not hang on this point, so asking about falsifiability here doesn't do any work. Of course it's not falsifiable.

Y'know what else is not falsifiable? "That AI doesn't understand what it's doing".

  > can you name one functional difference between an AI that understands, and one that merely behaves correctly in its domain of expertise?

No one seems to have changed their opinion about anything in the wake of AIs routinely passing the Turing Test. They are fooled by the chatbot passing as a human, and then ask about ducks instead. The most celebrated and seriously considered quacks like a duck argument has been won by the AIs and no-one cares.

By the way, the ornithologists' criteria for duck is probably genetic and not much to do with behavior. A dead duck is still a duck.

And because we know what a duck is, no-one is yelling at ducks that 'they don't really duck' and telling duck makers they need a revolution in duck making and they are doomed to failure if they don't listen.

Not so with 'understanding'.

  > Y'know what else is not falsifiable? "That AI doesn't understand what it's doing".

  > seriously considered quacks like a duck argument has been won by the AIs and no-one cares.

Often when people find that their initial beliefs are wrong or not precise enough then they update their beliefs. You seem to be calling this a flaw. It's not like the definitions are dramatically changing, they're refining. There's a big difference

My first post here is me explaining that I have a non-standard definition of what ‘understanding’ means, which helps me avoid an apparently thorny issue. I’m literally here offering a refinement of a definition.

This is a weird conversation.

  > This is a weird conversation.

Maybe what was interpreted is different than what you meant to convey, but certainly my interpretation was not unique. I'm willing to update my responses if you are willing to clarify but we'll need to work together on that. Because unfortunately just because the words make perfect sense to you doesn't mean they do to others.

I'll even argue that this is some of the importance of understanding. Or at least what we call understanding.

so your definition of "understand" is "able to develop the QC test (or explain tests already developed)"

I hate to break it to you, but the LLMs can already do all 3 tasks you outlined

It can be argued for all 3 actors in this example (the QC operator, the PhD chemist and the LLM) that they don't really "understand" anything and are iterating on pre-learned patterns in order to complete the tasks.

Even the ground-breaking chemist researcher developing a new test can be reduced to iterating on the memorized fundamentals of chemistry using a lot of compute (of the meat kind).

The mythical Understanding is just a form of "no true Scotsman"

  > that does not have a technical meaning

It's a hard problem, but we are making progress on it. We will probably get there, but it's going to end up being very nuanced and already it is important to recognize that the word means different things in vernacular and in even differing research domains. Words are overloaded and I think we need to recognize this divergence and that we are gravely miscommunicating by assuming the definitions are obvious. I'm not sure why we don't do more to work together on this. In our field we seem to think we got it all covered and don't need others. I don't get that.

  > In this view, if a machine performs a task as well as a human, it understands it exactly as much as a human.

Okay, you might take an out by saying the calculator can't do abstract math like I can, right? Well we're going to run into that same problem. You can't test your way out of it. We've known this in hard sciences like physics for centuries. It's why physicists do much more than just experiments.

There's the classic story of Freeman Dyson speaking to Fermi, which is why so many know about the 4 parameter elephant[0], but it is also just repeated through our history of physics. Guess what? Dyson's experiments worked. They fit the model. They were accurate and made accurate predictions! Yet they were not correct. People didn't reject Galileo just because the church, there were serious problems with his work too. Geocentricism made accurate predictions, including ones that Galileo's version of Heliocentrism couldn't. These historical misunderstandings are quite common, including things like how the average person understands things like Schrodinger's Cat. The cat isn't in a parallel universe of both dead and alive lol. It's just that we, outside the box can't determine which. Oh, no, information is lossy, there's injective functions, the universe could then still be deterministic yet we wouldn't be able to determine that (and my name comes into play).

So idk, it seems like you're just oversimplifying as a means to sidestep the hard problem[1]. The lack of a good technical definition of understanding should tell us we need to determine one. It's obviously a hard thing to do since, well... we don't have one and people have been trying to solve it for thousands of years lol.

  > Just my opinion, but my professional opinion from thirty-plus years in AI.

[0] https://www.youtube.com/watch?v=hV41QEKiMlM

[1] I'm perfectly fine tabling a hard problem and focusing on what's more approachable right now, but that's a different thing. We may follow a similar trajectory but I'm not going to say the path we didn't take is just an illusion. I'm not going to discourage others from trying to navigate it either. I'm just prioritizing. If they prove you right, then that's a nice feather in your hat, but I doubt it since people have tried that definition from the get go.

> It's a hard problem

So people say.

I’m not sidestepping the Hard Problem. I am denying it head on. It’s not a trick or a dodge! It’s a considered stance.

I'm denying that an idea that has historically resisted crisp definition, and that the Stanford Encyclopedia of Philosophy introduces as 'protean', needs to be taken seriously as an essential missing part of AI systems, until someone can explain why.

In my view, the only value the Hard Problem has is to capture a feeling people have about intelligent systems. I contend that this feeling is an artifact of being a social ape, and it entails nothing about AI.

Regardless of whether you think understanding is important, it’s clear from this thread that a lot of people find understanding valuable. In order to trust an AI with decisions that affect people, people will want to believe that the AI “understands” the implications of its decisions, for whatever meaning of “understand” those people have in their head. So indeed I think it is important that AI researchers try to get their AIs to understand things, because it is important to the consumers that they do.

I agree with this. I contend that as the AIs improve in performance, the designation of understanding will accrete to them. I predict there will never be a component, module, training process, or any other significant piece of an AI that is the ‘understanding’ piece that some believe is missing today.

Also, the widespread human belief that something is valuable has absolutely no entailments to me other than treating the believers with normal respect. It’s very easy to think of things that are important to billions that you believe are not true or relevant to a reality-driven life.

It's a sidestep if your stance doesn't address critiques.

  > needs to be taken seriously as an essential missing part of AI systems, until someone can explain why.

While I agree with you in the main, I also take seriously the "until someone can explain why" counterpoint.

Though I agree with you that your calculator doesn't understand math, one might reasonably ask, "why should we care?" And yeah, if it's just a calculator, maybe we don't care. A calculator is useful to us irrespective of understanding.

If we're to persuade anyone (if we are indeed right), we'll need to articulate a case for why understanding matters, with respect to AI. I think everyone gets this on an instinctual level- it wasn't long ago that LLMs suggested we add rocks to our salads to make them more crunchy. As long as these problems can be overcome by throwing more data and compute at them, people will remain incurious about the Understanding Problem. We need to make a rigorous case, probably with a good working alternative, and I haven't seen much action here.

  > "why should we care?"

I don't think anyone is saying that the calculator isn't a useful tool. But certainly we should push back when people are claiming it understands math and can replace all mathematicians.

  > If we're to persuade anyone, we'll need to articulate a case for why understanding matters

I'll say that a major motivating reason of why I went into physics in the first place is because I found that a deep understanding was a far more efficient way of learning how to do things. I started as an engineer and even went into engineering after my degree. Physics made me a better engineer, and I think a better engineer than had I stayed in engineering. Understanding gave me the ability to not just take building blocks and put them together, but to innovate. Being able to see things at a deeper level allowed me to come to solutions I otherwise could not have. Using math to describe things allowed me to iterate faster (just like how we use simulations). Understanding what the math meant allowed me to solve the problems where the equations no longer applied. It allowed me to know where the equations no longer applied. It told me how to find and derive new ones.

I often found that engineers took an approach of physical testing first, because "the math only gets you so far." But that was just a misunderstanding of how far their math took them. It could do more, just they hadn't been taught that. So maybe I had to take a few days working things out on pen and paper, but that was a cheaper and more robust solution than using the same time to test and iterate.

Understanding is a superpower. Problems can be solved without understanding. A mechanic can fix an engine without knowing how it works. But they will certainly be able to fix more problems if they do. The reason to understand is because we want things to work. The problem is, the world isn't so simple that every problem is the same or very similar to another. A calculator is a great tool. It'll solve calculations all day. Much faster than me, with higher accuracy, but it'll never come up with an equation on its own. That isn't to call it useless, but I need to know this if I want to get things done. The more I understand what my calculator can and can't do, the better I can use that tool.

Understanding things, and the pursuit to understand more is what has brought humans to where they are today. I do not understand why this is even such a point of contention. Maybe the pursuit of physics didn't build a computer, but it is without a doubt what laid the foundation. We never could have done this had we not thought to understand lightning. We would have never been able to tame it like we have. Understanding allows us to experiment with what we cannot touch. It does not mean a complete understanding nor does it mean perfection, but it is more than just knowledge.

Super late to this, sorry.

> I'm not the one claiming that a calculator thinks. The burden of proof lies on those that do. Claims require evidence and extraordinary claims require extraordinary evidence.

You're right, I may have misconstrued the original claim. I took the parent to be saying something like "calculators understand math, but also, understanding isn't particularly important with respect to AI" but I may have gotten some wires crossed. This isn't the old argument about submarines that swim, I don't think.

> Understanding is a superpower.

Thanks, this is all well-put.

Critiques should come with some argument if they want to be taken seriously.

If I say it’s not real intelligence because the box isn’t blue, how much does anyone owe that critique? How about if a billion people say that blueness is the essence missing from AIs?

Tell me why blue matters and we have a conversation.

Only problem is this time enough money is being burned that if AGI does not come, it will probably be extremely painful/fatal for a lot of people that had nothing to do with this field or the decisions being made. What will be the consequences if that comes to pass? So many lives were permanently ruined due to the GFC.

I'm not sure. There's a view that, as I understand it, suggests that language is intelligence. That language is a requirement for understanding.

An example might be kind of the contrary—that you might not be able to hold an idea in your head until it has been named. For myself, until I heard the word gestalt (maybe a fitting example?) I am not sure I could have understood the concept. But when it is described it starts to coalesce—and then when named, it became real. (If that makes sense.)

FWIW, Zeitgeist is another one of those concepts/words for me. I guess I have to thank the German language.

Perhaps it is why other animals on this planet seem to us lacking intelligence. Perhaps it is their lack of complex language holding their minds back.

  > There's a view that suggests that language is intelligence. 

  > That language is a requirement for understanding.

I think you must then revisit your definition of language and ensure that it matches to all the creatures that you consider intelligent. At least by doing this you'll make some falsifiable claims and can make progress. I think an ant is intelligent, but I also think ants do things far more sophisticated than the average person thinks. It's an easy trap, not knowing what you don't know. But if we do the above we get some path to aid in discovery, right?

  > that you might not be able to hold an idea in your head until it has been named

It is the condition where a person does not have an internal monologue. They think without words. The definition of language is still flexible enough that you can probably still call that language, just like in your example, but it shows a lack of precision in the definition, even if it is accurate.

  > Perhaps it is why other animals on this planet seem to us lacking intelligence

Perhaps language is a necessary condition for advanced intelligence, but not intelligence alone. Perhaps it is communication and societies, differentiating from an internalized language. Certainly the social group can play an influence here, as coalitions can do more than the sum of the individuals (by definition). But the big question is if these things are necessary. Getting the correct causal graph, removing the confounding variables, is no easy task. But I think we should still try and explore differing ideas. While I don't think you're right, I'll encourage you to pursue your path if you encourage me to pursue mine. We can compete, but it should be friendly, as our competition forces us to help see flaws in our models. Maybe the social element isn't a necessary condition, but I have no doubt that it is a beneficial tool. I'm more frustrated by those wanting to call the problem solved. It obviously isn't, as it's been so difficult to get generalization and consensus among experts (across fields).

> It is the condition where a person does not have an internal monologue.

These people are just nutjobs that misinterpreted what internal monologue means, and have trouble doing basic introspection.

I know there are a myriad of similar conditions, aphantasia, synaesthesia, etc. But someone without internal monologue simply could not function in our society, or at least not pass as someone without obvious mental diminishment.

If there really were some other, hidden code in the mind, that could express "thoughts" in the same depth as language does - then please show it already. At least the tiniest bit of a hint.

I know some of these people. We've had deep conversations about what is going on in our thought processes. Their description significantly differs from mine.

These people are common enough that you likely know some. It's just not a topic that frequently comes up.

It is also a spectrum, not a binary thing (though full anendophasia does exist, it is just on the extreme end). I think your own experiences should allow you to doubt your claim. For example, I know when I get really into a fiction book I'm reading that I transition from a point where I'm reading the words in my head to seeing the scenes more like a movie, or more accurately like a dream. I talk to myself in my head a lot, but I can also think without words. I do this a lot when I'm thinking about more physical things like when I'm machining something, building things, or even loading dishwasher. So it is hard for me to believe that while I primarily use an internal monologue that there aren't people that primarily use a different strategy.

On top of that, well, I'm pretty certain my cat doesn't meow in her head. I'm not certain she has a language at all. So why would it be surprising that this condition exists? You'd have to make the assumption that there was a switch in human evolution. Where it happened all at once or all others went extinct. I find that less likely than the idea that we just don't talk enough about how we think to our friends.

Certainly there are times where you think without a voice in your head. If not, well you're on the extreme other end. After all, we aren't clones. People are different, even if there's a lot of similarities.

I’m like that more often than not. Words and language always seemed like a “translation layer” to express myself to other people, not something essential that needs to happen in my head. Especially when thinking deeply about some technical problem there’s no language involved, just abstract shapes and seeing things “in my mind’s eye”.

We might just be rehashing that silly internet meme about “shape rotators”, but there could be a correlation here where people whose minds work this way are more dismissive of LLMs.

I suggest you revisit the subject with your friends, with two key points:

1. Make it clear to them that with "internal monologue" you do not mean an actual audible hallucination

2. Ask them if they EVER have imagined themselves or others saying or asking anything

If they do, which they 100% will unless they lie, then you have ruled out "does not have an internal monologue", the claim is now "does not use his internal monologue as much". You can keep probing them what exactly that means, but it gets washy.

Someone that truly does not have an internal dialogue could not do the most basic daily tasks. A person could grab a cookie from the table when they feel like it (oh, :cookie-emoji:!), but they cannot put on their shoes, grab their wallet and keys, look in the mirror to adjust their hair, go to the supermarket, to buy cookies. If there were another hidden code that can express all huge mental state pulled by "buy cookies", by now we would at least have an idea that it exists underneath. We must also ask, why would we translate this constantly into language, if the mental state is already there? Translation costs processing power and slows down. So why are these "no internal monologue" people not geniuses?

I have no doubt that there is a spectrum, on that I agree with you. But the spectrum is "how present is (or how aware is the person of-) the internal monologue". E.g. some people have ADHD, others never get anxiety at all. "No internal monologue" is not one end of the spectrum for functioning adults.

The cat actually proves my point. A cat can sit for a long time before a mouse-hole, or it can hide to jumpscare his brother cat, and so on. So to a very small degree there is something that let's it process ("understand") very basic and near-future event and action-reactions. However, a cat could not possibly go to the supermarket to buy food, obviating anatomical obstacles, because: it has no language and therefore cannot make a complex mental model. Fun fact: whenever animals (apes, birds) have been taught language, they never ask questions (some claim they did, but if you dig in you'll see that the interpretation is extremely dubious).

  > 1. Make it clear to them that with "internal monologue" you do not mean an actual audible hallucination

And yes, this has been discussed in depth. It was like literally the first thing...

But no, they do not have conversations in their heads like I do. They do not use words as their medium. I have no doubt that their experience is different from mine.

  > 2. Ask them if they EVER have imagined themselves or others saying or asking anything

  > The cat actually proves my point.

  > Fun fact: whenever animals (apes, birds) have been taught language, they never ask questions (some claim they did, but if you dig in you'll see that the interpretation is extremely dubious).

It's pretty self explanatory: there's actual voice heard with your ears, there's the internal monologue, and then there's a hallucination.

> Yes, they have imagined normal interactions. But frequently those imaginary conversations do not use words.

And you did not dig in deeper? How exactly do you imagine a conversation without words?

  > there's actual voice heard with your ears, there's the internal monologue, and then there's a hallucination.

  >> I hear my voice in my head. I can differentiate this from a voice outside my head, but yes, I do "hear" it.

  > And you did not dig in deeper?

  >>>> I know some of these people. ***We've had deep conversations about what is going on in our thought processes.***

I think you need to consider that you may just be wrong. You are trying very hard to defend your belief, but why? The strengths of our beliefs should be proportional to the evidence that supports them. I am not trying to say that your logic is bad, let's make that clear. But I think your logic doesn't account for additional data. If you weren't previously aware of this data then how could you expect the logic to reach the correct conclusion? I want to make this clear because I want to distinguish correctness from intelligence (actually relevant to the conversation this stemmed from). You can be wrong without being dumb, but you can also be right and dumb. I think on this particular issue you fall into the former, not the latter. I respect that you are defending your opinion and beliefs, but this is turning as you are rejecting data. Your argument now rests on the data being incorrect, right? Because that's the point. Either the data is wrong or your model is wrong (and let's distinguish that a model is derived through logic to explain data).

I want to remind you that this idea is testable too. I told you this because it is a way to convince yourself and update the data up have available to you. You can train yourself to do this in some cases. Not all and obviously it won't be an identical experience to these people, but you can get yourself to use lower amounts of language when thinking through problems. You had also mentioned that people with aphantasia couldn't function, but think about that too. These topics are quite related actually, considering how we've discussed anendophasia you should be able to reason that these people are really likely to have low aphantasia. Notice I said low, as this is a spectrum. You can train the images in your mind to be stronger too. The fact that some images are stronger than others should lead you to believe that this is a spectrum and that it is likely people operate at different base levels. It should also lead you to reason that this is likely trainable in an average person. The same goes for anendophasia. Don't make this binary, consider it a spectrum. That's how the scientific literature describes the topic too. But if you pigeonhole it to being binary and only true in the extreme cases then your model isn't flexible enough as it also isn't considering the variances in people.

Go talk with your friends. Get detailed. When you imagine an apple in your head how much do you see? As the person if their process involves words or if it is purely imagery. If words, how many? Is it a red apple? Green? Yellow? Can they smell it? Can they taste it? What's it smell and taste like? I will bet you every single person you talk to will answer these differently. I will even wager that each time you do the exercise you yourself will answer differently, even if the variance is much smaller. But that's data, and your model needs to be able to explain that data too. While I think you have the right thought process I don't think you are accounting for this variance, instead treating it as noise. But noise can be parameterized and modeled too. Noise is just the statistical description of uncertainty.

Let me be clear: yes, I know I might be wrong. I hope I'm not dumb and wrong, or at least not dumb. I am also not writing here as some kind of debate exercise. I do because I find this topic extremely interesting and insightful. What if language is the intelligence? What if "guessing the next word" really was all that was there, to peak human intelligence, knowledge, and understanding of our world? I am not hyped by AI, it's rather that I find this possibility somewhat sad.

I've made up a model, an idea, and I don't think the data opposing it is trustworthy. My first problem is that there are many people that claim that they have NO internal monologue, which means NEVER constructing a sentence from theirselves or others in their head (except directly as verbal speech), and this seems outright impossible. When pressed, these people usually either admit that they do have some monologue, just "much less". Or they misunderstood it for something similar to schizophrenia, actual hallucinations. If they don't admit to actually, sometimes, having them, then they fail to explain where exactly the line between "thinking of someone or themselves saying something" and the internal monologue/dialogue is. As if they had been caught lying by the detective, they end the conversation. Or at least that's how I feel, I really don't know how to ask more questions here before making them feel too interrogated, or like someone that has self-diagnosed being told that they are imagining things.

With "absolutely none" group out of the way, it leaves us with people who claim to perceive the internal monologue very scarcely, and claim that they do not need to "think" or "do". How can we possibly test this scientifically? The data is all self-reported. Or at least I don't know if this can or has been neurologically researched.

Consider also that all self-reported data about internal monologue is "poisoned": we are trying to get objective data with the data itself as a vehicle. We are not asking if someone feels pain, or if they can solve a puzzle in a timeframe. We cannot measure electric activity with some instruments, nor evaluate yes-or-no questions.

What if it is true that some people do not perceive their internal monologue? I certainly don't remember it "popping" into my head at a certain age, and I think nobody does. When we learn language, we become conscious with it, because it allows to model the world, beyond putting things in our mouth and screaming. So it could be that not everybody perceives it equally, a spectrum like you said, and that some people rationalize it retroactively as not being there - just "thoughts", ideas, feelings. We reconstruct past events via a narration, filling in details by guessing, so why wouldn some people guess that they are not narrating in their head? It is not something that is taught in school or from or parents, you either perceive it as "internal monologue", or as "just thinking", because, well, it's the thinking doing it's thing.

Somewhat out of my league in this thread but,I think I am.one of these people. I do remember a time before I had an internal monologue , in fact I remember the day in elementary school when I learned, after having been explained to me by my teacher, that everyone else was "talking to themselves in their head". I think I spent the next month or so obsessing over this new found ability. But before that day I was perfectly capable of thought, and conversation, and writing. Even now I can "switch modes" and have coherent thoughts occur, with no labeling or accompanying narrative. I can distinctly identify concepts and transitions between them but there are no words involved until I open my mouth. So I dont know if it was just a hidden background process before that day. But it definitely "feels' different when its in the foreground or back, or nor there.

  > What if language is the intelligence? 

  > What if "guessing the next word" really was all that was there, to peak human intelligence, knowledge, and understanding of our world?

-> I can make accurate predictions about coin flips without any understanding of physics or how the coin is being flipped. All I need to do is be lucky. Or we can take many mechanical objects like a clock that can predict time.

Or a horse can appear to do math if I tell it how many times to stomp its foot. It made accurate predictions yet certainly has no understanding.

Ehh I'll give you a more real example. Here's a model that gives accurate predictions for turn by turn taxi directions where the authors extract the world model and find it is not only inaccurate but find that it significantly diverges. Vafa has a few papers on the topic, I suggest reading his work.

<- You can understand all the physics to a double pendulum and not predict the movement for any arbitrary amount of time moving forward if you do not also know the initial conditions. This is going to be true for any chaotic system.

I said we've seen this in the history of science. {Geo,Helio}centrism is a great example. Scientists who had no affiliation with the church still opposed Galileo because his model wasn't making accurate predictions for certain things. Yet the heliocentric model is clearly a better understanding and more accurate as a whole. If you want to dive deeper into this topic I'd highly recommend both the podcast "An Opinionated History of Math" and the book "Representing and Intervening" by Ian Hacking. They're both very approachable. FWIW, metaphysics talks about this quite a lot.

  > My first problem is that there are many people that claim that they have NO internal monologue

The problem here is one of language and semantics, not effect. I completely believe that someone will say "I have no internal monologue" if >90% of their thinking is without an internal monologue. Just like how a guy who's 5'11.75" will call themselves 6'. Are they a liar? I wouldn't say so, they're >99% accurate. Would you for someone 5'11"? That's probably more contextually dependent.

So you distrust the data. That's fine. Let's assume poisoned. We should anyways since noise is an important part of any modeling[2]. It is standard practice...

So instead, do you distrust that there's a distribution into how much of an internal monologue individuals use? Or do you presume they all use them the same.

I'd find it hard to believe you distrust the spectrum. But if you trust the spectrum then where is the threshold for your claim? 0%? That's really not a useful conversation even if heavy tailed.

You are hyper-fixated on the edge case but its result isn't actually consequential to your model. The distribution is! You'll have to consider your claims much more carefully when you consider a distribution. You need to then claim a threshold, in both directions. Or if you make the claim that we're all the same (I'd find that quite surprising tbh, especially given the nature of linguistics), you need to explain that too and your expected distribution that would claim that (narrow).

All I can tell you is that my friend and I have had this conversation multiple times over many years and it seems very constant to me. I have no reason to believe they are lying and if they are they are doing so with an extreme level of consistency, which would be quite out of the norm.

[0] Arguing the relationship still requires addressing trivial relationships.

[1] https://arxiv.org/abs/2406.03689

[2] Even if there are no liars (or "lizardmen"[3]) we still have to account for miscommunication and misunderstandings.

[3] https://en.wiktionary.org/wiki/Lizardman%27s_Constant

> We have a lot of different measures for intelligence when it comes to animals.

But there is an abismal difference between animal intelligence and human intelligence.

> predict next word <--> understanding

Yes, and I could say a stone understands the world because its state reflects the world: it gets hot, cold, wet, dry, radiated, whatever. Perhaps its internal state can even predict the world: if it's rolling downhill, it can predict that it will stop soon. But the stone is not conscious like a human, and neither is a clock nor a horse that can count to ten. The stone obviously is "reducing to the absurd" - a horse can actually "guess" to some degree, but nothing like a human. It cannot ask a question, and it cannot answer itself a question.

> I cannot stress that we should not represent this as a binary setting.

That was kind of my point, to eliminate the binary "no", leaving us with a spectrum.

My initial claim "these are just nutjobs" - my apologies for the phrasing - was addressing this: there are no people "without internal monologue AT ALL".

Since we seem to actually agree on this point, our difference is that I believe that the people with "little internal monologue" are simply not aware of it.

Let me phrase string it this way: If language is the understanding, then the internal monologue is not some quirky side effect. To understand something at the human level, we need to describe it with language, the rest are primitive instincts and "feelings".

We can model the past and the future. We can model ourselves in 10 years. And what is one of the most important things we would model? What we would say or think then - thinking being "saying something out silently in our head". Not really just feelings: "I would love my partner", sure but why? "Because . . .".

When we are utilizing language, the internal monologue, to construct the model, we cannot be "aware of it" constantly. That is, the bandwidth is taken by the tasks at hand that we are dealing with, it would be detrimental if every other phrase would be followed with "btw did I notice that I just understand this via a string of words?". The more complex actions or idea we process, the less aware we are that we are using language for it. That is "being in the flow". We can reconstruct it when done, and here, if there is a lack of awareness of internal monologue, it will be rationalized as something else.

> Or if you make the claim that we're all the same (I'd find that quite surprising tbh, especially given the nature of linguistics), you need to explain that too and your expected distribution that would claim that (narrow).

My explanation (without proof), is that it's just a matter of awareness.

> All I can tell you is that my friend and I have had this conversation multiple times over many years and it seems very constant to me. I have no reason to believe they are lying and if they are they are doing so with an extreme level of consistency, which would be quite out of the norm.

Can you think of some kind of tests question (or string of questions) that could prove either? I have been thinking about it obviously, but I can't come up with any way to empirically test that there is or is no internal monologue. Consistency could simply mean that their rationalization is consistent.

I'll leave you this article, which I found quite interesting: https://news.ycombinator.com/item?id=43685072 The person lost language, and lost what we could consider human-level consciousness at the same time, and then recovered both at the same rate. Of course, there was brain damage, so it's not an empirical conclusion.

Also this book https://en.wikipedia.org/wiki/The_Origin_of_Consciousness_in... while partially debunked and being pop-sci to begin with, has wildly interesting insights into the internal monologue and at least draws extremely interesting questions.

There is us a book written by a woman who suffered a stroke. She lost the ability to speak and understand language. Yet she remained conscious. It took her ten years to fully recover. The book is called "A stroke of insight".

Conscious, like an animal or a baby. She could not function at all like a normal adult. Proves my point.

> It's to recite (or even apply) knowledge. To understand does actually require a world model.

This is a shell game, or a god of the gaps. All you're saying is that the models "understand" how to recite or apply knowledge or language, but somehow don't understand knowledge or language. Well what else is there really?

  > Well what else is there really?

I'd say there's a difference between a database and understanding. If they're the same, well I think Google created AGI a long time ago.

A database doesn't recite or apply knowledge, it stores knowledge.

It sure recites it when I query it

It makes perfect sense to say that the database understands your query. It also makes sense to say that the database's factorization of domain knowledge + domain queries exhibit at least a static domain understanding (which still isn't general ala AGI). This is the standard systems response to the Chinese Room.

The "general" part comes from whether that static aspect can be made dynamic and extensible. In what sense is a system that can be arbitrarily extended to "recite" or "apply" knowledge not AGI?

I would say that the database processes my query, not understands it. You reference the Chinese Room, but that's the thing. The Chinese Room processes the inputs without understanding them.

> The Chinese Room processes the inputs without understanding them.

I disagree, hence my reference to the systems response.

> We always had the math to show that scale wasn't enough

Math, to show that scale (presumably of LLMs) wasn't enough for AGI?

This sounds like it would be quite a big deal, what math is that?

As someone who is invested in researching said math, I can say with some confidence that it does not exist, or at least not in the form claimed here. That's the whole problem.

I would be ecstatic if it did though, so if anyone has any examples or rebuttal, I would very much appreciate it.

You're right that there is no purely mathematical argument; it's almost non-sensical to claim such.

Instead you can simply make the rather reasonable observation that LLMs are a product of their training distribution, which only contains partial coverage of all possible observable states of the world. Some highly regular observable states are thus likely missing, but an embodied agent (like a human) would be able to correctly reason about these states due to other sensory modes that observe these states. It's very clear that the number of possible states far exceeds any text describing those states (the number of states transcribed to text is vanishingly small in comparison, but they are locally coherent and quite regular when observed through other sensory modes like vision). That's the closest you can make to a mathematical argument.

As a result one common failure case for an LLM might involve describing a specific physical environment and associated set of actions that are in an area of low (or no) probability mass in the training data and asking the LLM to reason about it. Here's an example where I asked ChatGPT the following:

  imagine i walked into a room through a doorway, then turned around and walked right back out of the room. from the perspective of a person in the room facing the door, which side would my right arm be? please explain.

  Let’s break this down step by step.

  1. *You walk into the room through the doorway*:

   * As you're entering, the person in the room is facing the door. So, you're entering from the **outside**.
   * Your **right arm** will be on the **right side** of your body, just as it is normally when you're standing.

  2. *You turn around and walk right back out*:

   * When you turn around to walk out, you’ll be facing **away** from the room, looking back toward the doorway.
   * Since you're walking **back out**, the person inside the room is still facing the door, while you are walking away from them, so your **right arm** is still on the right side of your body.

  So, from the perspective of the person in the room, as you’re exiting, your *right arm* will still be on their *left side*. That’s because they’re facing the door, and your right side is to their left when you’re turning around and leaving.

  To visualize:

  * Imagine a door in front of them.
  * You walk through and turn around, so you're now facing away from them.
  * Your right arm is on your right side, but to them, it's on their left side since you’re now moving in the opposite direction.

My claim is more about that data processing is not enough. I was too vague and I definitely did not convey myself accurately. I tried to clarify a bit in a sibling comment to yours but I'm still unsure if it is sufficient tbh.

For embodiment, I think this is sufficient but not necessary. A key part to the limitation is that the agent cannot interact with its environment. This is a necessary feature for distinguishing competing explanations. I believe we are actually in agreement here, but I do think we need to be careful how we define embodiment. Because even a toaster can be considered a robot. It seems hard to determine what does not qualify as a body when we get to the itty gritty. But I think in general when people are talking about embodiment they are discussing the capability of being interventional.

By your elaboration I believe we agree since part of what I believe to be necessary is the ability to self-analyze (meta-cognition) to determine low density regions of its model and then to be able to seek out and rectify this (intervention). Data processing is not sufficient for either of those conditions.

Your prompt is, imo, more about world modeling, though I do think this is related. I asked Claude Sonnet 4.5 with extended thinking enabled and it also placed itself outside the room. Opus 4.1 (again with extended thinking), got the answer right. (I don't use a standard system prompt, though that is mostly to make it not syncopathic and to try to get it to ask questions when uncertain and enforce step by step thinking)

  From the perspective of the person in the room, your right arm would be on their right side as you walk out.
  
  Here's why: When you initially walk into the room facing the person, your right arm appears on their left side (since you're facing each other). But when you turn around 180 degrees to walk back out, your back is now toward them. Your right arm stays on your right side, but from their perspective it has shifted to their right side.

  Think of it this way - when two people face each other, their right sides are on opposite sides. But when one person turns their back, both people's right sides are now on the same side.

I don't think this proves a world model though. Misses are more important than hits, just as counter examples are more important than examples in any evidence or proof setting. But fwiw I also frequently ask these models variations on river crossing problems and the results are very shabby. A few appear spoiled now but they are not very robust to variation and that I think is critical.

I think an interesting variation of your puzzle is as follows

  Imagine you walked into a room through a doorway. Then you immediately turn around and walk back out of the room. 

  From the perspective of a person in the room, facing the door, which side would your right arm be? Please explain.

  Your right arm would be on their right side.
  When you turn around to walk back out, you're facing the same direction as the person in the room (both facing the door). Since you're both oriented the same way, your right side and their right side are on the same side.

It definitely isn't robust to red herrings[2], and I think that's a kicker here. It is similar to failure results I see in any of these puzzles. They are quite easy to break with small variations. And we do need to remember that these are models trained on the entire internet (including HN comments), so we can't presume this is a unique puzzle.

[0] http://0x0.st/K158.txt

[1] http://0x0.st/K15T.txt

[2] http://0x0.st/K15m.txt

Let me clarify. I was too vague and definitely did not express things accurately. That is on me.

We have the math to show that it can be impossible to distinguish two explanations through data processing alone. We have examples of this in science, a long history of it in fact. Fundamentally there is so much that we cannot conclude from processing data alone. Science (the search of knowledge) is active. It doesn't require just processing existing data, it requires the search for new data. We propose competing hypotheses that are indistinguishable from the current data and seek out the data which distinguishes them (a pain point for many of the TOEs like String Theory). We know that data processing alone is insufficient for explanation. We know it cannot distinguish confounders. We know it cannot distinguish causal graphs (e.g. distinguish triangular maps. We are able to create them, but not distinguish them through data processing alone). The problem with scaling alone is that it makes the assertions that data processing is enough. Yet we have so much work (and history) telling us that data processing is insufficient.

The scaling math itself also shows a drastic decline in performance with scale and often do not suggest convergence even with infinite data. They are power laws with positive concavity, requiring exponential increase in data and parameters for marginal improvements on test loss. I'm not claiming that we need zero test loss to reach AGI, but the results do tell us that if this is strongly correlated then we'll need to spend an exponential amount more to achieve AGI even if we are close. By our measures, scaling is not enough unless we are sufficiently close. Even our empirical results align with this as despite many claiming that scale is all we need, we are making significant changes to the model architectures and training procedures (including optimizers). We are making these large changes because throwing the new data at the old models (even when simply increasing the number of parameters) does not work out. It is not just the practicality, it is the results. The scaling claim has always been a myth used to drive investments since it is a nice simple story that says that we can get there by doing what we've already been doing, just more. We all know that these new LLMs aren't dramatic improvements off their previous versions, despite being much larger, more efficient, and having processed far more data.

[side note]: We even have my namesake who would argue that there are truths which are not provably true with a system that is both consistent and efficient (effectively calculable). But we need not go that far, as omniscience is not a requirement for AGI. Though it is worth noting for the limits of our models, since at the core this matters. Changing our axioms changes the results, even with the same data. But science doesn't exclusively use a formal system, nor does it use a single one.

My apologies for the much delayed reply as I have recently found myself with little extra time to post adequate responses. Your critiques are very interesting to ponder, so I thank you for posting them. I did want to respond to this one though.

I believe all of my counterarguments center around my current viewpoint that given the rapid rate of progress involved on the engineering side, it is no longer reasonable in deep learning theory to consider what is possible, and it is more interesting to try to outline hard limitations. This emposes a stark contrast between deep learning and classical statistics, as the boundaries in the latter are very clear and are not shared by the former.

I want to stress that at present, nearly every conjectured limitation of deep learning over the last several decades has fallen. This includes many back of the napkin, "clearly obvious" arguments, so I'm wary of them now. I think the skepticism all along has been fueled in response to hype cycles, so we must be careful not to make the same mistakes. There is far too much empirical evidence available to counter precise arguments against the claim that there is an underlying understanding within these models, so it seems we must resort to the imprecise to continue the debate.

Scaling, along one axis, suggests a high polynomial degree of additional compute (not exponential) is required for increasing improvements, this is true. But the progress over the last few years has occurred due to the discovery of new axes to scale on, which further reduces the error rate and improves performance. There are still many potential axes left untapped. What is significant about scaling to me is not how much additional compute is required, but the fact that the predicted bottom at the moment is very, very low, far lower than anything else we have ever seen, and that doesn't require any more data than we currently have. That should be cause for concern until we find a better lower bound.

> We all know that these new LLMs aren't dramatic improvements off their previous versions

No, I don't agree. This may be evident to many, but to some, the differences are stark. Our perceived metrics of performance are nonlinear and person-dependent, and these major differences can be imperceptible to most. The vast majority of attempts at providing more regular metrics or benchmarks that are not already saturated have shown that LLM development is not slowing down by any stretch. I'm not saying that LLMs will "go to the moon". But I don't have anything concrete to say they cannot either.

> We have the math to show that it can be impossible to distinguish two explanations through data processing alone.

Actually, this is a really great point, but I think this highlights the limitations of benchmarks and the requirements of capacity-based, compression-based, or other types of alternative data-independent metrics. With these in tow, it can be possible to distinguish two explanations. This could be a fruitful line of inquiry.

Fantastic comment!

I think current AI is a human language/behavior mirror. A cat might believe they see another cat looking in a mirror, but you can’t create a new cat by creating a perfect mirror.

Model based reinforcement learning is a thing and it is kind of a crazy idea. Look up temporal difference model predictive control.

The fundamental idea behind temporal difference is that you can record any observable data stream over time and predict the difference between past and present based on your decision variables (e.g. camera movement, actuator movement, and so on). Think of it like the Minecraft clone called Oasis AI. The AI predicts the response to a user provided action.

Now imagine if it worked as presented. The data problem would be solved, because you are receiving a constant stream of data every single second. If anything, the RL algorithms are nowhere near where they need to be and continual learning has not been solved yet, but the best known way is through automatic continual learning ala Schmidhuber (co-inventor of LSTMs along with Hochreiter).

So, model based control is solved right? Everything that can be observed can be controlled once you have a model!

Wrong. Unfortunately. You still need the rest of reinforcement learning: an objective and a way to integrate the model. It turns out that reconstructing the observations is too computationally challenging and the standard computational tricks like U-Nets learn a latent representation that is optimized for reconstruction rather than for your RL objectives. There is a data exchange problem that can only realistically be solved by throwing an even bigger model at it, but here is why that won't work either:

Model predictive control tries to find the best trajectory over a receding horizon. It is inherently future oriented. This means that you need to optimize through your big model and that is expensive to do.

So you're going to have to take shortcuts by optimizing for a specific task. You reduce the dimension of the latent space and stop reconstructing the observations. The price? You are now learning a latent space for your particular task, which is less demanding. The dream of continual learning with infinite data shatters and you are brought down to earth: it's better than what came before, but not that much better.

The thing is, achieving say, 99.99999% reliable AI would be spectacularly useful even if it's a dead end from the AGI perspective.

People routinely conflate the "useful LLMs" and "AGI", likely because AGI has been so hyped up, but you don't need AGI to have useful AI.

It's like saying the Internet is dead end because it didn't lead to telepathy. It didn't, but it sure as hell is useful.

It's beneficial to have both discussions: whether and how to achieve AGI and how to grapple with it, and how to improve a reliability, performance and cost of LLMs for more prosaic use cases.

It's just that they are separate discussions.

> The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to understand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.

That's the basic success of LLMs. They don't have much of a model of the world, and they still work. "Attention is all you need". Good Old Fashioned AI was all about developing models, yet that was a dead end.

There's been some progress on representation in an unexpected area. Try Perchance's AI character chat. It seems to be an ordinary chatbot. But at any point in the conversation, you can ask it to generate a picture, which it does using a Stable Diffusion type system. You can generate several pictures, and pick the one you like best. Then let the LLM continue the conversation continue from there.

It works from a character sheet, which it will create if asked. It's possible to start from an image and get to a character sheet and a story. The back and forth between the visual and textural domains seems to help.

For storytelling, such system may need to generate the collateral materials needed for a stage or screen production - storyboards, scripts with stage directions, character summaries, artwork of sets, blocking (where everybody is positioned on stage), character sheets (poses and costumes) etc. Those are the modeling tools real productions use to keep a work created by many people on track. Those are a form of world model for storytelling.

I've been amazed at how good the results I can get from this thing are. You have to coax it a bit. It tends to stay stuck in a scene unless you push the plot forward. But give it a hint of what happens next and it will run with it.

[1]https://perchance.org/ai-character-chat

Absolutely. AGI isn't a matter of adding more 9s. It's a matter of solving more "???"s. And those require not just work but also a healthy serving of luck.

As I understand it, to the breadth of LLMs was also something that was stumbled on kinda by accident, I understand they got developed as translators and were just 'smarter' than expected.

Also, to understand the world you don't need language. People don't think in language. Thought is understanding. Language is knowledge transfer and expression.

I think this a useful challenge to our normal way of thinking.

At the same time, "the world" exists only in our imagination (per our brain). Therefore, if LLMs need a model of a world, and they're trained on the corpus of human knowledge (which passed through our brains), then what's the difference, especially when LLMs are going back into our brains anyway?

Language isn't thought. It's a representation of thought.

Something to think about (hah!) is there are people without an internal monologue i.e. no voice inside their head they use when working out a problem. So they're thinking and learning and doing what humans do just fine with no little voice no language inside their head.

It's so weird that people literally seem to have a voice in their head they cannot control. For me personally my "train of thought" is a series of concepts, sometimes going as far as images. I can talk to myself in my head with language if I make a conscious effort to do so, just as I can breathe manually if I want. But if I don't, it's not really there like some people seem to have.

Probably there are at least two groups of people and neither really comprehends how the other thinks haha.

I think there are significantly more than 2, when you start to count variations through the spectrum of neurodiversity.

Spatial thinkers, for example, or the hyperlexic.

Meaning for hyperlexics is more akin to finding meaning in the edges of the graph, rather than the vertices. The form of language contributing a completely separate graph of knowledge, alongside its content, creating a rich, multimodal form of understanding.

Spatial thinkers have difficulty with procedural thinking, which is how most people are taught. Rather than the series of steps to solve the problem, they see the shape of the transform. LLMs as an assistive device can be very useful for spatial thinkers in providing the translation layer between the modes of thought.

[dead]

Are the particles that make up thoughts in our brain not also a representation of a thought? Isn't "thought" really some kind of Platonic ideal that only has approximate material representations? If so, why couldn't some language sentences be thoughts?

The sentence is the result of a thought. The sentence in itself does not capture every process that went into producing the sentence.

> The sentence in itself does not capture every process that went into producing the sentence.

A thought does not capture every process that went into producing the thought either.

I guess? I would argue that a thought doesn't "capture" anything though, because unlike language it's not a representational tool.

A painting of a landscape can capture details of the landscape it's representing. The landscape itself doesn't capture anything, it just is.

> The landscape itself doesn't capture anything, it just is.

Sure, but the landscape is something, namely an aggregate of particles. A thought in principle isn't its physical expression, but its information content, and it's represented in a human brain by some aggregate of particles. So no matter how you slice it, thoughts can only manifest within representations, and so calling language a representation of thought isn't some kind of dunk, because it also proves that human brains don't have thoughts.

It's not clear whether the information content of all possible human thoughts can be captured by language, but clearly at least some language expressions have the same information content as human thoughts.

Its very interesting to see how many people struggle to understand this.

We are paying the price now for not teaching language philosophy as a core educational requirement.

Most people have had no exposure to even the most basic ideas of language philosophy.

The idea all these people go to school for years and don't even have to take a 1 semester class on the main philosophical ideas of the 20th century is insane.

Language philosophy is not relevant, and evidently never was. It predicted none of what we're seeing and facilitated even less.

One must imagine Sisyphus happy and Chomsky incoherent with rage.

If it were that simple, LLMs wouldn't work at all.

I think it explains quite well why LLMs are useful in some ways but stupid in many other ways.

LLMs clearly think. They don't have a sense of object permanence, at least not yet, but they absolutely, indisputably use pretrained information to learn and reason about the transient context they're working with at the moment.

Otherwise they couldn't solve math problems that aren't simple rephrasings of problems they were trained on, and they obviously can do that. If you give a multi-step undergraduate level math problem to the human operator of a Chinese room, he won't get very far, while an LLM can.

So that leads to the question: given that they were trained on nothing but language, and given that they can reason to some extent, where did that ability come from if it didn't emerge from latent structure in the training material itself? Language plus processing is sufficient to produce genuine intelligence, or at least something indistinguishable from it. I don't know about you, but I didn't see that coming.

They very clearly do not think. If they did, they wouldn't be able to be fooled by so many simple tests that even a very small (and thus, uneducated) human would pass.

Are you really claiming that something doesn't think if it's possible to fool it with simple tricks?

Seriously?

A world model can not exist, the context windows aren't even near big enough for that. Weird that every serious scientist agrees on AGI not being a thing in the next decades. LLMs are good if you train them for a specific thing. Not so much if you expect them to explain the whole world to you. This is not possible yet.

> LLMs seem to udnerstand language therefore they've trained a model of the world.

This isn’t the claim, obviously. LLMs seem to understand a lot more than just language. If you’ve worked with one for hundreds of hours actually exercising frontier capabilities I don’t see how you could think otherwise.

> This isn’t the claim, obviously.

This is precisely the claim that leads a of lot people to believe that all you need to reach AGI is more compute.

What I mean here is that this is certainly not what Dwarkesh would claim. It’s a ludicrous strawman position.

Dwarkesh is AGI-pilled and would base his assumption of a world model on much more impressive feats than mere language understanding.

Watching the video it seems that Dwarkesh doesn't really have a clue what he's confidently talking about yet running fast with his personal half-baked ideas, to the points where it gets both confusing and cringe when Karpathy apparently manages to make sense of it and yes-anding the word salad AK. Karpathy is supposedly there to clear up misunderstanding yet lets all the nonsense Dwarkesh is putting before him slide.

"ludicrous" sure but I wouldn't be so certain about "strawman" or that Dwarkesh has a consistent view.

To me, it's a matter of a very big checklist - you can keep adding tasks to the list, but if it keeps marching onwards checking things off your list, some day you will get there. whether it's a linear or asymptotic march, only time will tell.

I don't know if you will get there, that's far from clear at this stage.

Did you see the recent video by Nick Beato [1] where he asks various models about a specific number? The models that get it right are the models that consume youtube videos, because there was a youtube video about that specific number. It's like, these models are capable of telling you about very similar things that they've seen, but they don't seem like they understand it. It's totally unclear whether this is a quantitative or qualitative gap.

[1] https://www.youtube.com/watch?v=TiwADS600Jc

That's like saying that if we image every neuron in the brain we will understand thinking. We can build these huge databases and they tell us nothing about the process of thinking.

What if we copy the functionality of every neuron? what if we simply copy all the skills that those neurons compute?

Do we even know the functionality of every neuron?

Not yet.

[dead]

What "9" do you add to AGI? I don't think we even have the axes defined, let alone a way to measure them. "Mistakes per query?" It's like Cantor's diagonal test, where do we even start?

Problem is that these models feels like they are 8 and getting more 8's

(maybe 7)

I don’t have a deep understand of LLMs but don’t they fundamentally work on tokens and generate a multi-dimensional statistical relationship map between tokens?

So it doesn’t have to be LLM. You could theoretically have image tokens (though I don’t know in practice, but the important part is the statistical map).

And it’s not like my brain doesn’t work like that either. When I say a funny joke in response to people in a group, I can clearly observe my brain pull together related “tokens” (Mary just talked about X, X is related to Y, Y is relevant to Bob), filter them, sort them and then spit out a joke. And that happens in like less than a second.

Yes! Absolutely. And this is likely what would be necessary for anything approaching actual AGI. And not just visual input, but all kinds of sensory input. The problem is that we have no ability, not even close, to process that even near the level of a human yet, much less some super genius being.

yeah that "model of the world" would mean:

babies are already born with "the model of the world"

but a lot of experiments on babies/young kids tell otherwise

> yeah that "model of the world" would mean: babies are already born with "the model of the world"

No, not necessarily. Babies don't interact with the world only by reading what people wrote wikipedia and stackoverflow, like these models are trained. Babies do things to the world and observe what happens.

I imagine it's similar to the difference between a person sitting on a bicycle and trying to ride it, vs a person watching videos of people riding bicycles.

I think it would actually be a great experiment. If you take a person that never rode a bicycle in their life and feed them videos of people riding bicycles, and literature about bikes, fiction and non-fiction, at some point I'm sure they'll be able to talk about it like they have huge experience in riding bikes, but won't be able to ride one.

We’ve been thinking about reaching the singularity from one end, by making computers like humans, but too little thought has been given to approaching the problem from the other end: by making babies build their world model by reading Stack Overflow.

The “Brave New World meets OpenAI” model where bottle-born babies listen to Stack Overflow 24 hours a day until they one day graduate to Alphas who get to spend Worldcoin on AI-generated feelies.

That's it. Now you've done it! I will have stackoverflow Q&A, as well as moderator comments and closings of questions playing 24/7 to my first not yet born child! Q&A for the knowledge and the mod comments for good behavior, of course. This will lead to singularity in no time!

It's a lot more complicated than that.

You have instincts, right? Innate fears? This is definitely something passed down through genetics. The Hawk/Goose Effect isn't just limited to baby chickens. Certainly some mental encoding passes down through genetics as how much the brain controls, down to your breathing and heartbeat.

But instinct is basic. It's something humans are even able to override. It's a first order approximation. Inaccurate to do meaningfully complex things, but sufficient to keep you alive. Maybe we don't want to call the instinct a world model (it certainly is naïve) but can't be discounted either.

In human development, yeah, the lion's share of it happens post birth. Human babies don't even show typical signs of consciousness, even really till the age of 2. There's many different categories of "awareness" and these certainly grow over time. But the big thing that makes humans so intelligent is that we continue to grow and learn through our whole lifetimes. And we can pass that information along without genetics and have very advanced tools to do this.

It is a combination of nature and nurture. But do note that this happens differently in different animals. It's wonderfully complex. LLMs are quite incredible but so too are many other non-thinking machines. I don't think we should throw them out, but we never needed to make the jump to intelligence. Certainly not so quickly. I mean what did Carl Sagan say?

One of the biggest mysteries of humans Vs LLMs is that LLMs need an absurd amount of data during pre training, then a little bit of data during fine tuning to make them behave more human. Meanwhile humans don't need any data at all, but have the blind spot that they can only know and learn about what they have observed. This raises two questions. What is the loss function of the supervised learning algorithm equivalent? Supposedly neurons do predictive coding. They predict what their neighbours are doing. That includes input only neurons like touch, pain, vision, sound, taste, etc. The observations never contain actions. E.g. you can look at another human, but that will never teach you how to walk because your legs are different from other people's legs.

How do humans avoid starving to death? How do they avoid leaving no children? How do they avoid eating food that will kill them?

These things require a complicated chain of actions. You need to find food, a partner and you need to spit out poison.

This means you need a reinforcement learning analogue, but what is going to be the reward function equivalent? The reward function can't be created by the brain, because it would be circular. It would be like giving yourself a high, without even needing drugs. Hence, the reward signal must remain inside the body but outside the brain, where the brain can't hack it.

The first and most important reward is to perform reproduction. If food and partners are abundant, the ones that don't reproduce simply die out. This means that reward functions that don't reward reproduction disappear.

Reproduction is costly in terms of energy. Do it too many times and you need to recover and eat. Hunger evolved as a result of the brain needing to know about the energy state of the body. It overrides reproductive instincts.

Now let's say you have a poisonous plant that gives you diarrhea, but you are hungry. What stops you from eating it? Pain evolves as a response to a damaged body. Harmful activities signal themselves in the form of pain to the brain. Pain overrides hunger. However, what if the plant is so deadly that it will kill you? The pain sensors wouldn't be fast enough. You need to sense the poison before it enters your body. So the tongue evolves taste and cyanide starts tasting bitter.

Notice something? The feelings only exist internally inside the human body, but they are all coupled with continued survival in one way or another. There is no such thing for robots or LLMs. They won't accidentally evolve a complex reward function like that.

  > Meanwhile humans don't need any data at all

1) Certainly the data I discussed exists. No creature comes out a blank slate. I'll be bold enough to say that this is true even for viruses, even if we don't consider them alive. Automata doesn't mean void of data and I'm not sure why you'd ascribe this to life or humans.

2) humans are processing data from birth (technically before too but that's not necessary for this conversation and I think we all know that's a great way to have an argument and not address our current conversation). This is clearly some active/online/continual/ reinforcement/wherever-word-you-want-to-use learning.

It's weird to suggest an either or situation. All evidence points to "both". Looking at different animals even see both but also with different distributions.

I think it's easy to over simplify the problem and the average conversation tends to do this. It's clearly a complex with many variables at play. We can't approximate with any reasonable accuracy by ignoring or holding them constant. They're coupled.

  > The reward function can't be created by the brain, because it would be circular.

  > It would be like giving yourself a high, without even needing drugs

Just because there are externally driven or influenced goals doesn't mean internal ones can't exist. Our emotions can be driven both externally and internally.

  > Notice something?

> babies are already born with "the model of the world"

> but a lot of experiments on babies/young kids tell otherwise

I believe they are born with such a model? It's just that model is one where mummy still has fur for the baby to cling on to? And where aged something like 5 to 8 it's somehow useful for us to build small enclosures to hide in, leading to a display of pillow forts in the modern world?

damn I guess I had to be more specific:

"LLM-level world-detail knowledge"

I think I'm even more confused now about what you mean…

Lots of experiments show that babies develop import capabilities at roughly the same times. That speaks to inherited abilities.

love the intentional use of udnerstand, brilliant!

"just a matter of adding more 9s" is a wild place to use a "just" ...

I have a very surface level understanding of AI, and yet this always seemed obvious to me. It's almost a fundamental law of the universe that complexity of any kind has a long tail. So you can get AI to faithfully replicate 90% of a particular domain skill. That's phenomenal, and by itself can yield value for companies. But the journey from 90%-100% is going to be a very difficult march.

The nines comment was in the context of self driving cars which I can see because you are never perfect driving and accidents can be fatal.

Some AI is like chess though, where they steadily advance in ELO ranking.

The last mile problem is inescapable!

Reminds me of a time-honored aphorism in running:

A marathon consists of two halves: the first 20 miles, and then the last 10k (6.2mi) when you're more sore and tired than you've ever been in your life.

This is 100% unrelated to the original article but I feel like there's an underreported additional first half. As a bigger runner who still loves to run, the first two or three miles before I have enough endorphins to get into the zen state that makes me love running is the first half, then it's 17 miles of this amazing meditative mindset. Then the last 10k sucks.

Just, ftr, endorphins cannot pass the blood brain barrier

http://hopkinsmedicine.org/health/wellness-and-prevention/th...

So a runner’s high is more like a literal high then? Interesting

Ha! Endorphins are "endogenous opioid peptides produced by the pituitary and hypothalamus glands that function as the body's natural painkillers and mood regulators".

"They are part of the endogenous opioid system", so either way was talking about literal highs.

The endocannabinoid system (I hope that I have the spelling correct) is a relatively recent discovery (1980s on), and is quite fascinating on how integral to the human body it is

I suspect that is true for many difficult physical goals.

My dad told me that the first time you climb a mountain, there will likely be a moment not too distant from the top when you would be willing to just sit down and never move again, even at the risk to your own life. Even as you can see the goal not far away.

He also said that it was a dangerous enough situation that as a climb leader he'd start kicking you if he had to, if you sat down like that and refused to keep climbing. I'm not a climber myself, though, so this is hearsay, and my dad is long dead and unable to remind me of what details I've forgotten.

I think I hated life most after 20 miles. Especially in training.

Why just run 20 miles then?

Because it would be 16 miles of bliss and 4 miles of torture then. The point is the last section of the run is always significantly harder - it’s even the same for 5k

Because then it wouldn't be a challenge and nobody would care about the achievement.

I’m curious do ultramarathoners feel the same way about the rest of the race past 20 miles?

I've heard it claimed that an ultramarathon is fundamentally a different experience because while it definitely requires excellent physical stamina, it has a large mental component to it, as well as a much bigger focus on nutrition. Very different sort of race, I guess.

there are multiple cycles from highs to lows and back and then typically a larger dominant split similar what was discussed here for the marathon but scaled to the distance.

Yes. I've run numerous 50Ks, 50 milers, 100ks and 100 milers. I felt like crap after 20 miles in almost all of them. Most of getting better at ultramarathons is learning to keep going when feeling like crap. Oddly, the one race that was an exception is probably the hardest one of them I did on paper - in that case I was going so slowly from the beginning that I never really hit a 20 mile wall.

the split would be first 80 and las t 20 miles +-10 miles.

This makes no sense.

20 miles is still a challenge, and how many people run marathons because someone else is impressed if you run 26 miles, but couldn't care less if you run 20?

because that'd be quitting the race with 6.2 miles left to go

You could run a half marathon.

yeah but anyone can do that

FWIW, Karpathy literally says, multiple times, that he thinks we never left the exponential - that all human progress over last 4+ centuries averages out to that smooth ~2% growth rate exponential curve, that electricity and computing and AI are just ways we keep it going, and we'll continue on that curve for the time being.

It's the major point of contention between him and the host (who thinks growth rate will increase).

The thing about this, though - cars have been built before. We understand what's necessary to get those 9s. I'm sure there were some new problems that had to be solved along the way, but fundamentally, "build good car" is known to be achievable, so the process of "adding 9s" there makes sense.

But this method of AI is still pretty new, and we don't know it's upper limits. It may be that there are no more 9s to add, or that any more 9s cost prohibitively more. We might be effectively stuck at 91.25626726...% forever.

Not to be a doomer, but I DO think that anyone who is significantly invested in AI really has to have a plan in case that ends up being true. We can't just keep on saying "they'll get there some day" and acting as if it's true. (I mean you can, just not without consequences.)

While you are right about the broader (and sort of ill defined) chase toward 'AGI' - another way to look at it is the self driving car - they got there eventually.And, if you work on applications using LLMs you can pretty easily see that Karpathy's sentiment is likely correct. You see it because you do it. Even simple applications are shaped like this, albeit each 9 takes less time than self driving cars for a simple app.. it still feels about right.

> another way to look at it is the self driving car - they got there eventually

Current self driving cars only work in American roads. Maybe Canada too, not sure how their roads are. Come to Europe/anywhere else and every other road would be intractable. Much tighter lanes, many turns you have a little mirror to see who's coming on the other side, single car at a time lanes that you need to "understand" who goes first, mountain roads where you sometimes need to reverse for 100m when another car is coming so it's wide enough that they can pass before you can keep going forward, etc.

Many things like this that would require another 2 or 3 "nines" as the guy put it than acceptable quality in American huge roads.

https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQ4NWIt...

Waymo has promised to launch In London and Tokyo next year. New York, London, Tokyo probably covers the entire spectrum of difficulty for self driving cars, maybe we need to include Mumbai as the final boss but I would be happy saying self driving is solved if the above 3 cities have a working 24/7 self driving fleet

The final boss could be something like Scottland mountain roads, or some of the million beaches on a cliff in Greece where this "you have to first reverse" kinda situation happens every 30 seconds.

No no, the final boss is a dozen of them have to race around such roads, no crashes allowed, and they must all finish at exactly the same time. Blindfolded.

Give the Waymo guys some credit - San Francisco isn't the suburbs of Houston. It might not be quite the same as a 1000 year old city in Europe, but it's no snack either.

> another way to look at it is the self driving car - they got there eventually.

No they did not. Elon has been saying Tesla will get there “next year” since 2015. He is still saying that, and despite changing definitions, we still are not there.

They = Waymo

Karpathy talked about Waymo, and he said they aren't there yet. They still have humans in the loop via telemetry and there are parts of cities they won't go to.

Karpathy is biased when it comes to self driving. Example: You can't both have humans teleoperating like he claimed, AND have cones disabling Waymos. Waymo's mistakes such as one where they were going around a parking lot honking at each other tells you they don't have humans in the loop except in the most extreme cases. He's likely correct that it's not self driving 100% of the time, but what if it's 99.999% and in the 0.001% the humans have to tell the Waymo how to get out of a very tricky situation?

Just go ride in one and make the decision for yourself. By any reasonable person's definition they are self driving.

[dead]

Mm hmm, you probably think airplanes aren't real too.

i guess the comment you replied proves the actual point "we may never get there, but it will be enough for the market".

sigh, i guess it's time to laugh on that video compilation of elon saying "next week" for 10yrs straight and then cry seeing how much he made of doing that.

It's a good way to think about lots of things. It's Pareto efficiency. The 80/20 rule

20% of your effort gets you 80% of the way. But most of your time is spent getting that last 20%. People often don't realize that this is fractal like in nature, as it draws from the power distribution. So of that 20% you still have left, the same holds true. 20% of your time (20% * 80% = 16% -> 36%) to get 80% (80% * 20% => 96%) again and again. The 80/20 numbers aren't actually realistic (or constant) but it's a decent guide.

It's also something tech has been struggling with lately. Move fast and break things is a great way to get most of the way there. But you also left a wake of destruction and tabled a million little things along the way. Someone needs to go back and clean things up. Someone needs to revisit those tabled things. While each thing might be little, we solve big problems by breaking them down into little ones. So each big problem is a sum of many little ones, meaning they shouldn't be quickly dismissed. And like the 9's analogy, 99.9% of the time is still 9hrs of downtime a year. It is still 1e6 cases out of 1e9. A million cases is not a small problem. Scale is great and has made our field amazing, but it is a double edged sword.

I think it's also something people struggle with. It's very easy to become above average, or even well above average at something. Just trying will often get you above average. It can make you feel like you know way more but the trap is that while in some domains above average is not far from mastery in other domains above average is closer to no skill than it is to mastery. Like how having $100m puts your wealth closer to a homeless person than a billionaire. At $100m you feel way closer to the billionaire because you're much further up than the person with nothing but the curve is exponential.

https://youtu.be/bpiu8UtQ-6E?si=ogmfFPbmLICoMvr3

"I'm closer to LeBron than you are to me."

I also quite like the way he puts it. However, from a certain point onward, the AI itself will contribute to the development—adding nines—and that’s the key difference between this analogy of nines in other systems (including earlier domain‑specific ML ones) and the path to AGI. That's why we can expect fast acceleration to take off within two years.

I don't think we can be confident that this is how it works. It may very well be that our level of intelligence has a hard limit to how many nines we can add, and AGI just pushes the limit further, but doesn't make it faster per se.

It may also be that we're looking at this the wrong way altogether. If you compare the natural world with what humans have achieved, for instance, both things are qualitatively different, they have basically nothing to do with each other. Humanity isn't "adding nines" to what Nature was doing, we're just doing our own thing. Likewise, whatever "nines" AGI may be singularly good at adding may be in directions that are orthogonal to everything we've been doing.

Progress doesn't really go forward. It goes sideways.

Intuition of someone who has put in a decade or two of wondering openly can't me discounted as easily as someone who might be a beginner to it.

AGI to encompass all of humanity's knowledge in one source and beat every human on every front might be a decade away.

Individual agents with increased agency adequately covering more and more abilities consistently? Seems like a steady path that can be seen into the horizon to put one foot in front of the other.

For me, the grain of salt I'd take Karpathy with is much, much, smaller than average, only because he tries to share how he thinks and examines his own understanding and changes it.

His ability to explain complex things simply is something that for me helps me learn and understand things quicker and see if I arrive at something similar or different, and not immediately assume anything is wrong, or right without my understanding being present.

Adding nines to nature is exactly what humans are doing. We are nature. We are part of the natural order.

Anything that exists is part of nature, there can be no exceptions.

If I go burn a forest down on purpose, that is in fact nature doing it. No different than if a dolphin kills another animal for fun or a chimp kills another chimp over a bit of territory. Insects are also every bit as 'vicious' in their conquests.

It's also assuming that all advances in AI just lead to cold hard gains, people have suggested this before but would a sentient AI get caught up in philosophical, silly or religious ideas? Silicone investor types seem to hope it's all just curing diseases they can profit from, but it might also be, "let's compose some music instead"?

AI doesn't have hopes and desires or something it would rather be doing. It has a utility function that it will optimise for regardless of all else. This doesn't change when it gets smarter, or even when it gets super-intelligence.

Isn't that one of the measures of when it becomes an AGI? So that doesn't help you with however many nines we are away from getting an AGI.

Even if you don't like that definition, you still have the question of how many nines we are away from having an AI that can contribute to its own development.

I don't think you know the answer to that. And therefore I think your "fast acceleration within two years" is unsupported, just wishful thinking. If you've got actual evidence, I would like to hear it.

AI has been helping with the development of AI ever since at least the first optimising compiler or formal logic circuit verification program.

Machine learning has been helping with the development of machine learning ever since hyper-parameter optimisers became a thing.

Transformers have been helping with the development of transformer models… I don't know exactly, but it was before ChatGPT came out.

None of the initials in AGI are booleans.

But I do agree that:

> "fast acceleration within two years" is unsupported, just wishful thinking

Nobody has any strong evidence of how close "it" is, or even a really good shared model of what "it" even is.

AGI is when it is general. a narrow AI trained only on coding and training AIs would contribute to the acceleration without being AGI itself.

I doubt this. General intelligence will be a step change not a gentle ramp. If we get to an architecture intelligent enough to meaningfully contribute to AI development, we'll have already made it. It'll simply be a matter of scale. There's no 99% AGI that can help build 100% AGI but for some reason can't drive a car or cook a meal or work an office job.

> However, from a certain point onward, the AI itself will contribute to the development—adding nines—and that’s the key difference between this analogy of nines in other systems (including earlier domain‑specific ML ones) and the path to AGI.

There's a massive planet-sized CITATION NEEDED here, otherwise that's weapons grade copium.

It's a possibility, but far from certainty.

If you look at it differently, assembly language may have been one nine, compilers may have been the next nine, successive generations of language until ${your favorite language} one more nine, and yet, they didn't get us noticeably closer to AGI.

I think the 9's include this assumption.

Ha, I often speak of doing the first 90% of the work, and then moving on to the following 90% of the work...

> The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.

— Tom Cargill, Bell Labs (September 1985)

https://dl.acm.org/doi/pdf/10.1145/4284.315122

I use "The project is 90% ready, now we only have to do the other half"

92% is half actually - RuneScape Players

I think a ton of people see a line going up and they think exponential. When in Reality, the vast majority of the time it’s actually logistic.

Given the physical limits of the universe and our planet in particular, yeah, this is pretty much always true. The interesting question is: what is that limit, and: how many orders of magnitude are we away from leveling off?

For reasoning the data suggests models are in the logistic domain?

Grok 3 - Grok 3 reasoning: 15% increase in training compute for a 25% uplift in inteligence

Grok 3 reasoning - Grok 4: 80% increase in training compute for a 15% uplift in inteligence.

Inteligence: Source Atrifical Analysis

Training Compute: Source https://youtu.be/MtYsUdfZPMA?t=162

I mean the cost line does look somewhat exponential…

I think the point Andrej was making here is that in some areas, such as self driving, the cost of failure is extremely high (maybe death), so 99.9% reliable doesn't cut it, and therefore doesn't mean you are almost done, or have done 99.9% of the work. It's "The last 10% is 90% of the work" recursively applied.

He was also pointing out that the same high cost of failure consideration applies to many software systems (depending on what they are doing/controlling). We may already be at the level where AI coding agents are adequate for some less critical applications, but yet far away from them being a general developer replacement. I see software development as something that uses closer to 100% of your brain than 10% - we may well not see AI coding agents approach human reliability levels until we have human level AGI.

The AI snake oil salesmen/CEOs like to throw out competitive coding or math olympiad benchmarks as if they are somehow indicative of the readiness of AI for other tasks, but reliability matters. Nobody dies or loses millions of dollars if you get a math problem wrong.

The thing is, the example of the "march of nines" is self-driving cars. These deal with roads and roads are interface between the chaos of the overall world and a system that has quite well-defined rules.

I can imagine other task on a human/rules-based "frontier" would have a similar quality. But I think there are others that are going to be inaccessible entirely "until AGI" (or something). Humanoid robots moving freely in human society would an example I think.

This is an amazing quote that really applies to all software development

Drawn from Karpathy killing a bunch of people by knowingly delivering defective autonomous driving software instead of applying basic engineering ethics and refusing to deploy the dangerous product he was in charge of.

In Texas, it's a $5K a day fine to call yourself an engineer, even in a job title on LinkedIn, and actually not be a certified engineer.

has anyone ever been convicted/fined based on this statute?

Well, maybe not all. I’ve definitely built CRUD UIs that were linear in effort. But certainly anything technically challenging or novel.

When I worked at Facebook they had a slogan that captured this idea pretty well: “this journey is 1% finished”.

Copied from Amazon's "Day 1".

like leveling to 99 in old school runescape

The first 92% and the last 92%, exactly.

Or Diablo 2

i don't remember the end-game of the original Diablo; however, in diablo III and IV everyone i've tried to play the game gets bored in the run up to max level. I always tell them "i skip that part as much as possible, because that's not the game. That's just the story!"

Once you hit max level in III and IV, the game actually "begins."

and to explain the Diablo 2 Reference, the amount of time/effort it takes to go from level 98 to level 99 (the max level), is the same amount of time it takes to go from level 1 to level 98. I've heard "2 weeks" as a rough estimate of "unhealthy playtime", at least solo.

Perfect analogy.

Infinitely big little numbers

Academia has rediscovered itself

Signal attenuation, a byproduct of entropy, due to generational churn means there's little guarantee.

Occam's Razor; Karpathy knows the future or he is self selecting biology trying to avoid manual labor?

His statements have more in common with Nostradamus. It's the toxic positivity form of "the end is nigh". It's "Heaven exists you just have to do this work to get there."

Physics always wins and statistics is not physics. Gamblers fallacy; improvement of statistical odds does not improve probability. Probability remains the same this is all promises of some people who have no idea or interest in doing anything else with their lives; so stay the course.

>> Heaven exists you just have to do this work to get there.

Or perhaps Karpathy has a higher level understanding and can see a bigger picture?

You've said something about heaven. Are you able to understand this statement, for example: "Heaven is a memeplex, it exists." ?

Also note Karpathy notes the problems with agents are tractable but hard.

He's vague on the paths being explored to resolve them. His "higher level" view is probably awareness the solutions to software problems are hardware based fixes, but he cannot say that to software developers. Which has been the back and forth of tech since I was a kid in the 80s writing Basic; new state management unlocked by old software logic being embedded into new hardware.

Two main problems to solve for: too many people bought in to a status quo. And much simpler, the actual engineering of new hardware. One is only resolved by generational churn without resorting to all out police state action. So tech jobs as we know them will fade away slowly to not upset too many, and younger generations will not care as they will never experience anything else.

"Higher" than an EE with an MSc in elastic structures, ~30 years industry experience, now working with PhDs across the spectrum on energy models to embed in chips? Energy models in part, inferred from categorization of LLM contents and compression of those contents into geometric functions like I described?

"Higher level" implies acceptance of geometric structure. You place tokens like a Chomsky diagrams at each step up and down, where you should see parameters to transform geometry of the structure.

My team works "above" the contrived state management of software workers to more efficiently sync memory matrix to display matrix. LLMs are a form of compression [1]. My team is working on compressing them further into sets of points that make up each glyph and functions to recreate them.

Electromagnetic geometry transforms hardcoded[2] into hardware so reduce energy use of all the outdated string mangling of software dev as most know it.

What's higher level, relative to our machines, than design and implementation of the machine?

DnD dungeon master versus WOTC game designer.

Notice outside how there are no words and philosophy? Just color gradient and geometry?

Notice inside the human body no philosophy or words?

Language is not intelligence it's an emergent phenomena of geometry created by fundamental forces of physics organizing matter at various speeds relative to light.

You've read too much into an ultimately arbitrary statement meant to invoked a subtext, a subtle emotion context. You think of language as Legos, when it is music to feel.

[1] https://arxiv.org/abs/2309.10668 [2] https://iopscience.iop.org/article/10.1088/1742-6596/2987/1/...

Notably the scaling law paper shows result graphs on log-scale

In my experience with AI it's steeper than that: the jump from 90% to 99% is much harder than the jump from 0 to 90%

if it works 90% of the time that means it fails 10% of the time, to get to 1% failure rate is a 10x improvement and from 1% failure rate to a 0.1% failure rate is also a 10x improvement

First time being hearing it be called "march of nines", did Tesla make the term, I thought it was an Amazon thing

something that replaces humans doesn’t need to be 99.9999% reliable, it just has to be better than the humans it replaces.

But to be accepted by people, it has to be better than humans in the specific ways that humans are good at things. And less bad than humans in the ways that they're bad at things.

When automated solutions fail in strange alien ways, it understandably freaks people out. Nobody wants to worry about if a car will suddenly serve into oncoming traffic because of a sensor malfunction. Comparing incidents-per-miles-driven might make sense from a utilitarian perspective, just isn't good enough for humans to accept replacement tech psychologically, so we do have to chase those 9s until they can handle all the edge cases at least as well as humans.

Waymo has been growing rapidly. It still makes mistakes, but leas often than humans, and its riders are willing to accept the trade off given the benefits.

The question is how many nines are humans.

Humans adapt and become more nines the more they learn about something. Humans also are liable in a lawful sense. This is a huge factor in any AI use case.

So, it's not really nines, but the lack of continuous learning and legal issues.

Continuous learning seems like one of real criteria of AGI

So when you say first 9, you mean like Anthropic's uptime on models, right?

You know what they say, a Silicon Valley 9 is a 10 anywhere else. Or something like that.

I assume you're describing the fact that Silicon Valley culture keeps pushing out products before they're fully baked?

It was a half-baked joke along the lines of "A New York 6 is a Utah 10" or something like that, one of those that crops up in a sitcom every once in a while. Like a silicon valley product, I should have let it develop further before pushing it out for public consumption.

[dead]

[dead]

[flagged]

This is not exactly new information[1]. You may have a point that it was not presented to customers this way though.

[1] https://x.com/elonmusk/status/1382458022367162370

The lie is older than that:

https://web.archive.org/web/20161020091022/https://tesla.com...

https://motherfrunker.ca/fsd/

The fact that lie is old only makes it worse that Musk, Karpathy, and Tesla generally have still not taken responsibility for the lie. They are still not willing to refund the money they took for something they did not deliver.

One of the most brilliant AI minds on the planet, and he's focused on education. How to make all the innovation of the last decade accessible so the next generation can build what we don't know how to do today.

No magical thinking here. No empty blather about how AI is going to make us obsolete with the details all handwaved away. Karpathy sees that, for now, better humans are the only way forward.

Also, speculation as to why AI coders are "mortally terrified of exceptions": it's the same thing OpenAI recently wrote about, trying to get an answer at all costs to boost some accuracy metric. An exception is a signal of uncertainty indicating that you need to learn more about your problem. But that doesn't get you points. Only a "correct answer" gets you points.

Frontier AI research seems to have yet to operationalize a concept of progress without a final correct answer or victory condition. That's why AI is still so bad at Pokemon. To complete open-ended long-running tasks like Pokemon, you need to be motivated to get interesting things to happen, have some minimal sense of what kind of thing is interesting, and have the ability to adjust your sense of what is interesting as you learn more.

It's nice seeing commentary from someone who is both knowledgable in AI and NOT trying to pump the AI bag.

Right now the median actor in the space loudly proclaims AGI is right around the corner, while rolling out pornbots/ads/in-chat-shopping, which generally seems at odds with a real belief that AGI is close (TAM of AGI must be exponentially larger than the former).

Zvi made this point the other day, and then this counterpoint which I agree with more - if you think AGI is soon but you need to keep up the exponential datacenter growth for 2-3 years (or whatever “around the corner” means for the company in question) then a land-grab on consumer ARR is a faster way to short-term revenue (and therefore higher valuations at your next round).

OAI is also doing F100 and USG work; it takes longer to book the revenue though.

By selling porn and shopping you are in some sense weakening your position with regulators which you'll need when AGI starts displacing jobs - but you can also imagine thinking that this is a second order problem and winning the race is way more urgent.

Well if they think AGI is soon with no evidence then that's not worth a whole lot

If they think AGI is soon AND they can back it up, why can't they convince investors and have them make up the difference for the next 2-3 years? Seems like a much better strategy than the pornbots and a weaker position with regulators

I mean if anything this makes it seem like they're not confident they'll get to AGI that soon.. they're not optimizing for that eventual outcome

Why does it have to be a dichotomy? Raise money from AGI-pilled investors with an AGI pitch. Raise more money from AGI-skeptics with a B2C growth story.

Moar money == moar GPUs.

What exactly is a "pornbot"?

>An exception is a signal of uncertainty indicating that you need to learn more about your problem

No, that would be a warning. Ab exceprion is a signal something failed and it was impossible to continue

Many exceptions are recoverable. This sometimes depends on the context, and on how well polished the software is

Yes. Note how I didn't say impossible to recover, just impossible to continue.

The execution couldn't continue in one path due to an error it needed to be caught in another path.

The difference with standard conditional mechanisms like if loops is mostly semantical. Exceptions are unforeseen errors, (technically they are sets of errors, which can have size 1, but the syntax is designed for catching groups of errors, if you want to react to a single error case you could also just use a condition with a return value and it ceases being an exception. )

Imagine a PhD mortally terrified of exceptions!

Now I see why Karpathy was talking of RL up-weights as if they were a destructive straw-drawn line of a drug for an LLM's training.

It looks like Andrej's definition of "agent" here is an entity that can replace a human employee entirely - from the first few minutes of the conversation:

When you’re talking about an agent, or what the labs have in mind and maybe what I have in mind as well, you should think of it almost like an employee or an intern that you would hire to work with you. For example, you work with some employees here. When would you prefer to have an agent like Claude or Codex do that work?

Currently, of course they can’t. What would it take for them to be able to do that? Why don’t you do it today? The reason you don’t do it today is because they just don’t work. They don’t have enough intelligence, they’re not multimodal enough, they can’t do computer use and all this stuff.

They don’t do a lot of the things you’ve alluded to earlier. They don’t have continual learning. You can’t just tell them something and they’ll remember it. They’re cognitively lacking and it’s just not working. It will take about a decade to work through all of those issues.

He’s not just talking about agents good enough to replace workers. He’s talking about whether agents are currently useful at all.

>Overall, the models are not there. I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it’s not. It’s slop. They’re not coming to terms with it, and maybe they’re trying to fundraise or something like that. I’m not sure what’s going on, but we’re at this intermediate stage. The models are amazing. They still need a lot of work. For now, autocomplete is my sweet spot. But sometimes, for some types of code, I will go to an LLM agent.

>They kept trying to mess up the style. They’re way too over-defensive. They make all these try-catch statements. They keep trying to make a production code base, and I have a bunch of assumptions in my code, and it’s okay. I don’t need all this extra stuff in there. So I feel like they’re bloating the code base, bloating the complexity, they keep misunderstanding, they’re using deprecated APIs a bunch of times. It’s a total mess. It’s just not net useful. I can go in, I can clean it up, but it’s not net useful.

I don't think he is saying agents are not useful at all, just that they are not anywhere near the capability of human software developers. Karpathy later says he used agents to write the Rust translation of algorithms he wrote in Python. He also explicitly says that agents can be useful for writing boilerplate or for code that can be very commonly found online. So I don't think he is saying they are not useful at all. Instead, he is just holding agents to a higher standard of working on a novel new codebase, and saying they don't pass that bar.

Tbh I think people underestimate how much software development work is just writing boilerplate or common patterns though. A very large percentage of the web development work I do is just writing CRUD boilerplate, and agents are great at it. I also find them invaluable for searching through large codebases, and for basic code review, but I see these use-cases discussed less even though they're a big part of what I find useful from agents.

I’m not saying he’s saying agents aren’t useful at all. It’s literally in the quotes I provided that he says they are useful for some subset of tasks.

I’m saying that he is answering the question “are agents useful at all”. not “can agents replace humans”.

His answer is mostly not. He generally prefers autocomplete. But they are useful for some limited tasks.

Saying "He’s talking about whether agents are currently useful at all" is negatively loaded. It is very easy to take that and assume the answer is "no" based on the "at all".

If you wanted to be more neutral, you could have said something like "He's also questioning how useful agents really are today". That wouldn't have implied that they're not useful at all, but instead that they're less useful than people are claiming.

That question doesn’t do enough to highlight how wrong the OP’s interpretation was. He’s going far beyond just stating that agents are less useful than people are claiming. Less useful than people are claiming fits the OP’s interpretation.

> I’m not saying he’s saying agents aren’t useful at all

I'm not saying you're saying he's saying agents aren't useful at all

You’re not the person I’m replying to.

The person I’m replying to said

>I don't think he is saying agents are not useful at all, just that they are not anywhere near the capability of human software developers.

Implying I was supporting the first clause.

My biggest takeaway is that agents/LLMs in general are super helpful when paired together with a human who knows the inside and out of software development, who uses it side-by-side with their normal work.

They start being less useful when you start treating them as "I can send them ill-specified stuff, ignore them for 10 minutes and merge their results", as things spiral out of control. Basically "vibe-coding" as a useful concept doesn't work for projects you need to iterate on, only for things you feel OK with throwing away eventually.

Augmenting the human intellect with LLMs? Usually a increase in productivity. Replacing human coworkers with LLMs? Good luck, have fun.

It does seem pretty clear that an individual who possess super high quality human capital, paired with something like an LLM (provided the LLM is good enough relative to the individual) can be a powerful combination.

The issues are:

1) There isnt enough supply of those individuals 2) Such an LLM of that kind doesnt exist (at least not in consistent nature) 3) The amount invested into what is going on will not yield returns commensurate to the required rate of return

Interestingly enough, I believe Andrej Karpathy is also focusing on education (levelling up the supply of human capital) - I came to the above conclusion about a month ago. And it 'feels' right to me.

I am just some shmoe, but I agree with that assessment. My biggest take-away is that we got super lucky.

At least now we have a slight chance to prepare for the potential economic and social impacts.

I am thinking the same.

And we should start considering on what makes us humans and how we can valorize our common ground.

This. I believe it’s the most important question in the world right now. I’ve been thinking long and hard about this from an entirely practical perspective and have surprised myself that the answer seems to be our capacity to love. The idea is easily dismissed as romantic but when I say I’m being practical I really mean it. I’m writing about it here https://giftcommunity.substack.com/

You’ve probably already listened to it but in the event you haven’t: https://podcasts.apple.com/us/podcast/freakonomics-radio/id3...

He seems to share your sentiment

The link is broken - could you repost please?

Probably they meant this: https://freakonomics.com/podcast/how-can-we-break-our-addict...

My ever growing reporting chain is incredibly invested in having autonomous agents next year.

Because that's the definition that is leading to all these investments, the promise that very soon they will reach it. If Altman said plainly that LLMs will never reach that stage, there would be a lot less investment into the industry.

Hard disagree. You don’t need AGI to transform countless workflows within companies, current LLMs can do it. A lot of the current investments are to help with the demand with current generation LLMs (and use cases we know will keep opening up with incremental improvements). Are you aware of how intensely all the main companies that host leading models (azure, aws, etc) are throttling usage due to not enough data center capacity? (Eg. At my company we have 100x more demand than we can get capacity for, and we’re barely getting started. We have a roadmap with 1000x+ the current demand and we’re a relatively small company.)

AGI would be more impactful of course, and some use cases aren’t possible until we have it, but that doesn’t diminish the value of current AI.

> Eg. At my company we have 100x more demand than we can get capacity for, and we’re barely getting started. We have a roadmap with 1000x+ the current demand and we’re a relatively small company.

OpenAI's revenue is $13bn with 70% of that coming from people just spending $20/mo to talk to ChatGPT. Anthropic is projecting $9bn in revenue in 2025. For nice cold splash of reality, fucking Arizona Iced Tea has $3bn in revenue (also that's actual revenue not ARR)

You might have 100x more demand than you can get capacity for, but if that 100x still puts you at a number that in absolute terms is small, it's not very impressive. Similarly if you're already not profitable and achieving 100x growth requires 1,000x in spend, that's also not a recipe for success. In fact it's a recipe for going bankrupt in a hurry.

I have no idea if OpenAI’s valuation is reasonable. All I’m saying is I’m convinced the demand is there, even without AGI around the corner. You do not need AGI to transform countless industries.

And we are profitable on our AI efforts while adding massive value to our clients.

I know less about OpenAI’s economics, I know there are questions on whether their model is sustainable/for how long. I am guessing they are thinking about it and have a plan?

This is correct, it should burn the retinas of anyone thinking that OAI or Anthropic are in any way worth their multi-billion dollar valuations. I liked AK’s analysis of AI for coding here (it’s overly defensive, lacks style and functionality awareness, is a cargo cultist, and/or just does it wrong a lot) but autocomplete itself is super valuable, as is the ability to generate simple frontend code and let you solve the problem of making a user interface without needing a team of people with those in-house skills.

There are many more use cases that aren't fully realised yet. With regards to coding, LLMs have shortcomings. However, there's a lot of work that can be automated. Any work that requires interaction with a computer can eventually be automated to some extent. To what extent is something only time can tell.

Sure, but you don’t need AI to automate computer work. You can make a career out of formalizing the kinds of excel-jockeying that people do for reports or data entry

This is a relatively reasonable take. Unfortunately, that's not what most AI investors or non-technical punters think. Since GPT 1 it's been all about unlocking 100%+ annual GDP growth by wholesale white collar automation. I agree with AK that the actually effect on GDP will be more or less negligible, which will be an unmitigated disaster for us economically given how much cash has already been incinerated

Oh look, people with skin in the AI game insist AI is not a massive bubble. More news at 11.

We’re a regular old SaaS company that has figured out how to add massive value using AI. I am making no statements about valuations and bubbles. I’m actually guessing there is some bubble / overhype. That doesn’t mean it isn’t still incredibly valuable.

Link? And explain in detail, incrementally, what you have done so we can analyse it for ourselves?

[dead]

Quite telling -- thanks for the insightful comment as always, Simon. Didn't know that, even though I've been discussing this on and off all day on Reddit.

He's a smart man with well-reasoned arguments, but I think he's also a bit poisoned by working at such a huge org, with all the constraints that comes with. Like, this:

  You can’t just tell them something and they’ll remember it.

LLMs are a necessary part of AGI(/"agents") due to their ability to avoid the Frame Problem[1], but they're far from the only needed thing. We're pretty dang good at "remembering things" with computers already, and connecting that with LLM ensembles isn't going to take anywhere close to 10 years. Arguably, we're already doing it pretty darn well in unified systems[2]...

If anyone's unfamiliar and finds my comment interesting, I highly recommend Minsky's work on the Society of Mind, which handled this topic definitively over 20 years ago. Namely;

A short summary of "Connectionism and Society of Mind" for laypeople at DARPA: https://apps.dtic.mil/sti/tr/pdf/ADA200313.pdf

A description of the book itself, available via Amazon in 48h or via PDF: https://en.wikipedia.org/wiki/Society_of_Mind

By far my favorite paper on the topic of connectionist+symbolist syncreticism, though a tad long: https://www.mit.edu/~dxh/marvin/web.media.mit.edu/~minsky/pa...

[1] https://plato.stanford.edu/entries/frame-problem/

[2] https://github.com/modelcontextprotocol/servers/tree/main/sr...

> You can’t just tell them something and they’ll remember it.

I find it fascinating that this is the problem people consistently think we're a decade away on.

If you can't do this, you don't have employee-like AI agents, you have AI-enhanced scripting. It's basically the first thing you have to be able to do to credibly replace an actual human employee.

Do you have a comment? Most of what you've said here is a quote.

This is part of my wider hobby of collecting definitions of "agents" - you can see more in my collection here: https://simonwillison.net/tags/agent-definitions/

In this case the specific definition matters because the title of the HN submission is "it will take a decade to work through the issues with agents."

Andrej Karpathy seems to me like a national (world) treasure.

He has the ability to explain concepts and thoughts with analogies and generalizations and interesting sayings that allow you to keep interest in what he is talking about for literally hours - in a subject that I don't know that much about. Clearly he is very smart, as is the interviewer, but he is also a fantastic communicator and does not come across as arrogant or pretentious, but really just helpful and friendly. Its quite a remarkable and amazing skillset. I'm in awe.

Agreed. I'd also add he's intellectually honest enough to not overhype what's happening just to hype whatever he's working on or appear to be a thought leader. Just very clear, pragmatic, and intellectually honest thought about the reality of things.

It's almost like having more money than you'll ever know what to do with lets you say and do what you _actually_ want to do.

Most people don’t take this opportunity, though.

His old Youtube guides on Rubiks cube are widely regarded as awesome, so he has that kind of gift for sure.

(Link: https://www.youtube.com/user/badmephisto)

It's good to see experts with similar scepticism about agents that I have. I don't doubt they will be useful in some settings, but they lean into all the current weak points of large language models and make them worse. Security, reproducibility, hallucinations, bias, etc etc.

With all these issues already being hard to manage, I just don't believe businesses are going to delegate processes to autonomous agents in a widespread manner. Literally anything that matters is going to get implemented in a crontrolled workflow that strips out all the autonomy with human checkpoint at every step. They may call them agents just to sound cool but it will be completely controlled.

Software people are all fooled by what is really a special case around software development : outcomes are highly verifiable and mistakes (in development) are almost free. This is just not the case out there in the real world.

> Literally anything that matters is going to get implemented in a crontrolled workflow that strips out all the autonomy with human checkpoint at every step.

Yea, there aren't a ton of problems (that I can see) in my current domain that could be solved by having unattended agents generating something.

I work in healthcare and there are a billion use cases right now, but none that don't require strict supervision. For instance, having an LLM processing history and physicals from potential referrals looking for patient problems/extracting historical information is cool, but it's nowhere near reliable enough to do anything but present that info back to the clinician to have them verify it.

Fully autonomous agents are marketing fluff right now, but there is like $10T of TAM from promoting most knowledge workers to a manager and automating the boring 80% of their work, and this doesn’t require this full autonomy.

Karpathy’s definition of “agent” here is really AGI (probably somewhere between expert and virtuoso AGI https://arxiv.org/html/2311.02462v2). In my taxonomy you can have non-AGI short-task-timeframe agents. Eg in the METR evals, I think it’s meaningful to talk about agent tasks if you set the thing loose for 4-8h human-time tasks.

If the transcript is accurate, Karpathy does not actually ever, in this interview, say that AGI is a decade away, or make any concrete claims about how far away AGI is. Patel's title is misleading.

Hmm good point. I skimmed the transcript looking for an accurate, representative quote that we could use in the title above. I couldn't exactly find one (within HN's 80 char limit), so I cobbled together "It will take a decade to get agents to work", which is at least closer to what Karpathy actually said.

If anyone can suggest a more accurate and representative title, we can change it again.

Edit: I thought of using "For now, autocomplete is my sweet spot", which has the advantage of being an exact quote; but it's probably not clear enough.

Edit 2: I changed it to "It will take a decade to work through the issues with agents" because that's closer to the transcript.

Anybody have a better idea? Help the cause of accuracy out here!

You could go with the title from the associated YouTube video (https://www.youtube.com/watch?v=lXUZvyajciY)?

Andrej Karpathy — “We’re summoning ghosts, not building animals”

It's a good suggestion, but where the 'autocomplete' quote is scoped too narrowly, this one is maybe scoped too broadly. Neither really represent what the article is about.

To be fair to the OP of the thread, he's just using Patel's title word-for-word. It's Patel who is being inaccurate.

Oh that's clear, and the submitter didn't do anything wrong. It's just that on HN the idea is to find a different title when the article's own title is misleading or linkbait (https://news.ycombinator.com/newsguidelines.html).

The best way to do that of course is to find a more representative phrase from the article itself. That's almost always possible but I couldn't quite swing it in this case.

dang!! I have so much respect for this ironic situation where we are discussing the superpowers of AI while a very human, very decent being ponders deeply on how to compose a few words to make a suitable title. Please can we have a future world where such events can always happen every so often.

He says re agents:

>They don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this stuff. They don't do a lot of the things you've alluded to earlier. They don't have continual learning. You can't just tell them something and they'll remember it. They're cognitively lacking and it's just not working.

>It will take about a decade to work through all of those issues. (2:20)

Him saying that it will take a decade to work through agents' issues isn't the same as him saying that there will be AGI in a decade, though

Couldn't have even been bothered watching ~ 2 minutes of an interview before commenting.

And the decade timeframe is mostly based on his intuition and previous experience with such leaps.

Did the same with Sutton (LLMs are a dead end) when Sutton never said this in the conversation.

He didn't say those words exactly but he did say

"The scalable method is you learn from experience. You try things, you see what works. No one has to tell you. First of all, you have a goal. Without a goal, there’s no sense of right or wrong or better or worse. Large language models are trying to get by without having a goal or a sense of better or worse. That’s just exactly starting in the wrong place."

and a bunch of similar things implying LLMs have no hope of reaching AGI

There's a lot of salt here

> Hey, podcast bro needs to get clicks

Please don't cross into personal attack. It's not what this site is for, and destroys what it is for.

Edit: please don't edit comments to change their meaning once someone has replied. It's unfair to repliers whose comments no longer make sense, and it's unfair to readers who can no longer understand the thread. It's fine, of course, to add to an existing comment in such a case, e.g. by saying "Edit:" or some such and then adding what else you want to say.

Huh, I'm surprised that he goes from "No AI" to "AI autocomplete" to "Vibecoding / Agents" (which I assume means no human review per his original coinage of the term.) This seems to preclude the chat-oriented / pair-programming model which I find most effective. Or even the plan-spec-codegen-review approach, which IME works extremely well for straightforward CRUD apps.

Also they discuss the nanochat repo in the interview, which has become more famous for his tweet about him NOT vibe-coding it: https://www.dwarkesh.com/i/176425744/llm-cognitive-deficits

Things are more nuanced than what people have assumed, which seems to be "LLMs cannot handle novel code". The best I can summarize it as is that he was doing rather non-standard things that confused the LLMs which have been trained on vast amounts on very standard code and hence kept defaulting to those assumptions. Maybe a rough analogy is that he was trying to "code golf" this repo whereas LLMs kept trying to write "enterprise" code because that is overwhelmingly what they have been trained on.

I think this is where the chat-oriented / pair-programming or spec-driven model shines. Over multiple conversations (or from the spec), they can understand the context of what you're trying to do and generate what you really want. It seems Karpathy has not tried this approach (given his comments about "autocomplete being his sweet spot".)

For instance, I'm working on some straightforward computer vision stuff, but it's complicated by the fact that I'm dealing with small, low-resolution images, which does not seem well-represented in the literature. Without that context, the suggestions any AI gives me are sub-optimal.

However, after mentioning it a few times, ChatGPT now "remembers" this in its context, and any suggestion it gives me during chat is automatically tailored for my use-case, which produces much better results.

Put another way (not an AI expert so I may be using the terms wrong), LLMs will default to mining the data distribution they've been trained on, but with sufficient context, they should be able to adapt their output to what you really want.

Agency. If one studied the humanities they’d know how incredible a proposal “agentic” AI is. In the natural world, agency is a consequence of death: by dying, the feedback loop closes in a powerful way. The notion of casual agency (I’m thinking of Jensen Huang’s generative > agentic > robotic insistence) is bonkers. Some things are not easily speedrunned.

(I did listen to a sizable portion of this podcast while making risotto (stir stir stir), and the thought occurred to me: “am I becoming more stupid by listening to these pundits?” More generally, I feel like our internet content (and meta content (and meta meta content)) is getting absolutely too voluminous without the appropriate quality controls. Maybe we need more internet death.)

> In the natural world, agency is a consequence of death: by dying, the feedback loop closes in a powerful way.

I don't follow. If we, in some distant future, find a way to make humans functionally immortal, does that magically remove our agency? Or do we not have agency to begin with?

If your position on the "free will" question is that it doesn't exist, then sure I get it. But that seems incompatible with the death prerequisite you have put forward for it, because if it doesn't exist then surely it's a moot point to talk prerequisites anyway.

When I think of the term "agency" I think of a feedback loop whereby an actor is aware of their effect and adjusts behavior to achieve desired effects. To be a useful agent, one must operate in a closed feedback loop; an open loop does not yield results.

Consider the distinction between probabilistic and deterministic reasoning. When you are dealing with a probabilistic method (eg, LLMs, most of the human experience) closing the feedback loop is absolutely critical. You don't really get anything if you don't close the feedback loop, particularly as you apply a probabilistic process to a new domain.

For example, imagine that you learn how to recognize something hot by hanging around a fire and getting burned, and you later encounter a kettle on a modern stove-top and have to learn a similar recognition. This time there is no open flame, so you have to adapt your model. This isn't a completely new lesson, the prior experience with the open flame is invoked by the new experience and this time you may react even faster to that sensation of discomfort. All of this is probabilistic; you aren't certain that either a fire or a kettle will burn you, but you use hints and context to take a guess as to what will happen; the element that ties together all of this is the fact of getting burned. Getting burned is the feedback loop closing. Next time you have a better model.

Skillful developers who use LLMs know this: they use tests, or they have a spec sheet they're trying to fulfill. In short, they inject a brief deterministic loop to act as a conclusive agent. For the software developer's case it might be all tests passing, for some abstract project it might be the spec sheet being completely resolved. If the developer doesn't check in and close the loop, then they'll be running the LLM forever. An LLM believes it can keep making the code better and better, because it lacks the agency to understand "good enough." (If the LLM could die, you'd bet it would learn what "good enough" means.)

Where does dying come in? Nature evolved numerous mechanisms to proliferate patterns, and while everyone pays attention to the productive ones (eg, birth) few pay attention to the destructive (eg, death). But the destructive ones are just as important as the productive ones, for they determine the direction of evolution. In terms of velocity you can think of productive mechanisms as speed and destructive mechanisms as direction. (Or in terms of force you can think of productive mechanisms as supplying the energy and destructive mechanisms supplying the direction.) Many instances are birthed, and those that survive go on and participate in the next round. Dying is the closed feedback loop, shutting off possibilities and defining the bounds of the project.

I see your perspective about the inevitability of death causing a forcing-function directedness for agents, but that's a much much weaker claim than (emphasis mine):

> In the natural world, agency is a consequence of death: by dying, the feedback loop closes in a powerful way.

My original question was why could agency not exist without death, not why it was hampered without it. For clarity, I'm coming at from an analytic philosophy angle, not its more rhetorical counterpart that I struggle to wrap my head around.

I don't really view death or evolution as a necessity for agency. Nebulous AGI predictions aside: if a self-aware, conscious and intelligent being, capable of affecting consequential changes to its environment, becomes functionally immortal, it doesn't somehow lose its agency. I'd actually go further and say losing the forcing function of inevitable death is the biggest freedom a species can aim for. Without it, our agency is limited to solving problems of survival, in one form or another.

The existence of death is ultimately arbitrary and random, as random as our existence in the first place. The "direction" we get for evolution as a result of it, is another random function on top, also taking: the random circumstances the soup of organic molecules live in, as another parameter. Only once this random inevitability is conquered can we truly shape our lives and environments in ways that are a true reflection of who we are. Only then are we genuinely free. And "agency" without freedom is impotent at best.

(Addendum: I know positing "Immortality is good actually" can cause negative associations with "billionaires who want to cryopreserve themselves". This association has melded with the general romanticization of death in various philosophical and religious beliefs that has existed since millennia, further empowering the distaste against trying to reverse aging and eventually remove death as moral goals. While I personally have no plans (or means) to cryopreserve myself when I get old, I do believe it's a goal worth fighting for. One of the more important ones, alongside ensuring we have a planet to live on in the interim)

I love the discussion — thank you.

Your comment makes me more bullish on death. Death isn’t arbitrary as you claim: it is a direct expression of an entity in its environment, it epitomizes contextualization. (I argue that honoring context is the opposite of arbitrariness.)

Further, death encapsulates multiple layers of abstraction. When an entity dies, it dies on every level (eg both instincts and socially learned heuristics). The death reaches deep down inside the hierarchy of its own form to eliminate possibilities. That is some seriously strong directionality; it’s not like “taking your second left” or some other mono-dimensional vector. Layers and layers of genes and learning are discarded. It is truly an incredibly powerful feedback-loop closure.

Models die too - the less agentic ones are out-competed by the more agentic ones.

Every AI lab brags how "more agentic" their latest model is compared to the previous one and the competition, and everybody switches to the new model.

Yes but the point is that models must be imminently aware of their impending death to force the calculation of tradeoffs.

I don't agree but I did laugh

One inherent limitation of current LLM/AI is that they are primarily trained on abstracted data that focuses primarily on mimicking our logical and reasoning prefrontal cortex portion of the mind. However most humans make decisions based on activity in the limbic regions of the brain which are essentially emotional and intuition based. So we often will do something before we actually know why we did it, however to maintain a sense of self and sanity we will then use our prefrontal cortex to create a cohesive narrative on why we do what we do (despite it often being inaccurate).

In a nutshell we are mimicking neural activity in a certain region based on certain abstracted data which is quite removed from how we as humans process reality.

In the same sense that split brain patients [1] will make up a reasonable explanation for what the other half did.

And why witnesses are preferably interviewed very shortly after they witnessed a crime. Before their brains start to 'fill in the blanks'

1: https://en.wikipedia.org/wiki/Split-brain

He is an absolute treasure, I have watched all his videos more than 4 times and I don't think I would've been able to have a good mental model about deep learning without them, regardless of the amount of Bengio, Goodfellow etc lectures I have seen, none of them come even close.

He is singlehandedly enabling millions of people to understand what is going on, what + and * do, actually demystifying the "wires".

I just wish he start thinking of himself as more than 'collapsing weights', regardless if it turns out to be true.

I agree, I think I learned the most on this topic from his videos. And before that (a while ago), it was Andrew Ng coursera's class. The latter had hands-on project, which is much better than just listening in term of retention.. I don't know if Andrej Karpathy has more structured classes somewhere.

This is a good one.

https://karpathy.ai/zero-to-hero.html

On vibe coding vs using auto complete:

> The models have so many cognitive deficits. One example, they kept misunderstanding the code because they have too much memory from all the typical ways of doing things on the Internet that I just wasn’t adopting.

> I also feel like it’s annoying to have to type out what I want in English because it’s too much typing. If I just navigate to the part of the code that I want, and I go where I know the code has to appear and I start typing out the first few letters, autocomplete gets it and just gives you the code.

> They keep trying to make a production code base, and I have a bunch of assumptions in my code, and it’s okay. I don’t need all this extra stuff in there. So I feel like they’re bloating the code base, bloating the complexity, they keep misunderstanding, they’re using deprecated APIs a bunch of times. It’s a total mess. It’s just not net useful. I can go in, I can clean it up, but it’s not net useful.

I'm pretty content to say this may be true, but may well prove quite wrong.

Why? Because humans—including the smartest of us—are continuously prone to cognitive errors, and reasoning about the non-linear behavior of complex systems is a domain we are predictably and durably terrible at, even when we try to compensate.

Personally I consider the case of self-driving cars illustrative and a go-to reminder for me of my own very human failure in this case. I was quite sure that we could not have autonomous vehicles in dynamic messy urban areas without true AGI; and that FSD would in the fashion of the failed Tesla offering, emerge first in the much more constrained space of the highway system. Which would also benefit from federal regulation and coordination.

No Waymos have eaten SF, and their driving is increasingly nuanced; and last night a friend and very early adopter relayed a series of anecdotes about some of the strikingly nuanced interactions he'd been party to recently, including being in a car that was attacked late at night, and, how one did exactly the right thing when approached head-on in a narrow neighborhood street that required backing out. Etc.

That's just one example, and IMO we are only beginning to experience the benefits of "network effects" so popular in tails of singularity take-off.

Ten years is a very, very, very long time under current conditions. I have done neural networks since the mid-90s (academically: published, presented, etc.) and I have proven terrible in anticipating how quickly "things" will improve. I have now multiple times witnessed my predictions that X or Y would take "5-8" or "8-10" years or "too far out to tell," instead arrive within 3 years.

Karpathy is smart of course but he's no smarter in this domain than any of the rest of us.

Are scaled tuned transformers with tack-ons going to give us AGI in 18 months? "No" is a safe bet. Is no approach going to give us AGI inside of 5 years? That is absolutely a bet I would never make. Not even close.

I find it strange AGI is the goal. The label AI is off and irrelevant. A language model is not AI, even a large language model. But language models are still extremely useful and potentially revolutionary. Labelling language models as AI is both under and overstating the value. It's not AI (insert sad trombone), but that doesn't mean it's amazing technology (insert thunderous applause).

Without defining AGI as a goal, current AI companies would not be able to amass the amount of money they want

This terminology is confusing. Historically, AI was always used to mean any kind of machine intelligence, including the most basic novice chess AI, or an image classifier, or a video game character's AI. Now a lot of people seem to be using it as a synonym for AGI - a human-level intelligence.

Greg Brockman talks about it on Lex’ podcast I think. He thinks ultimately AGI won’t be anything more than token generation, if I remember correctly

I knew this once I heard OpenAI was going to get into the porn bot business. If you have AGI you don't need porn bots.

Or Sora...

Why not?

Most likely because you'll be filthy reach from selling AGI and won't need to go after secondary revenue sources.

>Most likely because you'll be filthy reach from selling AGI

Why? If AGI costs more than a human or operates slower than one, it may not be economical for people to buy it. By the time it becomes economical, competitors may have also cracked it reducing your ability to charge high margins on it.

If OpenAI had anything even resembling AGI they'd be milking the shit of that, even if only for marketing.

Cost decreases with time

Humans can work on a problem 8 hours a day? You can run inference 24/7

It decreases, but decreasing from $1 million per token to $0.9 million per token after a year is still a decrease, but it still is not viable. Paying an AGI a $100 billion dollars for it to work 24/7 for a year is worse than hiring 10 people for $30k a year to work shifts to do the same work 24/7.

Maybe I'm being too simplistic, but I think we're mixing two distinct debates.

Today we have an extraordinary invention—comparable to the wheel in its time. That invention is: predictive inference over all human knowledge. Period. I don't like calling it "Artificial Intelligence" because it's not intelligence; it's a prediction system that can project responses by illuminating patterns across all human knowledge encapsulated in text, audio, and video. What companies like OpenAI call "reasoning" models is simply that predictive process, but in a loop packaged as a product—one of the first marvelous uses of this fascinating invention: predictive inference over all human knowledge.

When the wheel was invented, no one could have imagined that, combined with hundreds of subsequent technologies, it would enable an electric car powered by solar energy. The wheel wasn't autonomous transportation—it was a fundamental component.

I see two debates getting mixed up here:

- The debate about the current invention: A tool that makes encyclopedias "speak" by connecting patterns across all human knowledge. As a tool, that's what it is—nothing more, nothing less. Tremendously useful, but a tool.

- The debate about the future dream: What this invention might enable when combined with hundreds of technologies that don't yet exist—similar to imagining an electric car when you only have the wheel.

It seems many experts are taking positions and getting "upset" because they're mixing these two debates. Some evaluate the wheel as if it should already be a solar electric car. Others defend the wheel by saying it already IS a solar electric car. Both are right in their observations, but they're talking about different things.

LLMs are a fundamental breakthrough—the "wheel" of the information age. But discussing whether they "understand" or have "world models" is like asking whether the wheel "comprehends transportation."

On the danger of confusing capabilities: Conflating the tool with the end goal leads us to poor decisions—from over-investment to under-utilization. When we expect AGI from what is fundamentally a pattern-matching engine, we set ourselves up for disappointment and misallocation of resources. No magic, just reality.

The temporal factor: The AGI debate is a debate about the future—about what might emerge from combinations of technologies we haven't yet invented.

> I don't like calling it "Artificial Intelligence" because it's not intelligence

A pattern I noticed in a AI[sic] discussions: Handwavily declaring what intelligence is not, while not explaining what is.

> Handwavily declaring what intelligence is not, while not explaining what is.

That goes in the other direction too. Declaring it intelligent without explaining what it is. Or even worse, if any explanations are offered, they are often half truths or exaggerated.

You are right, I thought maybe something interesting in these debates is more education about how an LLM works. I don’t like calling it artificial intelligence because precisely we don’t understand well what “intelligence” is. What we do understand is how we came to build an LLM. Good point, I will keep that in mind for next time; it’s better to give more details and, above all, remove the “no” from assertions and clarify more. Thanks :)

I think this comparison is all wrong. The internet is more closer to the notion of a wheel - the internet has done amazing stuff just as the wheel has and nobody foresaw the impact the internet would have and how the underlying technologies that power it have evolved.

Just like how a wheel moves stuff, the internet is the medium through which bits are transmitted and received.

Thanks for your view! My analogy was intentional—I wanted to talk about revolutionary tools that extend human capabilities, not about the foundational infrastructure itself. Of course, the internet is a fundamental platform like the wheel, but I’m focusing on what’s built on top of that—how new tools like predictive inference change the landscape again. Analogies can work at different layers. I just chose the tool, not the medium.

Very good point. With one caveat, though. Even though I was not there, I imagine that debates about the wheel were less heated than those we’re having about AI. I think this is because the latter is much more abstract, too close from our own consciousness etc. Wheels never challenged our place in the universe.

Totally agree with you. It makes me think that the wheel is a tool—technologically simple yet incredibly powerful—that helps humans overcome their limitations. Similarly, predictive inference is also a tool that extends our cognitive capacity by connecting all human knowledge. This tool is built upon other tools, all designed with the fundamental purpose of facilitating and empowering humans. The refinement of these aids is such that sometimes it evokes a mix of awe and a certain unease, due to how closely and powerfully these tools can influence our world and decisions. It is natural for such intensity to generate suspicion because the assistance becomes extremely sophisticated and gives the illusion of something “intelligent.”

Am I dating myself by thinking Kurzweil is still relevant?

2029: Human-level AI

2045: The Singularity - machine intelligence 1 billion times more powerful than all human intelligence

Based on exponential growth in computing. He predicts we'll merge with AI to transcend biological limits. His track record is mixed, but 2029 looks more credible post-GPT-5. The 2045 claim remains highly speculative.

It's curious that Kurzweil's predictions about transcending biology align so closely with his expected lifespan. Reminds me of someone saying, if you ask a researcher for a timeline of a breakthrough they'll give you the expected span of their career.

Hegel thought history ended with the Prussian state, Fukuyama thought it ended in liberal America, Paul thought judgement day was so close you need not bother to marry, the singularity always comes around when the singularians get old. Funny how that works

The biggest problem I've had with Kurzweil and the exponential growth curve is that the elbow depends entirely on how you plot and scale the axis. With a certain vantage point we have arguably been on an exponential curve since the advent of Homo Sapiens.

I lost all respect for him after reading about his views on medical immortality. His argument is that over time human life expectancy has been constantly increasing * and he calculated that based on some arbitrary rate of acceleration, that science would be expanding human life expectancy by more than a year, per year - medical immortality in other words, and all expected to happen just prior to the time he's reaching his final years.

The overwhelming majority of all gains in human life expectancy have come due to reductions in infant mortality. When you hear about things like a '40' year life expectancy in the past it doesn't mean that people just dropped dead at 40. Rather if you have a child that doesn't make it out of childhood, and somebody else that makes it to 80 - you have a life expectancy of ~40.

If you look back to the upper classes of old their life expectancy was extremely similar to those of today. So for instance in modern history, of the 15 key Founding Fathers, 7 lived to at least 80 years old: John Adams, John Quincy Adams, Samuel Adams, Jefferson, Madison, Franklin, John Jay. John Adams himself lived to 90. The youngest to die were Hamilton who died in a duel, and John Hancock who died of gout of an undocumented cause - it can be caused by excessive alcohol consumption.

All the others lived into their 60s and 70s. So their overall life expectancy was pretty much the same as we have today. And this was long before vaccines or even us knowing that surgeons washing their hands before surgery was a good thing to do. It's the same as you go back further into history. A study [1] of all men of renown in Ancient Greece was 71.3 [1], and that was from thousands of years ago!

Life expectancy at birth is increasing, but longevity is barely moving. And as Kurzweil has almost certainly done plentiful research on this topic, he is fully aware of this. Cognitive dissonance strikes again.

[1] - https://pubmed.ncbi.nlm.nih.gov/18359748/

This is true, and I tend to believe that indefinite human lifespan extension will come too late for anyone who is already an adult today including myself. But I do think that it will come, mostly as a consequence of advanced AI accelerating medical research. It may be wishful thinking to believe that it will happen within our lifetimes, but that doesn't mean it won't ever happen.

While it'd be absurd to say it's impossible, the one thing I'd observe is that it's almost certain that a precursor to anything like this would be achieving something comparable in a simpler species. And that would likely come long before we might be able to see something similar in humans. For instance the fruit fly has been studied and experimented on extensively, particularly for aging, for over a century now.

But the results remain modest. The biggest breakthrough was in the 80s when somebody was able to roughly double their life expectancy from 2 months to 4 through artificial selection. But the context there is that fruit flies are a textbook 'quantity over quality' species, meaning that survival is not generally selected for, whereas humans are an equally textbook 'quality over quantity' species meaning that survival is one of the key things we select for. In other words, there was likely a lot more genetic low hanging fruit for survivability with fruit flies than there is for humans.

So I don't know. We need some serious acceleration and I'm not seeing much of anything when looked at with a critical eye.

This is backward looking. Future advances don't have to work like this

Example: 20ish years ago, stage IV cancer was a quick death sentence. Now many people live with various stage IV cancers for many years and some even "die of sending else" these advancements obviously skew towards helping older people.

Your claim doesn't argue against the issue. Even if we accept that you're correct there, you're again speaking of more people getting to their 'expiration date' rather than expanding that date itself. If you cure cancer, heart disease, and everything else - we're still not going to be living to a 100, or even near it, on average.

The reason humans die of 'old age' is not because of any specific disease but because of advanced senescence. Your entire body just starts to fail. At that point basically anything can kill you. And sometimes there won't even be any particular cause, but instead your heart will simply stop beating one night while you sleep. This is how you can see people who look like they're in great shape for their age, yet the next month they're dead.

> He predicts we'll merge with AI to transcend biological limits.

The merge with a machine 1 million times more intelligent than us is the same as letting AI use our bodies. I'd rather live in cave. Iirc, the 7th episode of Black Mirror starts with this plot line.

Redefinitions aside, fully capable AI is right up there with commercially viable fusion power, cost effective quantum completing, and fully capable self-driving cars, as a technology that is quickly advancing yet always a decade or two away.

Waymo's self-driving cars are scaling quickly. With some inaccuracy it can be said that the problem is solved, we have the technology for a full-scale deployment, we just need to do the boring work to deploy it everywhere.

> Also, when you look at these cars and there’s no one driving, I actually think it’s a little bit deceiving because there are very elaborate teleoperation centers of people kind of in a loop with these cars. I don’t have the full extent of it, but there’s more human-in-the-loop than you might expect. There are people somewhere out there beaming in from the sky. I don’t know if they’re fully in the loop with the driving. Some of the time they are, but they’re certainly involved and there are people. In some sense, we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them.

The car can ask for remote assistance, which doesn't happen often.

What was the last example where humans succeeded at a hard problem like that?

Space flight?

Even if it's not some staggering triumph of human achievement, I'd argue that Ozempic (etc.) is similar. A magic weight loss drug has always captured the public's imagination, and it feels like I've been hearing about new weight loss drug studies in the news for my entire life that never went anywhere.

That was a stroke of luck. It's synthetic gila monster poison.

We've "succeeded" at space flight about as much as we've "succeeded" at AI. Yay, man on the moon! Over half a century later, and it turns out that the "next small step" - man on Mars - isn't so small and still hasn't been achieved. Anything remotely resembling sci-fi-style ubiquitous space travel remains exactly that - sci-fi!

Flying a plane and intercontinental flight are different levels of the same remarkable achievement.

A man on the moon, or the SpaceX rockets that land and can rapidly relaunch, both feel like hard problems that have been solved, although it’s not the next hard step of intergalactic space travel.

Waymo, which works and is scaling quickly.

re: fusion: https://commons.wikimedia.org/wiki/File:U.S._historical_fusi...

Fusion power seems closer than ever. And plenty of experts just five years ago thought AGI would still be decades away. A credible expert suggesting AGI is ten years away is a sign of real progress.

A credible expert suggesting AGI is ten years away is a sign of good marketing.

Except he didn't actually do that; the title is misleading.