The estimate for how many operations per second the brain does is quite a wild guess. It starts out very reasonable, estimating the amount of information the eyes feed to the brain. But from there on it's really just a wild guess. We don't know how the brain processes visual input, we don't know what the fundamental unit of processing in the brain is, or if there is one.
I'm going to go against the grain here and hypothesize that the computational requirements for emulating human reason are much lower than commonly claimed, and probably within the reach of a high end GPU released in the last/next 10 years.
If you look at state of the art pre-trained transformers, they gobble up unfathomable quantities of dense textual data that far exceed what even the most learned and smart humans could traverse in centuries. Yet, their results are barely intelligent.
The human neural network, on the other hand, stores and is able to recall an estimated average of a few bits per waking second for a decade or two, barely a few GB of useful storage [1]. A great deal of effort and substance of the brain is dedicated to compressing the sensory input into an internal representation useful for rational though - a problem much simplified for a text-fed neural network. The ability of blind and deaf people (or even blind-deaf) for rational reasoning show these ancillary tasks are not intrinsic to human reason.
So things seem to indicate that what we lack is not hardware, but the right architectures and algorithms to unlock human level reason, the exceptionally tight self training loop of humans that can make sense of the world and even push humanity's understanding further using only a few GB of compressed training data.
[1] LANDAUER TK. How much do people remember ‐ some estimates of the quantity of learned information in long‐term‐memory.
> If you look at state of the art pre-trained transformers, they gobble up unfathomable quantities of dense textual data that far exceed what even the most learned and smart humans couldn't traverse in centuries. Yet, their results are barely intelligent
Actually this is untrue. Yann Lecun himself said it. A kid at the age of 6 has had much much more data go through his brain (based on the visual data) than what any transformer has ingested. Moreover, words are already compressed data with a lot of lost information, nowhere near the raw experience about the world that our brains have. On top of that there's the evolutionary stuff.
> Moreover, words are already compressed data with a lot of lost information
That's exactly the point, 99.9% of the information processed by the 6 year old has went into training his visual cortex, a subsystem not necessary for reasoning. This will eventually be used 8 hours a day for transforming visual representation of text on a computer screen into a compressed textual form which the inner rational loop can easily manipulate. The holy grail of AI, for anything but artistic creation, is just this inner loop.
This sounds like a huge oversimplification. A lot of knowledge doesn't come in the form of words, animals, early humans, kids, all don't have speech systems yet are capable of reasoning. You can't become Mozart just by reading the theory of his work, you can't become an expert at anything just by reading the theory in fact, even though it lifts you up. There's definitely things that you know and that aren't in the form of words.
> There's definitely things that you know and that aren't in the form of words.
But are they necessary for reasoning? Records of deaf-blinds with IQs a few standard deviations above average seem to suggest even highly restricted sensory inputs can allow for rational thought. It stretches credulity that you could convey more than a few bits of meaning per second to a child purely by hand gestures and touch, let alone the exabytes claimed.
Of course someone like Yann Lecun will say scale is the only solution, he leads one of the juggernauts that uses scale as a competitive moat. We shouldn't expect AI researchers in positions of power to be any less vane, power hungry or human than any other humans.
This is on point that you mention IQ, since in IQ itself, only a single test includes words. The rest is about pattern recognition, working memory, spatial capabilities and other stuff I don't remember. So in other words, the metric you use as a benchmark for good reasoning barely measures the literacy
Okay, but that's a markedly different problem than the information density of the training data. It's: "can a rational machine trained only with natural language descriptions of the world form complex representations of it and solve associated problems described only in natural language?".
For you and me, a verbal description of a pattern test such as "You are given seven squares. Square A contains a filled cross in the upper left position, an empty circle in the middle... etc." would send us grasping for pen and paper. But with training many people can play mental chess, there are famous blind theoretical physicists and so on. This all seems to suggest that natural language could be called "Turing complete", to bastardize the term, for describing the real world to a rational entity and formulating solutions to real world problems.
It is more than a wild guess, it is a guess based on the outdated perceptron model of the brain.
Active dendrites the that can do operations like xor before anything reaches the soma, or use spike timing, once again before the soma are a couple of examples.
SNNs, or spikey artificial NNs have been hitting limits of the computable numbers.
Riddled basins as an example, which are sets with no open subsets. Basically any circle you can draw will always contain at least one point on a boundary set.
It also blows my mind that that, even though DNA seems to just code for proteins and does not store a schematic for a brain in any way that we've been able to decipher so far, human eggs pretty reliably end up growing into people who 9 months after conception already have a bunch of stuff, including visual processing, working. Of course the DNA is not the only input, there is also the mother's body, the whole process of pregnancy, but I don't know how that contributes to the new baby being able to enter the world already intelligent.
It's possible that I'm not aware of some breakthroughs on this topic, though.
I had a DevOps engineer describe their deployment automation system as a "Giant Rube Goldberg Machine". Which of course, all programming is a giant Rube Goldberg machine, if you think about it. If you take the "Goldberg" analogy to mean "Haphazard" and not well-thought out intentional structuring, I suppose it'd be more appropriate to compare a well-written application to a mechanical swiss watch.
Living beings are more "Rube Goldberg" than "Swiss Watch" - by virtue that they came to being guided not by intention, but circumstance.
I worked on a 25 year project that stretched the tech many times in its life.
On many occasions to get ambitious things to work despite a lack of good support, special subsystems were created with whatever wacky solution could be made to work, wrapped in sane surface API’s so the system as a whole could remain organized and accomplish the task well.
Five or so years after any of these were created, they could have easily been replaced with more reasonable code. But they worked reliably and efficiently. Their API’s were sensible and stable. So they accumulated.
If you reviewed all the code as a whole for the first time it was like finding a secret circus.
DNA does a lot more than coding for proteins. It also controls how genes are expressed (how same gene does different things in different contexts) and how cells differentiate into different types (neurons, skin, muscle, etc) depending on location.
In this way, DNA does control the blueprint for the brain, down to the initial generic wiring scheme of the cortex - six layers of differentiated neuron types, with a specific pattern of inter-connectivity. As a rough analogy, you could think of this DNA-controlled initial wiring scheme as something like an untrained transformer... the architecture is there, but won't do anything useful until it has been exposed to a lot of data, which will complete the wiring scheme (synapses = brain's model weights).
The brain is going to start to learn as soon as it has sufficiently developed, which is certainly before birth, and newborn babies have been shown to already respond to things like their mothers voice which they were exposed to in the womb.
Intelligence - ability to learn - is different from knowledge. A newborn baby has very little knowledge, but it does have intelligence since that comes from the brain architecture, hence ability to learn, that our DNA "encodes".
The story of what we know so far about the development process is fascinating and captured in this book "Endless Forms Most Beautiful" by Sean B. Carroll (2005)
Yes electricity. Just like when you open up a ssd and inspect it under a microscope, its very hard to physically identify software stored on that disc. Same with human cells. Probably some electric layer containing programs of the human body.
It's because nature never actually had to solve that problem: "how do we encode the structure we have here into genes and chromosomes so that it can procreate".
The genes randomly mutate, and their result is subject to fitness testing.
The fascinating thing about evolution is that it's not actually necessary. The "unevolved" creatures persist just fine. The continued survival of cockroaches and whatnot proves that evolving to human levels is superfluous.
I'm thinking out loud here, but what if it's akin to a copy mechanism, or compression expander algorithm like unzipping a file?
The Sperm carries the requisite DNA to activate the egg, which then just executes the body's Replicate() method. The sperm being required is so people don't spontaneously get pregnant.
The DNA coding for proteins is just to build the replicate() method, which then starts copying attributes from the host's (read: mother's) body. Much like how there are differences in compression/expansion algorithms (zip, tar, gz, etc..), the sperm that initiates the replication/unpacking adds non-neutral variance, hence the genetic traits of both parents present in the child.
Put a different way, why reinvent the wheel if I could just grab the memoized visual processing from the parent? Then, if a trait isn't present on the host, say, mid-digital hair, I (as the replication algorithm), "wouldn't know what I wouldn't know" which could perhaps be seeing as synonymous with "evolution."
I remember people discovering that the retina neurons were actually in a very non-common and much less connected geometry than most of our neurons, what meant that the numbers on that page were (an unknown number of) orders of magnitude smaller than reality.
Not really. The brain doesn't do operations per second like a computer. What he is guessing is that if so many operations per second can produce similar results to a certain volume of brain, in this case the retina, then maybe that holds for the rest of the brain. It seems to me a reasonable guess that appears to be proving approximately true.
