If you pick a random combination, there is a very good chance that the combination and the product do not exist anywhere. So the LLM has to "create" it somehow.
It sure goes through a lot (hundreds of lines of self-reflection) but it successfully does the math.
I don't think it is the same kind of "reasoning" as humans, but there is an emergent kind of structure happening here that is allowing for this reasoning.
I think it is very human-like reasoning. I reason exactly like this when doing numerical calculations in my head, and I'm a mathematician (no, I can't work with numbers this big in my head).
It's quite funny where late in the piece it says it's checking with a calculator, which a human would do if possible (if they didn't start out with that) but then its statements are pretty much the same as before, and it probably didn't actually use a calculator.
That is like saying a calculator is reasoning since it spends many cycles thinking about the problem before answering. You could say yes a calculator is reasoning, but most would say a calculator isn't really reasoning.
> there is an emergent kind of structure happening here that is allowing for this reasoning.
Yes, but that structure doesn't need to be more complicated than a calculator. The complicated bit is the fuzzy lookup that returns fuzzy patterns (every LLM does that though so its old), and then how it recourses that, but seeing how that can result in it recursing into a solution for a multiplication is no harder than understanding how a calculator works.
So to add basic reasoning like this you just have to add "functions" for the LLM to fuzzy lookup, and then when looking up those "functions" it will eventually recourse into a solution. The hard part is finding a set of such functions that solves a wide range of problems, and that was solved here.
I think that demonstrates how far we are from reasoning and from true self-reflection. If either were happening it would know that it has the capability to multiply four numbers in nanoseconds and know that it doesn't even matter the order it multiplies them. The first reasoning step, I have 4 numbers to multiply, should be the only one necessary.
Here is R1 trying to multiply a large number (successfully): https://gist.github.com/omarabid/038678cc269a3f2db756a7e0825...
If you pick a random combination, there is a very good chance that the combination and the product do not exist anywhere. So the LLM has to "create" it somehow.
It sure goes through a lot (hundreds of lines of self-reflection) but it successfully does the math.
I don't think it is the same kind of "reasoning" as humans, but there is an emergent kind of structure happening here that is allowing for this reasoning.