This is similar to the problem with some of the things people have been doing with o1 and o3. I've seen people share "PhD level" results from them... but if I don't have a PhD myself in that subject it's almost impossible for me to evaluate their output and spot if it makes sense or not.
I get a ton of value out of LLMs as a programmer partly because I have 20+ years of programming experience, so it's trivial for me to spot when they are doing "good" work as opposed to making dumb mistakes.
I can't credibly evaluate their higher level output in other disciplines at all.
This begs the question, is this wave of LLM AI anything more than a fancy mirror? They're certainly very good at agreeing with people and following along, but, as many have noted, not really useful for anything acting on their own.
I get a ton of value out of LLMs as a programmer partly because I have 20+ years of programming experience, so it's trivial for me to spot when they are doing "good" work as opposed to making dumb mistakes.
I can't credibly evaluate their higher level output in other disciplines at all.