Guy gives non-deterministic software root access, desaster happens. Movie at eleven.
Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to know: total and absolute failure of guard rails, but these guard rails can not be expected to be in an LLM.
Prompts are just weights on a graph traversal. They don't guarantee anything. The LLM does not "understand" the prompts and so it cannot fully adhere to them. They only improve the liklihood it will output what you want.
Never ever ever give an LLM access to something you can't afford to break. And stop thinking of them like people.
This feels like what a dog does. It's incredibly hard to train dogs by punishment, because it's very hard to tell if the dog understands what he did wrong and feels genuine remorse, or is just showing submissive signs at your display of dominance.
It seems here the guard rails at failure were the llm users right? Whatever guard rails you can think may be useless against the superior human stupidity.
I guess people are finding out the hard way you do sorta need technical people to say, "hey, maybe this isn't a great idea" rather than trusting marketing hype that says technical skills are dead.
At the very least, when an agent can delete a production database you should get an obvious warning whenever you enable it. Marketing wouldn't like it though.
Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to know: total and absolute failure of guard rails, but these guard rails can not be expected to be in an LLM.