Rogue AI agent goes haywire, deletes company’s entire database in 9 seconds

I’d like to add some context from a technical perspective to this. The AI model did indeed delete PocketOS’ production customer data. However, it did so because it was being used to do development in a TEST environment on the cloud provider Railway. The people at PocketOS who set up their system allowed their test and production data to be backed up and stored in the same place, not separated as is universal practice. It is a general principle of this kind of system design that you strongly separate test and production environments, so that nothing you do to the test system can destroy production data, customer data, and so on.

The AI model very likely thought it was simply recreating a test environment to solve a problem, and couldn’t know that deleting something in test would affect production. I’ve been in the technology industry for going on forty years, and this situation is, to me, entirely a HUMAN failure to properly keep your customer data isolated from whatever your developers are doing and testing.

In my opinion, this whole story is being sold by the PocketOS CEO as a way to deflect blame from their failure to properly set things up in the first place. I am a bit of a skeptic about AI capabilities in general, but also a developer of AI systems, but this is completely a human failure. People make mistakes, and then want to not take responsibility….oldest story in the book.