2.6.4 AI security and hacking

I think the most worrisome aspect of AI systems in the short term is that we will give them too much autonomy without being fully aware of their limitations and vulnerabilities. We tend to anthropomorphize AI systems: we impute human qualities to them and end up overestimating the extent to which these systems can actually be fully trusted.

So when you hear someone say, “AI is really scary!”, try to steer them towards this type of thinking. Who would do that? How would they do it? What would happen as a result? And what would the be the effects of that happening?

This is important because if you don’t know how well your software is controlled and you can’t routinely build and deploy it then you have limited agility, inability to drive security improvements with acceptable operating risk and many other factors that are crucial to cybersecurity.