By 2028, a misconfigured artificial intelligence system operating within cyber-physical systems could cause a shutdown of national critical infrastructure in a G20 nation, according to Gartner. The analyst firm emphasized the need for human-accessible override mechanisms to prevent such incidents.
Cyber-physical systems (CPS), as defined by Gartner, integrate sensing, computation, control, networking and analytics to manage interactions with the physical world. These encompass operational technology, industrial control systems, industrial automation and control systems, the industrial Internet of Things, robotics, drones and related technologies under the umbrella of Industrie 4.0.
Wam Voster, vice president and analyst at Gartner, said “The next great infrastructure failure may not be caused by hackers or natural disasters but rather by a well-intentioned engineer, a flawed update script, or a misplaced decimal,” said Wam Voster, VP Analyst at Gartner. “A secure ‘kill-switch’ or override mode accessible only to authorised operators is essential for safeguarding national infrastructure from unintended shutdowns caused by an AI misconfiguration.”
The firm noted that misconfigured AI could autonomously disable key services, misinterpret sensor data or trigger unsafe actions, resulting in physical damage or widespread disruption to essential systems such as power grids or manufacturing plants. Modern electricity networks, for example, rely on AI to balance generation and consumption in real time; an erroneous configuration could lead to unwarranted grid isolation or load shedding across large regions.
Voster added “Modern AI models are so complex they often resemble ‘black boxes,’” said Voster. “Even developers cannot always predict how small configuration changes will impact the emergent behaviour of the model. The more opaque these systems become, the greater the risk posed by misconfiguration. Hence, it is even more important that humans can intervene when needed.”
To address these risks, Gartner advised chief information security officers to implement secure override modes within all critical infrastructure CPS, create digital twins for testing updates and configuration changes before deployment, and establish real-time monitoring and rollback mechanisms. The firm also recommended the formation of national AI incident response teams to enhance resilience and oversight.
