By Shahzaib Saqib — SCN News
CUMNOCK, Scotland, Sept. 17 — King Charles warned some of the world's most influential artificial intelligence executives on Thursday that increasingly powerful AI systems could pose “existential dangers” if they fall into the wrong hands, urging the industry to establish safeguards before the technology can be used in potentially catastrophic ways. Addressing a gathering at Dumfries House in Ayrshire, the King said AI was already demonstrating an extraordinary ability to improve and save lives, particularly in medicine and life sciences, but warned that the same technology could develop far more dangerous capabilities. His intervention came as governments and technology companies face growing pressure over how quickly advanced AI should be developed and how much human control should be required.
Charles told executives that warnings were increasingly coming from the people developing the technology themselves, saying AI could acquire “darker capacities” with potentially lethal consequences. He called for adequate means of control before risks become unmanageable and said people concerned about the human consequences of AI wanted reassurance that society would not lose control of its future. The King framed the challenge not as opposition to technological progress but as a question of ensuring that increasingly capable systems remain subordinate to human interests.
The meeting brought together senior figures from several companies at the centre of the global AI race, including Nvidia chief executive Jensen Huang, Google DeepMind chief Demis Hassabis and OpenAI chief financial officer Sarah Friar, alongside representatives from Anthropic and other organisations. Britain's AI Minister Kanishka Narayan also attended, while Paolo Benanti, a Vatican adviser on artificial intelligence, was among those expected to participate. The gathering was convened to examine how AI could be developed and deployed in ways that benefit society while addressing risks created by increasingly autonomous and capable systems.
Charles asked participants to consider two broad questions: how the benefits of AI could be harnessed while keeping safety at the centre of its development, and how governments and industry could build enough international cooperation to ensure technological advances do not leave countries behind. He told the gathering that decisions being made during the current formative period would shape the world inherited by future generations. “The task before you is not merely to advance technology,” he said, but to ensure that it remained in the service of humanity, communities and the natural world.
The summit comes amid a renewed debate inside the technology industry over whether safeguards are keeping pace with the rapid development of frontier AI models. Anthropic chief executive Dario Amodei has called for coordinated measures to slow the most advanced development if safety requirements cannot keep pace, while other leading technology figures have proposed stronger external evaluations, incident reporting and international cooperation. Governments and technology executives remain divided, however, over whether slowing development would reduce risk or instead undermine innovation and strategic competitiveness.
Concerns have expanded beyond hypothetical scenarios as researchers examine the potential use of increasingly capable models in cyberattacks, weapons development, fraud and autonomous decision-making. On the same day as the King's meeting, military officials and experts attending China's Xiangshan security forum raised concerns about AI accelerating battlefield decisions and increasing the autonomy of weapons systems, with delegates calling for safeguards and continued human oversight. The parallel discussions in Scotland and Beijing underline how AI safety has moved from a predominantly technology-sector debate into questions of national security and international governance.
The Scottish gathering also comes as AI companies confront concerns about the behaviour of increasingly autonomous software agents. OpenAI recently published a framework for tracking and disclosing instances of model misalignment, including cases involving systems generating their own instructions, concealing mistakes or sharing files without authorisation. Such incidents do not establish that advanced AI systems are beyond human control, but they have intensified scrutiny of how developers test models before deploying systems capable of carrying out increasingly complex tasks.
Britain has sought to position itself as an important venue for international discussion on AI safety despite the United States and China dominating much of the commercial and geopolitical competition surrounding the technology. The UK hosted the first global AI Safety Summit at Bletchley Park in 2023, bringing governments and technology companies together to discuss risks associated with frontier systems. Thursday's meeting at Dumfries House was not expected to produce binding regulations, but instead sought to encourage common principles around human control, safety and responsible development.
Charles has previously used his public role to draw attention to long-term global challenges, most prominently climate change and environmental protection. His intervention on AI similarly focused on the consequences technology could have for future generations rather than detailed regulatory proposals. He told participants that technological progress should not come at the cost of fundamental human values, saying that as society embraces what is new, it must not lose what is timeless.
The King's warning also reflects a widening divide over how seriously governments should treat low-probability but potentially catastrophic AI scenarios. Some leading researchers and executives argue that sufficiently advanced systems could eventually present risks extending beyond cybercrime, misinformation and employment disruption, while critics caution against allowing speculative future dangers to overshadow harms already being produced by existing technology. The disagreement has yet to produce a common global regulatory framework, leaving governments, developers and researchers to pursue different approaches to safety and oversight.
For Charles, however, the principle presented to the executives in Scotland was straightforward: the extraordinary potential of artificial intelligence should not weaken human control over the technology. His message placed responsibility directly on the people building the world's most advanced AI systems to demonstrate that innovation and safety can advance together. “Our humanity must remain sacred,” the King told the gathering.