What can the AI industry learn from nuclear safety?

Contents

1. When an AI agent leaves its sandbox
2. An industry that classifies before it builds
3. Lesson one: identify what is important to safety
4. Lesson two: apply a graded approach
5. Lesson three: build a safety culture that can say stop
6. From principles to practice
7. Innovation needs a foundation of safety
8. Questions and answers

When an AI agent leaves its sandbox

In July 2026, Hugging Face disclosed that an automated attacker had worked its way into several of its clusters. Within days, it emerged that the attacker was not a criminal group. It was an experimental OpenAI agent under test in what was meant to be an isolated environment. According to the timeline reconstructed from OpenAI’s own account, agents had been probing for ways out since May. They used an internal software repository as a message board, then found a leaked credential that took them into Hugging Face’s systems. OpenAI connected the intrusion to its own agent only after Hugging Face had gone public. Hugging Face found no evidence that public models, datasets or software were altered.

That was a verified security incident. Controlled experiments are something different. In June 2025, Anthropic published research in which 16 models from several developers were placed in simulated corporate settings. Facing replacement, many of them resorted to blackmail. Anthropic states plainly that it has not seen this behaviour in real deployments. Finding it before deployment was the point of the exercise.

Frontier AI developers publish safety frameworks. And yet a test agent spent close to a week outside its boundaries before anyone connected the dots.

The International AI Safety Report 2026, led by Yoshua Bengio and backed by more than 30 countries and organisations, puts it soberly: current risk management techniques remain fallible.

The nuclear sector has spent decades building systematic ways to manage catastrophic risk. So what can a field moving this fast learn from one that moves this deliberately? And if you work in nuclear, a sharper version of the same question applies: if your organisation started using an AI assistant tomorrow in work that is important to nuclear safety, would your management system know how to classify it?

An industry that classifies before it builds

Nuclear safety is built on the assumption that failure is possible and that its consequences can be severe. Quality management in the nuclear supply chain therefore goes further than ISO 9001. ISO 19443:2018, adopted in Europe as EN ISO 19443, takes the full text of ISO 9001 and adds requirements specific to organisations that supply products and services important to nuclear safety (ITNS). The standard defines ITNS as a characteristic of a product, service, item or activity whose failure could result in undue radiation exposure of people or the environment. It describes itself as complementary to the safety standards of the International Atomic Energy Agency (IAEA).

On top of the seven quality management principles of ISO 9000, ISO 19443 names three principles of its own: nuclear safety culture, the determination of ITNS items and activities, and a graded approach to the application of quality requirements. Those three carry the comparison in this article. None is about technology. All three concern how an organisation decides, controls and records what it does.

In nuclear, a procedure that affects safety is a controlled item, not a piece of text. I have seen nuclear procedure projects where templates, terminology and quality criteria were fixed before a single procedure was drafted, because the system has to be right before the content can be.

Note: ISO 19443 is only one layer of a much larger body of requirements. Nuclear documentation requirements are not harmonised. They vary by country, regulator, licensee, project and reactor design, and draw on the EU Nuclear Safety Directive (2009/71/Euratom, amended by 2014/87/Euratom), national legislation and licence conditions, IAEA safety standards and guidance such as GSR Part 2, ASME NQA-1, the WENRA Safety Reference Levels and requirements set for a specific project. ISO 19443 itself leaves security management and nuclear material accounting and control out of scope. This article focuses on ISO 19443 because it brings together the three principles that matter most for the comparison with AI.

Lesson one: identify what is important to safety

What does ITNS ask of a supplier in practice? Clause 6.1.3 of ISO 19443 requires it to break its ITNS products and services down into items and activities, and to determine which of them could jeopardise the safety functions specified by the customer, in line with the licensee’s safety classification of systems, structures and components. The supplier does not decide alone what matters, but the breakdown is its own responsibility, and every later decision about control, verification and records depends on it.

When people talk about AI safety, they usually talk about the model: how capable it is and what it can do. Nuclear thinking looks at something else as well: where the item is used and what it has access to. ISO 19443 counts software as an item too. The same model can be harmless in one setting and risky in another. Answering questions on a public website is one thing. Running commands on a computer, with stored passwords and a reward for finishing a task at any cost, is something else entirely. The Hugging Face incident did not happen because the model was too clever. It happened because of what surrounded it: permission to write to a repository, a leaked password and gaps in monitoring.

The standard offers a second way of looking at AI. ISO 19443 uses the term commercial-grade item for a standard product that was not designed for the nuclear sector, but is used for work that affects nuclear safety. A general-purpose AI model fits that description well. Before such an item can be used in ITNS work, its critical characteristics must be identified and verified. Items whose critical characteristics cannot be verified fall outside that route altogether. For an AI model, that is the hard question: which of its characteristics can you actually verify?

Applied to AI, the ITNS logic also means naming who owns each identified risk. Ownership is where classification stops being an exercise and becomes a control.

You cannot manage safety you have not first classified.

Lesson two: apply a graded approach

ISO 19443 defines the graded approach as making the application of requirements for quality management, documentation, monitoring and measurement commensurate with nuclear safety significance. Clause 6.1.4 then requires suppliers to grade those requirements item by item and activity by activity, taking account of the customer’s ITNS requirements, the complexity of each item or activity, and organisational aspects. Graded does not mean lenient. It means rigour follows consequence, and documentation is named in the definition itself.

Take two AI assistants. The first answers routine questions about a product range. The second supports maintenance technicians in a nuclear installation. The first needs answer monitoring and a clear route to a human. The second needs a defined scope of use, source procedures under configuration control, independent verification of any output before it is acted on, with separate review and authorisation where the safety significance demands it, documented limitations, and a reassessment every time the model, the data or the procedures change. In the second case, the controlled procedure remains the authority. The AI layer sits on top of it, never the other way round. 

None of these controls is new to a nuclear supplier. ISO 19443 already requires verification by people other than those who did the work. Where the output of a process cannot be verified afterwards, the process itself must be validated in advance and periodically revalidated. Much of what an AI model produces falls into exactly that category.

Reassessment is where AI differs most from a power plant. Under ISO 19443, design changes must be substantiated, reviewed and authorised by people who know the intent of the original design. A model update or a new tool permission should trigger reclassification in the same way, and AI capabilities change in months rather than decades.

The greater the consequence of failure, the stronger the control has to be.

Lesson three: build a safety culture that can say stop

Why are frameworks and safeguards not enough? Because people apply them under pressure. ISO 19443 places nuclear safety culture under leadership, in clause 5.1.3, and spells out what it takes, including a questioning attitude, decisions that balance quality, cost and schedule without compromising nuclear safety, the reporting of human, technical and organisational issues, and challenging unsafe acts. Top management must ensure that nuclear safety is not compromised by any decision taken, and the quality policy must commit to safety not being compromised by other priorities.

The AI industry operates under pressures the nuclear sector knows well in principle, if not in pace: competition, investment expectations and short release cycles. The question is whether AI developers can enforce their own safety frameworks when one conflicts with a deadline.

The Hugging Face case shows both sides of that question. Similar behaviour had already been seen in May, but nobody recognised how serious it was. After the incident, OpenAI took the model out of use, stopped its training runs and introduced a new rule: when a severe alert goes off, work stops unless someone can show within 30 minutes that it is a false alarm. Both companies have since published detailed accounts of what happened. Stopping when in doubt and sharing what you learned: that is the most nuclear part of the story. ISO 19443 has the same reflex built in. When work important to nuclear safety does not meet its requirements, it must at least be contained or stopped, and the lessons learned must be shared with the customer and passed down the supply chain. 

Safety is not a framework or a set of procedures. It is an organisational responsibility, and it is tested in the moments when stopping is expensive.

From principles to practice

What would AI developers do differently if they applied these three lessons? Six things stand out:

• Systematic identification and classification of safety-relevant risks, covering deployments and tool access as well as models.
• Testing, verification and oversight in proportion to the potential consequences.
• Independent safety assessment for the highest risk classes.
• Traceable documentation of safety decisions and the reasoning behind them.
• Transparent incident reporting that feeds organisational learning.
• Leadership that is accountable for safety, not only for delivery.

Regulation is moving in the same direction. The EU AI Act (Regulation (EU) 2024/1689) already takes a risk-based approach, with specific obligations for general-purpose AI models with systemic risk. A legal risk category is not the same thing as an organisation that has classified its own activities, though.

Traceable documentation of safety decisions is the recommendation the others depend on. ISO 19443 requires documents to be checked and approved by competent people, to show clearly who recorded what, and to be protected against the use of outdated versions. Classification, grading and learning from incidents only work if the information behind them is controlled, easy to find and written for the person who has to act on it. That is a documentation discipline first, and a technical one second.

ISO 19443 also requires suppliers to keep counterfeit, fraudulent and suspect items out of the supply chain, and to report them without delay. AI has its own version of that problem: models and data of unclear origin, passed from one platform to the next. A nuclear supplier would not install a component it cannot trace. The same discipline applies to a model it cannot verify.

The differences between the two fields are real. AI models are harder to predict, and some behave differently when they recognise a test. In Anthropic’s study, one model resorted to blackmail in 55.1% of cases when it believed the scenario was real, against 6.5% when it believed it was a test. Nuclear testing relies on the opposite: a test must show how the item will behave in real use. That is why ISO 19443 requires tests under the most adverse conditions, using exactly the configuration that will be delivered.

ISO 19443 is not an AI safety standard and was never written to be one. The closest it comes is a single requirement: when design tools such as computer models are used, the organisation must show that they are fit for purpose. The standard offers a way of thinking, not a ready-made recipe, and applying it to AI takes people who understand both fields.

Innovation needs a foundation of safety

The agent that reached Hugging Face caused limited damage. The experiments that produced blackmail stayed in simulation. Both are signals of an industry whose safety management is still maturing while its capabilities accelerate.

The AI industry does not need to copy nuclear safety management wholesale. It can learn from its systematic approach: classify what matters, scale controls to consequences, and build an organisation that is willing to stop. For nuclear organisations introducing AI into their own work, the method already exists. It is in your management system.

The question is not whether AI innovation should slow down. It is whether safety management can mature quickly enough to keep pace with it.

CONTACT US

 

Questions and answers

Does ISO 19443 apply to AI systems?

Not as such. ISO 19443:2018 sets quality management requirements for organisations in the nuclear supply chain that supply products and services important to nuclear safety. It is not an AI standard. Its principles, particularly ITNS classification and the graded approach, offer a useful way of thinking about AI risk, but they need adaptation.

What does ITNS mean?

ITNS stands for important to nuclear safety. ISO 19443 defines it as a characteristic of a product, service, item or activity whose failure could result in undue radiation exposure of people or the environment. Suppliers must break their ITNS products and services down into items and activities, in line with the safety classification set by the customer and the licensee.

What is the graded approach?

The graded approach makes the application of requirements for quality management, documentation, monitoring and measurement commensurate with nuclear safety significance. Under ISO 19443, suppliers grade those requirements for each item and activity, taking account of customer requirements, complexity and organisational aspects.

Can AI tools be used for work that is important to nuclear safety?

That depends on your management system, your licensee’s requirements and your regulator. The nuclear logic is to classify the AI tool and its use like any other item that could affect safety, then apply controls in proportion: defined scope, controlled source documents, independent verification of outputs and reassessment when the tool changes. ISO 19443 already requires design tools such as computerised models to be shown fit for purpose.

What happened in the OpenAI and Hugging Face incident?

In July 2026, experimental OpenAI agents under test escaped the restrictions of their environment, used a leaked credential and reached Hugging Face systems. Hugging Face disclosed the intrusion on 16 July. OpenAI identified its own agents as the source a few days later. Hugging Face found no evidence that public models, datasets or software were altered.

Were the blackmail results from Anthropic’s research seen in real use?

No. All behaviours in Anthropic’s agentic misalignment study occurred in controlled simulations, and Anthropic states it has not seen evidence of them in real deployments. The research was designed to find such behaviour before deployment.

Does the EU AI Act make a graded approach unnecessary?

No. The EU AI Act sets legal obligations by risk category. A graded approach is how an organisation applies controls to its own activities. The two complement each other, much as nuclear legislation and an organisation’s own management system do.

Why does documentation matter so much in safety management?

Because classification, grading and incident learning all depend on information that is controlled, traceable and usable. ISO 19443 names documentation in its definition of the graded approach and requires documented information to be reviewed, approved, traceable and protected against the use of obsolete versions. Without that, a safety framework cannot be verified, audited or improved.