Doing AI Governance | Edition #62: The Six Design Rules that AI Safety Forgot


This newsletter provides practical guidance, tools and resources for the real work of governing safe, secure and lawful AI.

Newsletter #62 - August 2026

Feature Article

The Six Design Rules that AI Safety Forgot

By James Kavanagh


Designing an agentic system that is safe and secure is really hard. As an industry, we're solving it by improvisation, trial-and-error and often getting it wrong. Yet the principles we need already exist. They just come from elsewhere.

I've been working on how to translate the established principles and mindset of safety engineering to the practice of building safe agentic systems. It's a discipline I learned as a chemical engineer and a way of thinking that is second nature to anyone who has worked in safety-critical design. Yet I find it's largely absent in AI systems design, even in some of the most sophisticated labs and largest AI companies. Despite the increasing safety risks we're all observing, they're making classic safety errors like failing to separate controls from the system being protected, failing to verify at the boundary, failing in permissive states. Multiple incidents in the last few months bear this out.

There's a deep literature underneath it too, running through Leveson, Rasmussen, Hollnagel and Dekker. Almost none of it is being brought to bear on systems that now hold credentials, call tools, move money and act without a human in the path. We're rediscovering in public, at speed, what was settled decades ago in industries where the feedback was less forgiving.

If you're trying to make sense of the threats to agentic AI systems, and translate those into governance and engineering practices, then these six rules can really help:

  • Separate the control from the thing it constrains.
  • Verify everything that crosses a boundary.
  • Never rely on a single control for a safety-critical property.
  • Design every component for how it fails, not just how it works.
  • Ensure every action is observable and attributable.
  • Every control needs a feedback signal that drives adaptation.

Serious Vampires Rarely Drink Orange Fanta. Not the best mnemonic, but it works.

Guided Practitioner Cohorts

What Practitioner Cohort #1 built in eight weeks

video preview

James on what fifteen practitioners built over eight weeks through a complex hospital case study that had governance for everything except AI.

Cohort #2 starts August 25

Live Sessions: August 25, September 8, September 22, October 6.

Group 1: 8:00 AM Tuesday Sydney. Best for the Americas, Australia and NZ.

Group 2: 8:00 PM Tuesday Sydney. Best for the UK, Europe, ME, India and Asia.

Cost: US$899 OR US$700 if you own the Foundation Track (code FT-COHORT2)

Enrollment closes August 23 or when all places are filled. Only 15 per group.

Frequently Asked Questions

What is a guided practitioner cohort?

It complements our practitioner self-paced courses with a small guided group working through practical implementation in realistic scenarios. Think of it as an immersion in what it's really like to do AI Governance work. It builds on the same four courses, wrapped in a structure with a fixed start and finish, no more than fifteen people, live workshops with me, and a single complex case study the group works from beginning to end. The rhythm is a fortnight of self-paced study on one course, supported by a 90-minute workshop where we apply it to the case along with discussions along the way.

You play the incoming AI governance lead at a children's hospital, working your first hundred days. An AI module alerting nurses, live for twelve months has never been evaluated. A research model drifting into clinical decisions about vulnerable adolescents. A chatbot handling post-discharge follow-up. Patient data flowing to tools nobody approved. Clinical governance committees, risk registers and incident reporting, all of it working, and none of it seeing AI. Over eight weeks you map the harms, set the principles, build the inventory, work out which mechanisms are broken and design the ones that are missing, then write the policies. The challenge increases to the final capstone project and assessment for the AI Career Pro Practitioner Award.

Who's it for?

Anyone working to turn their accelerate their knowledge into build practical skills. You might be new to AI Governance or have already completed a credential like the AIGP. You may already be practicing professional looking to expand your skills

Cohort #1 drew a more senior room than I expected. Fifteen practitioners across five countries, from law, audit, data leadership. Three already held the AIGP. Two were ISO 42001 lead implementers or auditors. Some were already highligh proficient in other aspects of risk and compliance. Others were new to AI Governance, but coming from accomplished careers.

It suits people who learn best by debating and challenging themselves alongside peers, and people who want a credential that reflects assessed capability rather than course completion. It's open whether you already own the Foundation Track or you're starting from scratch, since the price includes the four courses.

Why would someone do it? What problem is it solving?

Two problems, and they're different.

The first is completion. Self-paced study can compete with everything else in your week and it usually loses. A fixed schedule, a group moving at the same pace, and people expecting you changes those odds considerably.

The second is the distance between knowing and doing. Reading how to build a harms register and building one for an organization with competing pressures and awkward constraints are different skills, and only one of them is what an employer is buying. You finish with artifacts you built yourself, configured in tools and competence to take to a new role.

For the more senior participants there's a third reason, which is the room. Fourteen other practitioners from different sectors working the same problem, guided by myself as a practitioner who has built governance and engineering programs in some of the most challenging environments.

What's included?

  • The four Foundation Track courses
  • Four live 90-minute workshops with me, one every fortnight
  • Two 30-minute 1:1 sessions with me, scheduled to suit you
  • Special-topic discussions with guest practitioners on the alternate weeks, including a session with the founder of VerifyWise
  • Direct email access to me throughout
  • A private cohort forum that continues after the program ends
  • The continuous case study and every artifact you build through it
  • A graded, applied final assignment
  • The Practitioner Award, AI Governance Foundations, on successful completion

What's the commitment?

The self-paced Track teaches you the material, and you can start that as soon as you sign up, even before the cohort sessions begin. In the cohort workshop, you then apply what you've learned. You should plan for 4-6 hours per week over the 8 weeks, including the workshops.

Can my whole team do it?

Yes. Several organizations have already put multiple people through the Foundation Track together, and team enrollment is priced case by case depending on numbers and how you want it structured. Reply to this email and we'll work it out with you.

How do I secure my spot?

Enroll at the cohort page. If you already own the Foundation Track, apply FT-COHORT2 at checkout. Both groups have a few spaces left at the moment, so there's no need to pick one at checkout. But they close for certain at 15 people. We'll confirm your group with you once you're enrolled.

What if I have more questions?

Reply to this email. It comes to me, and I'll answer.


What we're working on

As more and more people prepare for and then succesfully pass their AIGP Certification exams with us, we've been listening to feedback and ideas. I believe in the value of feedback and continuous improvement, so we've been making the practice exams tougher, and we've added more explainers and resources on the areas people consistently wanted more on - like AI technologies and US laws. This isn't work with an end date. We're listening to everything we receive and we'll keep sharpening the course. You'll find our AIGP exam preparation course here.

I'm also running two live webinars this month for people who've completed either of our assessments, the Practitioner Capability assessment or the Organizational Capacity one. Plenty of you have taken these and told asked to discuss in more depth the outcomes and pathways forward. So that's what these sessions are for: what the dimensions actually measure, how to build from knowledge into practice and good judgment. Come with questions.

Webinar #1 · best for the USA, Canada, Latin America, Australia and New Zealand
🗓️ Tuesday 11 August, 5:00 PM Pacific · 8:00 PM Eastern
🗓️ Wednesday 12 August, 9:00 AM Japan · 10:00 AM AEST

Webinar #2 · best for the UK, Europe, the Middle East and Singapore
🗓️ Thursday 13 August, 11:00 AM UK · 12:00 PM Central Europe · 2:00 PM UAE · 6:00 PM SGT · 8:00 PM AEST

Register here, and if you haven't taken an assessment yet, start there and then join us.

As always, reach out if you have questions about any of this, or if you're trying to work out your next move professionally and want a second opinion. We're learning alongside you, and the questions you send genuinely shape what we build next.

PS. You're receiving this as a subscriber to communications from AI Career Pro. We respect your privacy, so please unsubscribe through the link below if you do not wish to receive these communications in the future.

PO BOX 7087, Redhead, NSW 2290
Unsubscribe · Preferences

Doing AI Governance

Join over 5,000 subscribers and learn about the real work of AI governance. Moving beyond theory, we focus on the practical application of AI governance in real-world organizations with case studies, tools, templates and guidance. Led by James Kavanagh - the AI governance practitioner who led governance at both AWS and Microsoft.

Read more from Doing AI Governance

This newsletter provides practical guidance, tools and resources for the real work of governing safe, secure and lawful AI. Newsletter #63 - August 2026 Feature Article The Gap Between Job Ads and the Real Work Being Done in AI Governance Roles By James Kavanagh "When people ask me what the job of being an AI governance practitioner really looks like, beyond the list of tasks, and stripped of the language in job ads, I've landed on a fairly steady answer. We do 8 things." I read a lot of AI...

This newsletter provides practical guidance, tools and resources for the real work of governing safe, secure and lawful AI. Newsletter #61 - July 2026 Feature Article 5 Ways to Take the Practice Leap into AI Governance By James Kavanagh Every AI governance practitioner faces three capability leaps over the course of their career: the Practice Leap, the Adaptive Leap, and the Leadership Leap. The Practice Leap moves a practitioner from knowing what governance should look like, to actually...

This newsletter provides practical guidance, tools and resources for the real work of governing safe, secure and lawful AI. Newsletter #60 - July 2026 Feature Article #1 Building an AI Inventory is your First Governance Intervention By James Kavanagh Your first meaningful governance intervention is most likely building an AI inventory. It's the first time anyone in the organisation has to sit down and agree, out loud, on what AI is really running and what it's for. That's not an admin task...