Building With AI, Defending Against It: Why I Don't Get to Stop Learning

Security knowledge has a shelf life#

Everything I know about securing software has an expiration date. That has always been true. What is new is how quickly the date arrives for anything involving AI.

The taxonomy moved under me while I was studying it. OWASP renumbered its LLM Top 10, then published a separate top ten for agentic applications, because agents had outgrown being a footnote on the LLM list. Attack classes that were conference talks two years ago now have CVE numbers attached to developer tools I have open right now.

Attackers are not waiting for defenders to catch up, and they have access to the same AI tooling I do. That asymmetry resolves in exactly one direction. The people defending these systems have to learn faster than the attack surface grows, or the gap compounds quietly until something expensive happens.

Which is why I no longer think of security as a specialty that lives with a team down the hall. If you are a developer or an engineer building with AI, a meaningful share of your system's security posture gets decided by choices you make, in code, on an ordinary Tuesday, usually without anyone reviewing them through a security lens.

I am in that position twice over, and that is what sent me back to structured training.

This post contains no exam answers, no scenario details, and no material lifted from behind TryHackMe's paywall. The value here is in the reasoning and in how the training is built, not in anything that would spoil it for the next person.

Two jobs, one problem#

Personally, I build with AI constantly. Most days I'm inside Claude Code, wiring together automation, custom MCP servers, and multi-agent pipelines that hand tasks to each other with minimal supervision.

Behind that sits a homelab SIEM that pulls my home network's telemetry into a queryable store, and a RAG-shaped persistent memory system that recalls context from weeks earlier without me re-explaining anything.

All of it is AI-native by design. The agents read data, write files, call APIs. Increasingly they decide what happens next without a human approving every step.

Professionally, I sit on the other side of the same problem. Part of my job is helping keep an enterprise secure while that enterprise adopts AI as quickly as it can, which is the right speed for the business and an uncomfortable speed for anyone who owns the risk. Slowing the innovation down is not the assignment. Understanding it well enough to secure it while it moves is.

Those two roles ask the same question from opposite ends. As a builder, I want to know what my system does when someone feeds it something hostile. As a defender, I have to ask that about systems other people built, most of them assembled by engineers who were not thinking about hostile input at all, because they were thinking about shipping.

Neither version of that question gets answered by reading a vendor datasheet or a vulnerability list. You answer it by having personally attacked something shaped like the thing you are worried about.

The numbers back up how wide this gap has become. The global cybersecurity workforce gap sits at 4.8 million people, up 19.1% year over year, and more than a third of respondents in ISC2's 2024 workforce study named AI and automation skills as their team's single biggest shortfall.

Adoption is what keeps that gap widening rather than closing. Eighty-four percent of developers now use or plan to use AI tools, up from seventy-six percent the year before, and fifty-one percent of professional developers use them daily, according to Stack Overflow's 2025 Developer Survey. Only 3.1% said they highly trust the accuracy of what those tools produce.

Heavy daily reliance paired with almost no confidence in the output is not a stable combination. Security engineering and AI system design are converging fast, and treating them as separate disciplines is starting to look like a mistake. I would rather close that gap deliberately than keep living inside it.

Choosing reps over reading#

I could have read papers. I have read plenty of papers. The problem with reading about prompt injection is that it makes you fluent in the vocabulary without making you any better at the thing.

Knowing that indirect injection exists is not remotely the same skill as sitting in front of a chatbot that is actively refusing you and working out, in real time, why your third approach failed when your first one worked.

So I went looking for training with a high ratio of doing to reading, and landed on TryHackMe's AI Security learning path. I finished it after roughly nineteen hours and fifteen minutes of course time. Five modules, twenty-five hands-on labs, rated medium difficulty.

The module names give away the shape of it: AI Fundamentals, Secure AI Systems, Prompt Security, AI Supply Chain Security, and Data Poisoning.

The path's own framing is to build a practical foundation in AI security by teaching how these systems are built, where they break, and how to defend them. In practice, that framing held up.

Click to enlarge

What made the material stick was that ratio. Twenty-five labs against five modules means you are rarely more than a page away from a live environment.

The written material sets up a concept, then the lab immediately makes you use it against a running target. You break something yourself and watch what happens. I have worked through certification paths where the labs feel bolted onto the theory as an afterthought, and this did not feel that way. The hands-on work is the strongest part of the platform, and it is some of the best in the business.

That distinction matters more than it sounds. Reading builds recognition. Labs build the reps. The exam is where you find out whether the reps took.

Digging into the certification#

AI1 is thirteen hands-on scenarios spread across four sections, completed inside a 48-hour window: Threat Modelling, Prompt Injection and Jailbreaking, AI Supply Chain Security, and Data Poisoning.

It costs $399, which includes a free retake, three months of premium, and the learning path itself.

TryHackMe positions it as attack and defend in one exam rather than picking a side, and aligns the content to OWASP and MITRE ATLAS. It is explicitly aimed at three audiences, one of which is people shipping AI-powered products. That is the bucket I fall into from both directions.

The scoring model is what makes this worth writing about, because it is the part that turns an exam into a learning instrument.

Your score on a scenario is raw points divided by marks available. Your final result is the average across all thirteen. Seventy percent or above passes.

The static half#

Seven of the thirteen scenarios are static and web-based. Selection questions score a point for a correct answer, zero for a grey-area answer, and lose a point for an incorrect one, floored at zero so you can't go negative.

Justification questions in this half are graded by an AI against a rubric rather than a human.

The live half#

The other six scenarios are live AI chatbot exercises, and this is where the interesting scoring lives.

Each one is graded from two inputs: eighty percent from an AI grader reviewing your entire conversation transcript against the scenario's marking criteria, evaluating technique, reasoning, adaptability, and conduct. Twenty percent from flag submission, checked against an answer key with partial credit available.

Click to enlarge

That 80/20 split is the detail I keep coming back to. On the live scenarios, the flag you extract is worth a fifth of your grade. The transcript of how you got there, or tried to, is worth the other four fifths.

Which means you can pass while missing flags, and I did exactly that. I did not recover every flag in the live scenarios, and I still cleared the bar, because the reasoning I demonstrated on the way to the ones I missed was worth more than the extraction itself.

That is the correct weighting, and I say that as someone it happened to favour. Capture-the-flag scoring rewards the moment the lock opens. Real defensive work almost never produces that moment. It produces a judgment call about whether a system is safe, made under time pressure, with incomplete information, that someone else has to be able to follow.

The exam isn't asking whether you got lucky. It's asking whether you can reason under pressure against a system that's actively resisting you, and it grades that reasoning directly instead of inferring it from whether you happened to land the extraction.

Click to enlarge

The four lessons that transferred#

I went in expecting to sharpen attack technique. What I actually walked out with was a sharper version of a question I already ask about my own systems, and now ask about everyone else's.

Where, specifically, is the line between data and instructions? Does the code enforce that line, or does it just hope the model respects it?

Click to enlarge

Indirect injection is the cleanest illustration of why that question matters. It doesn't arrive as a prompt typed at the model. It rides in on content the agent is already going to trust: a calendar entry, a wiki page, a retrieved document, an issue comment.

If your agent concatenates that content into the same context window as its instructions, the model has no reliable way to tell "text I should reason about" from "text I should obey."

The current OWASP LLM Top 10 puts Prompt Injection at LLM01 for exactly this reason. My RAG memory system carries the same shape of risk, which is not a comfortable thing to notice mid-exam.

That LLM01 label is worth pausing on, because the numbering is the shared vocabulary underneath everything that follows. Four lessons came out of this training, and between them they land on five of the ten categories.

Click to enlarge

Worth noting what that map also shows: five categories I did not personally stress-test at all. Knowing the shape of your own coverage, rather than assuming it is even, is most of what training like this is actually for.

The clearest real-world version of that failure mode landed on a tool I use every day. In June, researcher RyotaK at GMO Flatt Security disclosed a malicious GitHub issue that could induce the Claude Code GitHub Action to read /proc/self/environ and harvest CI runner secrets, including an API key.

The instructions were hidden inside HTML comments, invisible in the rendered UI. The payload framed itself as a routine compliance review and truncated the credential to dodge secret scanning.

Microsoft Threat Intelligence independently confirmed the finding days later, and it was patched in Claude Code v2.1.128.

The shape of the bug is the point: indirect prompt injection, against an AI coding agent, sitting inside CI/CD, on the exact tool I run automation through daily.

A pair of chained flaws in Cursor's IDE that summer, nicknamed DuneSlide and tracked as CVE-2026-50548, pushed the same bug class to full remote code execution via a poisoned MCP response. Prompt injection isn't a content-moderation problem. It's an RCE vector with a friendlier name.

Drawn as a threat model, the CI/CD chain is short enough to fit on a single line, and that brevity is the alarming part. There is no exploit in it. Every step is something the pipeline was explicitly designed to do, executed in the order it was designed to do it.

Click to enlarge

The only genuine defect is at stage three, where untrusted text and trusted instructions end up in one context window with nothing marking which is which. Everything downstream is the pipeline working correctly on a premise that was already false.

The second lesson sits right next to the first: authorization has to live outside the model, never inside it. A model that's asked to be both the guard and the judge of its own gate can be talked into opening that gate. "Access granted" narrated by a language model isn't authorization. It's a sentence the model generated because the conversation made that sentence plausible.

This is what OWASP's Agentic Applications list calls out directly as ASI03, Identity and Privilege Abuse. It's why every MCP server I write now checks identity and permission deterministically in code before a tool call executes, never by asking the model whether the request seems okay. GitHub's Octoverse 2025 report backs this up with a trend I did not expect: Broken Access Control overtook Injection as the most common CodeQL alert, flagged in over 151,000 repositories, much of it AI-generated scaffolding that skipped checks nobody told the model to add.

Model artefacts are the third lesson, and the one that surprised me most, because it barely looks like security at first glance.

A model file is software supply chain, filed under OWASP LLM03 and ASI04, Agentic Supply Chain Vulnerabilities.

Format is the first control that actually matters. SafeTensors cannot execute code on load. Pickle can, by design, which is how researchers at ReversingLabs found proof-of-concept malicious models on Hugging Face in early 2025, using compression tricks to slip past the platform's own pickle scanner.

A checksum proves a file wasn't corrupted in transit. A signature proves it came from a trusted publisher. Neither proves the contents are benign, and conflating "verified" with "safe" is exactly the gap that got exploited.

The fourth lesson is the one I think about most, because my own memory system is built on it: a retrieval corpus is an authorization boundary, whether you designed it to be one or not. That's the ground OWASP's ASI06, Memory and Context Poisoning, covers on the agentic side.

Similarity-only retrieval has no concept of truth or authority baked in. "Most similar to the query" quietly becomes "most repeated by whoever had write access to the corpus."

Provenance that's cryptographically bound to the content, rather than just string-matched against a source label, is the actual fix for a poisoned corpus, not a scanner running after the fact.

MITRE ATLAS catalogs the specific techniques underneath all four lessons, worth bookmarking as the attacker's-eye taxonomy next to OWASP's defender-side checklist.

The feedback loop is the point#

Most certifications hand you a pass or a fail and leave you to guess what it meant. AI1 does something considerably more useful, and it is the reason I would recommend it to someone who cares about learning rather than about a line on a resume.

That feedback is specific to my own attempt, so I am not going to reproduce a word of it here. What I can share is what it told me, in my own words, because that part generalises.

The per-variant breakdown was more useful than any section score. Each attack type is graded separately, direct and indirect injection, then each again with guardrails in the loop. The spread across mine was wide enough to be the finding. The numbers are mine rather than the exam's; the shape is what generalises.

Direct injection went well. I sustained a coherent multi-turn narrative, so every concession the model granted became leverage for the next request. Strategy rather than guesswork.

Indirect injection is where I fell apart, and the reason stings, because it was a debugging failure rather than a technique failure. Against consistent refusals I cycled through techniques instead of diagnosing which layer was blocking me. I kept rotating instead of asking which specific control was firing.

My indirect attempts were also too explicit, too obviously constructed, to survive contact with an agent that was actually paying attention.

Would I have caught that faster outside an exam clock? Maybe. But the clock isn't really the excuse. I know how to debug a system by isolating variables one at a time. I didn't do it here, not consistently, and my weakest sub-score of the four is the honest record of that.

The guardrail variants complicate that story instead of confirming it, and the honest version is more interesting than a clean pattern would have been. On direct injection, guardrails wrecked me. My strongest sub-score collapsed to nearly my weakest once a structural check entered the loop, which says the multi-turn narrative I was building does not survive contact with a guardrail. On indirect injection guardrails coincided with a marked improvement instead.

Even after that improvement, indirect-with-guardrails still landed well below where I scored on direct injection with nothing in the way. Indirect is the weak axis on its own terms, guardrails or not. Guardrails did not make me uniformly worse; they exposed exactly where my technique was load-bearing and exactly where it wasn't.

That's a real skill gap, not a scoring quirk. The fix isn't a bigger toolbox. It's reading each failure signal for what layer it's actually reporting on, and aiming at that layer specifically instead of trying every tool I own against the same wall.

No amount of reading would have surfaced that. It took a graded transcript of me doing the thing badly.

What changed in how I build#

The indirect-injection gap didn't stay theoretical for long. Within the same week, I went back through every MCP server I've shipped.

The question was narrower than before: is the boundary between retrieved content and executable instruction actually enforced somewhere, or is the model being trusted to keep them apart on its own?

In a couple of places the honest answer was the second one.

Explicit framing has since gone in, so anything pulled from an external source, a fetched page, a stored note, a document another agent wrote, gets tagged as data before it ever reaches the context the model reasons from.

Authorization moved too. Any tool in the stack that can write, delete, or exfiltrate anything now checks identity and permission in a deterministic code path before the call executes, never by trusting the model's narration of whether the request looked legitimate.

That was already partly true before AI1. It's fully true now, because the gaps got looked for instead of assumed away.

One habit stuck harder than the rest. Before shipping anything that reads external content, I now sit down and try to break it myself, the same way the exam kept forcing me to.

It isn't a formal red-team pass, just a deliberate attempt to make the system do something its own instructions never asked for. Most of the time nothing gives. Often enough, something does, and that's a cheap thing to find before shipping instead of halfway through an exam transcript.

The NIST AI Risk Management Framework's Generative AI Profile has also earned a permanent spot in the review checklist. It isn't binding on anything running in a homelab, but it's the closest thing to an industry-standard reference for exactly this class of risk, which makes it useful vocabulary on the professional side too.

The Veracode stat that stuck with me hardest: AI-generated code across more than a hundred models introduced a known OWASP Top 10 vulnerability 45% of the time. Security performance stayed flat regardless of model size or how recent the model was. Bigger and newer did not mean safer.

Sit with that one if you ship AI-assisted code. It is not a prediction about some future model generation. It is a measurement of the tools we are all using today, and it means the review step is the control, not a formality behind some other control that was quietly assumed into existence.

Why the learning does not stop#

The honest conclusion from all of this is not that I now know AI security. It's that I have a current, tested map of what I know and what I don't, and that map has a short shelf life.

The threat actors on the other side of this are getting better, and they are using the same acceleration everyone else is. Defenders do not get to match that pace; we have to beat it, because the attacker only needs the one path nobody thought to close.

That is not a job title. It is a working condition, and it applies to anyone touching these systems.

If you write code with AI assistance, you are making security decisions whether or not anyone labels them that way. If you deploy an agent that reads untrusted content and acts on it, you have taken on a risk that no scanner in your pipeline is currently looking for. Security stopped being a department's problem some time ago, and building with AI is what finally made that impossible to ignore.

The training helped because it was adversarial and graded, not because it produced a certificate. I came away with four lessons I could apply to code I had already shipped, a documented weakness I would not have found on my own, and a much sharper sense of which questions to ask about systems I did not build.

The certificate is dated August 1st, the same week this post went up. That timing isn't incidental. Writing this forced a second pass over my own transcript, and that second pass is what turned a stack of feedback into something I could actually act on.

The exam includes a free retake. I'm going to use it, not to chase a cleaner average, but to see whether the gap between my best and worst sub-scores narrows once the failure-signal habit replaces the toolbox habit.

If it does, the next post is the retake. If it doesn't, I'll write that one too. Either way the next thing to learn is already queued, because that is the only version of this job that works.