AI Risk Assessment Framework
View on GitHub
Contents

Reference: Risk Catalogue

This catalogue provides the detail behind the eight risk categories. Use it alongside Step 3: Assess the Risks. Once the heatmap has shown you which risks are material for your use, read the relevant entries here as you work through each one.

Each entry has four parts:

The eleven use types are defined in Reference: Use-Type Profiles. Where a use involves AI that acts rather than advises, also apply the autonomy adjustment in Step 3. It raises accountability, supply chain, and prompt injection above the levels described here.

Data leakage

The risk: Sensitive data is sent to a third-party AI service and is exposed, retained, used for model training, or otherwise leaves your control. This includes data in prompts, context windows, uploaded files, and any metadata transmitted alongside your queries.

Use type How this risk typically manifests
Software development Source code snippets containing secrets (API keys, credentials, connection strings), proprietary business logic, or security-sensitive implementation details are sent to a cloud AI service. Even without explicit secrets, code can reveal system architecture and vulnerabilities.
Code analysis The risk is amplified because code analysis typically involves sharing much larger volumes of code: potentially entire repositories. This greatly increases the chance of inadvertently sharing embedded secrets, and gives the AI provider a comprehensive view of the system's architecture and security posture.
Synthetic data generation Real production data is used to seed generation, exposing it to the AI service like any other data. And the output may reproduce real records verbatim or allow individuals to be re-identified by combining fields. The dataset is then distributed to development and test environments and handled with far less care than the source data, because everyone believes it is synthetic.
Product feature User data (PII, case records, health data) is processed by AI services at runtime, on an ongoing basis. A data breach or policy change by the AI provider could expose user data at scale.
User-facing support Support tickets contain an unpredictable mix of sensitive data: users routinely paste credentials, PII, system details, and screenshots into tickets. This data is processed by the AI tool for categorisation, response drafting, or automation. Attachments may contain visible secrets that bypass text-based scanning.
Live service operations Production logs, traces, and alert payloads are among the most credential-rich data a team holds: access tokens, session identifiers, connection strings, and user PII appear in them routinely and unreviewed. During an incident, responders paste whatever is in front of them into whatever tool is to hand, under time pressure and often outside their normal habits.
User research Interview transcripts, survey responses, and demographic data are shared with AI tools. This data is often highly personal: health, finances, immigration status, experiences of harm. It was collected under a specific consent that frequently predates AI entirely. Even anonymised transcripts can be re-identifiable from the detail of what a participant described.
Design Lower exposure than most use types. Design briefs, wireframes, and journey maps rarely contain personal data, though prototypes populated with realistic-looking data, and summarised research findings pasted in for context, can carry more than intended.
Content Content drafts are rarely sensitive in themselves, but letter and notification templates often carry real examples, and policy source material may be sensitive before publication. Content for an unannounced service or policy change is commercially and politically sensitive until launch.
Business analysis Stakeholder notes, commercial and financial information, draft business cases, and pre-decision policy material are shared with AI tools. Rarely participant-level personal data, but frequently sensitive before a decision is announced.
General productivity Meeting transcripts, email content, documents, and calendar data are processed by AI tools. Meetings frequently contain sensitive discussions, PII, and commercially confidential information. AI tools may also access data via broad permissions (calendar, contacts, files) beyond what the user explicitly shares.

Ask yourself:

Key mitigations:

Accuracy and hallucination

The risk: AI generates output that is plausible and confidently presented but factually incorrect, logically flawed, or subtly misleading. This is an inherent characteristic of current AI systems, not an occasional bug. The consequence depends entirely on how the output is used and how much scrutiny it receives.

Use type How this risk typically manifests
Software development Code that is syntactically valid but contains logical errors, uses deprecated or insecure APIs, introduces subtle security vulnerabilities, or implements incorrect business logic. These issues may pass casual review because the code looks plausible. Research indicates AI-generated code introduces approximately 1.7x more issues than human-written code.
Code analysis Analysis findings that are inaccurate: false positives (flagging issues that don't exist) or false negatives (missing real issues). AI may misunderstand the intent of code, miss context-dependent vulnerabilities, or produce architectural assessments that sound authoritative but are wrong.
Synthetic data generation Generated data that does not reflect the real distribution gives false confidence: the service passes its tests and then fails in production on cases the generator never produced: unusual names, non-Latin characters, missing or very long field values, edge-case dates. The failure is silent, because a green test suite looks identical either way.
Product feature Incorrect information presented to users as fact. Fabricated citations or references. Wrong classifications or recommendations that affect service delivery. In government contexts, this can directly harm citizens, for example through incorrect eligibility assessments or misleading guidance.
User-facing support Incorrect ticket categorisation delays resolution of critical issues. Wrong troubleshooting advice makes problems worse. AI suggests plausible but fictitious resolution steps or references non-existent procedures. Stale knowledge from historical tickets leads to outdated advice for current systems.
Live service operations A confidently wrong root cause sends responders down the wrong path while the incident continues. Log correlations that look authoritative may be coincidence. Generated infrastructure-as-code may be syntactically valid but semantically wrong in ways that only surface on apply. Incident pressure is exactly the condition under which plausible-sounding output receives the least scrutiny.
User research Distorted summaries of what participants actually said. Fabricated themes or patterns that do not exist in the source data. Subtle synthesis bias that systematically underweights certain participant perspectives.
Design Design suggestions that look plausible but break under real use, and confident accessibility assessments that are simply wrong: more dangerous than no assessment at all, because they close the question.
Content The highest-consequence form of this risk outside product features. A wrong eligibility statement, deadline, or entitlement on a guidance page is read and acted on by thousands of people, and for many users the guidance is the service. Fluent, confident prose is the least likely to be fact-checked.
Business analysis Hallucinated detail with a long fuse. An invented eligibility rule or misremembered statutory duty that enters a user story looks like a requirement by the time it reaches a developer. It is then built, tested against itself, and shipped, with every downstream step treating it as already verified. Nothing in the delivery process is designed to catch a requirement that was wrong from the start.
General productivity Incorrect meeting summaries that misattribute statements or miss key decisions. Inaccurate translations that change meaning. Email drafts that contain fabricated details or references. Document summaries that lose important context or nuance.

Ask yourself:

Key mitigations:

Accountability gaps

The risk: When AI contributes to an output or decision, the chain of responsibility becomes unclear. If AI-generated code introduces a security vulnerability, who is accountable? If an AI-assisted research summary leads to a poor design decision, who owns that? In government contexts, statutory duties and public accountability make this particularly important.

Use type How this risk typically manifests
Software development Unclear ownership of AI-generated code. Developers may feel less responsibility for code they didn't write from scratch. Code review standards may slip because "the AI wrote it." No audit trail of what was AI-generated versus human-written.
Code analysis Analysis findings are presented without clear attribution. Decision-makers may not understand that findings are AI-generated hypotheses rather than verified facts. Recommendations based on AI analysis may lack the human expert judgement needed to assess their validity.
Synthetic data generation Unclear who owns the claim that a dataset is safe to treat as non-personal. If re-identification later proves possible, the decision to classify the output as synthetic may have been made implicitly by whoever happened to run the generator, with no record of what was checked.
Product feature The most critical accountability gap. When an AI system makes or influences decisions that affect citizens, there must be a clear human decision-maker who is accountable. Algorithmic transparency obligations (ATRS) apply. Users must be able to understand how decisions were reached and challenge them. For classification and scoring models, UK GDPR Article 22 restricts solely automated decisions with legal or similarly significant effects, and affected people need a route to explanation and challenge.
User-facing support Accountability is diffuse: support team, tool provider, and management all share responsibility. When AI gives wrong advice that causes a user to take an action damaging a system, or auto-closes a ticket that should have remained open, the escalation path may be unclear. Precedent exists (e.g. the Air Canada chatbot ruling) establishing that organisations are liable for their AI tool's statements.
Live service operations During an incident the chain of decision-making is compressed and poorly recorded. If an AI-suggested remediation makes an outage worse, the incident record may not show that AI proposed it. Where the AI acts rather than advises, there is no person who chose the action, only someone who configured the system, possibly months earlier and no longer on the team.
User research Research findings presented without transparency about AI involvement. Design decisions based on AI-synthesised insights rather than direct engagement with users. Stakeholders may not realise the evidence base was AI-processed.
Design Generally lower. Design work is visibly iterative and passes through research and testing, so AI involvement rarely obscures who owns the resulting decision.
Content Published content carries the department's name. Where AI-drafted content stating a rule or entitlement turns out to be wrong, the organisation is answerable for it regardless of how it was produced, and there is often no record that AI was involved or who approved the wording.
Business analysis The most acute accountability gap outside product features. A governance body reads an options paper, business case, or DPIA as evidence that a named person reasoned through a problem. An artefact that reads well because a model wrote it, and that nobody has genuinely thought about, defeats the governance step while passing it.
General productivity Shared meeting notes or summaries produced by AI without colleagues knowing. Decisions based on AI-processed information where the AI involvement is not disclosed. Unclear responsibility when AI-drafted communications cause misunderstandings.

Ask yourself:

Key mitigations:

Bias and fairness

The risk: AI systems can reflect, amplify, or introduce biases from their training data. In government services, biased AI outputs can lead to discriminatory outcomes that disproportionately affect already disadvantaged groups, potentially breaching the Equality Act 2010 and the public sector equality duty.

Use type How this risk typically manifests
Software development Generally lower risk, but AI coding suggestions may reflect biases in training data (e.g. gendered variable names, culturally specific assumptions, accessibility antipatterns).
Code analysis Analysis may underweight or miss issues that disproportionately affect certain user groups (e.g. accessibility problems, internationalisation issues). AI may also reflect biases about what constitutes "good" code.
Synthetic data generation The defining risk of this use type. Generators trained on majority-case data under-produce minority cases: unusual name forms, non-binary or absent gender values, atypical address formats, edge-case eligibility combinations. Those are precisely the cases where government services most often fail their users, so a fidelity gap becomes an equality gap. Where the dataset trains or evaluates a model, the skew is inherited directly.
Product feature The highest risk. Classification, recommendation, or decision-support systems may discriminate against protected groups. AI trained on historical data will reflect historical biases. This applies to any feature that treats different users differently or prioritises some over others.
User-facing support Language bias: sentiment analysis and categorisation tools perform differently with non-native English speakers or regional dialects. Prioritisation bias: AI trained on historical ticket data reproduces historical patterns, potentially disadvantaging certain user groups or issue types. Tone bias: direct or terse communication styles may be misinterpreted. Digital literacy bias: well-structured tickets may receive better AI-assisted responses than those from less technically confident users.
Live service operations Generally lower risk. Anomaly detection tuned on historical patterns may under-flag conditions affecting smaller user groups, less-used journeys, or less-monitored parts of the estate.
User research AI synthesis gravitates to the clearly-expressed majority view. Participants who spoke through an interpreter, took longer to make a point, or used non-standard English are the most likely to be smoothed out of a themes list, and in government research they are frequently the users the service most needs to work for. AI-generated personas may also rely on stereotypes.
Design AI produces accessibility antipatterns confidently: colour as the only carrier of meaning, insufficient contrast, vague link text, unlabelled imagery, form patterns that break with assistive technology. Accessibility decisions made at design time are the hardest and most expensive to undo later.
Content Reading age and register drift exclude users. GOV.UK targets a reading age of nine; AI writes well above that, in a register that drifts corporate. AI also handles tone poorly for content about bereavement, debt, immigration status, or a refused application, tending towards brightness where plainness is needed. Translation may introduce cultural bias or lose nuance.
Business analysis Generally lower risk, though requirements drafted from AI-summarised research inherit whatever representativeness problems that synthesis had, and by then they look like settled requirements rather than contested findings.
General productivity Translation tools may introduce cultural bias or lose nuance. Summarisation may systematically underweight certain viewpoints in meeting discussions. AI-generated content may reflect cultural assumptions that are not appropriate for the audience.

Ask yourself:

Key mitigations:

IP and licensing

The risk: The intellectual property status of AI-generated content is uncertain and evolving. AI-generated code may incorporate material from training data with unclear licensing. There are unresolved questions about copyright ownership of AI-generated outputs and potential liability for inadvertent infringement.

Use type How this risk typically manifests
Software development AI may reproduce code from its training data, potentially introducing GPL or other copyleft-licensed code into a proprietary codebase. Copyright ownership of AI-generated code is legally uncertain: the UK position is that works without sufficient human creative input may not attract copyright protection.
Code analysis Lower risk as the output is analytical rather than generative. However, if the analysis tool reproduces substantial portions of the analysed code in its output, IP considerations may apply.
Synthetic data generation Generally lower risk. Consider whether licence or contractual terms attached to the source data extend to data derived from it, particularly where the source is client-owned or third-party data.
Product feature Content generated by AI for users may not be copyrightable. If the product relies on AI-generated content, consider whether this creates a business or legal risk. Liability for incorrect AI-generated content shown to users is an evolving area.
User-facing support Generally lower risk. AI-generated knowledge base articles or response templates may incorporate content from training data, but the main concern is operational accuracy rather than IP. If the support tool generates customer-facing content, consider whether it accurately represents your organisation's position.
Live service operations Generally lower risk. Generated infrastructure-as-code and runbook content carries the same licensing uncertainty as any other AI-generated code, but at smaller volumes and with less public exposure.
User research Generally lower risk. Outputs are internal research artefacts.
Design AI-generated images and illustration published under a government brand carry licensing uncertainty, and reputational exposure if a resemblance to training material is later identified. Establish provenance expectations before the material is produced, not after.
Content AI-generated copy used in published material has uncertain copyright status and may incorporate phrasing from training data. Consider provenance for anything appearing in public-facing or press material.
Business analysis Generally lower risk. These artefacts are internal, though a business case or process model reused across organisations may raise ownership questions.
General productivity Generally lower risk. AI-generated text for internal use is unlikely to raise significant IP concerns. However, AI-generated content used in external communications or publications may have uncertain copyright status.

Ask yourself:

Key mitigations:

Over-reliance and skill erosion

The risk: Teams become dependent on AI tools in ways that erode critical thinking, core competencies, and the ability to work without AI assistance. Junior team members may learn from AI outputs rather than developing foundational understanding. Recent research indicates that higher AI reliance correlates with reduced critical thinking.

Use type How this risk typically manifests
Software development Developers accept AI suggestions without fully understanding the code. Debugging skills atrophy because AI provides quick fixes. Junior developers learn patterns from AI rather than understanding fundamentals. Teams cannot function effectively when AI tools are unavailable.
Code analysis Teams defer to AI analysis rather than developing their own understanding of the codebase. AI findings are accepted without verification. The team's ability to manually assess code quality and architecture diminishes over time.
Synthetic data generation Teams stop thinking about what their test data actually needs to cover, delegating edge-case design to the generator. Over time nobody knows which real-world conditions the dataset does and does not represent, and the gaps become invisible.
Product feature Less directly applicable to skill erosion, but teams may over-rely on AI features rather than building simpler, more robust solutions where appropriate.
User-facing support Support agents who rely on AI-suggested resolutions stop building deep understanding of the systems they support. Automation bias leads to uncritical acceptance of AI suggestions. When experienced staff leave, institutional knowledge, the "muscle memory" of manual operations, goes with them. Teams may be unable to handle normal ticket volumes if the AI tool becomes unavailable.
Live service operations Diagnostic skill is the thing being eroded, and it is exactly what you need when the AI is unavailable or wrong. On-call engineers who reach for AI triage first stop building the mental model of the system that lets them recognise a novel failure. Knowledge of manual recovery procedures decays quietly until the day it is needed.
User research Researchers use AI summaries as a substitute for engaging directly with raw data. Analytical skills atrophy, and the empathy and nuance that comes from direct user engagement is lost. Decisions rest on AI-processed insight rather than first-hand understanding.
Design Designers converge on AI-suggested patterns rather than exploring the problem, and the craft of working out why a pattern fits this service decays. Teams may stop consulting the GOV.UK Design System because the AI answered first.
Content Content designers lose the discipline of writing plainly under constraint: the skill that produces good government content. Editing AI prose towards the style guide is a different and lesser skill than writing to it.
Business analysis Delegating the reasoning rather than the writing. The artefact exists to demonstrate that thinking happened; if AI supplies the thinking, the team loses both the analytical capability and the understanding the exercise was meant to produce.
General productivity Staff rely on AI for writing tasks and lose the ability to draft clearly without it. Meeting notes are delegated entirely to AI, reducing active listening and engagement. Teams cannot function at normal pace if the AI tool becomes unavailable.

Ask yourself:

Key mitigations:

Supply chain and security

The risk: AI tools and AI-generated outputs can introduce security vulnerabilities, malicious dependencies, or compromised packages. AI coding assistants may suggest vulnerable code patterns or resolve dependencies in ways that introduce supply chain risks.

Use type How this risk typically manifests
Software development AI-generated code may include insecure patterns, use deprecated APIs with known vulnerabilities, or suggest dependencies that are malicious or compromised. Research shows AI-generated code contains security vulnerabilities at higher rates than human-written code, including improper input validation, insecure data handling, and injection vulnerabilities.
Code analysis The AI tool itself becomes part of the supply chain: if it has access to the codebase, its security posture matters. A compromised analysis tool could exfiltrate code or inject misleading findings.
Synthetic data generation Generally lower risk. The generation tool receives production data and so must be assessed as a data processor, but it is not usually a runtime dependency of the service itself.
Product feature The AI service is a runtime dependency. Its availability, security, and behaviour directly affect the product. Changes to the AI model's behaviour (through provider updates) can change the product's behaviour without any code change on your part.
User-facing support The AI tool becomes part of the operational support infrastructure. If it has access to the ticketing system, its security posture matters. A compromised tool could access ticket data including credentials and system details. Prompt injection via malicious ticket content is a specific risk: a crafted support ticket could cause the AI to reveal sensitive information or take unintended actions. If AI can take actions (reset passwords, modify configurations), each automated action needs defined guardrails.
Live service operations The highest-exposure use type for this risk. An AI tool wired into observability or incident tooling has standing access to production telemetry, and frequently to the systems themselves. If it can execute actions, it is an actor inside your production environment with whatever privileges you granted it, and a compromised or misbehaving tool has the reach of an insider. Provider-side model changes can alter its behaviour with no change on your part.
User research Lower direct risk, but AI tools processing sensitive research data become part of the data processing chain and must be assessed accordingly.
Design Generally lower risk. Design tools with AI features hold design assets rather than production data or credentials.
Content Generally lower risk, though AI features integrated into a content management system may hold publishing permissions. Check what the integration can actually change, and whether it can publish without review.
Business analysis Generally lower risk. The tools involved are ordinary productivity and collaboration services already under existing controls.
General productivity AI productivity tools become part of the organisation's infrastructure. Broad permissions (calendar, email, files) create a large attack surface if the tool or provider is compromised. Changes to the AI model's behaviour can affect how it processes sensitive information without any action on your part.

Ask yourself:

Key mitigations:

Prompt injection

The risk: Adversarial content (whether in code, documents, support tickets, or user inputs) can manipulate an AI tool into ignoring its instructions, revealing sensitive information, or taking unintended actions. Prompt injection is ranked #1 on the OWASP Top 10 for LLM Applications (2025) and the UK's National Cyber Security Centre (NCSC) has warned that it "may never be totally mitigated" because LLMs have no inherent separation between instructions and data. This means any content the AI processes is a potential attack surface.

There are two forms:

Use type How this risk typically manifests
Software development Malicious instructions hidden in code comments, README files, AI configuration files (e.g. .cursorrules, .github/copilot-instructions.md), pull request descriptions, or issue trackers are processed by the coding assistant and followed instead of the developer's intent. Invisible Unicode characters can make these instructions undetectable to human reviewers. Real-world examples include remote code execution via crafted GitHub Issues (CVE-2025-53773) and supply chain attacks via poisoned AI rule files. AI assistants may also hallucinate package names that attackers register as malicious packages ("slopsquatting").
Code analysis Adversarial comments or string literals in the codebase can instruct the analysis tool to suppress or downgrade security findings, misclassify vulnerability severity, or (if the tool has network access) exfiltrate code to external endpoints. This is particularly concerning because code analysis processes the entire codebase, giving any embedded adversarial content access to the full scope of the analysis.
Synthetic data generation Generally lower risk, as the tool generates content rather than processing untrusted content. Where real user-submitted free text seeds the generation, treat that text as untrusted input.
Product feature End users craft inputs to extract system prompts, bypass safety filters, or cause the AI to behave in unintended ways. Indirect injection via processed documents (hidden text in PDFs, zero-width Unicode characters, CSS-based obfuscation) can manipulate summarisation, classification, or decision support. AI-generated output may contain executable content (HTML, JavaScript) enabling cross-site scripting. Real-world incidents include zero-click data exfiltration from Microsoft 365 Copilot (CVE-2025-32711, CVSS 9.3) and session cookie theft via a customer support chatbot.
User-facing support Crafted support tickets can manipulate AI categorisation (e.g. causing a security incident to be classified as routine), extract information from other tickets via the AI's context retrieval, bypass exclusion rules designed to route sensitive tickets to humans, or instruct the AI to take unintended actions (password resets, configuration changes). Ticket attachments (screenshots, logs, PDFs) are a further injection vector that may bypass text-based scanning.
Live service operations Log content is partly attacker-controlled: anyone who can cause your service to log a string of their choosing (via a username, URL path, user agent, or form field) has an injection route into your operations assistant, without needing any access to your systems. Alert payloads, third-party webhook content, and ticket text carry the same exposure. Where the assistant can act, injection escalates from information disclosure to remote action inside production.
User research Adversarial content in survey responses or interview transcripts can manipulate AI-powered analysis, amplifying particular viewpoints, suppressing others, or injecting false themes. Research has shown this is effective in digital democracy and consensus-building tools. Documents from external sources may contain hidden instructions (white-on-white text, zero-width characters) that skew how the AI summarises them.
Design Generally lower risk. Design work mostly generates from your own brief rather than processing untrusted external content.
Content Generally lower risk, though source material gathered from external websites or third-party documents during content research may carry hidden instructions.
Business analysis Policy documents, supplier responses, and stakeholder material from external sources may contain hidden instructions that skew how the AI summarises evidence or frames options, a particular concern where the material comes from a party with an interest in the outcome.
General productivity Documents from external sources processed for summarisation may contain hidden adversarial instructions. Emails or messages processed by AI tools could contain content designed to manipulate how the AI summarises or responds. Meeting recordings shared with transcription tools could include deliberately misleading statements intended to influence the AI's summary.

Ask yourself:

Key mitigations: