AI-Generated Content Risks for Online Safety

AI-generated content introduces a distinct category of online safety risk that differs structurally from traditional cybersecurity threats: rather than exploiting software vulnerabilities, it exploits human perception, institutional trust, and information ecosystems. This page covers the definition and regulatory scope of AI-generated content risks, the technical mechanisms through which harm is produced, the operational scenarios most commonly encountered across personal and organizational contexts, and the decision boundaries that distinguish AI-specific risk from adjacent categories of digital harm. The Online Safety Providers provider network maps service providers and professional bodies operating in this space.

Definition and scope

AI-generated content risk refers to the class of online safety harms that arise when synthetic media — text, images, audio, or video produced by machine learning systems — is used to deceive, manipulate, harass, or defraud individuals or institutions. The Federal Trade Commission (FTC) has flagged AI-generated impersonation as a priority enforcement area under its impersonation rule (16 CFR Part 461), which took effect in 2024 and expressly covers AI tools used to impersonate government agencies and businesses.

The scope of AI-generated content risk spans four primary content types:

1. Synthetic text — Large language model outputs used in phishing, fraud, disinformation, and automated harassment
2. Synthetic images — Generative image model outputs used in fake identity documents, non-consensual intimate imagery (NCII), and manipulated visual evidence
3. Synthetic audio (voice cloning) — AI-replicated voice used in vishing attacks, extortion, and unauthorized impersonation
4. Synthetic video (deepfakes) — Composite video used in fraud, political manipulation, and reputational harm

The National Institute of Standards and Technology (NIST) Artificial Intelligence Risk Management Framework (NIST AI RMF 1.0) identifies trustworthiness failures in AI systems — including outputs that are misleading, biased, or harmful — as a structured risk category requiring organizational governance.

Distinguishing AI-generated content risk from general misinformation is a classification boundary that matters for regulatory and service-sector purposes: AI-specific risk involves automated generation at scale, low production cost relative to convincingness, and attributional ambiguity that complicates platform enforcement.

How it works

AI-generated content risks operate through a pipeline with three discrete phases: generation, distribution, and exploitation.

Generation occurs when a threat actor uses a foundation model — a large language model, diffusion model, or voice synthesis tool — to produce convincing synthetic content. The barrier to production has dropped significantly; as of 2023, the majority of commercially available voice cloning services required audio samples of under 30 seconds to produce convincing replicas (documented in the FTC's February 2024 consumer alert on voice cloning scams).

Distribution uses existing digital infrastructure — email, SMS, social media platforms, and messaging applications — to deliver the synthetic content to targets. Platforms governed by the Online Safety Providers service categories are primary distribution vectors, and platform-level detection is inconsistent.

Exploitation is the phase in which harm materializes: financial fraud is completed, reputational damage occurs, a victim's behavior is altered by manipulated evidence, or a political narrative is shaped by synthetic media. The exploitation phase is often separated in time from generation, making attribution and mitigation harder.

The principal technical feature that distinguishes AI-generated risk is scalability with plausibility: a single threat actor can produce thousands of individually personalized phishing emails or fabricated images at a cost and pace that no human-labor model could match.

Common scenarios

Documented operational scenarios across federal enforcement records and academic research cluster into five categories:

1. AI-powered phishing and business email compromise (BEC) — Synthetic text used to craft contextually accurate, grammatically clean phishing messages. The FBI Internet Crime Complaint Center (IC3) reported BEC losses exceeding $2.9 billion in 2023 (FBI IC3 2023 Internet Crime Report), with AI-assisted generation identified as an accelerant.

2. Voice cloning fraud — Synthetic audio replicating a family member's or executive's voice to authorize fraudulent wire transfers or extort victims. The FTC issued a dedicated consumer alert on this attack class in 2024.

3. Non-consensual intimate imagery (NCII) — AI-generated sexualized images of real individuals produced without consent. As of 2024, 48 U.S. states have enacted or introduced legislation specifically addressing deepfake NCII, according to the Cyber Civil Rights Initiative.

4. Synthetic identity fraud — AI-generated face images used to pass liveness checks on identity verification systems, enabling fraudulent account creation.

5. Disinformation and influence operations — Coordinated inauthentic behavior using synthetic personas, AI-written content, and deepfake video to manipulate political or commercial narratives, documented by the Stanford Internet Observatory.

The Online Safety Provider Network Purpose and Scope page provides context for how these risk categories map to service-sector classifications within this reference framework.

Decision boundaries

AI-generated content risk is distinct from — but frequently overlapping with — three adjacent categories:

The operative distinction for regulatory classification is whether the harm-producing content was generated by an automated system rather than authored by a human. This matters for platform liability analysis under Section 230 of the Communications Decency Act (47 U.S.C. § 230), which courts and regulators are actively reexamining in the context of AI-generated content.

For professional service navigation, How to Use This Online Safety Resource explains how the provider network structure maps to specific risk categories, including those covered here.