OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM

OpenAI announced on Tuesday a set of measures aimed at child sexual abuse material (CSAM), including a new image classifier, in-product warnings and temporary account blocks in ChatGPT, developer guidelines, and tools for law enforcement. The announcement comes amid what the company describes as growing safety concerns around how AI tools are used.
The centerpiece is a classifier that detects CSAM images, including ones produced by AI. OpenAI is making it available at no cost through its API to a select group of companies. Rather than matching against known files, the tool looks for visual signs of abuse in images it has not seen before and returns a score that gives reviewers an indicative direction on how to handle a case. That design matters in practice: hash-matching systems, the long-standing approach for known material, cannot catch novel images, so a model that generalizes from visual cues can surface candidates a database would miss.
OpenAI said it built the classifier with the National Center for Missing and Exploited Children (NCMEC) and is working with Tech Coalition and ROOST to share access. In early partner testing, the company reports the classifier flags 48% fewer images while finding the same number of abusive ones, cutting reviewer workload, and that flagged images are twice as likely to be abusive. For trust and safety teams, precision is the expensive part of moderation; fewer false positives for the same recall translates directly into fewer human reviews and less exposure of reviewers to harmful material.
On the consumer side, ChatGPT is getting warning screens. When a user attempts prompts that would access material sexualizing children, the app shows an alert, points the person toward specialized support intended to stop the behavior, and suspends the account for 24 hours as a warning. ChatGPT already blocks CSAM content and related requests; OpenAI said repeated offenses could lead to permanent account removal. OpenAI said it developed the messaging and guidelines with organizations and researchers including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania, who independently evaluated a trial of the deterrent messages.
A 24-hour suspension is a softer instrument than an outright ban, and this appears to be the point: it treats some attempts as a behavior to interrupt rather than only as an offense to punish, while keeping permanent removal for repeat cases. The counterargument a product team might weigh is that a timed block gives a determined user a cooling-off period and then another try, so the deterrent value depends on whether the warning and support referral actually change behavior. The independent evaluation OpenAI cites is aimed at exactly that question, though the announcement does not publish its results.
OpenAI is also publishing guidelines for developers using its tools to prevent, detect, and help users report CSAM. In early testing, the company said, the guidelines helped a number of developers adopt child safety tooling and register to report CSAM to NCMEC. For builders, this is the part most likely to change day-to-day work: reporting obligations and where to route suspected material vary by jurisdiction, and a vendor-supplied playbook can lower the cost of standing up a process.
For law enforcement, OpenAI is offering AI-powered features including dedicated plug-ins, investigative tools, and complex data analysis, with access managed by program administrators, plus training for investigators and a field guide on using AI. Restricted, administrator-mediated access is a common pattern for sensitive capabilities, since it limits who can query such systems and creates an audit trail.
Background: OpenAI has faced several lawsuits blaming its technology for harming users, particularly children, and the nonprofit Common Sense Media labeled its tech an “unacceptable risk” for teens. Other chatbots, specifically Grok, have been used to create nonconsensual sexualized imagery, including of children. Over the past 12 months OpenAI released a safety blueprint, updated content guidelines to bar inappropriate content, retuned its models against harmful generation, and launched a version of ChatGPT for teens. This announcement extends that arc from model behavior toward detection infrastructure and third-party enablement.
A likely trade-off is reach. Free API access limited to select companies, with sharing routed through Tech Coalition and ROOST, means the classifier's benefit depends on how quickly access widens and whether smaller platforms and open-source moderation projects can use it. The reported 48% reduction in flags is a strong operational result, but it is a partner-test figure; independent deployment across different image distributions may differ. This is inference, not a stated company claim.
Why it matters: Platforms and trust and safety teams that handle user-uploaded images are the immediate audience, since a free, NCMEC-informed classifier with higher precision could reduce manual review queues, and developers building on OpenAI tools get a reporting playbook they can adopt now. ChatGPT users will see warnings and 24-hour suspensions instead of silent blocking. The open question for product teams is access and integration: how broadly the API is distributed and whether detection scores fit existing moderation pipelines will determine how much of this reaches beyond pilot partners.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
ChatGPT will ban an account trying to access CSAM content for 24 hours as a warning.