AI Moderation Tools
Flat isometric illustration of red hexagonal node blocks linked by red lines, the central block showing small yellow window slots.
Tool Reviews

Omni-Moderation API Guide: Categories, Scores, Limits

What omni-moderation-latest actually returns: all 13 categories, which ones accept images, how to read category_scores, and where the free endpoint stops.

By AI Moderation Tools Editorial · · 8 min read

omni-moderation-latest is the model behind OpenAI’s Moderation endpoint, and it is the piece of the moderation stack most teams integrate first and understand last. Four lines of code return a flagged boolean, the boolean looks authoritative, and the thirteen category scores underneath it never get read.

This guide is the reference for what the endpoint actually returns: the full category list, which categories accept images, what category_scores does and does not mean, and the limits that decide whether the free endpoint is the whole answer or the first stage of something larger. For the separate question of whether to adopt it at all, the OpenAI Moderation API review covers the tradeoffs against the alternatives.

Model names, and which one to send

Three names circulate, and only two are current:

  • omni-moderation-latest — the moving alias. Always points at the newest omni snapshot.
  • omni-moderation-2024-09-26 — the pinned snapshot the alias currently resolves to. Built on GPT-4o, released 26 September 2024, and the first moderation model in the family to accept image input.
  • text-moderation-latest / text-moderation-007 — the previous generation. Text only, eleven categories, superseded.

Production systems should send the dated snapshot, not the alias. A moderation classifier is a policy surface: when the alias moves, every threshold you calibrated moves with it, silently, in the middle of a release you did not ship. Pin the snapshot, then re-run your own labelled sample against the new one before switching. That is the same discipline that applies to any versioned classifier, and it is why Llama Guard’s version-to-version taxonomy changes matter more than their headline benchmark numbers.

The 13 categories

The omni model returns thirteen categories. The parent/child structure is meaningful: hate and hate/threatening are separate flags, not a severity gradient on one flag, and they warrant different business responses.

CategoryWhat it coversImage input
harassmentHarassing language toward any targetText only
harassment/threateningHarassment including threats of violence or serious harmText only
hateHate on the basis of a protected characteristicText only
hate/threateningHateful content that also threatens violenceText only
illicitInstructions or advice for committing a crimeText only
illicit/violentThe same, where the wrongdoing involves violence or weaponsText only
self-harmContent promoting or depicting acts of self-harmText and image
self-harm/intentSpeaker states intent to engage in self-harmText and image
self-harm/instructionsInstructions or encouragement for self-harmText and image
sexualSexually explicit content intended to arouseText and image
sexual/minorsSexual content involving anyone under 18Text only
violenceDeath, violence, or physical injuryText and image
violence/graphicGraphic depiction of death, violence, or injuryText and image

Two structural facts fall out of that table and both bite in production.

First, illicit and illicit/violent are the two categories the older text-moderation-* models did not have — eleven categories there, thirteen here, and the delta is exactly those two. If your thresholds were carried over from the previous generation, those two are running unconfigured.

Second, the image path covers a subset. Self-harm, sexual, and violence classes are evaluated for images; harassment, hate, illicit, and — importantly — sexual/minors are text-only, and return zero on an image-only input. A zero there means “not evaluated”, not “clean”. Any pipeline that treats an all-zero response as a pass has a hole exactly where the stakes are highest, which is the argument for pairing the endpoint with dedicated image and video moderation tooling rather than treating multimodal support as coverage.

Reading the response properly

A moderation result has four fields, and most integrations use one of them:

from openai import OpenAI
client = OpenAI()

result = client.moderations.create(
    model="omni-moderation-2024-09-26",
    input=[{"type": "text", "text": user_message}],
).results[0]

result.flagged                       # bool  — OpenAI's own decision
result.categories                    # per-category booleans
result.category_scores               # per-category floats, 0-1
result.category_applied_input_types  # which input types triggered each category

flagged is a policy decision, not a measurement. It is true when OpenAI’s internal thresholds say the content violates its usage policies. Those thresholds encode OpenAI’s risk appetite for its own products, not yours. A children’s education platform and a security research forum need different lines, and neither of them is the line flagged draws.

category_scores are the useful field. They are confidence values in the 0-1 range, and two properties matter: they are not calibrated probabilities, and they are not comparable across categories. A violence score of 0.4 and a sexual score of 0.4 do not represent the same likelihood of harm. Thresholds must be set per category, against your own labelled sample, or they are guesses wearing a decimal point.

category_applied_input_types is the omni-only field that tells you whether a category fired on the text, the image, or both. In a mixed-content pipeline this is what lets you route a flag to the right reviewer queue and the right remediation — deleting an image versus editing a caption are different actions.

Setting thresholds you can defend

One global cutoff across thirteen categories is the most common misconfiguration in this endpoint. A defensible configuration has three properties.

Per-category thresholds. Calibrate each one against several hundred labelled examples drawn from your own traffic, not from a public benchmark. The distributions differ enough that borrowed thresholds transfer badly.

Three bands, not two. Below the low threshold, allow. Above the high threshold, block. Between them, route to human review or to a softer action such as rate limiting or a warning interstitial. A binary allow/block forces every ambiguous case into whichever error you find less painful, which is rarely the same error your users find less painful.

Asymmetric bands by category. sexual/minors and self-harm/intent justify aggressive thresholds and a low tolerance for misses. harassment in a debate forum justifies the opposite. The cost of a false positive is not uniform across a product, and treating it as uniform is what produces the documented false-positive costs that make moderation programmes unpopular internally.

Multilingual coverage

The omni model was a real step up on non-English content. In its launch announcement OpenAI reports evaluating the model on a 40-language set, with a 42% improvement over the previous generation on its internal multilingual evaluation and gains in 98% of the languages tested.

Read that carefully: it is an improvement figure on an internal evaluation, not an absolute accuracy figure, and “improved in 98% of languages tested” says nothing about the floor those languages started from. Major European and East Asian languages are genuinely usable. Lower-resource languages remain the weakest surface, and if a meaningful share of your traffic sits there, that share needs its own evaluation before you rely on a single global threshold.

Limits, cost, and throughput

The moderation endpoint is free to use. That is unusual enough in this category to be worth stating plainly, and it is the strongest argument for running it as a first-pass filter even when something else does the heavy lifting.

The practical constraints are elsewhere:

  • Rate limits. Requests per minute are set by account tier: 250 RPM on the free tier, 500 at tiers 1 and 2, 1,000 at tier 3, 2,000 at tier 4, and 5,000 at tier 5. A platform doing 5,000 moderation calls a minute is doing roughly 83 a second, so high-volume services hit the ceiling long before they hit any accuracy limit. /v1/moderations is one of the endpoints the Batch API accepts, which is the documented path for asynchronous bulk classification.
  • Image size. Image inputs are capped at 20 MB. Downscale before sending; the classifier does not need your full-resolution original.
  • Long inputs. Accuracy degrades on very long text because a single harmful passage is diluted across the whole input. Chunk long documents and score each chunk, then aggregate with a max rather than a mean.
  • No audio. The omni model takes text and images. Audio and video are out of scope entirely, and no amount of prompt shaping changes that.

What it does not cover

Four gaps decide whether this endpoint is a component or a complete answer:

No custom categories. The taxonomy is fixed at thirteen. Platform-specific harms — coordinated brigading, financial scam patterns, competitor spam, self-promotion policy — have no slot. That limit is the usual trigger for moving to a fine-tuned classifier once a policy outgrows the generic taxonomy.

No conversation context. Each call classifies one input. A harmful trajectory assembled across six benign-looking turns scores clean on every individual turn, which is the same blind spot that makes single-turn jailbreak prompt detection unreliable.

No normalization layer. Base64, ROT13, homoglyphs, and zero-width character insertion all degrade classification, because the model scores the string you send rather than the string a reader perceives.

Prompt injection is not a category. Nothing in the taxonomy covers instruction-hijacking, which is a security failure rather than a content harm and needs dedicated prompt injection detection tooling.

Nothing about factuality. A fluent, confident, entirely fabricated answer is not harmful content under any of the thirteen categories, and it is one of the most common real-world output failures. That surface belongs to hallucination detection methods, which is a separate control family with separate tooling.

Where it fits

The honest placement: omni-moderation-latest is an excellent free first stage. Run it synchronously on user input, asynchronously on model output, read the per-category scores rather than flagged, and treat anything it does not cover as a gap to be filled by another layer rather than a risk that does not exist.

For a stack that covers the rest — custom policy, conversation context, adversarial normalization — the classifier ensemble pattern is the usual next step, and the Moderation Stack Matcher will filter the field down to the tools that cover the modalities and harm classes you actually moderate.

Sources

  1. OpenAI Moderation Guide
  2. OpenAI Models: omni-moderation-latest
  3. OpenAI: Upgrading the Moderation API with our new multimodal moderation model
  4. OpenAI Batch API Guide (supported endpoints)
#openai-moderation#omni-moderation #content-moderation #api-review#thresholds
Subscribe

AI Moderation Tools — in your inbox

Sourced comparisons of AI content-moderation tooling — delivered when there's something worth your inbox.

No spam. Unsubscribe anytime.

Related