All terms
Toxicity Classification
Labeling text or model output as harmful, offensive, or policy-violating so a model can learn to avoid or catch it.
Toxicity classification work asks reviewers to judge whether content crosses a line - harassment, hate speech, self-harm content, and similar categories - usually against a detailed policy document rather than personal judgment alone. It's closely related to red teaming but focused on labeling existing content rather than trying to provoke new failures.
Example
Reviewing a set of flagged chat messages and labeling each one against specific policy categories (harassment, hate speech, none) rather than a general "bad/not bad" call.